A mis-triggered row cannot be written as it arrived — v6 declares n_frames per
row in the header and has no per-row length field, so a short or long row
would shift every later row in the file. Until now the only policy was to
square it up, which keeps the scan running but leaves the affected row
indistinguishable from a good one afterwards: nothing in the file records that
it was padded.
strict_rows selects the other trade. On any frame-count mismatch the scan
stops instead of writing the row, so a data run either produces rows that mean
what the header says they mean or fails loudly. Default stays pad, so existing
behaviour is unchanged.
_warn_frame_delta becomes _check_frame_delta, since it now decides rather than
just reports. Both acquisition paths already call it before writing anything
for the row (CH1 leads SCAN_CHANNELS, and the burst path checks every row up
front), so an abort leaves the file on a whole-row boundary rather than a
half-written row — test_strict_row_packing_writes_nothing_for_the_failed_row
pins that.
Plumbed through QtScanController to a checkbox in the scan panel, persisted in
ScanDefaults alongside burst_mode. scan_format.md documents both policies and
notes that the choice is not recorded in the file.
The row-clipping setup in the padding test is now a _clip_one_row helper,
reused by the strict tests. 92 tests passing, ruff clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per-row acquisition pays a full arm/stop/transfer round trip for every row,
and the transfer is one IEEE-488.2 block read per frame (~16k frames a row).
Burst mode runs one FastFrame acquisition across as many complete rows as the
scope's frame memory holds and pulls each burst in a single CURVe?
transaction, amortising the round trip over the whole burst.
It is opt-in (ScanEngine(burst_mode=...), default False) and writes
byte-identical files to the per-row path — test_burst_and_serial_produce_
identical_files runs the same plan both ways and compares the bytes, which is
the property the whole feature rests on.
core/scope_burst.py — the new policy module. Everything that computes rather
than talks to hardware is a free function, so sizing and row-splitting are
testable without a rig: rows_per_burst() (rounds down, since a partial row
can't be written, and clamps to a transfer-buffer budget), split_row_counts(),
normalize_row(), frame_means_block().
The hard part is that a burst carries no row markers — the scope returns one
flat run of frames. Boundaries come from ACQuire:NUMFRAMESACQuired? sampled
after each acquiring pass while the stage gate is already low, rebased on a
baseline read back at RUN rather than assuming the counter resets. A counter
that goes backwards means the acquisition restarted mid-burst and is now a
hard error instead of silently misattributing every later row.
core/scan_engine.py — the row loop splits into _scan_rows_serial and
_scan_rows_burst. The wire is channel-major and the file is row-major with
channels inner, so _write_burst deinterleaves by writing one channel at a
time to strided offsets; peak memory stays at a single channel's burst
instead of the whole thing.
_gate_off_preflight is what makes this trustworthy on real hardware. The BBD
value that idles the trigger output low is not settled by the protocol docs
(see TRIGOUT_GATE_OFF), and getting it wrong fills every burst with flyback
frames that silently shift the file. The scope already measures the gate on
CH3, so the check needs no bench probe: one gated-off flyback must acquire
nothing, and one gated pass must acquire something — the second half is what
stops a dark laser from making the first half pass vacuously. It runs once
per scan and costs two row-times.
Two fixes fall out of this work and apply to both paths:
- Rows are now squared up to the declared n_frames (short rows zero-padded,
long rows truncated, both warned). v6 commits to n_frames per row in the
header and has no per-row length field, so an over- or under-triggered row
used to shift every later row in the file.
- The X trigger output is returned to idle in the run() finally block. The
per-row path left TRIGOUT_MAXV armed for the rest of the session, so the
gate line kept being driven on every later jog.
core/scope_sras.py — pins DATa:ENCdg RIBinary and DATa:WIDth 1 during setup
instead of inheriting front-panel state. The file header hardcodes
bytes_per_sample=1; a scope left on 2 bytes would have corrupted every frame
written. frames_acquired/frame_means move to scope_burst, where the offset-
based variants serve both paths.
tests/fakes.py — FakeStage and FakeScope are now wired together the way the
rig is: a gated X move at scan velocity feeds frames into a running
acquisition at the real 20 kHz / 100 mm/s rate, direction-agnostic. Both
paths therefore derive frame counts from one model, which is what makes the
byte-identity comparison meaningful, and a gate the engine forgets to drop
shows up as extra frames instead of passing silently. Frame content is a
function of (channel, index) alone, so the same frame sequence yields the
same bytes however it is chopped into transfers.
87 tests passing, ruff clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gui/qt_workers.py — one QueueWorker base replaces the per-device command
queue + dispatch + signal boilerplate. The loop blocks on the queue
instead of waking 10-20x/second forever (test_idle_worker_does_not_spin
asserts an idle worker burns ~no CPU). PollingQueueWorker adds
self-rescheduling polling: the next poll is queued only after the
previous finishes, so a device slower than the interval can't accumulate
a backlog (test_polling_never_overlaps_or_backs_up).
Helios responsiveness — the concrete bug that motivated the above: a free
running 1 s QTimer queued a status poll that took ~2 s, so the queue grew
for as long as the panel stayed connected.
- helios_laser._query reads until the CR terminator instead of sleeping a
fixed 0.05 + 0.2 s per query
- one _query_int() helper replaces five copies of parse-with-logging
- polling is now driven by the worker; HeliosWindow's QTimer is gone
- dropped __del__, which disabled the laser and wrote to the serial port
from the garbage collector at an unpredictable time
helios_test_app.py — the worker was moveToThread'd but every call site
invoked its methods directly, so all serial I/O (including the sleeps)
ran on the GUI thread; Query All froze the UI for ~2 s. Calls now go
through a queued signal to a pyqtSlot. Also: connect/disconnect cycles
leaked a QThread + worker + 9 connections each time; 16 copies of the
not-connected guard collapse to _require_connection(); the Query Power
button called a method that has never existed (AttributeError popup) and
is now disabled and documented in KNOWN_ISSUES.
DCBiasImageWidget preallocates its image and uses set_data/set_clim, so
the live preview stops rebuilding the array and the whole artist tree per
row (O(rows^2) over a scan).
bbd20x: connect() now raises when no bays respond instead of reporting
success on the wrong port; disconnect() joins with a timeout so a wedged
reader can't hang shutdown; one _channel_for() helper replaces four
copy-pasted axis mappings; hardcoded travel limits become TRAVEL_MM; the
joke error strings are gone.
gui/widgets.py adds the shared ConnectionBar / PortSelector / bounded
LogConsole / StatusGrid for the test benches to adopt. 65 tests passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The headline of the refactor. Scan orchestration no longer lives inside a
QObject that reaches through Qt workers for its hardware handles.
core/scan_engine.py — ScanEngine(stage, scope, rotator, plan, out_path,
resume, callbacks). Takes the concrete drivers, blocks in run(), reports
via plain callables, and prompts through an injected blocking callable.
No Qt import anywhere in the path (test_engine_imports_without_qt proves
it), so a simpler GUI or a CLI can drive the identical acquisition.
Supporting extractions, all Qt-free:
- core/scope_sras.py — SCPI policy: channel profiles, trigger programming,
background average, per-row FastFrame transfer
- core/rotation.py — RotationAxis + RotationSettings (the GR_* constants)
- core/scan_resume.py — frontier contiguity rule + settings compatibility
- gui/scan_bridge.py — QtScanController, exposing exactly the signal
surface the old ScanWorker had, so MainWindow's connections are unchanged
hardware/t3r_driver.py is now Qt-free: a plain Signal class, a threading
reader, and a polling thread instead of QObject/QThread/QTimer.
gui/qt_t3r.py re-emits its callbacks as queued Qt signals for the panels.
Fixes carried by the extraction:
- rotation waits on the driver's MOTION_DONE event instead of
time.sleep(estimate + 0.5)
- abort during an operator prompt now takes effect; the old
_prompt_event.wait() had no timeout and could not be interrupted
- the poll timer is a thread, so an I/O error tearing down the driver no
longer calls QTimer.stop() from the wrong thread
- T3RDriver.disconnect() renamed close(); it shadowed QObject.disconnect()
- per-frame DC means use np.frombuffer over the joined block instead of
struct.unpack per frame (~16k tuple allocations per row)
tests/fakes.py + test_scan_engine.py (14 tests) assert the exact command
sequence, file layout, resume seeking, abort/pause, and geometry
rejection before any hardware call; test_scan_resume.py covers the
frontier rule. 58 passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The viewer could only parse v2-v4 headers while the app has been writing
v6 for some time — it could not open ANY file the current app produces.
It now uses core.sras_format directly (v6 only, per user decision).
New core/sras_analysis.py (Qt-free): ChannelCalibration, image reducers,
and SawPipeline. sras_viewer.py keeps only Qt.
Memory (measured, 92 MB synthetic scan, separate processes):
old eager path +305 MB read()+slice-copy+astype+float32 mean
new mmap path + 31 MB zero-copy view + mean(dtype=)
-> identical DC image; old scaled at ~3.3x file size, new at image size
- load_angle() returns a read-only mmap view instead of reading the whole
data block, then copying it twice
- SAW sweeps keep one scalar per pixel (process_shot_metrics) instead of
retaining 5 full arrays x pixel count in a results list
- CH1 float32 materializes only for pixels passing the DC mask
- matched filter caches the template FFT instead of recomputing per pixel
- opening a new file drops every reference to the old one (compute/
template/diagnostic workers used to pin the previous multi-GB mapping)
Responsiveness:
- 250 ms debounce coalesces spinbox storms into one recompute
- grating change is a display-time scalar multiply, not a full FFT rerun
- colormap/clim reuse the AxesImage (set_data/set_clim) instead of
clf() + rebuilding the colorbar; draw_idle() throughout
- SAW diagnostics (21 pipeline runs) and CSV export moved off the GUI thread
Also: ragged per-angle geometry is respected (v6 angles differ in rows/
frames), truncated scans show only rows present on disk, dead decimation
path and v2 fallback branch removed, scipy added to viewer requirements.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- core/sras_format.py: THE v6 implementation — create_scan_file (writer,
byte-identical to the old one, enforced against the Phase-0 goldens),
SrasFile parser with frontier/truncation walk, and zero-copy mmap
load_angle/load_row views for multi-GB files
- core/scan_geometry.py: ScanPlan/AngleGeometry dataclasses, build_plan
(rotated-bbox trig from MainWindow._build_scan_params), travel-limit
validate_plan (limits now a StageLimits dataclass, not literals buried
in the worker), format_eta + EtaEstimator (bounded deque)
- core/config.py: ScanDefaults dataclass replaces the module-import-time
dict globals. FIXES: editing any main-window port used to rewrite
aui_defaults.json without helios_port, silently reverting the Helios
port every time (test_helios_port_survives_partial_update covers it).
Also drops the inert laser_freq_hz plumbing — scans always used the
LASER_FREQ_HZ constant.
- hardware/serial_util.py: shared 8N1 open + scored port enumeration
(promoted from t3r_control_panel); helios_laser and the panel use it
- sc3_aui_app.py and sras_scan_manager.py migrated onto core (three
format implementations down to one); ScanWorker now takes a ScanPlan
- tests: byte-identical writer vs golden, frontier over every truncation
variant, mmap==eager, geometry vs golden fixtures + invariants, config
round-trip. 28 passing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ruff config, offscreen smoke tests for all 7 GUI apps/panels
- golden v6 .sras fixtures (complete + 4 truncation variants) generated
by the pre-refactor writer, with expected header/frontier JSON
- golden geometry fixtures from the pre-refactor _build_scan_params
- consistency tests proving current code reproduces the goldens
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thomas Ales