The left panel was a plain fixed-width widget sized by the window, so once
the Scan Info notes grew past the window height Qt compressed every control
below its minimum: the DC threshold spin box was sliced in half, the ADC
hint collided with it, and the cache note was cut off mid-line. Word-wrapped
labels made it worse by advertising a single-line minimum height, which
clipped the filename and the format/cache notes.
- Put both side panels in fixed-width QScrollAreas so a short window scrolls
instead of crushing the controls.
- Add _wrap_label(), which enables height-for-width so wrapped labels report
their real height; use it for the filename, Scan Info rows, format/cache
notes, DC-precompute status, ADC hint and ROI readouts.
- Replace the ad-hoc QHBoxLayout label/field rows with a shared QFormLayout
(Angle, Channel, DC threshold, Grating size, Colormap, min, max) so labels
align and fields keep a usable minimum width.
- Add _group() for consistent group-box margins; widen the left column
260 -> 288 px so the channel combo and checkbox text stop eliding.
- Keep the canvases usable: non-collapsible splitter, minimum canvas heights
and a 960x560 window minimum.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects found by testing the parallel paths end to end.
Nested parallelism was unbounded. DcPrecomputeWorker and the alignment
driver both parallelise over angles, and each angle's compute_dc_image
then parallelised over rows again: 14 x 14 threads, and — worse — every
angle sized its chunk against the *whole* memory budget, so worst-case
live buffers were ~13.9 GB against a 1 GB budget. Capping the inner
worker count alone does not fix it; the chunk is still sized against the
full budget. plan_angle_level now hands each angle both max_workers=1
and its share of the budget, which bounds the real file at 1002 MB.
Chunk planning could not produce concurrency at all. _plan_chunks sized
chunk_rows first, and _chunk_rows_for spends whatever budget it is given
— so a single chunk consumed the lot and budget//chunk_bytes came back
as one worker. It now picks the worker count first and sizes the chunk to
it, giving 7-16 workers on the real file instead of 1.
Cancellation was too coarse to shut down. stop() was only checked between
angles, and one angle of the real scan is ~40 s, so closing the window
sat through it and returned with threads still running. should_stop is
now polled per row chunk and closeEvent signals every worker before
waiting: close went from 4.0 s with two live threads to 1.04 s with both
finished. A cancelled ComputeWorker emits None so a half-filled image is
never cached.
Also: BatchCacheWorker no longer reports every file as failed when the
process pool itself dies (unguarded __main__, frozen build, sandbox) —
it retries those files in-process. Batches below 512 MB skip the pool
entirely, since interpreter startup would otherwise make small jobs
slower. cache_file rejects an unknown mode instead of silently treating
it as "fft".
Measured on the real 496 GB scan, angle 3, 32 rows:
DC 2.94s -> 0.75s RF 6.10s -> 1.99s peak RSS 1.40 -> 1.93 GB
Batch convert, 24 files / 1 GB: 4.03s -> 2.36s.
Memory budget defaults to 1024 MB (SRAS_MEM_BUDGET_MB), the measured knee.
Testing: tools/test_refactor.py (70 checks: cache round-trip and block
carry-forward, partial-cache handling, parallel-vs-serial identity,
no-mask path, ROI mask vs full-grid, v2/v3/v4 parsing, sras_average
round-trip incl. remainder handling) and tools/test_gui.py (46 checks
driving the real widgets and workers headlessly). check_equivalence.py
still reports all 167 outputs identical to ed0eba4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Structure
sras_format.py parsing/writing, calibration, axes (numpy + struct)
sras_compute.py DC/FFT images, alignment, batch cache (+ scipy)
sras_workers.py Qt background workers
sras_viewer.py ROI, canvases, dialog, main window
format+compute import in 0.59s with no Qt or matplotlib (vs 2.96s for the
full app), which is what makes a spawn-based process pool worth using.
Deduplication
- SrasFile.cal() and compute.dc_image_mv() replace the ADC->mV
calibration incantation that appeared at ten call sites.
- _read_preambles/_read_background/_set_calibration are shared by the
legacy and v6 parsers instead of duplicated.
- v5 PREC images are normalised into the same ragged per-angle list
v6/v7 uses, removing four isinstance() branches and shortening
compute_rf_image's fast path.
- _run_worker replaces five copies of the QThread setup and five
near-identical teardown methods; the two lifetime hazards they
guarded against are now documented once, authoritatively.
- _on_channel_changed defers to _update_controls_enabled rather than
re-deriving the same six enable rules.
- _corners_in_ref_frame, _CHANNEL_DISPLAY dict, unified progress-dialog
helper, dead _on_roi_angle_edited stub removed.
- sras_average.py builds on SrasFile instead of carrying a second copy
of the header format, and streams per angle instead of loading the
whole file (was a ~3x file-size peak).
Executable lines: 2390 -> 2254, despite adding all of the below.
Parallelism
- compute_rf_image/compute_dc_image map row chunks over a thread pool.
Worker count is derived from the memory budget rather than the core
count: on a large scan chunk_rows is already floored at 1 row (~75 MB
of float32 at 7507x2500), so only the worker count can bound peak RAM.
- DcPrecomputeWorker computes angles on a pool, emitting each result
from its own QThread as it lands.
- BatchCacheWorker runs one process per file via cache_file(); only
paths and scalars cross the boundary. Per-process thread counts are
divided so the two levels don't oversubscribe. Drops the old pass 1,
which fully parsed every file just to weight a progress bar.
- dc_threshold_mv=None means "no mask" and skips the CH4 read entirely
— the batch FFT job previously read all of CH4 to compare against a
threshold of -1e9.
- Alignment mask and correlation stages map over angles; fft2/ifft2 use
workers=-1.
Fixes found on the way
- A partially-cached v5 file showed an all-zero image for uncached
angles: the fast-path check was file-wide, not per-angle.
- v7 cached images were read-only big-endian views; now native float32.
Verified: tools/check_equivalence.py produces byte-identical hashes for
167 outputs (DC/FFT across channels, angles, bg-sub, pad factors and
thresholds, plus the full alignment result) against ed0eba4, on both
synthetic files and a real 496 GB 17-angle v6 scan.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/make_test_sras.py writes small v6 files with per-angle-varying
geometry, distinct per-channel calibration, and deterministic waveform
content (predictable FFT peak per pixel, predictable DC mean per pixel).
tools/check_equivalence.py hashes DC/FFT/alignment outputs across
channels, angles, bg-sub, pad factors, and thresholds. It imports from
either the monolith or the split modules, so the same script captures
both sides of a refactor. Hashes canonicalise to native float64 so a
dtype or byte-order change that preserves values is not a false failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scans come off the scope as v6; a new trailing CACH section (independent
SDCB/SFFT blocks, sized per-angle from the existing geometry table) lets
computed DC and FFT images be cached in place, bumping the file to v7 the
first time either is stored. The Convert menu's "Batch Compute DC and
Store"/"Batch Compute FFT and Store" actions run this across multiple
files in the background. compute_rf_image and the DC compute workers now
reuse cached v7 data instead of recomputing it.
Removes "Pre-process and Save as v5", which never supported v6 sources
and is superseded by in-place v7 caching. Legacy v2-v5 reading, including
v5's PREC section, is unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a background worker that aligns every scan angle onto one shared,
zero-padded canvas using a rigid transform only (no scaling): rotation
is taken analytically from the known scan angle, and only the residual
translation is found via FFT phase correlation of each angle's
binarized CH4 dc-mask. An "Aligned View" toggle then redisplays the
currently selected angle/channel resampled onto that shared canvas,
with ROI, CSV export, and waveform-click inspection all kept working.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the FFT was always run unmasked (dc_threshold_mv=-1e9) so the
result could be cached independent of the threshold, with masking applied
afterward as a display-time step. That meant every pixel's CH1 waveform
was read and FFT'd even when most of the scan is background — wasted work
now that the DC map (and its threshold mask) is already known ahead of
time via the DC precompute cache.
compute_rf_image now takes the real threshold and an optional precomputed
dc4_mv array (reused from the DC cache, avoiding a redundant CH4 read),
and skips the FFT for masked-out pixels entirely, same as before this
threshold-caching detour was introduced. The masked-out CH1 samples are
also never read from disk: the boolean valid-pixel mask is applied to the
raw memmap slice before any dtype conversion, so numpy only pages in the
bytes for pixels that pass the threshold.
Threshold is therefore back to being part of the FFT cache key (changing
it now decides which pixels get computed at all, so it can't be satisfied
from a cache built for a different threshold) — but the recompute it
triggers reuses the cached DC4 image and skips both the FFT and the I/O
for masked pixels, so it's much cheaper than the original full-image
compute. Grating and colormap remain pure post-processing with no
recompute.
Verified bit-identical output against the un-optimized reference path on
synthetic data (including scattered, non-row-aligned masking), and timed
on the real 532 GB / 17-angle file: angle 0's FFT (75.2% of pixels above
the default 50 mV threshold) went from ~121s (fully unmasked) to 114s
(FFT skipped, I/O not skipped) to 90.2s with this change (both skipped).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Colormap, DC threshold, velocity grating, and FFT pad-factor changes were
all routing through a full recompute, which is unworkable on large v6
scans where a single angle's FFT can take minutes. These are now applied
as cheap post-processing on cached data instead:
- ComputeWorker computes CH1/Velocity as an unmasked raw FFT plus its DC4
mask image, cached per (angle, bg_sub, n_fft). Threshold masking and
grating scaling are applied to the cached array on redraw, so neither
needs a recompute; colormap changes only touch matplotlib.
- New DcPrecomputeWorker fills a per-angle DC (CH3/CH4) cache in the
background right after load, since DC images are cheap (mean, no FFT)
and make switching angles instant once cached. Progress is shown in the
Scan Info panel.
- New _refresh_display()/_show_image_now() route every settings-changed
handler through the cache first, falling back to the existing threaded
_start_compute() (with a progress popup, now FFT- vs DC-specific) only
on a genuine cache miss.
- File load now defaults to a DC channel instead of CH1/FFT, so opening a
large file shows something in seconds instead of minutes.
Verified against a real 532 GB / 17-angle file: DC compute for one angle
takes ~100s+ (I/O-bound over USB) the first time, but switching to an
already-cached angle takes 0.4s with no compute thread spun up at all.
Audited every widget signal connection for this pass; the spinboxes
already correctly used editingFinished rather than valueChanged — the
recompute-on-every-change bug was in the handlers, not the event type.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause: compute_dc_image/compute_rf_image chunked processing by a fixed
32-row count, sized for old small-format scans. A real v6 file with
spf=2500 and up to ~7500 frames/angle needed ~6-7 GB for a single chunk's
core buffers, on a 17 GB machine — pushing past available memory on every
angle and aborting the process (not a clean MemoryError) when switching to
a new angle piled more allocations on top. Chunk size is now computed from
actual scan dimensions to hit a fixed ~128 MB budget instead of a fixed row
count, cutting peak footprint by roughly 16-18x (verified against the real
532 GB / 17-angle dataset: ~400 MB peak, no crash).
Also hardens QThread lifecycle handling, found while chasing this crash:
- _on_compute_thread_finished/_on_load_thread_finished/
_on_preprocess_thread_finished now call wait() before dropping the last
reference to a finished QThread, avoiding "QThread: Destroyed while
thread is still running" aborts if the OS thread hasn't fully joined
when finished() fires.
- _start_compute/_load_file/_on_preprocess now claim their QThread
immediately after the busy-guard check, before any call that can pump
the Qt event loop (e.g. first QProgressDialog.show()), closing a
reentrancy window where a second call could start a thread that the
first call's own assignment would then clobber mid-run.
- faulthandler is enabled at startup so any future native crash prints a
real stack trace instead of a bare "Aborted".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drops the legacy Time Gate FFT-windowing feature and the SAW matched-filter
pipeline (SawPipeline, SawDiagnosticWindow, TemplateBuildWorker, and the
associated UI panels/channel modes) to simplify the viewer.
Adds native support for the SRAS v6 file format (scan_format.md), which
scans a different bounding box per angle instead of a uniform AABB. Scan
geometry (n_rows, n_frames, x_start) is now exposed per-angle in SrasFile,
with v2-v5 files populating those arrays uniformly so both formats share
one code path. Waveform data is memory-mapped per angle to handle v6's
ragged layout, and aborted/truncated v6 scans are handled gracefully.
"Pre-process and Save as v5" is disabled for v6 files since the flat v5
layout can't represent per-angle geometry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Thomas Ales