Its only unmerged commit, "pre merge cleanup" (1a81871), branched off
refactor/parallel-and-dedup after PR #1 (b0bfe6e) had already merged that
branch into main and was never rebased onto main's later work. Every change
it made was independently reimplemented, more completely, by main's own
subsequent commits:
- FFT pad-factor caching + the stored-cache display fast path
(cached_rf_image, precomputed_pad_factor, _stored_fft_image,
_stored_dc_image, PAD_FACTOR_MAX) -> 8348ad3, refined by c30c8b1.
- The FftOptionsDialog "stored cache unusable at this pad" warning
-> also 8348ad3.
- spin_angle redrawing on arrows/keyboard via setKeyboardTracking(False)
+ valueChanged, and the corresponding tests
(test_stepping_the_angle_spinbox_redraws,
test_typing_an_angle_does_not_compute_intermediate_angles) -> already
on main before the refactor/parallel-and-dedup merge point.
A real 3-way merge of 1a81871 confirmed this: it produced no conflicts in
sras_format.py/sras_compute.py only because both sides independently wrote
equivalent code, and silently duplicated a method definition
(_stored_fft_image) in sras_viewer/main_window.py where the two
implementations diverged textually. Recording this as a merge with main's
tree kept (-s ours) marks the branch as fully incorporated without
reintroducing its now-superseded, lower cache-version (CACH_VERSION 2 vs
main's 3, missing row-averaged-FFT support) implementation.
origin/main split sras_viewer.py into the sras_viewer/ package (dialogs,
main_window, common, canvases) after the batch-export feature was built
against the old monolith, so merging origin/main into main lost the feature
to the sras_viewer.py deletion. Re-add it in the new structure:
BatchExportDialog in dialogs.py, a Jobs.EXPORT job key in common.py, and
the menu action/worker wiring in main_window.py, following the Jobs-key
and dialog-class conventions the split already established rather than the
old ad hoc "main" progress-key naming the original commit used.
Also carries forward two test fixes bundled in the original batch-export
commit that the parallel pytest conversion (007089d) had ported from
tools/test_refactor.py without them: the int16-vs-float32 accumulator
expectation in test_sras_average, and asserting the sras_average.py
subprocess calls actually succeed.
Add Export -> Batch Export Images..., which renders and saves one PNG
per angle for whichever of CH1 (RF), CH3 (DC-A), CH4 (DC-B), and
Velocity the user selects, for the currently open file. Each channel
gets its own fixed min/max colorbar range, entered once in the dialog
and held constant across every exported angle, so the resulting images
are directly comparable to each other instead of each auto-scaling to
its own data (today's live-view default).
BatchExportDialog (sras_viewer.py) collects the output folder, file
prefix, and per-channel checkbox + range, seeded from whatever's
already cached (session DC/FFT caches, or the file's own v7 cache
blocks) so opening the dialog never triggers a fresh compute. Export
runs on a background thread via a new BatchExportWorker
(sras_workers.py), which computes each angle's channel image with the
existing dc_image_mv/compute_rf_image helpers (reusing one FFT per
angle for both RF and Velocity) and renders it with a headless
matplotlib Agg canvas.
Also includes several small correctness/robustness fixes that were
already staged in the working tree: an int16-vs-float32 accumulator
mismatch in sras_average.py's row averaging, a DC4-mask cache reuse and
atomic sidecar write in sras_compute.py, a dc3/dc4 pairing guard in
sras_format.py's v7 cache writer, and matching updates to the
tools/ test fixtures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lets a user export the current ROI as one CSV with a column per selected
angle's value (RF peak freq, Bias A, Bias B, or velocity), once angles share
a common (x, y) grid -- either via a live Fusion alignment result or because
the open file is itself a previous Alignment Wizard export whose angles
already share one grid on disk. Never triggers a background compute; angles
without cached/stored data for the chosen value type are simply unavailable
in the picker.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The display path hardcoded row_avg_n=0 when checking a file's stored FFT
cache, since the main window has no row-averaging control (only the
batch-only RowAverageFftOptionsDialog). Once a file was batch-computed with
row-averaging, every view switch saw a phantom provenance mismatch and
silently launched a full raw recompute, discarding the precomputed data.
The display now asks for the stored image at the settings it was actually
computed under (bg-sub/pad/row-averaging), rather than the window's live
controls, so a stored or already-computed image is always shown once
present. Those controls now only affect a first-time compute for an
uncached angle or an explicit batch recompute -- never what's already on
screen. DC threshold is unaffected: it stays live, since re-masking a
cached image is free.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Extract cache mismatch logic into reusable cache_mismatch_reasons() function
for cleaner validation and consistent miss reporting
- Expose memory_budget_bytes() for external callers (aligned exporter)
- Optimize rebuild_stack() with incremental layer updates when only one angle
parameters change, avoiding full reprojection overhead
- Remove unused seed_per_angle parameter from alignment wizard
- Simplify cached_rf_image() DC masking with cleaner early exit logic
- Clean up internal state tracking with explicit origin caching
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
tests/test_stored_cache.py exercised two features that were never built,
so six of its tests had been failing on main. Both are now implemented.
Pad-factor caching. A padded view could not use a stored FFT cache at all:
the store was pad 1 by definition and cached_rf_image rejected any n_fft
outright, so a user working at a pad factor got nothing from batch-computing
a file. CACH tail v3 records the pad the images were resolved at, cache_file
computes at a requested pad, and the batch actions pass the viewer's own pad
down — while still refusing a store resolved at a different pad, since a
padded FFT interpolates between the natural bins and so resolves genuinely
different peak frequencies. v1/v2 tails read as pad 1 and keep working.
Stored-cache dispatch. _refresh_display only ever consulted this window's
in-session dicts, so after a batch every angle change still queued a worker
and a progress popup for an image already on disk — the exact cost the batch
was run to avoid. It now checks the file's own DC/FFT blocks first, asking
with allow_dc_recompute=False so the GUI thread never touches I/O. When the
stored cache genuinely cannot serve the view, the scan info panel says why
rather than leaving the silent recompute a mystery.
Two bugs surfaced on the way:
- The angle spinbox was wired on editingFinished, which QAbstractSpinBox
emits only on Return or focus-out — never on a step. Clicking its arrows,
the ordinary way to walk a scan, moved the number and left the image
behind. Now valueChanged with keyboard tracking off, which fires on a step
and once on commit, but not per keystroke mid-typing. Every existing GUI
test called _on_view_changed() by hand and so could not have caught this;
two new tests pin it and fail against the old wiring.
- A plain "fft" batch over a file previously cached with fft_rowavg carried
the old row_avg_n forward, labelling raw images as row-averaged. It now
writes row_avg_n=0 explicitly.
Verified: 111 passed (was 103 passed / 6 failed), and
tools/check_equivalence.py is byte-identical to the pre-change baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Alignment was two disconnected menu actions. `Angle Alignment` ran the
registration with ref_angle_idx hard-coded to 0, no exposed parameters and
no way to retry; `Manual Alignment...` was a separate dialog that
deliberately refused to inherit the automatic result, so a bad fit meant
starting over by hand. Neither told you whether the alignment was any
good, and neither produced anything durable beyond an in-memory result.
Fusion -> Alignment Wizard... now covers all of it in three steps:
1. Correlate. Reference angle, DC threshold, source, rotation seed and
sign, search window, coarse step and fine grid are all on the page,
and Run/Re-run is repeatable. Angles start pre-rotated from the
stage angles in the file, so the page is informative before any
correlation runs. The verdict is a picture: every angle's DC mask
reprojected onto the shared canvas and summed, coloured by how many
angles cover each pixel, so a good alignment reads as one saturated
plateau and a bad one as a fringe of low-count halos. A per-angle
fit table flags angles that did not register or that disagree with
their stage angle by more than a degree.
2. Crop. An axis-aligned rectangle on that canvas, with numeric
canvas-pixel boxes synced both ways and a live size estimate.
"Fit to full overlap" uses a largest-rectangle sweep rather than a
bounding box: the overlap region of several rotated scans is roughly
a disc, whose bounding box has corners no angle covers.
3. Save. Writes the aligned, cropped stack to a new .sras.
Because the wizard replaces both actions it absorbs the old dialog's
by-eye nudge editor — without it, a scan the search cannot fit would
have no fallback at all. ManualAlignmentDialog is therefore deleted
rather than left orphaned, and its tests move to the wizard.
Two bugs found while driving it end to end and fixed here: QSpinBox
setRange clamps and emits valueChanged, which committed a 1x1 crop
before the default preset could run; and the mm round trip returns an
exact pixel boundary as 11.000000000000002, so a bare ceil() added a
spurious column on every rectangle edit.
Verified: 103 pass, the 6 test_stored_cache failures are pre-existing on
main, and tools/check_equivalence.py is byte-identical to main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alignment machinery never modified a scan: it produced an
AlignmentResult and every consumer resampled on the fly. That is right
for a viewer but means the aligned stack cannot leave the process, so no
other tool can read it and reopening the scan redoes the registration.
sras_align_export.write_aligned_sras bakes an alignment into a new v6
file: each angle is gathered onto the shared canvas by nearest neighbour,
so every output angle shares one grid and the file opens already aligned.
Registering the export against itself returns identity, which is the test
that pins the whole index chain.
Three details that are easy to get wrong and are now covered:
* Rounding must be floor(x + 0.5), not np.rint. scipy's order=0 rounds
halves away from zero, and the canvas is snapped to the reference's
pixel grid, so exact halves are common rather than hypothetical.
* Out-of-bounds must be tested on the fractional coordinate against
[0, n-1], not on the rounded index, or a one-pixel rim gets real data
everywhere the Aligned View shows padding.
* ...but with a tolerance, because the mm-space affine chain lands an
exactly-integer transform a few times 1e-13 off. A bare >= 0 drops
the *reference* angle's entire first row and last column.
Padding is the per-channel ADC code nearest 0 mV, not zero: zero ADC
decodes to ~+100 mV on real calibration and would masquerade as sample.
Source rows are served from sliding in-RAM bands. A rotated angle maps
one output row to a diagonal across the source, so indexing a memmap in
output order refaults nearly the whole angle per row — terabytes of
paging for a gigabyte of data.
Also in compute: crop_alignment_result (a crop is pure index translation,
so it folds into the affine's offset rather than becoming a second
transform), overlap_stats, largest_rect_at_least for a crop that stays
inside the overlap region, and seed/sign/refine knobs on
register_angle_to_reference whose defaults leave existing behaviour and
tests byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three block essays in sras_compute.py — memory budget / row
chunking, angle-alignment coordinate frames, and the sidecar placement
+ schema history — move to docs/design.md (joined by a new section on
the zoom FFT peak search), each replaced by a 2-4 line pointer. The
save_manual_alignment docstring no longer restates the JSON schema its
own eight lines of code construct.
Load-bearing trap notes (normalization=None, subpixel-score mixing,
memmap lazy reads) stay in place. Zero code changes — golden-hash diff
verified empty.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- One _apply_alignment_result() replaces the five copies of the
cache-clear / generation-bump / Aligned-View-checkbox dance, and
_on_manual_alignment_saved/_cleared collapse into a shared
_manual_alignment_changed.
- QSignalBlocker context managers replace every hand-rolled
blockSignals(True)/set/blockSignals(False) triple (11 sites).
- _make_dspin() replaces the 11 four-to-seven-line QDoubleSpinBox
constructions; _axes_extent() replaces the duplicated imshow-extent
formula; _is_fft_mode() replaces the repeated CH1-mode guard.
- Jobs constants replace the stringly-typed background-job keys.
- The two 160-line widget builders split along their section banners:
_build_left_panel -> file/info/view/roi groups, the manual-alignment
_build_ui -> six per-group builders + _connect_controls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2790-line module becomes sras_viewer/: common.py (constants +
layout helpers), canvases.py (RoiQuad, ImageCanvas, WaveformCanvas,
ManualAlignOverlayCanvas), dialogs.py (FftOptionsDialog,
ManualAlignmentDialog), main_window.py (SrasViewerWindow + main), with
__init__ re-exporting the public names and __main__ keeping
`python -m sras_viewer` working. pyproject gains a `sras-viewer`
console script.
Code moved verbatim; only import headers are new (pyflakes-clean).
tests/test_gui.py patch targets follow the classes to their new
modules.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- SrasFile: _parse_v6 now retains per-angle x_delta and the verbatim
preamble/background byte spans, and gains public data_offset,
y_pos_per_angle, and iter_angle_blocks() (which now owns the ragged
block-offset walk used three separate places before).
- sras_edit_scans: the 70-line re-parse of the v6 header sections
(_read_v6_sections/_reread_span/_v6_angle_offsets) collapses into a
_write_v6 that consumes SrasFile directly — verified byte-identical
round-trip on v6 int8/int16 and legacy files. Eight print-and-exit
pairs become _die().
- tools/make_test_sras imports the struct layouts from sras_format and
the rotation matrix from sras_compute instead of re-declaring them
(byte assembly stays independent of the reader).
- sras_compute: block_mean_2d, pixel_pitch_mm, nominal_delta_deg made
public (they were GUI-facing); registration_workers() and
default_max_workers() wrap the remaining private reach-throughs from
sras_workers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DcPrecomputeWorker, Ch4MaskWorker and CrossCorrelateWorker shared the
same submit/as_completed/emit/shutdown skeleton; they are now
constructors plus _plan/_items/_one/_emit hooks. All three inherit the
cancellation-aware shutdown (wait=not stopped, cancel_futures) that
only DcPrecomputeWorker had before, and the per-run budget attributes
are initialised in __init__ instead of appearing mid-run.
Also removes the viewer's write-only _dc_precompute_worker plumbing
(job tracking already owns worker lifetime via _jobs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- FFT backend and pad factor persist across sessions via QSettings
(IniFormat; tests redirect the settings path for hermeticity).
- The default backend was labelled "NumPy FFT" but always dispatched to
scipy.fft — rename the canonical value to "scipy" ("numpy" stays as a
legacy alias) and fix the dialog label.
- cache_file: DC caching fans out over angles via _parallel_map with
per-angle budgets (the DcPrecomputeWorker pattern); FFT caching stays
serial per angle because compute_rf_image now parallelises internally
over blocks. Documented that the v7 FFT cache is natural-resolution
(pad 1) by design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At pad 40 the old path materialised a ~9 GB padded spectrum per row,
which collapsed the chunk planner to one worker and one rfft call with
workers=1 — synthesis ran single-threaded, ~1 hour per angle on real
files.
The padded spectrum is never materialised now. Each block of 512
waveforms gets a coarse rfft at next_fast_len(2*spf); every coarse bin
within 0.7 of its row's max (plus the DC-adjacent window, which coarse
DC suppression would otherwise blind) is refined onto the exact n_fft
grid by a small complex gemm. The selected bin is bit-identical to the
full padded argmax — enforced by test_zoom_identity, a 25-seed fuzz
test over adversarial spectra, and a clean golden-hash diff against the
pre-rewrite baseline across pads {1,2,4,8,40}, masked/unmasked, bg
on/off, int8/int16, and both backends.
Blocks fan out over a persistent thread pool; pyFFTW runs through
per-thread FFTW_MEASURE builder plans with wisdom persisted to
~/.cache/sras-viewer, and threadpoolctl clamps BLAS under the pool.
compute_rf_image(exact=True) (or SRAS_FFT_EXACT=1) keeps the reference
padded path for audits.
tools/bench_fft.py measures: pad 40, 16 cores, 8192x2500 synthetic —
exact serial 717 wf/s -> zoom pool 25100 wf/s (35x, pyFFTW backend;
19x scipy), every variant verified equal to the reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- pyproject.toml replaces sras_viewer_requirements.txt (same pins) and
adds a dev extra with pytest.
- tools/test_refactor.py, test_alignment.py, test_gui.py become
tests/test_compute.py, tests/test_alignment.py, tests/test_gui.py with
assertions preserved verbatim. test_gui.py stays one ordered
integration sequence over a shared module-scoped window.
- check_equivalence.py: drop the dead pre-refactor monolith shim (and the
_compute_angle_alignment alias it consumed), extend the pad sweep to
(1, 2, 4, 8, 40), add legacy-v4 and big-endian int16 legs (new bps=2
option in make_test_sras) so the padded FFT path and the >i2 memmap
path are in the baseline before the FFT rewrite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SRAS_FORMAT.md/.html documented only v2-v5 and were superseded by
scan_format.md in July. Ignore .DS_Store, screenshots, pytest cache,
and test scratch outputs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Continues the Manual Alignment work: refines the FFT cross-correlation and
mask handling, adds sras_edit_scans.py (drop/renumber bad angle scans),
tools/test_alignment.py (registration ground-truth suite), and a rotating
test fixture in make_test_sras.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manual mode previously offered only Auto De-rotate (rotation from known
scan angles) plus keyboard nudging, so every angle's translation had to
be found entirely by eye — fine for a small correction, unworkable for
the tens-of-mm scatter a real many-angle scan can have between angles
before any translation search runs at all.
Add an "Auto Cross-Correlate (vs Reference)" action that, for every
non-reference angle, sets rotation to the known analytic angle (same as
Auto De-rotate) and translation to the FFT-phase-correlation best fit
against angle #0 (always the reference/ground truth). This is meant to
land every angle's outline roughly stacked on the others so keyboard
nudging only has to make small corrections afterward, per the intended
workflow: cross-correlate first, then nudge.
Exposes two tunable options, since real data may correlate better one
way than the other: "Correlate on" (raw CH4 signal, minus its own
minimum -- the default, robust to a shared threshold not suiting every
angle's real signal level -- or the thresholded binary mask), and
"Search margin" (how far a shift the phase correlation searches before
wraparound bias becomes a risk).
Runs on a background thread (new CrossCorrelateWorker in
sras_workers.py) since correlating many high-resolution angles can take
long enough to visibly freeze the dialog otherwise.
The underlying per-angle correlation math is factored out of
compute_angle_alignment's Step 2 into a new, reusable
correlate_translation_mm, so the existing automatic Fusion -> Angle
Alignment action and the new manual button now share one implementation
instead of two copies of the same phase-correlation logic.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The dialog was seeding rotation_deg/shift_mm from self._alignment_result
whenever it was non-None, with no way to tell whether that result came
from a user's own manual Save or from the automatic Fusion -> Angle
Alignment run. The automatic path's translation comes from FFT phase
correlation - the exact thing manual mode exists to work around - so
opening Manual Alignment after running the automatic pass silently
inherited its unreliable shifts on top of the (correct) analytic
rotation, reproducing the same scattered-scan symptom under a "manual"
label.
Manual mode should always start from identity (each angle's centroid
already coincides with every other's, per the prior pivot fix) unless
the user previously used this same dialog to save their own deliberate
translation - which is exactly what the sidecar file already records.
Seed exclusively from load_manual_alignment() and drop the
self._alignment_result branch entirely; the automatic result is never
an appropriate seed for a manual editing session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pivot was still derived from the binary CH4 >= dc_threshold_mv mask,
so a threshold that happens to leave a real angle's mask empty (signal
levels vary scan to scan across a many-angle acquisition) silently fell
back to the raw scan-window bbox center — reproducing the exact "aligned
to the scan window, not the sample" scatter the centroid pivot exists to
fix, without any visible error.
Replace the binary-mask centroid with an intensity-weighted centroid of
the continuous CH4 signal, which is always well-defined regardless of the
RF-mask threshold in effect. The threshold still controls what the manual
dialog's overlay and CH1 masking show; it no longer has any bearing on
where alignment pivots.
Also bump the sidecar schema version: a file saved before today's pivot
and rotation-sign fixes stores numbers for a since-corrected transform, so
loading it unchanged would reintroduce the same scatter. Old sidecars are
now treated as absent rather than silently reused.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rotation was pivoting on each angle's raw scan-window bbox center, which
ties alignment to wherever the window happened to sit in microscope/global
XY space rather than to the sample itself, leaving a residual orbital drift
between angles. Pivot on each angle's own CH4-threshold mask centroid
instead (its own sample footprint), for both the automatic phase-
correlation path and the manual dialog's Auto De-rotate/live preview.
Also negate the rotation used everywhere (_theta_deg): the GR stage's
reported angle increases in the opposite rotational sense from this
module's CCW math convention, so de-rotating by the raw angles_deg delta
was making misalignment worse rather than better.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Automatic alignment's phase-correlation translation search is unreliable, so
add a Fusion -> Manual Alignment dialog: every angle's CH4 threshold mask
overlaid at once with distinct colors/opacity, a selector for the active
angle, arrow keys to nudge translation and Q/E to nudge rotation, an Auto
De-rotate button that snaps to the known scan angle, and Save/Clear
controls backed by a JSON sidecar that's restored automatically on reload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The left panel was a plain fixed-width widget sized by the window, so once
the Scan Info notes grew past the window height Qt compressed every control
below its minimum: the DC threshold spin box was sliced in half, the ADC
hint collided with it, and the cache note was cut off mid-line. Word-wrapped
labels made it worse by advertising a single-line minimum height, which
clipped the filename and the format/cache notes.
- Put both side panels in fixed-width QScrollAreas so a short window scrolls
instead of crushing the controls.
- Add _wrap_label(), which enables height-for-width so wrapped labels report
their real height; use it for the filename, Scan Info rows, format/cache
notes, DC-precompute status, ADC hint and ROI readouts.
- Replace the ad-hoc QHBoxLayout label/field rows with a shared QFormLayout
(Angle, Channel, DC threshold, Grating size, Colormap, min, max) so labels
align and fields keep a usable minimum width.
- Add _group() for consistent group-box margins; widen the left column
260 -> 288 px so the channel combo and checkbox text stop eliding.
- Keep the canvases usable: non-collapsible splitter, minimum canvas heights
and a 960x560 window minimum.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects found by testing the parallel paths end to end.
Nested parallelism was unbounded. DcPrecomputeWorker and the alignment
driver both parallelise over angles, and each angle's compute_dc_image
then parallelised over rows again: 14 x 14 threads, and — worse — every
angle sized its chunk against the *whole* memory budget, so worst-case
live buffers were ~13.9 GB against a 1 GB budget. Capping the inner
worker count alone does not fix it; the chunk is still sized against the
full budget. plan_angle_level now hands each angle both max_workers=1
and its share of the budget, which bounds the real file at 1002 MB.
Chunk planning could not produce concurrency at all. _plan_chunks sized
chunk_rows first, and _chunk_rows_for spends whatever budget it is given
— so a single chunk consumed the lot and budget//chunk_bytes came back
as one worker. It now picks the worker count first and sizes the chunk to
it, giving 7-16 workers on the real file instead of 1.
Cancellation was too coarse to shut down. stop() was only checked between
angles, and one angle of the real scan is ~40 s, so closing the window
sat through it and returned with threads still running. should_stop is
now polled per row chunk and closeEvent signals every worker before
waiting: close went from 4.0 s with two live threads to 1.04 s with both
finished. A cancelled ComputeWorker emits None so a half-filled image is
never cached.
Also: BatchCacheWorker no longer reports every file as failed when the
process pool itself dies (unguarded __main__, frozen build, sandbox) —
it retries those files in-process. Batches below 512 MB skip the pool
entirely, since interpreter startup would otherwise make small jobs
slower. cache_file rejects an unknown mode instead of silently treating
it as "fft".
Measured on the real 496 GB scan, angle 3, 32 rows:
DC 2.94s -> 0.75s RF 6.10s -> 1.99s peak RSS 1.40 -> 1.93 GB
Batch convert, 24 files / 1 GB: 4.03s -> 2.36s.
Memory budget defaults to 1024 MB (SRAS_MEM_BUDGET_MB), the measured knee.
Testing: tools/test_refactor.py (70 checks: cache round-trip and block
carry-forward, partial-cache handling, parallel-vs-serial identity,
no-mask path, ROI mask vs full-grid, v2/v3/v4 parsing, sras_average
round-trip incl. remainder handling) and tools/test_gui.py (46 checks
driving the real widgets and workers headlessly). check_equivalence.py
still reports all 167 outputs identical to ed0eba4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Structure
sras_format.py parsing/writing, calibration, axes (numpy + struct)
sras_compute.py DC/FFT images, alignment, batch cache (+ scipy)
sras_workers.py Qt background workers
sras_viewer.py ROI, canvases, dialog, main window
format+compute import in 0.59s with no Qt or matplotlib (vs 2.96s for the
full app), which is what makes a spawn-based process pool worth using.
Deduplication
- SrasFile.cal() and compute.dc_image_mv() replace the ADC->mV
calibration incantation that appeared at ten call sites.
- _read_preambles/_read_background/_set_calibration are shared by the
legacy and v6 parsers instead of duplicated.
- v5 PREC images are normalised into the same ragged per-angle list
v6/v7 uses, removing four isinstance() branches and shortening
compute_rf_image's fast path.
- _run_worker replaces five copies of the QThread setup and five
near-identical teardown methods; the two lifetime hazards they
guarded against are now documented once, authoritatively.
- _on_channel_changed defers to _update_controls_enabled rather than
re-deriving the same six enable rules.
- _corners_in_ref_frame, _CHANNEL_DISPLAY dict, unified progress-dialog
helper, dead _on_roi_angle_edited stub removed.
- sras_average.py builds on SrasFile instead of carrying a second copy
of the header format, and streams per angle instead of loading the
whole file (was a ~3x file-size peak).
Executable lines: 2390 -> 2254, despite adding all of the below.
Parallelism
- compute_rf_image/compute_dc_image map row chunks over a thread pool.
Worker count is derived from the memory budget rather than the core
count: on a large scan chunk_rows is already floored at 1 row (~75 MB
of float32 at 7507x2500), so only the worker count can bound peak RAM.
- DcPrecomputeWorker computes angles on a pool, emitting each result
from its own QThread as it lands.
- BatchCacheWorker runs one process per file via cache_file(); only
paths and scalars cross the boundary. Per-process thread counts are
divided so the two levels don't oversubscribe. Drops the old pass 1,
which fully parsed every file just to weight a progress bar.
- dc_threshold_mv=None means "no mask" and skips the CH4 read entirely
— the batch FFT job previously read all of CH4 to compare against a
threshold of -1e9.
- Alignment mask and correlation stages map over angles; fft2/ifft2 use
workers=-1.
Fixes found on the way
- A partially-cached v5 file showed an all-zero image for uncached
angles: the fast-path check was file-wide, not per-angle.
- v7 cached images were read-only big-endian views; now native float32.
Verified: tools/check_equivalence.py produces byte-identical hashes for
167 outputs (DC/FFT across channels, angles, bg-sub, pad factors and
thresholds, plus the full alignment result) against ed0eba4, on both
synthetic files and a real 496 GB 17-angle v6 scan.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tools/make_test_sras.py writes small v6 files with per-angle-varying
geometry, distinct per-channel calibration, and deterministic waveform
content (predictable FFT peak per pixel, predictable DC mean per pixel).
tools/check_equivalence.py hashes DC/FFT/alignment outputs across
channels, angles, bg-sub, pad factors, and thresholds. It imports from
either the monolith or the split modules, so the same script captures
both sides of a refactor. Hashes canonicalise to native float64 so a
dtype or byte-order change that preserves values is not a false failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scans come off the scope as v6; a new trailing CACH section (independent
SDCB/SFFT blocks, sized per-angle from the existing geometry table) lets
computed DC and FFT images be cached in place, bumping the file to v7 the
first time either is stored. The Convert menu's "Batch Compute DC and
Store"/"Batch Compute FFT and Store" actions run this across multiple
files in the background. compute_rf_image and the DC compute workers now
reuse cached v7 data instead of recomputing it.
Removes "Pre-process and Save as v5", which never supported v6 sources
and is superseded by in-place v7 caching. Legacy v2-v5 reading, including
v5's PREC section, is unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a background worker that aligns every scan angle onto one shared,
zero-padded canvas using a rigid transform only (no scaling): rotation
is taken analytically from the known scan angle, and only the residual
translation is found via FFT phase correlation of each angle's
binarized CH4 dc-mask. An "Aligned View" toggle then redisplays the
currently selected angle/channel resampled onto that shared canvas,
with ROI, CSV export, and waveform-click inspection all kept working.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the FFT was always run unmasked (dc_threshold_mv=-1e9) so the
result could be cached independent of the threshold, with masking applied
afterward as a display-time step. That meant every pixel's CH1 waveform
was read and FFT'd even when most of the scan is background — wasted work
now that the DC map (and its threshold mask) is already known ahead of
time via the DC precompute cache.
compute_rf_image now takes the real threshold and an optional precomputed
dc4_mv array (reused from the DC cache, avoiding a redundant CH4 read),
and skips the FFT for masked-out pixels entirely, same as before this
threshold-caching detour was introduced. The masked-out CH1 samples are
also never read from disk: the boolean valid-pixel mask is applied to the
raw memmap slice before any dtype conversion, so numpy only pages in the
bytes for pixels that pass the threshold.
Threshold is therefore back to being part of the FFT cache key (changing
it now decides which pixels get computed at all, so it can't be satisfied
from a cache built for a different threshold) — but the recompute it
triggers reuses the cached DC4 image and skips both the FFT and the I/O
for masked pixels, so it's much cheaper than the original full-image
compute. Grating and colormap remain pure post-processing with no
recompute.
Verified bit-identical output against the un-optimized reference path on
synthetic data (including scattered, non-row-aligned masking), and timed
on the real 532 GB / 17-angle file: angle 0's FFT (75.2% of pixels above
the default 50 mV threshold) went from ~121s (fully unmasked) to 114s
(FFT skipped, I/O not skipped) to 90.2s with this change (both skipped).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Colormap, DC threshold, velocity grating, and FFT pad-factor changes were
all routing through a full recompute, which is unworkable on large v6
scans where a single angle's FFT can take minutes. These are now applied
as cheap post-processing on cached data instead:
- ComputeWorker computes CH1/Velocity as an unmasked raw FFT plus its DC4
mask image, cached per (angle, bg_sub, n_fft). Threshold masking and
grating scaling are applied to the cached array on redraw, so neither
needs a recompute; colormap changes only touch matplotlib.
- New DcPrecomputeWorker fills a per-angle DC (CH3/CH4) cache in the
background right after load, since DC images are cheap (mean, no FFT)
and make switching angles instant once cached. Progress is shown in the
Scan Info panel.
- New _refresh_display()/_show_image_now() route every settings-changed
handler through the cache first, falling back to the existing threaded
_start_compute() (with a progress popup, now FFT- vs DC-specific) only
on a genuine cache miss.
- File load now defaults to a DC channel instead of CH1/FFT, so opening a
large file shows something in seconds instead of minutes.
Verified against a real 532 GB / 17-angle file: DC compute for one angle
takes ~100s+ (I/O-bound over USB) the first time, but switching to an
already-cached angle takes 0.4s with no compute thread spun up at all.
Audited every widget signal connection for this pass; the spinboxes
already correctly used editingFinished rather than valueChanged — the
recompute-on-every-change bug was in the handlers, not the event type.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause: compute_dc_image/compute_rf_image chunked processing by a fixed
32-row count, sized for old small-format scans. A real v6 file with
spf=2500 and up to ~7500 frames/angle needed ~6-7 GB for a single chunk's
core buffers, on a 17 GB machine — pushing past available memory on every
angle and aborting the process (not a clean MemoryError) when switching to
a new angle piled more allocations on top. Chunk size is now computed from
actual scan dimensions to hit a fixed ~128 MB budget instead of a fixed row
count, cutting peak footprint by roughly 16-18x (verified against the real
532 GB / 17-angle dataset: ~400 MB peak, no crash).
Also hardens QThread lifecycle handling, found while chasing this crash:
- _on_compute_thread_finished/_on_load_thread_finished/
_on_preprocess_thread_finished now call wait() before dropping the last
reference to a finished QThread, avoiding "QThread: Destroyed while
thread is still running" aborts if the OS thread hasn't fully joined
when finished() fires.
- _start_compute/_load_file/_on_preprocess now claim their QThread
immediately after the busy-guard check, before any call that can pump
the Qt event loop (e.g. first QProgressDialog.show()), closing a
reentrancy window where a second call could start a thread that the
first call's own assignment would then clobber mid-run.
- faulthandler is enabled at startup so any future native crash prints a
real stack trace instead of a bare "Aborted".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drops the legacy Time Gate FFT-windowing feature and the SAW matched-filter
pipeline (SawPipeline, SawDiagnosticWindow, TemplateBuildWorker, and the
associated UI panels/channel modes) to simplify the viewer.
Adds native support for the SRAS v6 file format (scan_format.md), which
scans a different bounding box per angle instead of a uniform AABB. Scan
geometry (n_rows, n_frames, x_start) is now exposed per-angle in SrasFile,
with v2-v5 files populating those arrays uniformly so both formats share
one code path. Waveform data is memory-mapped per angle to handle v6's
ragged layout, and aborted/truncated v6 scans are handled gracefully.
"Pre-process and Save as v5" is disabled for v6 files since the flat v5
layout can't represent per-angle geometry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Thomas Ales [M S E]
tka