Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 48d560fb00 | |||
| 2937c017a7 | |||
| a8b962e317 | |||
| efecc154fe | |||
| ec0d322871 | |||
| aca501ae5c | |||
| 105b9514a5 | |||
| 4212b8c313 | |||
| 6976cbf767 | |||
| c54cce453c | |||
| 1caf6373cb | |||
| 1a8187138f |
+66
-37
@@ -32,45 +32,58 @@ share of the budget. Capping the workers alone is not enough: the chunk would
|
|||||||
still be sized against the whole budget, and N concurrent callers would each
|
still be sized against the whole budget, and N concurrent callers would each
|
||||||
allocate all of it.
|
allocate all of it.
|
||||||
|
|
||||||
## FFT peak search: block-parallel zoom refinement (`sras_compute.py`)
|
## FFT peak search: block-parallel direct transform (`sras_compute.py`)
|
||||||
|
|
||||||
The displayed RF value per pixel is the argmax of the zero-padded power
|
The displayed RF value per pixel is the argmax of the zero-padded power
|
||||||
spectrum of that pixel's CH1 waveform. At the pad factor of 40 needed for
|
spectrum of that pixel's CH1 waveform. This used to run through `_peak_bins_zoom`,
|
||||||
mapping resolution, materialising padded spectra is hopeless: ~9 GB per scan
|
a coarse-rfft-plus-local-fine-DFT refinement that avoided ever materialising
|
||||||
row, which is what used to collapse the old row-chunk planner to one worker
|
a padded spectrum — at the pad factor of 40 needed for mapping resolution, a
|
||||||
and make synthesis single-threaded.
|
full padded spectrum is ~9 GB per scan row, which used to collapse the old
|
||||||
|
row-chunk planner to one worker and make synthesis single-threaded. That
|
||||||
|
refinement was removed once pyFFTW became the sole, mandatory FFT backend
|
||||||
|
(SciPy dropped as a compute backend entirely): `_peak_bins` now always runs
|
||||||
|
the real full transform, and the memory problem zoom dodged is instead
|
||||||
|
solved by bounding *per-block* spectrum memory rather than avoiding full
|
||||||
|
spectra altogether.
|
||||||
|
|
||||||
`_peak_bins_zoom` never materialises the padded spectrum:
|
`_peak_bins` runs the full transform (`_block_rfft`, a cached pyFFTW
|
||||||
|
`builders.rfft` plan, FFTW_MEASURE, wisdom persisted under
|
||||||
|
`~/.cache/sras-viewer/`) and argmaxes the power spectrum, in blocks fanned
|
||||||
|
out task-parallel over a persistent thread pool (`_fft_pool()`) — each pool
|
||||||
|
thread runs one single-threaded transform at a time, so aggregate
|
||||||
|
parallelism equals the pool's worker count. This block-streaming structure
|
||||||
|
is not just about parallelism: it is also what keeps memory bounded, by
|
||||||
|
never materialising more than one block's worth of full padded spectrum at
|
||||||
|
a time, regardless of how many waveforms a chunk holds.
|
||||||
|
|
||||||
1. a coarse rfft at `next_fast_len(2*spf)` — 2× oversampled, so the padded
|
The block size is the part that has to adapt to pad factor.
|
||||||
power spectrum (a trig polynomial of degree spf−1) cannot hide its global
|
`_fft_block_for(spf, n_len)` derives waveforms-per-task from a fixed
|
||||||
max between coarse samples;
|
per-thread byte budget (`_FFT_PLAN_BYTES_BUDGET`, `SRAS_FFT_PLAN_BUDGET_MB`,
|
||||||
2. every coarse bin within `_ZOOM_CAND_RATIO` (0.7) of its row's coarse max
|
default 16 MB) rather than a fixed constant, because a cached pyFFTW plan's
|
||||||
becomes a refinement candidate. Quarter-natural-bin scalloping at the 2×
|
input+output buffers are *permanent* per-thread memory (the plan cache is
|
||||||
grid can understate a peak's power by at most ~19%, so 0.7 keeps a wide
|
never evicted) — with a fixed 512-waveform block, pad 40 at a 2500-sample
|
||||||
margin. The DC-adjacent window is always refined too: the coarse DC bin
|
frame costs ~210 MB per pool thread (~3.3 GB total across 16 threads);
|
||||||
is zeroed for suppression, which would otherwise blind the scan to fine
|
`_fft_block_for` bounds that to ~16 MB per thread (~260 MB across 16
|
||||||
bins closer to DC than the first coarse sample (where the leakage skirt
|
threads) at the same pad factor, while still reproducing the old tuned 512
|
||||||
of an un-subtracted offset peaks);
|
exactly at natural resolution (pad 1), where it cost nothing to begin with.
|
||||||
3. each candidate window (±`_ZOOM_HALFWIDTH` = 0.75 coarse spacings; every
|
`_FFT_BLOCK_MAX` (512) and `_FFT_BLOCK_MIN` (32) cap and floor the result:
|
||||||
fine bin lies within 0.5 spacings of its nearest coarse bin) is evaluated
|
the ceiling is the measured knee on a 16-core machine at natural resolution
|
||||||
on the exact `n_fft` grid by one small complex gemm, with np.argmax's
|
(smaller blocks serialise on GIL-held numpy dispatch, larger ones lose cache
|
||||||
lowest-bin tie-break preserved across windows.
|
residency and task granularity); the floor keeps task granularity from
|
||||||
|
collapsing at extreme pad factors, at the cost of exceeding the byte budget
|
||||||
|
there.
|
||||||
|
|
||||||
The selected bin is bit-identical to the full padded argmax — enforced by
|
The outer row-chunk sizing (`_plan_fft_rows`) needed no companion change.
|
||||||
`tests/test_compute.py::test_zoom_identity`, a fuzz test over adversarial
|
It only ever budgets the raw float32 waveform *read* buffer, which this
|
||||||
spectra, and the golden-hash harness (`tools/check_equivalence.py`), whose
|
change doesn't touch — spectrum memory is bounded independently by
|
||||||
baseline was captured on the old full-padded path.
|
`_fft_block_for`, and since the pool only ever runs as many blocks
|
||||||
|
concurrently as it has workers, peak transient spectrum memory during a
|
||||||
|
chunk's FFT phase is `_MAX_WORKERS * block * bytes_per_wf`, the same bound
|
||||||
|
whether the chunk holds 10 rows or 10,000. Queuing more rows into one chunk
|
||||||
|
to keep the read-row pool busy therefore can't blow up spectrum memory.
|
||||||
|
|
||||||
Work fans out over a persistent thread pool in `_FFT_BLOCK` = 512-waveform
|
`tools/check_equivalence.py`'s golden-hash harness remains the end-to-end
|
||||||
tasks: smaller blocks serialise on GIL-held numpy dispatch, larger ones lose
|
regression baseline for this path, unaffected by this change.
|
||||||
cache residency and task granularity (measured on a 16-core machine, where
|
|
||||||
this path runs ~35× faster than the old serial padded transform at pad 40).
|
|
||||||
pyFFTW runs through per-thread `builders` plans (FFTW_MEASURE, wisdom
|
|
||||||
persisted under `~/.cache/sras-viewer/`), and `threadpoolctl` clamps BLAS to
|
|
||||||
one thread under the pool so the refinement gemm cannot oversubscribe.
|
|
||||||
`compute_rf_image(exact=True)` (or `SRAS_FFT_EXACT=1`) keeps the reference
|
|
||||||
full-padded path for audits.
|
|
||||||
|
|
||||||
## Row-averaged FFT: same-row, distance-weighted SNR cleanup (`sras_compute.py`)
|
## Row-averaged FFT: same-row, distance-weighted SNR cleanup (`sras_compute.py`)
|
||||||
|
|
||||||
@@ -121,8 +134,8 @@ same edge case the masked convolution already handles.
|
|||||||
The averaging step doubles the live per-row scratch memory (a full-width
|
The averaging step doubles the live per-row scratch memory (a full-width
|
||||||
`(n_frames, spf)` buffer on top of the existing compacted `waves` buffer),
|
`(n_frames, spf)` buffer on top of the existing compacted `waves` buffer),
|
||||||
so `compute_rf_image` halves its byte budget when `row_avg_n > 0` before
|
so `compute_rf_image` halves its byte budget when `row_avg_n > 0` before
|
||||||
`_plan_fft_rows`/the exact-path sizing runs — see "Memory budget and row
|
`_plan_fft_rows` runs — see "Memory budget and row chunking" above. On the
|
||||||
chunking" above. On the largest real scans `_plan_chunks` is already
|
largest real scans `_plan_chunks` is already
|
||||||
clamped to its floor of one row regardless, so this costs no concurrency
|
clamped to its floor of one row regardless, so this costs no concurrency
|
||||||
where it matters most; it mainly protects moderate-sized scans from an
|
where it matters most; it mainly protects moderate-sized scans from an
|
||||||
unexpected regression.
|
unexpected regression.
|
||||||
@@ -144,6 +157,18 @@ by `cached_rf_image` before it hands the image back. Getting this wrong is
|
|||||||
not a slow display, it is a *wrong* display, which is why the check is a
|
not a slow display, it is a *wrong* display, which is why the check is a
|
||||||
single predicate in one place rather than spread across callers.
|
single predicate in one place rather than spread across callers.
|
||||||
|
|
||||||
|
The min peak frequency floor is the fourth recorded setting, and it sits
|
||||||
|
between the DC threshold (fully re-applicable at display time) and the
|
||||||
|
baked-in three: it is *tighten-only* re-applicable. A floor at or above the
|
||||||
|
stored one is served by masking stored pixels below it to the 0.0 "no valid
|
||||||
|
peak" sentinel — invalidated, not re-resolved; only a real recompute can
|
||||||
|
find their true above-floor peak. A floor *below* the stored one is a
|
||||||
|
genuine mismatch: the stored search never looked at those bins. Because a
|
||||||
|
served-then-masked image is a lossy stand-in for a real floored compute,
|
||||||
|
`cache_file` passes `use_stored=False` so a batch recompute always runs the
|
||||||
|
real FFT — otherwise re-batching a cached file would silently bake the
|
||||||
|
masked copy of its own old image back into the store.
|
||||||
|
|
||||||
Padding is the subtlest of the three, because a padded FFT looks like it
|
Padding is the subtlest of the three, because a padded FFT looks like it
|
||||||
should be a refinement of the unpadded one. It isn't: zero-padding
|
should be a refinement of the unpadded one. It isn't: zero-padding
|
||||||
interpolates between the natural bins, so it resolves a different peak
|
interpolates between the natural bins, so it resolves a different peak
|
||||||
@@ -170,8 +195,12 @@ recompute. The *display* path (`_stored_fft_image`) never asks
|
|||||||
bg-sub/pad/row-averaging controls — it asks whether the image matches its
|
bg-sub/pad/row-averaging controls — it asks whether the image matches its
|
||||||
*own* recorded settings (`sras.precomputed_bg_sub`/`precomputed_pad_factor`/
|
*own* recorded settings (`sras.precomputed_bg_sub`/`precomputed_pad_factor`/
|
||||||
`precomputed_row_avg_n`), which is always true whenever a stored image
|
`precomputed_row_avg_n`), which is always true whenever a stored image
|
||||||
exists. So presence alone decides whether it's shown; the live controls
|
exists. Two settings are taken live instead: the DC threshold and the min
|
||||||
never gate it. They still matter for two things: a genuinely never-computed
|
peak frequency floor, both cheaply re-appliable as masks on serve. So for
|
||||||
|
bg-sub/pad/row-avg, presence alone decides whether a stored image is shown;
|
||||||
|
the floor is the one case where a live control can gate it — a live floor
|
||||||
|
*below* the stored one can't be answered by masking, so `cached_rf_image`
|
||||||
|
declines and the caller falls through to a real compute. They still matter for two things: a genuinely never-computed
|
||||||
angle's first live compute, and an explicit batch recompute — both of which
|
angle's first live compute, and an explicit batch recompute — both of which
|
||||||
read the live controls and produce new stored data, at which point it's the
|
read the live controls and produce new stored data, at which point it's the
|
||||||
new data's *own* settings that get self-matched from then on. This is what
|
new data's *own* settings that get self-matched from then on. This is what
|
||||||
|
|||||||
+2
-3
@@ -14,10 +14,8 @@ dependencies = [
|
|||||||
"scipy==1.18.0",
|
"scipy==1.18.0",
|
||||||
# Angle alignment only: masked FFT phase correlation (skimage.registration).
|
# Angle alignment only: masked FFT phase correlation (skimage.registration).
|
||||||
"scikit-image==0.26.0",
|
"scikit-image==0.26.0",
|
||||||
# Faster rfft backend; the viewer falls back to scipy.fft without it.
|
# Mandatory rfft backend for the peak search (no SciPy fallback).
|
||||||
"pyFFTW==0.15.1",
|
"pyFFTW==0.15.1",
|
||||||
# Clamps BLAS threading under the FFT worker pool.
|
|
||||||
"threadpoolctl==3.6.0",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
@@ -30,6 +28,7 @@ sras-viewer = "sras_viewer.main_window:main"
|
|||||||
py-modules = [
|
py-modules = [
|
||||||
"sras_format",
|
"sras_format",
|
||||||
"sras_compute",
|
"sras_compute",
|
||||||
|
"sras_render",
|
||||||
"sras_workers",
|
"sras_workers",
|
||||||
"sras_align_export",
|
"sras_align_export",
|
||||||
"sras_average",
|
"sras_average",
|
||||||
|
|||||||
+31
-13
@@ -255,7 +255,7 @@ actions.
|
|||||||
| Offset | Size | Type | Field | Description |
|
| Offset | Size | Type | Field | Description |
|
||||||
|--------|------|------|-------|-------------|
|
|--------|------|------|-------|-------------|
|
||||||
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
|
||||||
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `3`; readers also accept `1` and `2` (each older tail simply lacks the fields added since — see the `SFFT` block below and [CACH tail version history](#cach-tail-version-history)). Any other value → treat the file as uncached (unlike v5's `PREC` section, which read but never validated its version byte). |
|
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `4`; readers also accept `1`–`3` (each older tail simply lacks the fields added since — see the `SFFT` block below and [CACH tail version history](#cach-tail-version-history)). Any other value → treat the file as uncached (unlike v5's `PREC` section, which read but never validated its version byte). |
|
||||||
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |
|
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |
|
||||||
|
|
||||||
### DC block `SDCB` (present iff `block_flags & 0x01`)
|
### DC block `SDCB` (present iff `block_flags & 0x01`)
|
||||||
@@ -292,13 +292,16 @@ never shifts an existing offset:
|
|||||||
- **`cach_version` 1**: 7 bytes, format `">4sBH"` — magic, flags, n_stored.
|
- **`cach_version` 1**: 7 bytes, format `">4sBH"` — magic, flags, n_stored.
|
||||||
- **`cach_version` 2**: 8 bytes, format `">4sBHB"` — + `row_avg_n`.
|
- **`cach_version` 2**: 8 bytes, format `">4sBHB"` — + `row_avg_n`.
|
||||||
- **`cach_version` 3**: 10 bytes, format `">4sBHBH"` — + `pad_factor`.
|
- **`cach_version` 3**: 10 bytes, format `">4sBHBH"` — + `pad_factor`.
|
||||||
|
- **`cach_version` 4**: 14 bytes, format `">4sBHBHI"` — + `min_freq_khz`.
|
||||||
Always written by current code.
|
Always written by current code.
|
||||||
|
|
||||||
An older tail is read with its absent fields taken as the only value such a
|
An older tail is read with its absent fields taken as the only value such a
|
||||||
tail can describe: `row_avg_n = 0` for a `cach_version` 1 tail, which
|
tail can describe: `row_avg_n = 0` for a `cach_version` 1 tail, which
|
||||||
predates row-averaged FFT caching, and `pad_factor = 1` for `cach_version`
|
predates row-averaged FFT caching, `pad_factor = 1` for `cach_version`
|
||||||
1 or 2, which predate padded caching and are therefore natural-resolution.
|
1 or 2, which predate padded caching and are therefore natural-resolution,
|
||||||
Files cached before either change keep working with no recompute.
|
and `min_freq_khz = 0` (no floor) for `cach_version` 1–3, which predate the
|
||||||
|
min peak frequency floor and therefore searched every bin above DC.
|
||||||
|
Files cached before any of these changes keep working with no recompute.
|
||||||
|
|
||||||
| Offset (rel) | Size | Type | Field | Description |
|
| Offset (rel) | Size | Type | Field | Description |
|
||||||
|--------------|------|------|-------|-------------|
|
|--------------|------|------|-------|-------------|
|
||||||
@@ -306,7 +309,8 @@ Files cached before either change keep working with no recompute.
|
|||||||
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bit 1 = `row_averaged` — `peak_freq_mhz` came from same-row, distance-weighted averaged CH1 waveforms rather than raw per-pixel ones; `row_avg_n` (below) is the neighbor half-width used. Bits 2–7 reserved. |
|
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bit 1 = `row_averaged` — `peak_freq_mhz` came from same-row, distance-weighted averaged CH1 waveforms rather than raw per-pixel ones; `row_avg_n` (below) is the neighbor half-width used. Bits 2–7 reserved. |
|
||||||
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
|
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
|
||||||
| 7 | 1 | `u8` | `row_avg_n` | *`cach_version` ≥ 2 only.* Same-row neighbor half-width, in pixels, that `peak_freq_mhz` was averaged over before its FFT; `0` = raw (unaveraged). Meaningful only when `flags` bit 1 is set — a `cach_version` 1 tail has no such byte and is always `row_avg_n = 0`. |
|
| 7 | 1 | `u8` | `row_avg_n` | *`cach_version` ≥ 2 only.* Same-row neighbor half-width, in pixels, that `peak_freq_mhz` was averaged over before its FFT; `0` = raw (unaveraged). Meaningful only when `flags` bit 1 is set — a `cach_version` 1 tail has no such byte and is always `row_avg_n = 0`. |
|
||||||
| 8 | 2 | `u16` | `pad_factor` | *`cach_version` 3 only.* Zero-padding factor the stored `peak_freq_mhz` was resolved at: `n_fft = pad_factor × samples_per_frame`, so `1` = natural resolution. Never `0`; a `cach_version` 1 or 2 tail has no such field and is always `pad_factor = 1`. |
|
| 8 | 2 | `u16` | `pad_factor` | *`cach_version` ≥ 3 only.* Zero-padding factor the stored `peak_freq_mhz` was resolved at: `n_fft = pad_factor × samples_per_frame`, so `1` = natural resolution. Never `0`; a `cach_version` 1 or 2 tail has no such field and is always `pad_factor = 1`. |
|
||||||
|
| 10 | 4 | `u32` | `min_freq_khz` | *`cach_version` ≥ 4 only.* Min peak frequency floor the stored peak search excluded bins below, fixed-point in units of 0.001 MHz (kHz); `0` = no floor. Fixed-point rather than `f32` so a value that round-trips through the file compares exactly against the same value re-requested by a reader (the viewer's floor control has 0.001 MHz granularity). A `cach_version` 1–3 tail has no such field and is always `min_freq_khz = 0`. |
|
||||||
|
|
||||||
followed by `n_stored` entries, each:
|
followed by `n_stored` entries, each:
|
||||||
|
|
||||||
@@ -341,14 +345,27 @@ Readers must fall back to real-time FFT computation (ignoring stored
|
|||||||
the reader is asking for: time-domain gating is active, the reader's
|
the reader is asking for: time-domain gating is active, the reader's
|
||||||
requested `n_fft` doesn't equal `pad_factor × samples_per_frame`, the
|
requested `n_fft` doesn't equal `pad_factor × samples_per_frame`, the
|
||||||
reader's background-subtraction setting doesn't match
|
reader's background-subtraction setting doesn't match
|
||||||
`flags.bg_sub_applied`, or the reader's requested `row_avg_n` doesn't match
|
`flags.bg_sub_applied`, the reader's requested `row_avg_n` doesn't match
|
||||||
the stored value exactly. A raw request must never be served a row-averaged
|
the stored value exactly, or the reader's requested min peak frequency
|
||||||
store, or vice versa; a request at one row-averaging window size must never
|
floor is *below* the stored `min_freq_khz`. A raw request must never be
|
||||||
be served a store at another; and a request at one padding must never be
|
served a row-averaged store, or vice versa; a request at one row-averaging
|
||||||
served a store at another, since a padded FFT interpolates between the
|
window size must never be served a store at another; and a request at one
|
||||||
natural bins and so resolves genuinely different peak frequencies. An
|
padding must never be served a store at another, since a padded FFT
|
||||||
`n_fft` that is not a whole multiple of `samples_per_frame` can never match
|
interpolates between the natural bins and so resolves genuinely different
|
||||||
any store, because only an integer `pad_factor` is representable.
|
peak frequencies. An `n_fft` that is not a whole multiple of
|
||||||
|
`samples_per_frame` can never match any store, because only an integer
|
||||||
|
`pad_factor` is representable.
|
||||||
|
|
||||||
|
The min peak frequency floor is the one asymmetric provenance field. A
|
||||||
|
request at a floor *below* the stored one cannot be served: the stored
|
||||||
|
search never looked at bins below its floor, so the stored numbers cannot
|
||||||
|
say what a lower-floored search would have found. A request at a floor at
|
||||||
|
or *above* the stored one **is** servable — the difference is re-applied at
|
||||||
|
display time by masking every pixel whose stored `peak_freq_mhz` is below
|
||||||
|
the requested floor to `0` (the same sentinel as the DC threshold mask;
|
||||||
|
a genuine peak can never be `0`, since bin 0 is always excluded from the
|
||||||
|
search). Such masked pixels are *invalid*, not re-resolved — only a real
|
||||||
|
recompute can recover the strongest peak above the floor for them.
|
||||||
|
|
||||||
### In-place write ordering
|
### In-place write ordering
|
||||||
|
|
||||||
@@ -374,6 +391,7 @@ which has stayed `7` since the Cache Tail was introduced — this is the inner
|
|||||||
| 1 | Initial Cache Tail: `SDCB` (DC) and `SFFT` (FFT, 7-byte header) blocks. |
|
| 1 | Initial Cache Tail: `SDCB` (DC) and `SFFT` (FFT, 7-byte header) blocks. |
|
||||||
| 2 | `SFFT` header grows one byte, `row_avg_n` — the same-row neighbor half-width the stored `peak_freq_mhz` was averaged over before its FFT, `0` = raw. Readers still accept a `cach_version` 1 tail, treated as `row_avg_n = 0` for every angle it stores, so files cached before this change keep working without a recompute. |
|
| 2 | `SFFT` header grows one byte, `row_avg_n` — the same-row neighbor half-width the stored `peak_freq_mhz` was averaged over before its FFT, `0` = raw. Readers still accept a `cach_version` 1 tail, treated as `row_avg_n = 0` for every angle it stores, so files cached before this change keep working without a recompute. |
|
||||||
| 3 | `SFFT` header grows a `u16` `pad_factor` — the zero-padding factor the stored `peak_freq_mhz` was resolved at, `1` = natural resolution. Before this, a padded view could never use a stored cache at all (the store was pad 1 by definition and readers rejected any `n_fft ≠ samples_per_frame`), so a user working at a pad factor got no benefit from batch-computing a file. Recording the factor lets such a view be served, while still refusing a store resolved at a *different* pad. Readers accept `cach_version` 1 and 2 tails as `pad_factor = 1`. |
|
| 3 | `SFFT` header grows a `u16` `pad_factor` — the zero-padding factor the stored `peak_freq_mhz` was resolved at, `1` = natural resolution. Before this, a padded view could never use a stored cache at all (the store was pad 1 by definition and readers rejected any `n_fft ≠ samples_per_frame`), so a user working at a pad factor got no benefit from batch-computing a file. Recording the factor lets such a view be served, while still refusing a store resolved at a *different* pad. Readers accept `cach_version` 1 and 2 tails as `pad_factor = 1`. |
|
||||||
|
| 4 | `SFFT` header grows a `u32` `min_freq_khz` — the min peak frequency floor the stored peak search excluded bins below, in 0.001 MHz units, `0` = no floor. The floor exists because a pixel that passes the DC-bias threshold but carries only weak real signal can otherwise resolve to the un-subtracted background's DC-leakage skirt — an implausibly-near-zero frequency (and so an implausibly slow velocity) for a pixel that has a genuine peak higher up. Recording the floor is what makes it enforceable against a store: without it, a stored image silently bypassed the floor entirely. Unlike the other provenance fields it is asymmetric — a *higher* requested floor is servable by masking stored pixels below it, only a *lower* one forces a recompute (see above). Readers accept `cach_version` 1–3 tails as `min_freq_khz = 0`. |
|
||||||
|
|
||||||
A reader that does not know a `cach_version` must treat the file as
|
A reader that does not know a `cach_version` must treat the file as
|
||||||
uncached — not attempt a partial parse — and the file still reads as an
|
uncached — not attempt a partial parse — and the file still reads as an
|
||||||
|
|||||||
+163
-63
@@ -3,7 +3,23 @@
|
|||||||
sras_average.py — Waveform-averaging utility for .sras files.
|
sras_average.py — Waveform-averaging utility for .sras files.
|
||||||
|
|
||||||
Reduces memory footprint by coherently averaging every N consecutive frames
|
Reduces memory footprint by coherently averaging every N consecutive frames
|
||||||
along the acquisition axis, writing a new .sras file with n_frames / N frames.
|
along the acquisition axis, writing a new v6 .sras file with (about)
|
||||||
|
n_frames / N frames per angle.
|
||||||
|
|
||||||
|
Handles v6/v7 only (per-angle ragged geometry). A v7 input's cache tail is
|
||||||
|
dropped — it's indexed by frame, which this changes — so the output is always
|
||||||
|
written as v6; the viewer recomputes DC/FFT on next open. Reads via memmap and
|
||||||
|
writes in row-chunks sized to a memory budget (default 1024 MB, override with
|
||||||
|
the SRAS_MEM_BUDGET_MB env var), so peak RAM stays bounded regardless of file
|
||||||
|
size — this is what makes the tool usable on multi-hundred-GB scans.
|
||||||
|
|
||||||
|
Averaging every N frames also coarsens the physical X spacing between output
|
||||||
|
frames (each frame is a distinct stage position — see scan_format.md's
|
||||||
|
Spatial Mapping section: x_k = x_start + k * velocity_mm_s / laser_freq_hz) —
|
||||||
|
so the output header's laser_freq_hz is divided by N to keep x_axis_mm()
|
||||||
|
correct on the averaged file. This means the GUI's "Laser freq" info label
|
||||||
|
will show that adjusted value rather than the scope's real setting for an
|
||||||
|
averaged file; v6/v7 has no separate field for effective pixel pitch.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
python sras_average.py input.sras output.sras --n 10
|
python sras_average.py input.sras output.sras --n 10
|
||||||
@@ -13,9 +29,15 @@ Options:
|
|||||||
--n INT Number of frames to average into one (required).
|
--n INT Number of frames to average into one (required).
|
||||||
--discard-remainder Drop trailing frames that don't fill a complete group.
|
--discard-remainder Drop trailing frames that don't fill a complete group.
|
||||||
Default: include a partial average for the last group.
|
Default: include a partial average for the last group.
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
SRAS_MEM_BUDGET_MB Ceiling on one row-chunk's working memory, in MB
|
||||||
|
(default 1024). Lower it on a memory-constrained
|
||||||
|
machine; the tool just takes more, smaller chunks.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import struct
|
import struct
|
||||||
import sys
|
import sys
|
||||||
@@ -23,14 +45,22 @@ from pathlib import Path
|
|||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from sras_format import HDR_FMT, HDR_SIZE, SrasFile
|
from sras_format import GEO_FMT_V6, HDR_FMT_V6, SrasFile
|
||||||
|
|
||||||
_SUPPORTED = (2, 3, 4)
|
_SUPPORTED = (6, 7)
|
||||||
|
_DEFAULT_BUDGET_MB = 1024
|
||||||
|
|
||||||
|
# Throttle per-chunk progress printing to roughly this many lines per angle,
|
||||||
|
# so a huge angle (thousands of chunks) doesn't flood stdout while a small
|
||||||
|
# one still gets to print every chunk.
|
||||||
|
_MAX_PROGRESS_LINES = 40
|
||||||
|
|
||||||
|
|
||||||
def parse_args():
|
def parse_args():
|
||||||
p = argparse.ArgumentParser(
|
p = argparse.ArgumentParser(
|
||||||
description="Average every N waveforms in a .sras file and write a new file."
|
description="Average every N waveforms in a .sras file and write a new file.",
|
||||||
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||||
|
epilog=__doc__,
|
||||||
)
|
)
|
||||||
p.add_argument("input", help="Input .sras file")
|
p.add_argument("input", help="Input .sras file")
|
||||||
p.add_argument("output", help="Output .sras file")
|
p.add_argument("output", help="Output .sras file")
|
||||||
@@ -41,21 +71,45 @@ def parse_args():
|
|||||||
return p.parse_args()
|
return p.parse_args()
|
||||||
|
|
||||||
|
|
||||||
def read_header_sections(sras: SrasFile) -> bytes:
|
def _memory_budget_bytes() -> int:
|
||||||
"""The raw bytes between the header and the waveform data (angle table,
|
return int(os.environ.get("SRAS_MEM_BUDGET_MB", _DEFAULT_BUDGET_MB)) * 1024 * 1024
|
||||||
row table, preambles, background), copied through verbatim so nothing is
|
|
||||||
lost in a re-encode."""
|
|
||||||
with open(sras.path, "rb") as f:
|
|
||||||
f.seek(HDR_SIZE)
|
|
||||||
return f.read(sras.data_offset - HDR_SIZE)
|
|
||||||
|
|
||||||
|
|
||||||
def average_rows(block: np.ndarray, n: int, discard_remainder: bool) -> np.ndarray:
|
def _plan_angle(n_frames_in: int, n: int, discard_remainder: bool) -> tuple[int, int, int]:
|
||||||
"""Average every N frames of one angle's (n_rows, n_ch, n_frames, spf)
|
"""(n_full, remainder, n_frames_out) for averaging one angle's frames."""
|
||||||
block. Returns int16 of shape (n_rows, n_ch, n_out, spf).
|
n_full = n_frames_in // n
|
||||||
|
remainder = n_frames_in % n
|
||||||
|
n_out = n_full + (1 if remainder and not discard_remainder else 0)
|
||||||
|
return n_full, remainder, n_out
|
||||||
|
|
||||||
Averaging is done in float32 and rounded on cast, matching numpy's mean
|
|
||||||
followed by an int16 cast in the original implementation.
|
def _chunk_rows(n_rows: int, n_channels: int, n_frames_in: int,
|
||||||
|
samples_per_frame: int, budget: int) -> int:
|
||||||
|
"""How many rows to hold in RAM at once so one chunk's working buffers
|
||||||
|
(source block, mean's float64 accumulator, output block) fit the budget.
|
||||||
|
|
||||||
|
The 4 bytes/sample below is a deliberate middle estimate, not a sum of
|
||||||
|
the three buffers: the int16 source and int16 output are 2 each, and
|
||||||
|
only mean()'s float64 result is 8, over the reduced frame axis rather
|
||||||
|
than the full block. Raise SRAS_MEM_BUDGET_MB if a machine still runs
|
||||||
|
tight at a large --n."""
|
||||||
|
bytes_per_row = max(1, n_channels * n_frames_in * samples_per_frame * 4)
|
||||||
|
return max(1, min(n_rows, budget // bytes_per_row))
|
||||||
|
|
||||||
|
|
||||||
|
def _average_block(block: np.ndarray, n: int, discard_remainder: bool,
|
||||||
|
bps: int) -> np.ndarray:
|
||||||
|
"""Average every N frames of a (rows, n_ch, n_frames, spf) block along
|
||||||
|
the frame axis, returning the result already encoded in the on-disk
|
||||||
|
dtype: int8 (clipped) if bps == 1, else big-endian int16.
|
||||||
|
|
||||||
|
Cast to native int16 (not float32) before calling .mean(): numpy's mean()
|
||||||
|
uses a float64 accumulator by default for integer input, matching the
|
||||||
|
original implementation exactly (which kept the whole file as int16 and
|
||||||
|
called .mean() directly). A float32 cast here would use a float32
|
||||||
|
accumulator instead — for large group sizes that can round the sum
|
||||||
|
differently than float64 and, after the int16 cast below, occasionally
|
||||||
|
land on a value 1 ADC count away from the original tool's output.
|
||||||
"""
|
"""
|
||||||
n_frames = block.shape[2]
|
n_frames = block.shape[2]
|
||||||
n_full = n_frames // n
|
n_full = n_frames // n
|
||||||
@@ -63,46 +117,96 @@ def average_rows(block: np.ndarray, n: int, discard_remainder: bool) -> np.ndarr
|
|||||||
|
|
||||||
parts = []
|
parts = []
|
||||||
if n_full:
|
if n_full:
|
||||||
full = block[:, :, :n_full * n, :].astype(np.float32)
|
full = block[:, :, :n_full * n, :].astype(np.int16)
|
||||||
full = full.reshape(block.shape[0], block.shape[1], n_full, n, block.shape[3])
|
full = full.reshape(block.shape[0], block.shape[1], n_full, n, block.shape[3])
|
||||||
parts.append(full.mean(axis=3).astype(np.int16))
|
parts.append(full.mean(axis=3).astype(np.int16))
|
||||||
if remainder and not discard_remainder:
|
if remainder and not discard_remainder:
|
||||||
tail = block[:, :, n_full * n:, :].astype(np.float32)
|
tail = block[:, :, n_full * n:, :].astype(np.int16)
|
||||||
parts.append(tail.mean(axis=2, keepdims=True).astype(np.int16))
|
parts.append(tail.mean(axis=2, keepdims=True).astype(np.int16))
|
||||||
|
|
||||||
if not parts:
|
if not parts:
|
||||||
return np.empty((*block.shape[:2], 0, block.shape[3]), dtype=np.int16)
|
averaged = np.empty((*block.shape[:2], 0, block.shape[3]), dtype=np.int16)
|
||||||
return parts[0] if len(parts) == 1 else np.concatenate(parts, axis=2)
|
else:
|
||||||
|
averaged = parts[0] if len(parts) == 1 else np.concatenate(parts, axis=2)
|
||||||
|
|
||||||
|
if bps == 1:
|
||||||
|
return np.clip(averaged, -128, 127).astype(np.int8)
|
||||||
|
return averaged.astype(">i2")
|
||||||
|
|
||||||
|
|
||||||
def write_averaged(out_path: Path, sras: SrasFile, mid_sections: bytes,
|
def write_v6_averaged(sras: SrasFile, out_path: Path, n: int,
|
||||||
n: int, discard_remainder: bool) -> int:
|
discard_remainder: bool, budget: int | None = None) -> list[int]:
|
||||||
"""Stream each angle through the averager, writing as we go so peak RAM
|
"""Write *sras* averaged every N frames to a new v6 .sras file, streaming
|
||||||
stays at one angle's block rather than the whole file."""
|
row-chunks per angle so peak RAM never holds more than one chunk (bounded
|
||||||
|
by *budget* bytes, default from SRAS_MEM_BUDGET_MB).
|
||||||
|
|
||||||
|
Stages to a sibling '.part' file and os.replace()s it into position on
|
||||||
|
success, per scan_format.md's requirement that any .sras writer must
|
||||||
|
never leave a half-written file visible under its final name (a short
|
||||||
|
v6 file opens successfully with the wrong angle/frame count rather than
|
||||||
|
failing loudly).
|
||||||
|
|
||||||
|
Returns the per-angle output frame counts.
|
||||||
|
"""
|
||||||
|
budget = _memory_budget_bytes() if budget is None else max(1, budget)
|
||||||
|
n_ch = sras.n_channels
|
||||||
|
spf = sras.samples_per_frame
|
||||||
bps = sras.bytes_per_sample
|
bps = sras.bytes_per_sample
|
||||||
n_frames_in = int(sras.n_frames[0])
|
|
||||||
n_out = n_frames_in // n
|
plans = [_plan_angle(int(sras.n_frames[a]), n, discard_remainder)
|
||||||
if n_frames_in % n and not discard_remainder:
|
for a in range(sras.n_angles)]
|
||||||
n_out += 1
|
n_out_per_angle = [p[2] for p in plans]
|
||||||
|
|
||||||
header = struct.pack(
|
header = struct.pack(
|
||||||
HDR_FMT, b"SRAS", sras.version, sras.n_angles, int(sras.n_rows[0]),
|
HDR_FMT_V6, b"SRAS", 6, sras.n_angles,
|
||||||
float(sras.x_start_mm[0]), float(sras.x_delta_mm),
|
sras.x_start_nominal_mm, sras.y_start_nominal_mm,
|
||||||
sras.velocity_mm_s, sras.laser_freq_hz,
|
sras.x_delta_nominal_mm, sras.y_delta_nominal_mm,
|
||||||
n_out, # updated frame count
|
sras.row_spacing_mm, sras.velocity_mm_s, sras.laser_freq_hz / n,
|
||||||
sras.samples_per_frame, sras.sample_rate_hz, bps, sras.n_channels,
|
spf, sras.sample_rate_hz, bps, n_ch,
|
||||||
)
|
)
|
||||||
|
|
||||||
with open(out_path, "wb") as f:
|
geo = bytearray()
|
||||||
f.write(header)
|
|
||||||
f.write(mid_sections)
|
|
||||||
for a in range(sras.n_angles):
|
for a in range(sras.n_angles):
|
||||||
averaged = average_rows(sras.data[a], n, discard_remainder)
|
geo += struct.pack(GEO_FMT_V6, float(sras.x_start_mm[a]),
|
||||||
if bps == 1:
|
float(sras.x_delta_mm_per_angle[a]),
|
||||||
f.write(np.clip(averaged, -128, 127).astype(np.int8).tobytes())
|
int(n_out_per_angle[a]), int(sras.n_rows[a]))
|
||||||
else:
|
|
||||||
f.write(averaged.astype(">i2").tobytes())
|
part_path = out_path.with_name(out_path.name + ".part")
|
||||||
return n_out
|
try:
|
||||||
|
with open(part_path, "wb") as f:
|
||||||
|
f.write(header)
|
||||||
|
f.write(sras.angles_deg.astype(">f4").tobytes())
|
||||||
|
f.write(bytes(geo))
|
||||||
|
for a in range(sras.n_angles):
|
||||||
|
f.write(sras.y_pos_per_angle[a].astype(">f4").tobytes())
|
||||||
|
f.write(sras.encoded_preambles())
|
||||||
|
f.write(sras.encoded_background())
|
||||||
|
|
||||||
|
for a in range(sras.n_angles):
|
||||||
|
n_rows = int(sras.n_rows[a])
|
||||||
|
n_frames_in = int(sras.n_frames[a])
|
||||||
|
chunk_rows = _chunk_rows(n_rows, n_ch, n_frames_in, spf, budget)
|
||||||
|
total_chunks = max(1, -(-n_rows // chunk_rows))
|
||||||
|
print_every = max(1, total_chunks // _MAX_PROGRESS_LINES)
|
||||||
|
|
||||||
|
data = sras.data[a]
|
||||||
|
for i, r0 in enumerate(range(0, n_rows, chunk_rows)):
|
||||||
|
r1 = min(r0 + chunk_rows, n_rows)
|
||||||
|
block = np.asarray(data[r0:r1])
|
||||||
|
out_block = _average_block(block, n, discard_remainder, bps)
|
||||||
|
f.write(out_block.tobytes())
|
||||||
|
if total_chunks > 1 and (i % print_every == 0 or r1 == n_rows):
|
||||||
|
print(f" angle {a + 1}/{sras.n_angles}: "
|
||||||
|
f"{r1}/{n_rows} rows", flush=True)
|
||||||
|
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
os.replace(part_path, out_path)
|
||||||
|
except BaseException:
|
||||||
|
part_path.unlink(missing_ok=True)
|
||||||
|
raise
|
||||||
|
|
||||||
|
return n_out_per_angle
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
@@ -126,15 +230,13 @@ def main():
|
|||||||
sras = SrasFile(str(in_path))
|
sras = SrasFile(str(in_path))
|
||||||
if sras.version not in _SUPPORTED:
|
if sras.version not in _SUPPORTED:
|
||||||
print(f"Error: unsupported .sras version: {sras.version} "
|
print(f"Error: unsupported .sras version: {sras.version} "
|
||||||
f"(this tool handles v{'/v'.join(map(str, _SUPPORTED))})",
|
f"(this tool handles v{'/v'.join(map(str, _SUPPORTED))} only)",
|
||||||
file=sys.stderr)
|
file=sys.stderr)
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
n_frames_in = int(sras.n_frames[0])
|
aborted_note = " (scan aborted; trailing angle(s) already excluded)" if sras.scan_aborted else ""
|
||||||
print(f" Version : v{sras.version}")
|
print(f" Version : v{sras.version}")
|
||||||
print(f" Angles : {sras.n_angles}")
|
print(f" Angles : {sras.n_angles}{aborted_note}")
|
||||||
print(f" Rows : {int(sras.n_rows[0])}")
|
|
||||||
print(f" Frames (actual): {n_frames_in}")
|
|
||||||
print(f" Channels : {sras.n_channels}")
|
print(f" Channels : {sras.n_channels}")
|
||||||
print(f" Samples/frame: {sras.samples_per_frame}")
|
print(f" Samples/frame: {sras.samples_per_frame}")
|
||||||
print(f" Bytes/sample : {sras.bytes_per_sample}")
|
print(f" Bytes/sample : {sras.bytes_per_sample}")
|
||||||
@@ -145,28 +247,26 @@ def main():
|
|||||||
print(f"Wrote {out_path}")
|
print(f"Wrote {out_path}")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
print(f"\n{'idx':>4} {'angle_deg':>10} {'rows':>6} {'frames_in':>10} {'frames_out':>11}")
|
||||||
|
for a in range(sras.n_angles):
|
||||||
|
n_frames_in = int(sras.n_frames[a])
|
||||||
|
n_full, remainder, n_out = _plan_angle(n_frames_in, args.n, args.discard_remainder)
|
||||||
|
print(f"{a:>4} {sras.angles_deg[a]:>10.4f} {int(sras.n_rows[a]):>6} "
|
||||||
|
f"{n_frames_in:>10} {n_out:>11}")
|
||||||
if args.n > n_frames_in:
|
if args.n > n_frames_in:
|
||||||
print(f"Warning: --n ({args.n}) exceeds available frames ({n_frames_in}). "
|
print(f" Warning: --n ({args.n}) exceeds angle {a}'s frames "
|
||||||
"The entire dataset will be averaged into a single frame.")
|
f"({n_frames_in}); it collapses to a single frame.")
|
||||||
|
|
||||||
print(f"\nAveraging every {args.n} frames ...", flush=True)
|
if sras.version == 7:
|
||||||
print(f"\nWriting {out_path} ...", flush=True)
|
print("\nNote: input has a v7 cache tail; it is indexed by frame count "
|
||||||
mid_sections = read_header_sections(sras)
|
"and will be dropped. The viewer will recompute DC/FFT on next open.")
|
||||||
n_frames_out = write_averaged(out_path, sras, mid_sections,
|
|
||||||
args.n, args.discard_remainder)
|
|
||||||
|
|
||||||
n_full = n_frames_in // args.n
|
print(f"\nAveraging every {args.n} frames and writing {out_path} ...", flush=True)
|
||||||
remainder = n_frames_in % args.n
|
n_out_per_angle = write_v6_averaged(sras, out_path, args.n, args.discard_remainder)
|
||||||
if remainder and not args.discard_remainder:
|
|
||||||
status = f"({n_full} full groups + 1 partial group of {remainder})"
|
|
||||||
elif remainder:
|
|
||||||
status = f"({n_full} full groups, {remainder} trailing frames discarded)"
|
|
||||||
else:
|
|
||||||
status = f"({n_full} full groups)"
|
|
||||||
print(f" {n_frames_in} frames -> {n_frames_out} frames {status}")
|
|
||||||
|
|
||||||
in_mb = in_path.stat().st_size / 1024**2
|
in_mb = in_path.stat().st_size / 1024**2
|
||||||
out_mb = out_path.stat().st_size / 1024**2
|
out_mb = out_path.stat().st_size / 1024**2
|
||||||
|
print(f"\n Frames out: {n_out_per_angle}")
|
||||||
print(f" Input size : {in_mb:.1f} MB")
|
print(f" Input size : {in_mb:.1f} MB")
|
||||||
print(f" Output size: {out_mb:.1f} MB ({out_mb / in_mb * 100:.1f}% of input)")
|
print(f" Output size: {out_mb:.1f} MB ({out_mb / in_mb * 100:.1f}% of input)")
|
||||||
print("Done.")
|
print("Done.")
|
||||||
|
|||||||
+178
-239
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Image computation and angle alignment for .sras scans.
|
"""Image computation and angle alignment for .sras scans.
|
||||||
|
|
||||||
Depends only on numpy/scipy (+ optional pyfftw) and sras_format, so a
|
Depends only on numpy/scipy/pyfftw and sras_format, so a multiprocessing
|
||||||
multiprocessing child can import it without loading Qt or matplotlib —
|
child can import it without loading Qt or matplotlib — which matters
|
||||||
which matters because Python 3.14 on macOS spawns rather than forks.
|
because Python 3.14 on macOS spawns rather than forks.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import atexit
|
import atexit
|
||||||
@@ -15,61 +15,34 @@ from dataclasses import dataclass
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
import pyfftw
|
||||||
import scipy.fft as scipy_fft
|
import scipy.fft as scipy_fft
|
||||||
import scipy.ndimage as scipy_ndimage
|
import scipy.ndimage as scipy_ndimage
|
||||||
|
|
||||||
from sras_format import (CH1_IDX, CH3_IDX, CH4_IDX, MAX_PAD_FACTOR, SrasFile,
|
from sras_format import (CH1_IDX, CH3_IDX, CH4_IDX, MAX_PAD_FACTOR, SrasFile,
|
||||||
adc_to_mv)
|
adc_to_mv)
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# FFT backend
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
try:
|
|
||||||
import pyfftw
|
|
||||||
PYFFTW_AVAILABLE = True
|
|
||||||
except ImportError:
|
|
||||||
PYFFTW_AVAILABLE = False
|
|
||||||
|
|
||||||
try:
|
|
||||||
from threadpoolctl import threadpool_limits
|
|
||||||
except ImportError:
|
|
||||||
threadpool_limits = None
|
|
||||||
|
|
||||||
_fft_backend = "scipy" # "scipy" or "pyfftw"; set via set_fft_backend()
|
|
||||||
|
|
||||||
|
|
||||||
def set_fft_backend(name: str):
|
|
||||||
"""Select the rfft implementation. Module-level state, so it must be set
|
|
||||||
explicitly inside each multiprocessing child — it does not survive a
|
|
||||||
spawn. "numpy" is accepted as a legacy alias for "scipy"."""
|
|
||||||
global _fft_backend
|
|
||||||
if name == "numpy":
|
|
||||||
name = "scipy"
|
|
||||||
_fft_backend = name if (name == "pyfftw" and PYFFTW_AVAILABLE) else "scipy"
|
|
||||||
|
|
||||||
|
|
||||||
def get_fft_backend() -> str:
|
|
||||||
return _fft_backend
|
|
||||||
|
|
||||||
|
|
||||||
def _do_rfft(x: np.ndarray, n: int | None = None, axis: int = -1,
|
|
||||||
workers: int = 1) -> np.ndarray:
|
|
||||||
"""Dispatch rfft to the selected backend with optional multithreading."""
|
|
||||||
if _fft_backend == "pyfftw" and PYFFTW_AVAILABLE:
|
|
||||||
return pyfftw.interfaces.numpy_fft.rfft(x, n=n, axis=axis, threads=workers)
|
|
||||||
return scipy_fft.rfft(x, n=n, axis=axis, workers=workers)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# FFT worker pool and per-thread pyFFTW plans
|
# FFT worker pool and per-thread pyFFTW plans
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
_FFT_BLOCK = 512 # waveforms per FFT task. The knee on a 16-core machine:
|
_FFT_BLOCK_MAX = 512 # ceiling on waveforms/task: the knee measured on a
|
||||||
# smaller blocks serialise on GIL-held numpy dispatch,
|
# 16-core machine at natural resolution (n_len == spf).
|
||||||
|
# Smaller blocks serialise on GIL-held numpy dispatch,
|
||||||
# larger ones lose cache residency and task granularity.
|
# larger ones lose cache residency and task granularity.
|
||||||
_ZOOM_MIN_PAD = 4 # zoom refinement engages at n_fft >= _ZOOM_MIN_PAD * spf
|
_FFT_BLOCK_MIN = 32 # floor, so task granularity never collapses at
|
||||||
_FFT_EXACT_ENV = bool(int(os.environ.get("SRAS_FFT_EXACT", "0") or 0))
|
# extreme pad factors — at the cost of exceeding
|
||||||
|
# _FFT_PLAN_BYTES_BUDGET there (see _fft_block_for).
|
||||||
|
_FFT_PLAN_BYTES_BUDGET = int(
|
||||||
|
os.environ.get("SRAS_FFT_PLAN_BUDGET_MB", 16)) * 1024 * 1024
|
||||||
|
# Per-thread ceiling on one cached pyFFTW plan's resident input+output
|
||||||
|
# buffers (see _fft_block_for). This is PERMANENT memory — the plan
|
||||||
|
# cache is never evicted — so the worst case across the whole pool for
|
||||||
|
# one distinct (spf, n_len) is _MAX_WORKERS * _FFT_PLAN_BYTES_BUDGET.
|
||||||
|
# Deliberately separate from _TOTAL_BYTES_BUDGET/SRAS_MEM_BUDGET_MB,
|
||||||
|
# which bounds transient concurrently-live chunk buffers, freed at the
|
||||||
|
# end of each chunk — this one bounds a permanent per-thread tax that
|
||||||
|
# compounds with worker count, not chunk concurrency.
|
||||||
|
|
||||||
_pool_lock = threading.Lock()
|
_pool_lock = threading.Lock()
|
||||||
_pool: ThreadPoolExecutor | None = None
|
_pool: ThreadPoolExecutor | None = None
|
||||||
@@ -115,23 +88,38 @@ def _save_wisdom():
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
|
|
||||||
def _fftw_block_rfft(waves: np.ndarray, n: int) -> np.ndarray:
|
def _fft_block_for(spf: int, n_len: int) -> int:
|
||||||
|
"""Waveforms per FFT task/plan at transform length n_len. Bounds a
|
||||||
|
cached pyFFTW plan's resident input+output buffers (permanent, one per
|
||||||
|
pool thread, never evicted) to _FFT_PLAN_BYTES_BUDGET regardless of pad
|
||||||
|
factor. Reproduces _FFT_BLOCK_MAX exactly at natural resolution
|
||||||
|
(n_len == spf) — see docs/design.md for the worked numbers at high pad."""
|
||||||
|
bytes_per_wf = 4 * spf + 8 * (n_len // 2 + 1)
|
||||||
|
block = _FFT_PLAN_BYTES_BUDGET // max(1, bytes_per_wf)
|
||||||
|
return int(min(_FFT_BLOCK_MAX, max(_FFT_BLOCK_MIN, block)))
|
||||||
|
|
||||||
|
|
||||||
|
def _block_rfft(waves: np.ndarray, n: int) -> np.ndarray:
|
||||||
"""rfft of a (B, spf) float32 block via a cached per-thread FFTW plan.
|
"""rfft of a (B, spf) float32 block via a cached per-thread FFTW plan.
|
||||||
|
|
||||||
Plans have a fixed (_FFT_BLOCK, spf) input shape so each worker thread
|
Plans have a fixed (block, spf) input shape, block = _fft_block_for(spf,
|
||||||
plans once per transform length; a remainder block runs through the same
|
n), so each worker thread plans once per (block size, transform length)
|
||||||
plan with its tail rows ignored. The returned array is the plan's output
|
pair; a remainder block runs through the same plan with its tail rows
|
||||||
buffer — consume it before the next call on the same thread.
|
ignored. Single-threaded (threads=1): outer parallelism comes from the
|
||||||
|
pool, so the transform itself must not spin up threads. The returned
|
||||||
|
array is the plan's output buffer — consume it before the next call on
|
||||||
|
the same thread.
|
||||||
"""
|
"""
|
||||||
n_wf, spf = waves.shape
|
n_wf, spf = waves.shape
|
||||||
|
block = _fft_block_for(spf, n)
|
||||||
plans = getattr(_fftw_local, "plans", None)
|
plans = getattr(_fftw_local, "plans", None)
|
||||||
if plans is None:
|
if plans is None:
|
||||||
plans = _fftw_local.plans = {}
|
plans = _fftw_local.plans = {}
|
||||||
key = (_FFT_BLOCK, spf, n)
|
key = (block, spf, n)
|
||||||
plan = plans.get(key)
|
plan = plans.get(key)
|
||||||
if plan is None:
|
if plan is None:
|
||||||
_load_wisdom_once()
|
_load_wisdom_once()
|
||||||
buf = pyfftw.empty_aligned((_FFT_BLOCK, spf), dtype="float32")
|
buf = pyfftw.empty_aligned((block, spf), dtype="float32")
|
||||||
# No overwrite_input: FFTW must not scribble on input_array, whose
|
# No overwrite_input: FFTW must not scribble on input_array, whose
|
||||||
# zero-padded tail (columns spf..n) is zeroed exactly once here.
|
# zero-padded tail (columns spf..n) is zeroed exactly once here.
|
||||||
plan = pyfftw.builders.rfft(buf, n=n, axis=-1, threads=1,
|
plan = pyfftw.builders.rfft(buf, n=n, axis=-1, threads=1,
|
||||||
@@ -144,132 +132,20 @@ def _fftw_block_rfft(waves: np.ndarray, n: int) -> np.ndarray:
|
|||||||
return plan()[:n_wf]
|
return plan()[:n_wf]
|
||||||
|
|
||||||
|
|
||||||
def _block_rfft(waves: np.ndarray, n: int) -> np.ndarray:
|
|
||||||
"""Single-threaded rfft of one block; outer parallelism comes from the
|
|
||||||
pool, so the transform itself must not spin up threads."""
|
|
||||||
if _fft_backend == "pyfftw" and PYFFTW_AVAILABLE:
|
|
||||||
return _fftw_block_rfft(waves, n)
|
|
||||||
return scipy_fft.rfft(waves, n=n, axis=-1, workers=1)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Zoom peak search: coarse rfft + local fine DFT around the winning bin
|
# Peak search: full transform, block-parallel
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
_ZOOM_HALFWIDTH = 0.75 # refinement window half-width, in coarse spacings.
|
def _peak_bins(waves: np.ndarray, n_len: int, min_bin: int = 1) -> np.ndarray:
|
||||||
# Every fine bin lies within 0.5 spacings of its
|
"""Peak bin per waveform via the full transform. *min_bin* zeroes every
|
||||||
# nearest coarse bin, and that bin is guaranteed to
|
bin below it (always >= 1, so true DC stays suppressed) before the
|
||||||
# be a candidate (see _ZOOM_CAND_RATIO), so 0.5
|
argmax, so the peak search never returns a bin below the caller's
|
||||||
# suffices; 0.75 adds rounding margin.
|
frequency floor. Called in _fft_block_for(spf, n_len)-sized blocks,
|
||||||
_ZOOM_CAND_RATIO = 0.7 # refine every coarse bin within this power ratio of
|
fanned out across _fft_pool() — see docs/design.md."""
|
||||||
# its row's coarse maximum. Quarter-natural-bin
|
|
||||||
# scalloping at the 2x-oversampled coarse grid can
|
|
||||||
# understate a peak by at most ~19% in power, so 0.7
|
|
||||||
# keeps a wide margin — near-contenders are resolved
|
|
||||||
# on the fine grid, never ranked from coarse samples.
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class _ZoomPlan:
|
|
||||||
"""Constants for the coarse+refine peak search, built once per
|
|
||||||
compute_rf_image call and shared across worker threads. *phases* is a
|
|
||||||
lazily-filled window-start -> phase-vector cache; a benign duplicate
|
|
||||||
compute under concurrency is cheaper than locking."""
|
|
||||||
n_fft: int
|
|
||||||
n_coarse: int
|
|
||||||
m: int # fine bins per refinement window
|
|
||||||
n_bins_fine: int
|
|
||||||
E: np.ndarray # (spf, m) complex64 fine-DFT matrix, relative bins
|
|
||||||
j: np.ndarray # (spf,) float64 sample indices
|
|
||||||
phases: dict
|
|
||||||
|
|
||||||
|
|
||||||
def _zoom_plan(spf: int, n_fft: int) -> _ZoomPlan:
|
|
||||||
# 2x-oversampled coarse grid: the padded power spectrum is a trig
|
|
||||||
# polynomial of degree spf-1, so at 2x sampling its global max cannot
|
|
||||||
# hide between coarse bins.
|
|
||||||
n_coarse = scipy_fft.next_fast_len(2 * spf, real=True)
|
|
||||||
n_bins_fine = n_fft // 2 + 1
|
|
||||||
m = int(np.ceil(2 * _ZOOM_HALFWIDTH * n_fft / n_coarse)) + 1
|
|
||||||
m = min(m, n_bins_fine - 1)
|
|
||||||
j = np.arange(spf, dtype=np.float64)
|
|
||||||
E = np.exp((-2j * np.pi / n_fft) * np.outer(j, np.arange(m))).astype(np.complex64)
|
|
||||||
return _ZoomPlan(n_fft, n_coarse, m, n_bins_fine, E, j, {})
|
|
||||||
|
|
||||||
|
|
||||||
def _window_start(k_c: np.ndarray, zp: _ZoomPlan) -> np.ndarray:
|
|
||||||
"""First fine bin of the refinement window around each coarse bin.
|
|
||||||
Clipped to [1, ...] so fine bin 0 stays excluded (DC suppression)."""
|
|
||||||
k0 = np.floor((k_c - _ZOOM_HALFWIDTH) * (zp.n_fft / zp.n_coarse)).astype(np.int64)
|
|
||||||
return np.clip(k0, 1, max(1, zp.n_bins_fine - zp.m))
|
|
||||||
|
|
||||||
|
|
||||||
def _refine_window(waves: np.ndarray, rows: np.ndarray, k0: int, zp: _ZoomPlan,
|
|
||||||
best_pow: np.ndarray, best_bin: np.ndarray):
|
|
||||||
"""Evaluate fine bins k0..k0+m-1 for the given rows and fold the result
|
|
||||||
into the per-row best (power, bin), preserving np.argmax's lowest-bin
|
|
||||||
tie-break."""
|
|
||||||
ph = zp.phases.get(k0)
|
|
||||||
if ph is None:
|
|
||||||
ph = np.exp((-2j * np.pi * k0 / zp.n_fft) * zp.j).astype(np.complex64)
|
|
||||||
zp.phases[k0] = ph
|
|
||||||
F = (waves[rows] * ph) @ zp.E
|
|
||||||
q = F.real ** 2
|
|
||||||
q += F.imag ** 2
|
|
||||||
i = np.argmax(q, axis=1)
|
|
||||||
p = q[np.arange(len(rows)), i]
|
|
||||||
b = k0 + i
|
|
||||||
upd = (p > best_pow[rows]) | ((p == best_pow[rows]) & (b < best_bin[rows]))
|
|
||||||
ridx = rows[upd]
|
|
||||||
best_pow[ridx] = p[upd]
|
|
||||||
best_bin[ridx] = b[upd]
|
|
||||||
|
|
||||||
|
|
||||||
def _peak_bins_zoom(waves: np.ndarray, zp: _ZoomPlan) -> np.ndarray:
|
|
||||||
"""Peak fine-bin per waveform without materialising the padded spectrum:
|
|
||||||
coarse rfft, then a small fine DFT (one gemm per shared window) on the
|
|
||||||
exact n_fft bin grid. Identity with the full padded argmax is enforced
|
|
||||||
by tests/test_compute.py::test_zoom_identity and the golden-hash sweep."""
|
|
||||||
n_wf = waves.shape[0]
|
|
||||||
S = _block_rfft(waves, zp.n_coarse)
|
|
||||||
P = S.real ** 2
|
|
||||||
P += S.imag ** 2
|
|
||||||
P[:, 0] = 0.0
|
|
||||||
p_c = np.max(P, axis=1)
|
|
||||||
|
|
||||||
best_pow = np.full(n_wf, -1.0, dtype=np.float32)
|
|
||||||
best_bin = np.full(n_wf, np.iinfo(np.int64).max, dtype=np.int64)
|
|
||||||
|
|
||||||
# For a clean signal this yields one or two windows; a noise spectrum
|
|
||||||
# (many near-equal peaks) yields a dozen or so — still a tiny fraction
|
|
||||||
# of the padded grid.
|
|
||||||
thr = np.where(p_c > 0, np.float32(_ZOOM_CAND_RATIO) * p_c,
|
|
||||||
np.float32(np.inf))
|
|
||||||
rows_c, bins_c = np.nonzero(P >= thr[:, None])
|
|
||||||
k0_c = _window_start(bins_c, zp)
|
|
||||||
# Zeroing the coarse DC bin (suppression) blinds the candidate scan to
|
|
||||||
# fine bins closer to DC than the first coarse sample — where the
|
|
||||||
# DC-leakage skirt of an un-subtracted offset peaks. Always refine the
|
|
||||||
# DC-adjacent window too.
|
|
||||||
rows_c = np.concatenate([rows_c, np.arange(n_wf)])
|
|
||||||
k0_c = np.concatenate([k0_c, np.ones(n_wf, dtype=np.int64)])
|
|
||||||
pair = np.unique(np.stack([rows_c, k0_c], axis=1), axis=0)
|
|
||||||
for k0 in np.unique(pair[:, 1]):
|
|
||||||
_refine_window(waves, pair[pair[:, 1] == k0, 0], int(k0), zp,
|
|
||||||
best_pow, best_bin)
|
|
||||||
|
|
||||||
# An all-zero spectrum must reproduce argmax-of-zeros = bin 0.
|
|
||||||
best_bin[p_c == 0.0] = 0
|
|
||||||
return best_bin
|
|
||||||
|
|
||||||
|
|
||||||
def _peak_bins_direct(waves: np.ndarray, n_len: int) -> np.ndarray:
|
|
||||||
"""Peak bin per waveform via the full transform — for pad factors too
|
|
||||||
small for the zoom search to pay."""
|
|
||||||
S = _block_rfft(waves, n_len)
|
S = _block_rfft(waves, n_len)
|
||||||
power = S.real ** 2
|
power = S.real ** 2
|
||||||
power += S.imag ** 2
|
power += S.imag ** 2
|
||||||
power[:, 0] = 0.0
|
power[:, :min_bin] = 0.0
|
||||||
return np.argmax(power, axis=1)
|
return np.argmax(power, axis=1)
|
||||||
|
|
||||||
|
|
||||||
@@ -300,9 +176,11 @@ def _chunk_rows_for(n_frames: int, samples_per_frame: int,
|
|||||||
|
|
||||||
def _plan_fft_rows(n_frames: int, samples_per_frame: int, budget: int) -> int:
|
def _plan_fft_rows(n_frames: int, samples_per_frame: int, budget: int) -> int:
|
||||||
"""Rows per outer chunk for the block-FFT path. Only the float32
|
"""Rows per outer chunk for the block-FFT path. Only the float32
|
||||||
waveform buffer scales with the chunk (in-flight block spectra total a
|
waveform read buffer scales with the chunk; spectrum memory is bounded
|
||||||
few MB across the whole pool), so budget it with 2x slack and let the
|
independently, by _fft_block_for, to _MAX_WORKERS * _FFT_PLAN_BYTES_BUDGET
|
||||||
block fan-out saturate the pool regardless of pad factor."""
|
regardless of chunk size or pad factor (see docs/design.md), so this
|
||||||
|
formula budgets the read buffer alone, with 2x slack, and lets the block
|
||||||
|
fan-out saturate the pool."""
|
||||||
bytes_per_row = max(1, n_frames * samples_per_frame * 4 * 2)
|
bytes_per_row = max(1, n_frames * samples_per_frame * 4 * 2)
|
||||||
return int(max(1, min(_CHUNK_ROWS_MAX, budget // bytes_per_row)))
|
return int(max(1, min(_CHUNK_ROWS_MAX, budget // bytes_per_row)))
|
||||||
|
|
||||||
@@ -508,7 +386,8 @@ def pad_factor_for(sras: SrasFile, n_fft: int | None) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def cache_mismatch_reasons(sras: SrasFile, *, n_fft: int | None,
|
def cache_mismatch_reasons(sras: SrasFile, *, n_fft: int | None,
|
||||||
apply_bg_sub: bool, row_avg_n: int) -> list[str]:
|
apply_bg_sub: bool, row_avg_n: int,
|
||||||
|
min_freq_mhz: float = 0.0) -> list[str]:
|
||||||
"""Why this file's stored FFT images can't answer a request, as human-
|
"""Why this file's stored FFT images can't answer a request, as human-
|
||||||
readable phrases; empty means they can.
|
readable phrases; empty means they can.
|
||||||
|
|
||||||
@@ -518,8 +397,23 @@ def cache_mismatch_reasons(sras: SrasFile, *, n_fft: int | None,
|
|||||||
serves a stored image iff this returns nothing, and callers that want to
|
serves a stored image iff this returns nothing, and callers that want to
|
||||||
*explain* the miss (rather than silently recompute) format these same
|
*explain* the miss (rather than silently recompute) format these same
|
||||||
strings, so the two can't drift apart.
|
strings, so the two can't drift apart.
|
||||||
|
|
||||||
|
The min peak frequency floor is the asymmetric case: a stored floor
|
||||||
|
*above* the request is a genuine mismatch (bins below the stored floor
|
||||||
|
were never searched, so the stored numbers can't answer a request that
|
||||||
|
wants them considered), but a stored floor at or *below* the request is
|
||||||
|
servable — the difference is re-applied as a display-time mask by
|
||||||
|
cached_rf_image (pixels whose stored peak falls below the requested
|
||||||
|
floor are marked invalid rather than re-resolved). Compared in whole
|
||||||
|
kHz, the header field's own fixed-point grid, so a value that
|
||||||
|
round-trips through the file can never miscompare against itself.
|
||||||
"""
|
"""
|
||||||
reasons = []
|
reasons = []
|
||||||
|
if round(sras.precomputed_min_freq_mhz * 1000) > round(min_freq_mhz * 1000):
|
||||||
|
reasons.append(
|
||||||
|
f"stored with a {sras.precomputed_min_freq_mhz:g} MHz "
|
||||||
|
f"min-peak-freq floor, requested {min_freq_mhz:g} MHz "
|
||||||
|
f"(bins below the stored floor were never searched)")
|
||||||
pad = pad_factor_for(sras, n_fft)
|
pad = pad_factor_for(sras, n_fft)
|
||||||
if pad != sras.precomputed_pad_factor:
|
if pad != sras.precomputed_pad_factor:
|
||||||
want = f"pad {pad}x" if pad else "a ragged n_fft"
|
want = f"pad {pad}x" if pad else "a ragged n_fft"
|
||||||
@@ -543,7 +437,8 @@ def cached_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
n_fft: int | None = None,
|
n_fft: int | None = None,
|
||||||
dc4_mv: np.ndarray | None = None,
|
dc4_mv: np.ndarray | None = None,
|
||||||
row_avg_n: int = 0,
|
row_avg_n: int = 0,
|
||||||
allow_dc_recompute: bool = True) -> np.ndarray | None:
|
allow_dc_recompute: bool = True,
|
||||||
|
min_freq_mhz: float = 0.0) -> np.ndarray | None:
|
||||||
"""The precomputed-cache fast path for compute_rf_image: a ready-to-
|
"""The precomputed-cache fast path for compute_rf_image: a ready-to-
|
||||||
display peak-frequency image if this file already has one matching
|
display peak-frequency image if this file already has one matching
|
||||||
every setting the caller cares about, else None (caller must run a
|
every setting the caller cares about, else None (caller must run a
|
||||||
@@ -571,10 +466,20 @@ def cached_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
channel on the caller's behalf; this returns None instead so a caller
|
channel on the caller's behalf; this returns None instead so a caller
|
||||||
that wants to stay off the I/O path (e.g. a GUI thread) can choose to
|
that wants to stay off the I/O path (e.g. a GUI thread) can choose to
|
||||||
fall through to a real compute rather than block.
|
fall through to a real compute rather than block.
|
||||||
|
|
||||||
|
*min_freq_mhz* above the stored floor is re-imposed here as a mask:
|
||||||
|
pixels whose stored peak is below it are set to 0.0 — the same sentinel
|
||||||
|
the DC mask uses, and unambiguous, since a genuine peak can never be
|
||||||
|
0.0 (bin 0 is always excluded from the search). Masked-below-floor
|
||||||
|
pixels are marked invalid, not re-resolved; only a real recompute can
|
||||||
|
recover the strongest peak *above* the floor for them. A request whose
|
||||||
|
floor is *below* the stored one can't be answered at all (see
|
||||||
|
cache_mismatch_reasons) and returns None like any other mismatch.
|
||||||
"""
|
"""
|
||||||
cached_freq = sras.precomputed_freq_mhz[angle_idx]
|
cached_freq = sras.precomputed_freq_mhz[angle_idx]
|
||||||
if cached_freq is None or cache_mismatch_reasons(
|
if cached_freq is None or cache_mismatch_reasons(
|
||||||
sras, n_fft=n_fft, apply_bg_sub=apply_bg_sub, row_avg_n=row_avg_n):
|
sras, n_fft=n_fft, apply_bg_sub=apply_bg_sub, row_avg_n=row_avg_n,
|
||||||
|
min_freq_mhz=min_freq_mhz):
|
||||||
return None
|
return None
|
||||||
dc4_img = None
|
dc4_img = None
|
||||||
if dc_threshold_mv is not None:
|
if dc_threshold_mv is not None:
|
||||||
@@ -594,6 +499,10 @@ def cached_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
freq_img = cached_freq.copy()
|
freq_img = cached_freq.copy()
|
||||||
if dc4_img is not None:
|
if dc4_img is not None:
|
||||||
freq_img[dc4_img < dc_threshold_mv] = 0.0
|
freq_img[dc4_img < dc_threshold_mv] = 0.0
|
||||||
|
if min_freq_mhz > 0.0:
|
||||||
|
# Strict <, matching the compute path: the first bin at or above
|
||||||
|
# the floor survives searchsorted there, so it must survive here.
|
||||||
|
freq_img[freq_img < min_freq_mhz] = 0.0
|
||||||
return freq_img
|
return freq_img
|
||||||
|
|
||||||
|
|
||||||
@@ -605,8 +514,9 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
max_workers: int | None = None,
|
max_workers: int | None = None,
|
||||||
budget: int | None = None,
|
budget: int | None = None,
|
||||||
should_stop=None,
|
should_stop=None,
|
||||||
exact: bool = False,
|
row_avg_n: int = 0,
|
||||||
row_avg_n: int = 0) -> np.ndarray:
|
min_freq_mhz: float = 0.0,
|
||||||
|
use_stored: bool = True) -> np.ndarray:
|
||||||
"""FFT of each CH1 waveform; pixel = peak frequency in MHz.
|
"""FFT of each CH1 waveform; pixel = peak frequency in MHz.
|
||||||
|
|
||||||
Pixels where CH4_dc < dc_threshold_mv are set to 0 — and the FFT is
|
Pixels where CH4_dc < dc_threshold_mv are set to 0 — and the FFT is
|
||||||
@@ -624,60 +534,78 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
DC-channel precompute cache), pass it as *dc4_mv* (mV, shape
|
DC-channel precompute cache), pass it as *dc4_mv* (mV, shape
|
||||||
(n_rows, n_frames)) to reuse it instead of re-reading CH4 here.
|
(n_rows, n_frames)) to reuse it instead of re-reading CH4 here.
|
||||||
|
|
||||||
At pad factors >= _ZOOM_MIN_PAD the padded spectrum is never
|
The peak search runs the full transform (_peak_bins) in
|
||||||
materialised: a coarse rfft finds each peak and a local fine DFT
|
_fft_block_for(spf, n_len)-sized blocks, fanned out across the
|
||||||
resolves it on the exact n_fft bin grid (_peak_bins_zoom). *exact*
|
pyFFTW-plan-caching thread pool — see docs/design.md for how the block
|
||||||
forces the reference full-padded transform instead — it exists for
|
size is bounded so this stays memory-safe at high pad factors.
|
||||||
tests, audits, and the SRAS_FFT_EXACT=1 escape hatch, and is slow and
|
|
||||||
memory-hungry at high pad.
|
|
||||||
|
|
||||||
*row_avg_n* > 0 averages each pixel's CH1 waveform with its up-to-n
|
*row_avg_n* > 0 averages each pixel's CH1 waveform with its up-to-n
|
||||||
same-row neighbors (distance-weighted, valid-neighbors-only per the
|
same-row neighbors (distance-weighted, valid-neighbors-only per the
|
||||||
same dc_threshold_mv mask) before the FFT runs — see
|
same dc_threshold_mv mask) before the FFT runs — see
|
||||||
_row_average_waveforms. 0 (default) is the raw, unaveraged behavior.
|
_row_average_waveforms. 0 (default) is the raw, unaveraged behavior.
|
||||||
|
|
||||||
|
*min_freq_mhz* excludes every bin below it (true DC, bin 0, is always
|
||||||
|
excluded regardless) from the peak search, on top of — not instead of —
|
||||||
|
the dc_threshold_mv mask. Without it, a pixel that passes the DC-bias
|
||||||
|
threshold but carries only weak real signal can still resolve to a
|
||||||
|
near-zero frequency: the un-subtracted background's DC-leakage skirt
|
||||||
|
then has more power than the genuine (but weak) signal peak. Raising
|
||||||
|
the floor above that skirt forces the search to report the strongest
|
||||||
|
peak that is plausibly real signal instead. 0.0 (default) disables it
|
||||||
|
(only true DC is excluded, the long-standing behavior).
|
||||||
|
|
||||||
Fast path: if the file has a precomputed peak-frequency image for this
|
Fast path: if the file has a precomputed peak-frequency image for this
|
||||||
angle (v5 PREC or v7 CACH) matching every one of the caller's settings
|
angle (v5 PREC or v7 CACH) matching every one of the caller's settings
|
||||||
— including row_avg_n exactly — the stored image is used directly, no
|
— including row_avg_n exactly — the stored image is used directly, no
|
||||||
FFT is run. See cached_rf_image.
|
FFT is run. See cached_rf_image. A *min_freq_mhz* at or above the
|
||||||
|
stored floor is part of that service: pixels whose stored peak falls
|
||||||
|
below it come back masked to 0.0 rather than re-resolved. A request
|
||||||
|
*below* the stored floor is a genuine mismatch and falls through to
|
||||||
|
the real FFT here.
|
||||||
|
|
||||||
|
*use_stored=False* skips the fast path entirely and always runs the
|
||||||
|
real FFT. Batch recompute (cache_file) needs this: served-then-masked
|
||||||
|
pixels written back into the store would permanently replace peaks a
|
||||||
|
real recompute re-resolves above the floor.
|
||||||
"""
|
"""
|
||||||
n_rows, n_frames = sras.image_shape(angle_idx)
|
n_rows, n_frames = sras.image_shape(angle_idx)
|
||||||
data = sras.data[angle_idx]
|
data = sras.data[angle_idx]
|
||||||
|
|
||||||
fast = cached_rf_image(sras, angle_idx, dc_threshold_mv, apply_bg_sub=apply_bg_sub,
|
if use_stored:
|
||||||
n_fft=n_fft, dc4_mv=dc4_mv, row_avg_n=row_avg_n)
|
fast = cached_rf_image(sras, angle_idx, dc_threshold_mv,
|
||||||
|
apply_bg_sub=apply_bg_sub, n_fft=n_fft,
|
||||||
|
dc4_mv=dc4_mv, row_avg_n=row_avg_n,
|
||||||
|
min_freq_mhz=min_freq_mhz)
|
||||||
if fast is not None:
|
if fast is not None:
|
||||||
return fast
|
return fast
|
||||||
|
|
||||||
# ---- Chunked FFT path --------------------------------------------------
|
# ---- Chunked FFT path --------------------------------------------------
|
||||||
exact = exact or _FFT_EXACT_ENV
|
|
||||||
spf = sras.samples_per_frame
|
spf = sras.samples_per_frame
|
||||||
n_len = n_fft if n_fft is not None else spf
|
n_len = n_fft if n_fft is not None else spf
|
||||||
|
block = _fft_block_for(spf, n_len)
|
||||||
freq32 = sras.freq_axis_mhz(n_fft).astype(np.float32)
|
freq32 = sras.freq_axis_mhz(n_fft).astype(np.float32)
|
||||||
|
# First fine bin at or above the floor; searchsorted is exact here since
|
||||||
|
# freq32 is the very axis the floor is expressed against.
|
||||||
|
min_bin = max(1, int(np.searchsorted(freq32, min_freq_mhz)))
|
||||||
img = np.zeros((n_rows, n_frames), dtype=np.float32)
|
img = np.zeros((n_rows, n_frames), dtype=np.float32)
|
||||||
background = sras.background if (apply_bg_sub and sras.background is not None) else None
|
background = sras.background if (apply_bg_sub and sras.background is not None) else None
|
||||||
cal4 = sras.cal(CH4_IDX)
|
cal4 = sras.cal(CH4_IDX)
|
||||||
|
# DC4 mask source, in the same priority order as the fast path above:
|
||||||
|
# already-cached DC block (skips reading CH4 raw waveforms entirely),
|
||||||
|
# then the caller-supplied image, else recompute per chunk below.
|
||||||
|
dc4_full = sras.cached_dc_mv(angle_idx, CH4_IDX) if dc_threshold_mv is not None else None
|
||||||
|
if dc4_full is None:
|
||||||
|
dc4_full = dc4_mv
|
||||||
row_avg_weights = _row_average_weights(row_avg_n) if row_avg_n > 0 else None
|
row_avg_weights = _row_average_weights(row_avg_n) if row_avg_n > 0 else None
|
||||||
|
|
||||||
zp = (_zoom_plan(spf, n_fft)
|
|
||||||
if not exact and n_fft is not None and n_fft >= _ZOOM_MIN_PAD * spf
|
|
||||||
else None)
|
|
||||||
total = _TOTAL_BYTES_BUDGET if budget is None else max(1, budget)
|
total = _TOTAL_BYTES_BUDGET if budget is None else max(1, budget)
|
||||||
if row_avg_n > 0:
|
if row_avg_n > 0:
|
||||||
# One extra same-sized transient buffer (the pre-averaging full-row
|
# One extra same-sized transient buffer (the pre-averaging full-row
|
||||||
# scratch array) is live per in-flight row; halve the budget so
|
# scratch array) is live per in-flight row; halve the budget so
|
||||||
# _plan_fft_rows/the exact-path sizing accounts for it rather than
|
# _plan_fft_rows accounts for it rather than relying on its existing
|
||||||
# relying on _plan_fft_rows's existing 2x slack to happen to cover it.
|
# 2x slack to happen to cover it.
|
||||||
total = max(1, total // 2)
|
total = max(1, total // 2)
|
||||||
cap = max_workers if max_workers is not None else _MAX_WORKERS
|
cap = max_workers if max_workers is not None else _MAX_WORKERS
|
||||||
if exact:
|
|
||||||
# The reference path materialises the full padded spectrum, so rows
|
|
||||||
# are budgeted against it (complex64 + power + temp per bin) and the
|
|
||||||
# chunk runs as one serial transform.
|
|
||||||
bytes_per_row = max(1, n_frames * (4 * spf + 16 * (n_len // 2 + 1)))
|
|
||||||
chunk_rows = int(max(1, min(_CHUNK_ROWS_MAX, total // bytes_per_row)))
|
|
||||||
cap = 1
|
|
||||||
else:
|
|
||||||
chunk_rows = _plan_fft_rows(n_frames, spf, total)
|
chunk_rows = _plan_fft_rows(n_frames, spf, total)
|
||||||
pool = _fft_pool() if cap > 1 else None
|
pool = _fft_pool() if cap > 1 else None
|
||||||
|
|
||||||
@@ -685,8 +613,8 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
if dc_threshold_mv is None:
|
if dc_threshold_mv is None:
|
||||||
valid = None
|
valid = None
|
||||||
else:
|
else:
|
||||||
if dc4_mv is not None:
|
if dc4_full is not None:
|
||||||
dc4_chunk = dc4_mv[r0:r1]
|
dc4_chunk = dc4_full[r0:r1]
|
||||||
else:
|
else:
|
||||||
dc4_chunk = adc_to_mv(
|
dc4_chunk = adc_to_mv(
|
||||||
data[r0:r1, CH4_IDX, :, :].astype(np.float32).mean(axis=-1), *cal4)
|
data[r0:r1, CH4_IDX, :, :].astype(np.float32).mean(axis=-1), *cal4)
|
||||||
@@ -742,23 +670,14 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
def fft_block(b0: int):
|
def fft_block(b0: int):
|
||||||
if should_stop is not None and should_stop():
|
if should_stop is not None and should_stop():
|
||||||
return
|
return
|
||||||
b1 = min(b0 + _FFT_BLOCK, n_wf)
|
b1 = min(b0 + block, n_wf)
|
||||||
w = waves[b0:b1]
|
out[b0:b1] = freq32[_peak_bins(waves[b0:b1], n_len, min_bin)]
|
||||||
bins = (_peak_bins_zoom(w, zp) if zp is not None
|
|
||||||
else _peak_bins_direct(w, n_len))
|
|
||||||
out[b0:b1] = freq32[bins]
|
|
||||||
|
|
||||||
if exact:
|
if pool is None:
|
||||||
spectrum = _do_rfft(waves, n=n_fft, axis=-1, workers=1)
|
for b0 in range(0, n_wf, block):
|
||||||
power = spectrum.real ** 2
|
|
||||||
power += spectrum.imag ** 2
|
|
||||||
power[:, 0] = 0.0 # suppress DC bin
|
|
||||||
out[:] = freq32[np.argmax(power, axis=1)]
|
|
||||||
elif pool is None:
|
|
||||||
for b0 in range(0, n_wf, _FFT_BLOCK):
|
|
||||||
fft_block(b0)
|
fft_block(b0)
|
||||||
else:
|
else:
|
||||||
list(pool.map(fft_block, range(0, n_wf, _FFT_BLOCK)))
|
list(pool.map(fft_block, range(0, n_wf, block)))
|
||||||
|
|
||||||
# Boolean scatter is row-major, matching the read pass's
|
# Boolean scatter is row-major, matching the read pass's
|
||||||
# concatenation order.
|
# concatenation order.
|
||||||
@@ -767,18 +686,10 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
else:
|
else:
|
||||||
img[r0:r1] = out.reshape(r1 - r0, n_frames)
|
img[r0:r1] = out.reshape(r1 - r0, n_frames)
|
||||||
|
|
||||||
# BLAS must not thread under the pool (the fine-DFT gemm would multiply
|
|
||||||
# against the pool's own workers).
|
|
||||||
limiter = (threadpool_limits(limits=1)
|
|
||||||
if pool is not None and threadpool_limits is not None else None)
|
|
||||||
try:
|
|
||||||
for r0 in range(0, n_rows, chunk_rows):
|
for r0 in range(0, n_rows, chunk_rows):
|
||||||
if should_stop is not None and should_stop():
|
if should_stop is not None and should_stop():
|
||||||
break
|
break
|
||||||
process(r0, min(r0 + chunk_rows, n_rows))
|
process(r0, min(r0 + chunk_rows, n_rows))
|
||||||
finally:
|
|
||||||
if limiter is not None:
|
|
||||||
limiter.unregister()
|
|
||||||
return img
|
return img
|
||||||
|
|
||||||
|
|
||||||
@@ -787,16 +698,17 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
||||||
fft_backend: str = "scipy", max_workers: int = 0,
|
max_workers: int = 0,
|
||||||
pad_factor: int = 1,
|
pad_factor: int = 1,
|
||||||
dc_threshold_mv: float | None = None,
|
dc_threshold_mv: float | None = None,
|
||||||
row_avg_n: int = 0) -> str:
|
row_avg_n: int = 0,
|
||||||
|
min_freq_mhz: float = 0.0) -> str:
|
||||||
"""Compute and store DC or FFT images for every angle of one file,
|
"""Compute and store DC or FFT images for every angle of one file,
|
||||||
converting v6 → v7 in place. Returns "" on success or an error message.
|
converting v6 → v7 in place. Returns "" on success or an error message.
|
||||||
|
|
||||||
Module-level and picklable so it can run in a ProcessPoolExecutor. The
|
Module-level and picklable so it can run in a ProcessPoolExecutor. The
|
||||||
FFT backend and worker cap are passed explicitly because module globals
|
worker cap is passed explicitly because module globals do not survive a
|
||||||
do not survive a spawn.
|
spawn.
|
||||||
|
|
||||||
mode is "dc", "fft" (raw per-pixel FFT, unmasked — masking applied at
|
mode is "dc", "fft" (raw per-pixel FFT, unmasked — masking applied at
|
||||||
display time), or "fft_rowavg" (same-row, distance-weighted CH1
|
display time), or "fft_rowavg" (same-row, distance-weighted CH1
|
||||||
@@ -812,6 +724,16 @@ def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
|||||||
using, since a pad-1 cache is dead weight to a padded view and vice
|
using, since a pad-1 cache is dead weight to a padded view and vice
|
||||||
versa (cached_rf_image refuses the mismatch rather than showing peaks
|
versa (cached_rf_image refuses the mismatch rather than showing peaks
|
||||||
resolved at the wrong resolution).
|
resolved at the wrong resolution).
|
||||||
|
|
||||||
|
*min_freq_mhz* is the peak-search floor (see compute_rf_image), also
|
||||||
|
recorded in the SFFT block; it is quantized up front to the header
|
||||||
|
field's 0.001 MHz grid so the stored images and the recorded floor
|
||||||
|
can never disagree. Both FFT modes force use_stored=False on
|
||||||
|
compute_rf_image: a batch recompute must always run the real FFT, never
|
||||||
|
serve this file's own existing cache back to itself — with a floor (or
|
||||||
|
a changed DC threshold in fft_rowavg mode) that would silently bake a
|
||||||
|
masked copy of the old image into the store in place of a genuine
|
||||||
|
recompute.
|
||||||
"""
|
"""
|
||||||
global _MAX_WORKERS
|
global _MAX_WORKERS
|
||||||
try:
|
try:
|
||||||
@@ -822,7 +744,12 @@ def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
|||||||
# between a bad argument costing nothing and costing the whole run.
|
# between a bad argument costing nothing and costing the whole run.
|
||||||
if not (1 <= pad_factor <= MAX_PAD_FACTOR):
|
if not (1 <= pad_factor <= MAX_PAD_FACTOR):
|
||||||
return f"pad_factor must be 1-{MAX_PAD_FACTOR}, got {pad_factor}"
|
return f"pad_factor must be 1-{MAX_PAD_FACTOR}, got {pad_factor}"
|
||||||
set_fft_backend(fft_backend)
|
if not (np.isfinite(min_freq_mhz) and min_freq_mhz >= 0):
|
||||||
|
return f"min_freq_mhz must be a finite value >= 0, got {min_freq_mhz}"
|
||||||
|
# Quantize to the SFFT header's fixed-point grid (whole kHz) before
|
||||||
|
# computing, so the floor the FFT actually ran with is exactly the
|
||||||
|
# floor the header records.
|
||||||
|
min_freq_mhz = round(min_freq_mhz * 1000) / 1000.0
|
||||||
if max_workers:
|
if max_workers:
|
||||||
_MAX_WORKERS = max_workers
|
_MAX_WORKERS = max_workers
|
||||||
|
|
||||||
@@ -854,13 +781,16 @@ def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
|||||||
# internally over blocks, so angles run one at a time with the
|
# internally over blocks, so angles run one at a time with the
|
||||||
# full budget.
|
# full budget.
|
||||||
freq = [compute_rf_image(sras, a, dc_threshold_mv=None,
|
freq = [compute_rf_image(sras, a, dc_threshold_mv=None,
|
||||||
apply_bg_sub=effective_bg, n_fft=n_fft)
|
apply_bg_sub=effective_bg, n_fft=n_fft,
|
||||||
|
min_freq_mhz=min_freq_mhz,
|
||||||
|
use_stored=False)
|
||||||
for a in range(n)]
|
for a in range(n)]
|
||||||
# new_row_avg_n=0 explicitly: these images are raw per-pixel FFTs,
|
# new_row_avg_n=0 explicitly: these images are raw per-pixel FFTs,
|
||||||
# and carrying forward a row_avg_n left by an earlier fft_rowavg
|
# and carrying forward a row_avg_n left by an earlier fft_rowavg
|
||||||
# write would label them as something they are not.
|
# write would label them as something they are not.
|
||||||
sras.write_v7_cache(new_freq_mhz=freq, new_bg_sub=effective_bg,
|
sras.write_v7_cache(new_freq_mhz=freq, new_bg_sub=effective_bg,
|
||||||
new_row_avg_n=0, new_pad_factor=pad_factor)
|
new_row_avg_n=0, new_pad_factor=pad_factor,
|
||||||
|
new_min_freq_mhz=min_freq_mhz)
|
||||||
else: # "fft_rowavg"
|
else: # "fft_rowavg"
|
||||||
if row_avg_n <= 0:
|
if row_avg_n <= 0:
|
||||||
return "row_avg_n must be a positive neighbor half-width for fft_rowavg mode"
|
return "row_avg_n must be a positive neighbor half-width for fft_rowavg mode"
|
||||||
@@ -870,11 +800,14 @@ def cache_file(path: str, mode: str, apply_bg_sub: bool,
|
|||||||
effective_bg = apply_bg_sub and sras.background is not None
|
effective_bg = apply_bg_sub and sras.background is not None
|
||||||
freq = [compute_rf_image(sras, a, dc_threshold_mv=dc_threshold_mv,
|
freq = [compute_rf_image(sras, a, dc_threshold_mv=dc_threshold_mv,
|
||||||
apply_bg_sub=effective_bg, n_fft=n_fft,
|
apply_bg_sub=effective_bg, n_fft=n_fft,
|
||||||
row_avg_n=row_avg_n)
|
row_avg_n=row_avg_n,
|
||||||
|
min_freq_mhz=min_freq_mhz,
|
||||||
|
use_stored=False)
|
||||||
for a in range(n)]
|
for a in range(n)]
|
||||||
sras.write_v7_cache(new_freq_mhz=freq, new_bg_sub=effective_bg,
|
sras.write_v7_cache(new_freq_mhz=freq, new_bg_sub=effective_bg,
|
||||||
new_row_avg_n=row_avg_n,
|
new_row_avg_n=row_avg_n,
|
||||||
new_pad_factor=pad_factor)
|
new_pad_factor=pad_factor,
|
||||||
|
new_min_freq_mhz=min_freq_mhz)
|
||||||
return ""
|
return ""
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return str(exc)
|
return str(exc)
|
||||||
@@ -1941,7 +1874,13 @@ def save_manual_alignment(sras: SrasFile, ref_angle_idx: int,
|
|||||||
for a, p in per_angle.items()
|
for a, p in per_angle.items()
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
path.write_text(json.dumps(payload, indent=2))
|
# Write to a temp file in the same directory then rename over the target
|
||||||
|
# — os.replace() is atomic on both POSIX and Windows, so a crash mid-write
|
||||||
|
# leaves either the old sidecar or the new one, never a truncated/corrupt
|
||||||
|
# file that load_manual_alignment would silently treat as "no alignment".
|
||||||
|
tmp_path = path.with_name(path.name + f".tmp{os.getpid()}")
|
||||||
|
tmp_path.write_text(json.dumps(payload, indent=2))
|
||||||
|
os.replace(tmp_path, path)
|
||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+102
-32
@@ -60,15 +60,18 @@ PREC_FLAG_BG_SUB = 0x01
|
|||||||
CACH_MAGIC = b"CACH"
|
CACH_MAGIC = b"CACH"
|
||||||
CACH_HDR_FMT = ">4sBB" # magic, cach_version, block_flags
|
CACH_HDR_FMT = ">4sBB" # magic, cach_version, block_flags
|
||||||
CACH_HDR_SIZE = struct.calcsize(CACH_HDR_FMT)
|
CACH_HDR_SIZE = struct.calcsize(CACH_HDR_FMT)
|
||||||
CACH_VERSION = 3 # written on every fresh write
|
CACH_VERSION = 4 # written on every fresh write
|
||||||
CACH_VERSIONS_READABLE = (1, 2, 3) # accepted on read — see
|
CACH_VERSIONS_READABLE = (1, 2, 3, 4) # accepted on read — see
|
||||||
# _read_sfft_block. Each bump only
|
# _read_sfft_block. Each bump only
|
||||||
# appended a field, and every older
|
# appended a field, and every older
|
||||||
# tail has a well-defined reading:
|
# tail has a well-defined reading:
|
||||||
# v1 predates row-averaged FFT
|
# v1 predates row-averaged FFT
|
||||||
# caching (row_avg_n=0) and v1/v2
|
# caching (row_avg_n=0), v1/v2
|
||||||
# predate padded caching, so both
|
# predate padded caching, so both
|
||||||
# are natural-resolution (pad 1).
|
# are natural-resolution (pad 1),
|
||||||
|
# and v1-v3 predate the min peak
|
||||||
|
# frequency floor (min_freq 0 =
|
||||||
|
# no floor).
|
||||||
CACH_FLAG_DC = 0x01
|
CACH_FLAG_DC = 0x01
|
||||||
CACH_FLAG_FFT = 0x02
|
CACH_FLAG_FFT = 0x02
|
||||||
|
|
||||||
@@ -79,11 +82,20 @@ SDCB_HDR_SIZE = struct.calcsize(SDCB_HDR_FMT)
|
|||||||
SFFT_MAGIC = b"SFFT"
|
SFFT_MAGIC = b"SFFT"
|
||||||
SFFT_HDR_FMT_V1 = ">4sBH" # magic, flags, n_stored (cach_version 1)
|
SFFT_HDR_FMT_V1 = ">4sBH" # magic, flags, n_stored (cach_version 1)
|
||||||
SFFT_HDR_FMT_V2 = ">4sBHB" # + row_avg_n (cach_version 2)
|
SFFT_HDR_FMT_V2 = ">4sBHB" # + row_avg_n (cach_version 2)
|
||||||
SFFT_HDR_FMT = ">4sBHBH" # + pad_factor (cach_version 3)
|
SFFT_HDR_FMT_V3 = ">4sBHBH" # + pad_factor (cach_version 3)
|
||||||
|
SFFT_HDR_FMT = ">4sBHBHI" # + min_freq_khz (cach_version 4)
|
||||||
SFFT_HDR_SIZE_V1 = struct.calcsize(SFFT_HDR_FMT_V1)
|
SFFT_HDR_SIZE_V1 = struct.calcsize(SFFT_HDR_FMT_V1)
|
||||||
SFFT_HDR_SIZE_V2 = struct.calcsize(SFFT_HDR_FMT_V2)
|
SFFT_HDR_SIZE_V2 = struct.calcsize(SFFT_HDR_FMT_V2)
|
||||||
|
SFFT_HDR_SIZE_V3 = struct.calcsize(SFFT_HDR_FMT_V3)
|
||||||
SFFT_HDR_SIZE = struct.calcsize(SFFT_HDR_FMT)
|
SFFT_HDR_SIZE = struct.calcsize(SFFT_HDR_FMT)
|
||||||
MAX_PAD_FACTOR = 0xFFFF # the H field above
|
MAX_PAD_FACTOR = 0xFFFF # the H field above
|
||||||
|
# min_freq_khz is fixed-point (u32, units of 0.001 MHz), not a float32:
|
||||||
|
# the viewer's floor spinbox has 0.001 MHz granularity, and
|
||||||
|
# round(mhz * 1000) / 1000.0 reproduces the exact float64 the user typed,
|
||||||
|
# so the accept rule in sras_compute.cache_mismatch_reasons can compare
|
||||||
|
# with plain integer ordering. A ">f" float32 of e.g. 20.1 would read back
|
||||||
|
# as 20.10000038… > 20.1 and report a spurious mismatch forever.
|
||||||
|
MAX_MIN_FREQ_KHZ = 0xFFFFFFFF # the I field above; 0 = no floor
|
||||||
SFFT_FLAG_BG_SUB = 0x01
|
SFFT_FLAG_BG_SUB = 0x01
|
||||||
SFFT_FLAG_ROW_AVG = 0x02 # peak_freq_mhz came from same-row,
|
SFFT_FLAG_ROW_AVG = 0x02 # peak_freq_mhz came from same-row,
|
||||||
# distance-weighted averaged CH1
|
# distance-weighted averaged CH1
|
||||||
@@ -134,6 +146,13 @@ def adc_to_mv(adc, ymult_mv: float = _FALLBACK_YMULT_MV,
|
|||||||
return (adc - yoff_adc) * ymult_mv + yzero_mv
|
return (adc - yoff_adc) * ymult_mv + yzero_mv
|
||||||
|
|
||||||
|
|
||||||
|
def _axes_extent(x_axis, y_axis, dx: float, dy: float) -> list[float]:
|
||||||
|
"""Matplotlib imshow extent with half-pixel margins, Y flipped so row 0
|
||||||
|
renders at the top."""
|
||||||
|
return [x_axis[0] - dx / 2, x_axis[-1] + dx / 2,
|
||||||
|
y_axis[-1] + dy / 2, y_axis[0] - dy / 2]
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Binary read helpers
|
# Binary read helpers
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -188,9 +207,10 @@ class SrasFile:
|
|||||||
always as ragged per-angle lists (``list[np.ndarray | None]``, one entry
|
always as ragged per-angle lists (``list[np.ndarray | None]``, one entry
|
||||||
per angle, ``None`` where that angle was never stored) regardless of
|
per angle, ``None`` where that angle was never stored) regardless of
|
||||||
source version. The scalars ``precomputed_bg_sub`` /
|
source version. The scalars ``precomputed_bg_sub`` /
|
||||||
``precomputed_row_avg_n`` / ``precomputed_pad_factor`` record the
|
``precomputed_row_avg_n`` / ``precomputed_pad_factor`` /
|
||||||
settings the stored FFT images were computed under, so a reader can tell
|
``precomputed_min_freq_mhz`` record the settings the stored FFT images
|
||||||
whether they answer the question it is actually asking.
|
were computed under, so a reader can tell whether they answer the
|
||||||
|
question it is actually asking.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, path: str):
|
def __init__(self, path: str):
|
||||||
@@ -253,6 +273,13 @@ class SrasFile:
|
|||||||
# FFT resolves peaks a padded view would, and only such a view can
|
# FFT resolves peaks a padded view would, and only such a view can
|
||||||
# be served from it — see sras_compute.cached_rf_image.
|
# be served from it — see sras_compute.cached_rf_image.
|
||||||
self.precomputed_pad_factor: int = 1
|
self.precomputed_pad_factor: int = 1
|
||||||
|
# Min peak frequency floor (MHz) the stored peak search excluded
|
||||||
|
# bins below; 0.0 = no floor. Unlike bg-sub/pad/row-avg it is
|
||||||
|
# tighten-only re-applicable: a *higher* floor can be re-imposed on
|
||||||
|
# a stored image by masking pixels below it, but bins below the
|
||||||
|
# stored floor were never searched, so a lower floor needs a real
|
||||||
|
# recompute — see sras_compute.cache_mismatch_reasons.
|
||||||
|
self.precomputed_min_freq_mhz: float = 0.0
|
||||||
|
|
||||||
def encoded_preambles(self) -> bytes:
|
def encoded_preambles(self) -> bytes:
|
||||||
"""This file's Preamble Blocks section, as bytes a writer can emit.
|
"""This file's Preamble Blocks section, as bytes a writer can emit.
|
||||||
@@ -595,30 +622,35 @@ class SrasFile:
|
|||||||
store[angle_idx] = _read_f32_image(f, shape)
|
store[angle_idx] = _read_f32_image(f, shape)
|
||||||
return flags
|
return flags
|
||||||
|
|
||||||
def _read_sfft_block(self, f, cach_version: int) -> tuple[int, int, int] | None:
|
def _read_sfft_block(self, f, cach_version: int) -> tuple[int, int, int, float] | None:
|
||||||
"""Read the SFFT block header — its layout depends on cach_version,
|
"""Read the SFFT block header — its layout depends on cach_version,
|
||||||
since each bump appended a trailing field (v2 row_avg_n, v3
|
since each bump appended a trailing field (v2 row_avg_n, v3
|
||||||
pad_factor) — then n_stored per-angle peak_freq_mhz entries
|
pad_factor, v4 min_freq_khz) — then n_stored per-angle
|
||||||
(unchanged across versions).
|
peak_freq_mhz entries (unchanged across versions).
|
||||||
|
|
||||||
Returns (flags, row_avg_n, pad_factor), or None if the block is
|
Returns (flags, row_avg_n, pad_factor, min_freq_mhz), or None if
|
||||||
malformed. The absent fields of an older tail take the value that
|
the block is malformed. The absent fields of an older tail take the
|
||||||
describes what such a tail can only have been: row_avg_n=0 for v1,
|
value that describes what such a tail can only have been:
|
||||||
which predates row-averaged FFT caching, and pad_factor=1 for v1/v2,
|
row_avg_n=0 for v1, which predates row-averaged FFT caching;
|
||||||
which predate padded caching and so are natural-resolution.
|
pad_factor=1 for v1/v2, which predate padded caching and so are
|
||||||
|
natural-resolution; and min_freq_mhz=0.0 for v1-v3, which predate
|
||||||
|
the min peak frequency floor and so searched every bin above DC.
|
||||||
"""
|
"""
|
||||||
hdr_fmt = {1: SFFT_HDR_FMT_V1, 2: SFFT_HDR_FMT_V2}.get(
|
hdr_fmt = {1: SFFT_HDR_FMT_V1, 2: SFFT_HDR_FMT_V2,
|
||||||
cach_version, SFFT_HDR_FMT)
|
3: SFFT_HDR_FMT_V3}.get(cach_version, SFFT_HDR_FMT)
|
||||||
raw = f.read(struct.calcsize(hdr_fmt))
|
raw = f.read(struct.calcsize(hdr_fmt))
|
||||||
if len(raw) < struct.calcsize(hdr_fmt):
|
if len(raw) < struct.calcsize(hdr_fmt):
|
||||||
return None
|
return None
|
||||||
row_avg_n, pad_factor = 0, 1
|
row_avg_n, pad_factor, min_freq_khz = 0, 1, 0
|
||||||
if cach_version == 1:
|
if cach_version == 1:
|
||||||
magic, flags, n_stored = struct.unpack(hdr_fmt, raw)
|
magic, flags, n_stored = struct.unpack(hdr_fmt, raw)
|
||||||
elif cach_version == 2:
|
elif cach_version == 2:
|
||||||
magic, flags, n_stored, row_avg_n = struct.unpack(hdr_fmt, raw)
|
magic, flags, n_stored, row_avg_n = struct.unpack(hdr_fmt, raw)
|
||||||
else:
|
elif cach_version == 3:
|
||||||
magic, flags, n_stored, row_avg_n, pad_factor = struct.unpack(hdr_fmt, raw)
|
magic, flags, n_stored, row_avg_n, pad_factor = struct.unpack(hdr_fmt, raw)
|
||||||
|
else:
|
||||||
|
(magic, flags, n_stored, row_avg_n, pad_factor,
|
||||||
|
min_freq_khz) = struct.unpack(hdr_fmt, raw)
|
||||||
if magic != SFFT_MAGIC:
|
if magic != SFFT_MAGIC:
|
||||||
return None
|
return None
|
||||||
for _ in range(n_stored):
|
for _ in range(n_stored):
|
||||||
@@ -627,7 +659,7 @@ class SrasFile:
|
|||||||
break
|
break
|
||||||
self.precomputed_freq_mhz[angle_idx] = _read_f32_image(
|
self.precomputed_freq_mhz[angle_idx] = _read_f32_image(
|
||||||
f, self.image_shape(angle_idx))
|
f, self.image_shape(angle_idx))
|
||||||
return flags, row_avg_n, pad_factor
|
return flags, row_avg_n, pad_factor, min_freq_khz / 1000.0
|
||||||
|
|
||||||
def _parse_cach_section(self, offset: int):
|
def _parse_cach_section(self, offset: int):
|
||||||
"""Parse the v7 CACH tail that holds precomputed DC/FFT images."""
|
"""Parse the v7 CACH tail that holds precomputed DC/FFT images."""
|
||||||
@@ -650,10 +682,13 @@ class SrasFile:
|
|||||||
result = self._read_sfft_block(f, cach_version)
|
result = self._read_sfft_block(f, cach_version)
|
||||||
if result is None:
|
if result is None:
|
||||||
return
|
return
|
||||||
flags, row_avg_n, pad_factor = result
|
flags, row_avg_n, pad_factor, min_freq_mhz = result
|
||||||
self.precomputed_bg_sub = bool(flags & SFFT_FLAG_BG_SUB)
|
self.precomputed_bg_sub = bool(flags & SFFT_FLAG_BG_SUB)
|
||||||
self.precomputed_row_avg_n = row_avg_n if (flags & SFFT_FLAG_ROW_AVG) else 0
|
self.precomputed_row_avg_n = row_avg_n if (flags & SFFT_FLAG_ROW_AVG) else 0
|
||||||
self.precomputed_pad_factor = max(1, pad_factor)
|
self.precomputed_pad_factor = max(1, pad_factor)
|
||||||
|
# No flag bit gates the floor: 0 (= no floor) is already the
|
||||||
|
# value every pre-v4 tail reads as.
|
||||||
|
self.precomputed_min_freq_mhz = min_freq_mhz
|
||||||
|
|
||||||
def write_v7_cache(self, *,
|
def write_v7_cache(self, *,
|
||||||
new_dc3_mv: list[np.ndarray | None] | None = None,
|
new_dc3_mv: list[np.ndarray | None] | None = None,
|
||||||
@@ -661,7 +696,8 @@ class SrasFile:
|
|||||||
new_freq_mhz: list[np.ndarray | None] | None = None,
|
new_freq_mhz: list[np.ndarray | None] | None = None,
|
||||||
new_bg_sub: bool | None = None,
|
new_bg_sub: bool | None = None,
|
||||||
new_row_avg_n: int | None = None,
|
new_row_avg_n: int | None = None,
|
||||||
new_pad_factor: int | None = None):
|
new_pad_factor: int | None = None,
|
||||||
|
new_min_freq_mhz: float | None = None):
|
||||||
"""Store computed DC and/or FFT images into this file's CACH tail,
|
"""Store computed DC and/or FFT images into this file's CACH tail,
|
||||||
in place, converting a v6 source to v7 (or updating an existing v7
|
in place, converting a v6 source to v7 (or updating an existing v7
|
||||||
file). Only the block(s) passed in are recomputed; whichever block
|
file). Only the block(s) passed in are recomputed; whichever block
|
||||||
@@ -681,6 +717,13 @@ class SrasFile:
|
|||||||
different pad resolves different peaks, so recording it is what lets
|
different pad resolves different peaks, so recording it is what lets
|
||||||
a reader refuse the cache instead of showing the wrong numbers.
|
a reader refuse the cache instead of showing the wrong numbers.
|
||||||
|
|
||||||
|
*new_min_freq_mhz* is the min peak frequency floor the passed
|
||||||
|
*new_freq_mhz*'s peak search excluded bins below (0.0 = no floor),
|
||||||
|
carried forward the same way. It is stored fixed-point (whole kHz),
|
||||||
|
so the value is quantized to 0.001 MHz on write and
|
||||||
|
``precomputed_min_freq_mhz`` is updated to the quantized value —
|
||||||
|
what a reload would see, never a float the header can't represent.
|
||||||
|
|
||||||
The waveform data itself is never touched: the cache tail always
|
The waveform data itself is never touched: the cache tail always
|
||||||
starts at ``_cache_tail_offset()``, a fixed offset derived from the
|
starts at ``_cache_tail_offset()``, a fixed offset derived from the
|
||||||
header and geometry table alone.
|
header and geometry table alone.
|
||||||
@@ -697,13 +740,29 @@ class SrasFile:
|
|||||||
else self.precomputed_row_avg_n)
|
else self.precomputed_row_avg_n)
|
||||||
final_pad_factor = (new_pad_factor if new_pad_factor is not None
|
final_pad_factor = (new_pad_factor if new_pad_factor is not None
|
||||||
else self.precomputed_pad_factor)
|
else self.precomputed_pad_factor)
|
||||||
|
final_min_freq_mhz = (new_min_freq_mhz if new_min_freq_mhz is not None
|
||||||
|
else self.precomputed_min_freq_mhz)
|
||||||
if not (0 <= final_row_avg_n <= 255):
|
if not (0 <= final_row_avg_n <= 255):
|
||||||
raise ValueError(f"row_avg_n must fit in a byte (0-255), got {final_row_avg_n}")
|
raise ValueError(f"row_avg_n must fit in a byte (0-255), got {final_row_avg_n}")
|
||||||
if not (1 <= final_pad_factor <= MAX_PAD_FACTOR):
|
if not (1 <= final_pad_factor <= MAX_PAD_FACTOR):
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"pad_factor must be 1-{MAX_PAD_FACTOR}, got {final_pad_factor}")
|
f"pad_factor must be 1-{MAX_PAD_FACTOR}, got {final_pad_factor}")
|
||||||
|
if not (np.isfinite(final_min_freq_mhz) and final_min_freq_mhz >= 0):
|
||||||
|
raise ValueError(
|
||||||
|
f"min_freq_mhz must be a finite value >= 0, got {final_min_freq_mhz}")
|
||||||
|
final_min_freq_khz = int(round(final_min_freq_mhz * 1000))
|
||||||
|
if final_min_freq_khz > MAX_MIN_FREQ_KHZ:
|
||||||
|
raise ValueError(
|
||||||
|
f"min_freq_mhz too large for the u32 kHz header field: "
|
||||||
|
f"{final_min_freq_mhz}")
|
||||||
|
|
||||||
dc_entries = [a for a in range(self.n_angles) if final_dc3[a] is not None]
|
# dc3/dc4 are always populated together by every current caller, but
|
||||||
|
# guard the per-angle pairing explicitly rather than assume it: an
|
||||||
|
# angle present in only one of the two arrays would otherwise crash
|
||||||
|
# below on final_dc4[a].astype(...) (or silently store the wrong
|
||||||
|
# dc3/dc4 pairing for that angle).
|
||||||
|
dc_entries = [a for a in range(self.n_angles)
|
||||||
|
if final_dc3[a] is not None and final_dc4[a] is not None]
|
||||||
fft_entries = [a for a in range(self.n_angles) if final_freq[a] is not None]
|
fft_entries = [a for a in range(self.n_angles) if final_freq[a] is not None]
|
||||||
|
|
||||||
block_flags = ((CACH_FLAG_DC if dc_entries else 0)
|
block_flags = ((CACH_FLAG_DC if dc_entries else 0)
|
||||||
@@ -724,7 +783,7 @@ class SrasFile:
|
|||||||
fft_flags |= SFFT_FLAG_ROW_AVG if final_row_avg_n else 0
|
fft_flags |= SFFT_FLAG_ROW_AVG if final_row_avg_n else 0
|
||||||
payload += struct.pack(SFFT_HDR_FMT, SFFT_MAGIC, fft_flags,
|
payload += struct.pack(SFFT_HDR_FMT, SFFT_MAGIC, fft_flags,
|
||||||
len(fft_entries), final_row_avg_n,
|
len(fft_entries), final_row_avg_n,
|
||||||
final_pad_factor)
|
final_pad_factor, final_min_freq_khz)
|
||||||
for a in fft_entries:
|
for a in fft_entries:
|
||||||
payload += struct.pack(">H", a)
|
payload += struct.pack(">H", a)
|
||||||
payload += final_freq[a].astype(">f4").tobytes()
|
payload += final_freq[a].astype(">f4").tobytes()
|
||||||
@@ -735,13 +794,23 @@ class SrasFile:
|
|||||||
f.truncate()
|
f.truncate()
|
||||||
f.flush()
|
f.flush()
|
||||||
os.fsync(f.fileno())
|
os.fsync(f.fileno())
|
||||||
# Version-byte flip last: if the process dies before this point,
|
# Version-byte flip last: when converting a v6 source (self.version
|
||||||
# the file is still readable as plain v6 (v6 parsing only
|
# was 6 on entry), if the process dies before this point the file
|
||||||
# bounds-checks per-angle offset+nbytes <= file_size, it never
|
# is still readable as plain v6 (v6 parsing only bounds-checks
|
||||||
# asserts exactly how many bytes follow the last angle) — so an
|
# per-angle offset+nbytes <= file_size, it never asserts exactly
|
||||||
# interrupted write can never corrupt the file, only leave
|
# how many bytes follow the last angle) — so an interrupted write
|
||||||
# harmless trailing bytes that the next successful write
|
# can never corrupt the file, only leave harmless trailing bytes
|
||||||
# overwrites via this same deterministic cache offset.
|
# that the next successful write overwrites via this same
|
||||||
|
# deterministic cache offset.
|
||||||
|
#
|
||||||
|
# That guarantee does NOT extend to updating an already-v7 file:
|
||||||
|
# the version byte here is already 7 before this call, so a crash
|
||||||
|
# during the payload write above (before flush/fsync/truncate)
|
||||||
|
# can leave a cache tail that mixes a prefix of the new payload
|
||||||
|
# with a stale suffix of the old one, and this method has no
|
||||||
|
# protection against that case (no atomic rename — the tail is
|
||||||
|
# rewritten in place to avoid copying the, potentially huge,
|
||||||
|
# waveform data that precedes it).
|
||||||
f.seek(4)
|
f.seek(4)
|
||||||
f.write(struct.pack("B", 7))
|
f.write(struct.pack("B", 7))
|
||||||
f.flush()
|
f.flush()
|
||||||
@@ -754,6 +823,7 @@ class SrasFile:
|
|||||||
self.precomputed_bg_sub = final_bg_sub
|
self.precomputed_bg_sub = final_bg_sub
|
||||||
self.precomputed_row_avg_n = final_row_avg_n
|
self.precomputed_row_avg_n = final_row_avg_n
|
||||||
self.precomputed_pad_factor = final_pad_factor
|
self.precomputed_pad_factor = final_pad_factor
|
||||||
|
self.precomputed_min_freq_mhz = final_min_freq_khz / 1000.0
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Axes helpers
|
# Axes helpers
|
||||||
|
|||||||
+204
@@ -0,0 +1,204 @@
|
|||||||
|
"""Pure-matplotlib rendering of a displayed SRAS image: imshow + colorbar +
|
||||||
|
axis/title labeling, shared by the interactive Qt canvas
|
||||||
|
(sras_viewer.canvases.ImageCanvas, which supplies its own already-Qt-backed
|
||||||
|
Figure/Axes) and the headless batch image-export worker (which builds a
|
||||||
|
throwaway Agg Figure per file and never touches Qt) -- so an exported PNG
|
||||||
|
can never quietly start looking different from what the GUI actually shows.
|
||||||
|
|
||||||
|
Deliberately no PyQt6 import anywhere in this module: BatchExportImagesWorker
|
||||||
|
(sras_workers.py) may run export_view_image inside a spawned
|
||||||
|
ProcessPoolExecutor subprocess, exactly like sras_compute.cache_file, and
|
||||||
|
importing anything under the sras_viewer package would run its __init__.py
|
||||||
|
and pull in the whole Qt widget tree for no reason.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import matplotlib as mpl
|
||||||
|
import numpy as np
|
||||||
|
from matplotlib.backends.backend_agg import FigureCanvasAgg
|
||||||
|
from matplotlib.figure import Figure
|
||||||
|
|
||||||
|
from sras_compute import compute_rf_image, dc_image_mv
|
||||||
|
from sras_format import CH4_IDX, CH_NAMES, SrasFile, _axes_extent
|
||||||
|
|
||||||
|
# Figure size (inches) to fall back on when a caller has no live view to
|
||||||
|
# match: the size ImageCanvas is constructed at, so a headless render with
|
||||||
|
# no canvas behind it still looks like the viewer's starting layout.
|
||||||
|
DEFAULT_FIGSIZE = (7.0, 5.0)
|
||||||
|
|
||||||
|
# The canvas size travels from the GUI as-is, so it can be degenerate (a
|
||||||
|
# collapsed splitter pane, a window minimized mid-batch) or, on a very wide
|
||||||
|
# multi-monitor window, big enough that width * dpi approaches matplotlib's
|
||||||
|
# 2**16 pixel limit. Clamp rather than fail: an export's aspect ratio is a
|
||||||
|
# presentation detail and must never be the reason a batch loses an image.
|
||||||
|
_MIN_FIG_IN = 1.0
|
||||||
|
_MAX_FIG_IN = 100.0
|
||||||
|
|
||||||
|
|
||||||
|
def sanitize_figsize(figsize) -> tuple[float, float]:
|
||||||
|
"""(width, height) in inches, clamped to something renderable.
|
||||||
|
|
||||||
|
*figsize* is None (use DEFAULT_FIGSIZE) or any 2-sequence of numbers --
|
||||||
|
including the float64 pair Figure.get_size_inches returns, which is how
|
||||||
|
the GUI hands over the live canvas's current size.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
w, h = float(figsize[0]), float(figsize[1])
|
||||||
|
except (TypeError, ValueError, IndexError, KeyError):
|
||||||
|
return DEFAULT_FIGSIZE
|
||||||
|
if not (np.isfinite(w) and np.isfinite(h)):
|
||||||
|
return DEFAULT_FIGSIZE
|
||||||
|
return (min(max(w, _MIN_FIG_IN), _MAX_FIG_IN),
|
||||||
|
min(max(h, _MIN_FIG_IN), _MAX_FIG_IN))
|
||||||
|
|
||||||
|
|
||||||
|
def draw_view_image(ax, fig, img: np.ndarray, extent: list[float], cmap,
|
||||||
|
vmin: float, vmax: float, xlabel: str, ylabel: str,
|
||||||
|
title: str, colorbar_label: str = "", cb_ticks=None,
|
||||||
|
norm=None, bad_color=None):
|
||||||
|
"""imshow + colorbar + labels onto an already-created (ax, fig) pair.
|
||||||
|
|
||||||
|
*cmap* may be a name or a Colormap instance. *norm* (which overrides
|
||||||
|
vmin/vmax) and *cb_ticks* let a caller draw a discrete integer image
|
||||||
|
with whole-number colorbar bands instead of a continuous shade.
|
||||||
|
*bad_color*, if given, is the fill for NaN pixels -- a copy of *cmap* is
|
||||||
|
made so a shared, registered instance is never mutated.
|
||||||
|
|
||||||
|
Shared by ImageCanvas.show_image (Qt-backed ax/fig) and
|
||||||
|
export_view_image (headless Agg ax/fig) so the two can never drift into
|
||||||
|
showing different things for the same settings.
|
||||||
|
"""
|
||||||
|
if bad_color is not None:
|
||||||
|
cmap = (cmap if hasattr(cmap, "with_extremes")
|
||||||
|
else mpl.colormaps[cmap]).with_extremes(bad=bad_color)
|
||||||
|
|
||||||
|
kw = ({"norm": norm} if norm is not None
|
||||||
|
else {"vmin": vmin, "vmax": vmax})
|
||||||
|
im = ax.imshow(
|
||||||
|
img, aspect="auto", origin="upper",
|
||||||
|
extent=extent, cmap=cmap, interpolation="nearest", **kw,
|
||||||
|
)
|
||||||
|
cb = fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04, ticks=cb_ticks)
|
||||||
|
if colorbar_label:
|
||||||
|
cb.set_label(colorbar_label)
|
||||||
|
|
||||||
|
ax.set_xlabel(xlabel)
|
||||||
|
ax.set_ylabel(ylabel)
|
||||||
|
ax.set_title(title)
|
||||||
|
return im
|
||||||
|
|
||||||
|
|
||||||
|
def export_view_image(path: str, *, out_dir: str, angle_idx: int, ch_idx: int,
|
||||||
|
is_fft_mode: bool, is_velocity: bool,
|
||||||
|
dc_threshold_mv: float, apply_bg_sub: bool,
|
||||||
|
pad_factor: int, min_freq_mhz: float, grating_um: float,
|
||||||
|
cmap: str, auto_scale: bool, vmin: float, vmax: float,
|
||||||
|
highlight_masked: bool, mode_str: str,
|
||||||
|
colorbar_label: str, mask_color: str = "magenta",
|
||||||
|
max_workers: int | None = None,
|
||||||
|
figsize: tuple[float, float] | None = None,
|
||||||
|
dpi: int = 150) -> tuple[str, str]:
|
||||||
|
"""One file's contribution to Batch Export View as Images: renders
|
||||||
|
(angle_idx, ch_idx) at the given display settings to a PNG under
|
||||||
|
*out_dir*, via draw_view_image -- so a batch export is a folder of what
|
||||||
|
ImageCanvas.show_image would have put on screen for these settings, not
|
||||||
|
a raw data dump.
|
||||||
|
|
||||||
|
Module-level and picklable, like sras_compute.cache_file, so it can run
|
||||||
|
in a ProcessPoolExecutor -- see BatchExportImagesWorker. Unlike
|
||||||
|
cache_file this never writes to *path*: export is a read of the file's
|
||||||
|
own data, not a cache conversion, so any version SrasFile can open
|
||||||
|
works, with no v6/v7 precondition.
|
||||||
|
|
||||||
|
*figsize* is the on-screen ImageCanvas's current size in inches, so the
|
||||||
|
PNG carries the aspect ratio the view was actually being read at. It
|
||||||
|
matters more here than it would for a plot with fixed data aspect:
|
||||||
|
draw_view_image uses aspect="auto", so the image stretches to whatever
|
||||||
|
box it is given -- rendering a view the user has sized wide into a
|
||||||
|
hard-coded 7x5 squeezes the map into a different shape than the one
|
||||||
|
they judged it by. Passing the full size, not just the ratio, also
|
||||||
|
keeps titles, tick labels and the colorbar in the same proportion to
|
||||||
|
the map as on screen; *dpi* alone then sets the output resolution.
|
||||||
|
|
||||||
|
*pad_factor* (not n_fft) travels across files deliberately: n_fft
|
||||||
|
depends on samples_per_frame, which can differ between files in the
|
||||||
|
same batch, so n_fft is derived per file, here, from *this* file's own
|
||||||
|
value -- the same reason sras_compute.cache_file does the same thing.
|
||||||
|
|
||||||
|
row_avg_n is per file for the same reason and so is not a parameter at
|
||||||
|
all: it comes from *this* file's stored cache, mirroring
|
||||||
|
SrasViewerWindow._stored_fft_image and _start_compute, which both ask at
|
||||||
|
the window size the store was written at. Leaving it at compute_rf_image's
|
||||||
|
"raw per-pixel" default would make a row-averaged cache a mismatch, so a
|
||||||
|
file the viewer displays from its store would export as a full raw
|
||||||
|
recompute instead -- a different (and much slower) image than the one the
|
||||||
|
batch was triggered to reproduce.
|
||||||
|
|
||||||
|
Returns (error, out_name). error is "" on success. out_name is the
|
||||||
|
filename this call targeted -- set as soon as it's known, even on most
|
||||||
|
failures -- so the caller can flag same-stem collisions across the
|
||||||
|
batch without any cross-process bookkeeping.
|
||||||
|
"""
|
||||||
|
out_name = ""
|
||||||
|
try:
|
||||||
|
sras = SrasFile(path)
|
||||||
|
if angle_idx >= sras.n_angles:
|
||||||
|
return (f"angle {angle_idx} out of range "
|
||||||
|
f"(file has {sras.n_angles} angle(s))", out_name)
|
||||||
|
|
||||||
|
out_name = f"{Path(path).stem}_angle{angle_idx}_{CH_NAMES[ch_idx]}.png"
|
||||||
|
|
||||||
|
if is_fft_mode:
|
||||||
|
n_fft = (sras.samples_per_frame * pad_factor
|
||||||
|
if pad_factor > 1 else None)
|
||||||
|
freq = compute_rf_image(
|
||||||
|
sras, angle_idx, dc_threshold_mv=dc_threshold_mv,
|
||||||
|
apply_bg_sub=apply_bg_sub, n_fft=n_fft,
|
||||||
|
row_avg_n=sras.precomputed_row_avg_n,
|
||||||
|
min_freq_mhz=min_freq_mhz, max_workers=max_workers)
|
||||||
|
img = freq * grating_um if is_velocity else freq
|
||||||
|
else:
|
||||||
|
img = dc_image_mv(sras, angle_idx, ch_idx, max_workers=max_workers)
|
||||||
|
|
||||||
|
display_img, bad_color = img, None
|
||||||
|
if highlight_masked and is_fft_mode:
|
||||||
|
# Mirrors the viewer's _redraw_image rule: DC-masked pixels and
|
||||||
|
# value-0 pixels (the "no valid peak" sentinel — DC-masked,
|
||||||
|
# below the min-freq floor, or empty spectrum; the grating
|
||||||
|
# multiply above preserves zeros, so this holds for Velocity
|
||||||
|
# too) both render in the highlight color.
|
||||||
|
dc4 = dc_image_mv(sras, angle_idx, CH4_IDX, max_workers=max_workers)
|
||||||
|
valid = dc4 >= dc_threshold_mv
|
||||||
|
if valid.shape == display_img.shape:
|
||||||
|
valid &= display_img != 0.0
|
||||||
|
display_img = display_img.astype(np.float32, copy=True)
|
||||||
|
display_img[~valid] = np.nan
|
||||||
|
bad_color = mask_color
|
||||||
|
|
||||||
|
if auto_scale:
|
||||||
|
v0, v1 = float(np.nanmin(display_img)), float(np.nanmax(display_img))
|
||||||
|
if not np.isfinite(v0):
|
||||||
|
v0, v1 = 0.0, 0.0 # every pixel masked out
|
||||||
|
else:
|
||||||
|
v0, v1 = vmin, vmax
|
||||||
|
|
||||||
|
x_axis = sras.x_axis_mm(angle_idx)
|
||||||
|
y_axis = sras.y_positions_mm(angle_idx)
|
||||||
|
dx = x_axis[1] - x_axis[0] if len(x_axis) > 1 else sras.pixel_x_mm
|
||||||
|
dy = float(y_axis[1] - y_axis[0]) if len(y_axis) > 1 else 1.0
|
||||||
|
extent = _axes_extent(x_axis, y_axis, dx, dy)
|
||||||
|
|
||||||
|
title = (f"{CH_NAMES[ch_idx]} | {mode_str} | "
|
||||||
|
f"{sras.angles_deg[angle_idx]:.1f}°")
|
||||||
|
|
||||||
|
fig = Figure(figsize=sanitize_figsize(figsize), tight_layout=True)
|
||||||
|
FigureCanvasAgg(fig) # Agg-only: never registered with pyplot
|
||||||
|
ax = fig.add_subplot(111)
|
||||||
|
draw_view_image(ax, fig, display_img, extent, cmap, v0, v1,
|
||||||
|
"X (mm)", "Y (mm)", title, colorbar_label,
|
||||||
|
bad_color=bad_color)
|
||||||
|
fig.savefig(str(Path(out_dir) / out_name), dpi=dpi)
|
||||||
|
return ("", out_name)
|
||||||
|
except Exception as exc:
|
||||||
|
return (str(exc), out_name)
|
||||||
+35
-17
@@ -12,6 +12,7 @@ from PyQt6.QtGui import QKeyEvent
|
|||||||
from PyQt6.QtWidgets import QSizePolicy
|
from PyQt6.QtWidgets import QSizePolicy
|
||||||
|
|
||||||
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES, SrasFile, adc_to_mv
|
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES, SrasFile, adc_to_mv
|
||||||
|
from sras_render import draw_view_image
|
||||||
|
|
||||||
def count_colormap(n_angles: int):
|
def count_colormap(n_angles: int):
|
||||||
"""(cmap, norm, ticks) for an integer "how many angles cover this pixel"
|
"""(cmap, norm, ticks) for an integer "how many angles cover this pixel"
|
||||||
@@ -166,11 +167,14 @@ class ImageCanvas(FigureCanvasQTAgg):
|
|||||||
|
|
||||||
def show_image(self, img: np.ndarray, extent: list[float], cmap,
|
def show_image(self, img: np.ndarray, extent: list[float], cmap,
|
||||||
vmin: float, vmax: float, xlabel: str, ylabel: str, title: str,
|
vmin: float, vmax: float, xlabel: str, ylabel: str, title: str,
|
||||||
colorbar_label: str = "", cb_ticks=None, norm=None):
|
colorbar_label: str = "", cb_ticks=None, norm=None,
|
||||||
|
bad_color=None):
|
||||||
"""*cmap* may be a name or a Colormap instance. *norm* (which overrides
|
"""*cmap* may be a name or a Colormap instance. *norm* (which overrides
|
||||||
vmin/vmax) and *cb_ticks* let a caller draw a discrete integer image —
|
vmin/vmax) and *cb_ticks* let a caller draw a discrete integer image —
|
||||||
the wizard's overlap-count view — with whole-number colorbar bands
|
the wizard's overlap-count view — with whole-number colorbar bands
|
||||||
instead of a continuous shade."""
|
instead of a continuous shade. *bad_color*, if given, is the fill for
|
||||||
|
NaN pixels — a copy of *cmap* is made so the shared, registered
|
||||||
|
instance is never mutated."""
|
||||||
self.figure.clf()
|
self.figure.clf()
|
||||||
self.ax = self.figure.add_subplot(111)
|
self.ax = self.figure.add_subplot(111)
|
||||||
# Patches and lines are destroyed by figure.clf(); drop stale refs.
|
# Patches and lines are destroyed by figure.clf(); drop stale refs.
|
||||||
@@ -179,20 +183,12 @@ class ImageCanvas(FigureCanvasQTAgg):
|
|||||||
self._extent = extent
|
self._extent = extent
|
||||||
self._img_shape = img.shape
|
self._img_shape = img.shape
|
||||||
|
|
||||||
kw = ({"norm": norm} if norm is not None
|
# Shared with the headless batch image-export worker (sras_render.py)
|
||||||
else {"vmin": vmin, "vmax": vmax})
|
# so an exported PNG can never quietly drift from what this canvas
|
||||||
im = self.ax.imshow(
|
# shows on screen for the same settings.
|
||||||
img, aspect="auto", origin="upper",
|
draw_view_image(self.ax, self.figure, img, extent, cmap, vmin, vmax,
|
||||||
extent=extent, cmap=cmap, interpolation="nearest", **kw,
|
xlabel, ylabel, title, colorbar_label, cb_ticks, norm,
|
||||||
)
|
bad_color)
|
||||||
cb = self.figure.colorbar(im, ax=self.ax, fraction=0.046, pad=0.04,
|
|
||||||
ticks=cb_ticks)
|
|
||||||
if colorbar_label:
|
|
||||||
cb.set_label(colorbar_label)
|
|
||||||
|
|
||||||
self.ax.set_xlabel(xlabel)
|
|
||||||
self.ax.set_ylabel(ylabel)
|
|
||||||
self.ax.set_title(title)
|
|
||||||
|
|
||||||
# Re-draw the ROI (if any) on top of the fresh image so it persists
|
# Re-draw the ROI (if any) on top of the fresh image so it persists
|
||||||
# unchanged across mode / angle / channel switches.
|
# unchanged across mode / angle / channel switches.
|
||||||
@@ -433,13 +429,22 @@ class WaveformCanvas(FigureCanvasQTAgg):
|
|||||||
|
|
||||||
def show_rf_waveform(self, sras: SrasFile, angle_idx: int,
|
def show_rf_waveform(self, sras: SrasFile, angle_idx: int,
|
||||||
row_idx: int, frame_idx: int,
|
row_idx: int, frame_idx: int,
|
||||||
apply_bg_sub: bool = True):
|
apply_bg_sub: bool = True,
|
||||||
|
min_freq_mhz: float = 0.0):
|
||||||
"""CH1 RF: time-domain + FFT spectrum.
|
"""CH1 RF: time-domain + FFT spectrum.
|
||||||
|
|
||||||
If apply_bg_sub is True and sras.background is not None, the background
|
If apply_bg_sub is True and sras.background is not None, the background
|
||||||
waveform is overlaid on the time-domain plot and the FFT is computed
|
waveform is overlaid on the time-domain plot and the FFT is computed
|
||||||
on the subtracted signal. The unsubtracted FFT is also shown faintly
|
on the subtracted signal. The unsubtracted FFT is also shown faintly
|
||||||
for comparison.
|
for comparison.
|
||||||
|
|
||||||
|
*min_freq_mhz* > 0 restricts the labeled peak to bins at or above
|
||||||
|
it — the same floor the image's peak search uses, so the label
|
||||||
|
explains the map pixel instead of contradicting it — and shades the
|
||||||
|
excluded band on the spectrum. The spectrum curves themselves stay
|
||||||
|
complete (they are the evidence for choosing the floor). The peak
|
||||||
|
can still legitimately differ from a padded or row-averaged map:
|
||||||
|
this panel is always a single waveform at natural resolution.
|
||||||
"""
|
"""
|
||||||
data = sras.data[angle_idx]
|
data = sras.data[angle_idx]
|
||||||
waveform = data[row_idx, CH1_IDX, frame_idx, :].astype(np.float32)
|
waveform = data[row_idx, CH1_IDX, frame_idx, :].astype(np.float32)
|
||||||
@@ -480,8 +485,21 @@ class WaveformCanvas(FigureCanvasQTAgg):
|
|||||||
# FFT of the (possibly subtracted) waveform
|
# FFT of the (possibly subtracted) waveform
|
||||||
power_sub = np.abs(np.fft.rfft(waveform_plot)) ** 2
|
power_sub = np.abs(np.fft.rfft(waveform_plot)) ** 2
|
||||||
power_sub[0] = 0.0
|
power_sub[0] = 0.0
|
||||||
|
# First bin at or above the floor, exactly as the image peak search
|
||||||
|
# picks it (bin 0 always excluded). If the floor excludes every bin,
|
||||||
|
# fall back to the unrestricted peak rather than indexing past the
|
||||||
|
# end — the label is informational, not a mask.
|
||||||
|
lo = max(1, int(np.searchsorted(f_mhz, min_freq_mhz)))
|
||||||
|
if lo < len(power_sub):
|
||||||
|
peak_mhz = f_mhz[lo + int(np.argmax(power_sub[lo:]))]
|
||||||
|
else:
|
||||||
peak_mhz = f_mhz[int(np.argmax(power_sub))]
|
peak_mhz = f_mhz[int(np.argmax(power_sub))]
|
||||||
|
|
||||||
|
if min_freq_mhz > 0.0:
|
||||||
|
self.ax_right.axvspan(0, min_freq_mhz, color="#888888",
|
||||||
|
alpha=0.15, zorder=0,
|
||||||
|
label=f"< {min_freq_mhz:g} MHz excluded")
|
||||||
|
|
||||||
if bg is not None:
|
if bg is not None:
|
||||||
# Also show the unsubtracted FFT for reference
|
# Also show the unsubtracted FFT for reference
|
||||||
power_raw = np.abs(np.fft.rfft(waveform)) ** 2
|
power_raw = np.abs(np.fft.rfft(waveform)) ** 2
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ from PyQt6.QtWidgets import (
|
|||||||
QScrollArea, QSizePolicy, QVBoxLayout, QWidget,
|
QScrollArea, QSizePolicy, QVBoxLayout, QWidget,
|
||||||
)
|
)
|
||||||
|
|
||||||
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX
|
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, _axes_extent # noqa: F401 (re-exported)
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Display constants
|
# Display constants
|
||||||
@@ -26,6 +26,10 @@ CH1_DERIVED_MODES = (CH1_IDX, VELOCITY_MODE_IDX)
|
|||||||
|
|
||||||
CMAPS = ["gray", "viridis", "plasma", "inferno", "hot", "jet", "RdBu_r", "seismic"]
|
CMAPS = ["gray", "viridis", "plasma", "inferno", "hot", "jet", "RdBu_r", "seismic"]
|
||||||
|
|
||||||
|
# Fill color for masked (below-DC-threshold) pixels when "Highlight masked
|
||||||
|
# pixels" is on, chosen to stand out against every colormap in CMAPS above.
|
||||||
|
_MASKED_HIGHLIGHT_COLOR = "magenta"
|
||||||
|
|
||||||
# (mode_str, status-bar unit, colorbar label) per channel index
|
# (mode_str, status-bar unit, colorbar label) per channel index
|
||||||
_CHANNEL_DISPLAY = {
|
_CHANNEL_DISPLAY = {
|
||||||
CH1_IDX: ("RF", "Peak frequency (MHz)", "MHz"),
|
CH1_IDX: ("RF", "Peak frequency (MHz)", "MHz"),
|
||||||
@@ -60,6 +64,7 @@ class Jobs:
|
|||||||
COMPUTE = "compute"
|
COMPUTE = "compute"
|
||||||
DC_PRECOMPUTE = "dc_precompute"
|
DC_PRECOMPUTE = "dc_precompute"
|
||||||
BATCH = "batch"
|
BATCH = "batch"
|
||||||
|
EXPORT = "export"
|
||||||
# The alignment wizard's three background steps: fetching each angle's CH4
|
# The alignment wizard's three background steps: fetching each angle's CH4
|
||||||
# image for the mask stack, registering the angles, and writing the aligned
|
# image for the mask stack, registering the angles, and writing the aligned
|
||||||
# export. Separate keys because a retry of one must not be blocked by
|
# export. Separate keys because a retry of one must not be blocked by
|
||||||
@@ -108,13 +113,6 @@ def _combo(items=(), *, min_chars: int = 10) -> QComboBox:
|
|||||||
return combo
|
return combo
|
||||||
|
|
||||||
|
|
||||||
def _axes_extent(x_axis, y_axis, dx: float, dy: float) -> list[float]:
|
|
||||||
"""Matplotlib imshow extent with half-pixel margins, Y flipped so row 0
|
|
||||||
renders at the top."""
|
|
||||||
return [x_axis[0] - dx / 2, x_axis[-1] + dx / 2,
|
|
||||||
y_axis[-1] + dy / 2, y_axis[0] - dy / 2]
|
|
||||||
|
|
||||||
|
|
||||||
def _wrap_label(text: str = "", css: str | None = None) -> QLabel:
|
def _wrap_label(text: str = "", css: str | None = None) -> QLabel:
|
||||||
"""A word-wrapped QLabel that reports its *wrapped* height to the layout.
|
"""A word-wrapped QLabel that reports its *wrapped* height to the layout.
|
||||||
|
|
||||||
|
|||||||
+141
-33
@@ -8,15 +8,19 @@ mask-overlay editor plus the crop and export steps.
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from PyQt6.QtWidgets import (
|
from PyQt6.QtWidgets import (
|
||||||
QButtonGroup, QCheckBox, QDialog, QDialogButtonBox, QFileDialog,
|
QButtonGroup, QCheckBox, QDialog, QDialogButtonBox, QDoubleSpinBox,
|
||||||
QGroupBox, QHBoxLayout, QLabel, QLineEdit, QPushButton, QRadioButton,
|
QFileDialog, QGridLayout, QGroupBox, QHBoxLayout, QLabel, QLineEdit,
|
||||||
QScrollArea, QSpinBox, QVBoxLayout, QWidget,
|
QMessageBox, QPushButton, QRadioButton, QScrollArea, QSpinBox,
|
||||||
|
QVBoxLayout, QWidget,
|
||||||
)
|
)
|
||||||
|
|
||||||
from sras_compute import PYFFTW_AVAILABLE
|
|
||||||
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES
|
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES
|
||||||
|
from sras_workers import ExportChannel
|
||||||
|
|
||||||
from .common import CH_LABELS, VELOCITY_MODE_IDX, _CSS_HINT, _CSS_WARN, _group, _make_dspin, _wrap_label
|
from .common import (
|
||||||
|
CH_LABELS, VELOCITY_MODE_IDX, _CHANNEL_DISPLAY, _CSS_HINT, _CSS_WARN,
|
||||||
|
_SPIN_MIN_W, _form, _group, _make_dspin, _wrap_label,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -24,7 +28,7 @@ from .common import CH_LABELS, VELOCITY_MODE_IDX, _CSS_HINT, _CSS_WARN, _group,
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
class FftOptionsDialog(QDialog):
|
class FftOptionsDialog(QDialog):
|
||||||
"""Configure FFT backend and zero-padding.
|
"""Configure FFT zero-padding.
|
||||||
|
|
||||||
Changes take effect only when the user clicks Apply. Cancel discards
|
Changes take effect only when the user clicks Apply. Cancel discards
|
||||||
all pending edits. The live 'frequency resolution' label updates as
|
all pending edits. The live 'frequency resolution' label updates as
|
||||||
@@ -33,7 +37,6 @@ class FftOptionsDialog(QDialog):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, parent=None, *,
|
def __init__(self, parent=None, *,
|
||||||
current_backend: str,
|
|
||||||
current_pad_factor: int,
|
current_pad_factor: int,
|
||||||
samples_per_frame: int | None,
|
samples_per_frame: int | None,
|
||||||
sample_rate_hz: float | None,
|
sample_rate_hz: float | None,
|
||||||
@@ -49,29 +52,6 @@ class FftOptionsDialog(QDialog):
|
|||||||
|
|
||||||
layout = QVBoxLayout(self)
|
layout = QVBoxLayout(self)
|
||||||
|
|
||||||
# ---- Backend ---------------------------------------------------
|
|
||||||
grp_backend = QGroupBox("FFT Backend")
|
|
||||||
bl = QVBoxLayout(grp_backend)
|
|
||||||
|
|
||||||
self._btn_scipy = QRadioButton("SciPy FFT (pocketfft) (always available)")
|
|
||||||
self._btn_pyfftw = QRadioButton(
|
|
||||||
"pyFFTW (faster for large arrays)" if PYFFTW_AVAILABLE
|
|
||||||
else "pyFFTW (not installed — run: pip install pyfftw)")
|
|
||||||
self._btn_pyfftw.setEnabled(PYFFTW_AVAILABLE)
|
|
||||||
|
|
||||||
self._backend_group = QButtonGroup(self)
|
|
||||||
self._backend_group.addButton(self._btn_scipy, id=0)
|
|
||||||
self._backend_group.addButton(self._btn_pyfftw, id=1)
|
|
||||||
|
|
||||||
if current_backend == "pyfftw" and PYFFTW_AVAILABLE:
|
|
||||||
self._btn_pyfftw.setChecked(True)
|
|
||||||
else:
|
|
||||||
self._btn_scipy.setChecked(True)
|
|
||||||
|
|
||||||
bl.addWidget(self._btn_scipy)
|
|
||||||
bl.addWidget(self._btn_pyfftw)
|
|
||||||
layout.addWidget(grp_backend)
|
|
||||||
|
|
||||||
# ---- Zero-padding ----------------------------------------------
|
# ---- Zero-padding ----------------------------------------------
|
||||||
grp_zp = QGroupBox("Zero-Padding")
|
grp_zp = QGroupBox("Zero-Padding")
|
||||||
zl = QVBoxLayout(grp_zp)
|
zl = QVBoxLayout(grp_zp)
|
||||||
@@ -134,9 +114,6 @@ class FftOptionsDialog(QDialog):
|
|||||||
f"Velocity bin: {vel_res_ms:.3f} m/s "
|
f"Velocity bin: {vel_res_ms:.3f} m/s "
|
||||||
f"(at grating = {self._grating_um:.2f} µm)")
|
f"(at grating = {self._grating_um:.2f} µm)")
|
||||||
|
|
||||||
def get_backend(self) -> str:
|
|
||||||
return "pyfftw" if self._btn_pyfftw.isChecked() and PYFFTW_AVAILABLE else "scipy"
|
|
||||||
|
|
||||||
def get_pad_factor(self) -> int:
|
def get_pad_factor(self) -> int:
|
||||||
return max(1, self._spin_pad.value())
|
return max(1, self._spin_pad.value())
|
||||||
|
|
||||||
@@ -240,6 +217,137 @@ class RowAverageFftOptionsDialog(QDialog):
|
|||||||
return self._spin_threshold.value()
|
return self._spin_threshold.value()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Batch export dialog (Export -> Batch Export Images...)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class BatchExportDialog(QDialog):
|
||||||
|
"""Configure a batch PNG export of DC/RF/Velocity maps across every angle
|
||||||
|
of the currently open file.
|
||||||
|
|
||||||
|
Each channel's colorbar range is entered here and held fixed for every
|
||||||
|
exported angle (rather than auto-scaled per image, today's live-view
|
||||||
|
default) so the exported images are directly comparable to each other.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# (ch_idx, is_velocity, label, colorbar unit, filename tag)
|
||||||
|
_ROWS = [
|
||||||
|
(CH1_IDX, False, CH_LABELS[CH1_IDX], _CHANNEL_DISPLAY[CH1_IDX][2],
|
||||||
|
CH_NAMES[CH1_IDX]),
|
||||||
|
(CH3_IDX, False, CH_LABELS[CH3_IDX], _CHANNEL_DISPLAY[CH3_IDX][2],
|
||||||
|
CH_NAMES[CH3_IDX]),
|
||||||
|
(CH4_IDX, False, CH_LABELS[CH4_IDX], _CHANNEL_DISPLAY[CH4_IDX][2],
|
||||||
|
CH_NAMES[CH4_IDX]),
|
||||||
|
(CH1_IDX, True, CH_LABELS[VELOCITY_MODE_IDX],
|
||||||
|
_CHANNEL_DISPLAY[VELOCITY_MODE_IDX][2], CH_NAMES[VELOCITY_MODE_IDX]),
|
||||||
|
]
|
||||||
|
# DC is cheap and precomputed on load; CH1/Velocity need a per-pixel FFT
|
||||||
|
# that can take minutes, so don't default to exporting them (same
|
||||||
|
# rationale as SrasViewerWindow._on_load_done's channel default).
|
||||||
|
_DEFAULT_CHECKED = {CH3_IDX, CH4_IDX}
|
||||||
|
|
||||||
|
def __init__(self, parent, *, default_dir: str, default_prefix: str,
|
||||||
|
default_ranges: dict[tuple[int, bool], tuple[float, float]]):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.setWindowTitle("Batch Export Images")
|
||||||
|
self.setModal(True)
|
||||||
|
self.setMinimumWidth(460)
|
||||||
|
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
|
||||||
|
# ---- Output ------------------------------------------------------
|
||||||
|
grp_out, ol = _group("Output")
|
||||||
|
dir_row = QHBoxLayout()
|
||||||
|
self._edit_dir = QLineEdit(default_dir)
|
||||||
|
btn_browse = QPushButton("Browse…")
|
||||||
|
btn_browse.clicked.connect(self._on_browse)
|
||||||
|
dir_row.addWidget(self._edit_dir)
|
||||||
|
dir_row.addWidget(btn_browse)
|
||||||
|
out_form = _form()
|
||||||
|
out_form.addRow("Folder:", dir_row)
|
||||||
|
self._edit_prefix = QLineEdit(default_prefix)
|
||||||
|
out_form.addRow("File prefix:", self._edit_prefix)
|
||||||
|
ol.addLayout(out_form)
|
||||||
|
layout.addWidget(grp_out)
|
||||||
|
|
||||||
|
# ---- Channels ------------------------------------------------------
|
||||||
|
grp_ch, cl = _group("Channels (fixed range, applied to every angle)")
|
||||||
|
grid = QGridLayout()
|
||||||
|
grid.setHorizontalSpacing(8)
|
||||||
|
grid.setVerticalSpacing(6)
|
||||||
|
grid.addWidget(_wrap_label("min:", _CSS_HINT), 0, 1)
|
||||||
|
grid.addWidget(_wrap_label("max:", _CSS_HINT), 0, 2)
|
||||||
|
|
||||||
|
self._rows: list[tuple[QCheckBox, QDoubleSpinBox, QDoubleSpinBox]] = []
|
||||||
|
for i, (ch_idx, is_velocity, label, unit, _tag) in enumerate(self._ROWS, 1):
|
||||||
|
chk = QCheckBox(label + (f" [{unit}]" if unit else ""))
|
||||||
|
chk.setChecked(not is_velocity and ch_idx in self._DEFAULT_CHECKED)
|
||||||
|
vmin, vmax = default_ranges.get((ch_idx, is_velocity), (0.0, 1.0))
|
||||||
|
spin_min, spin_max = QDoubleSpinBox(), QDoubleSpinBox()
|
||||||
|
for spin, val in ((spin_min, vmin), (spin_max, vmax)):
|
||||||
|
spin.setRange(-1e9, 1e9)
|
||||||
|
spin.setDecimals(4)
|
||||||
|
spin.setMinimumWidth(_SPIN_MIN_W)
|
||||||
|
spin.setValue(val)
|
||||||
|
grid.addWidget(chk, i, 0)
|
||||||
|
grid.addWidget(spin_min, i, 1)
|
||||||
|
grid.addWidget(spin_max, i, 2)
|
||||||
|
self._rows.append((chk, spin_min, spin_max))
|
||||||
|
cl.addLayout(grid)
|
||||||
|
layout.addWidget(grp_ch)
|
||||||
|
|
||||||
|
# ---- Buttons --------------------------------------------------
|
||||||
|
buttons = QDialogButtonBox()
|
||||||
|
buttons.addButton("Export", QDialogButtonBox.ButtonRole.AcceptRole
|
||||||
|
).clicked.connect(self.accept)
|
||||||
|
buttons.addButton("Cancel", QDialogButtonBox.ButtonRole.RejectRole
|
||||||
|
).clicked.connect(self.reject)
|
||||||
|
layout.addWidget(buttons)
|
||||||
|
|
||||||
|
def _on_browse(self):
|
||||||
|
d = QFileDialog.getExistingDirectory(
|
||||||
|
self, "Select Output Folder", self._edit_dir.text())
|
||||||
|
if d:
|
||||||
|
self._edit_dir.setText(d)
|
||||||
|
|
||||||
|
def accept(self):
|
||||||
|
"""Validate before closing — Cancel bypasses this entirely."""
|
||||||
|
if not self.get_prefix():
|
||||||
|
QMessageBox.warning(self, "Batch Export", "Enter a file prefix.")
|
||||||
|
return
|
||||||
|
if not self.get_output_dir():
|
||||||
|
QMessageBox.warning(self, "Batch Export", "Choose an output folder.")
|
||||||
|
return
|
||||||
|
selected = self.get_selected_channels()
|
||||||
|
if not selected:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "Batch Export", "Select at least one channel to export.")
|
||||||
|
return
|
||||||
|
for ch in selected:
|
||||||
|
if not ch.vmin < ch.vmax:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "Batch Export", f"{ch.label}: min must be less than max.")
|
||||||
|
return
|
||||||
|
super().accept()
|
||||||
|
|
||||||
|
def get_output_dir(self) -> str:
|
||||||
|
return self._edit_dir.text().strip()
|
||||||
|
|
||||||
|
def get_prefix(self) -> str:
|
||||||
|
return self._edit_prefix.text().strip()
|
||||||
|
|
||||||
|
def get_selected_channels(self) -> list[ExportChannel]:
|
||||||
|
result = []
|
||||||
|
for (chk, spin_min, spin_max), (ch_idx, is_velocity, label, unit, tag) in zip(
|
||||||
|
self._rows, self._ROWS):
|
||||||
|
if chk.isChecked():
|
||||||
|
result.append(ExportChannel(
|
||||||
|
ch_idx=ch_idx, is_velocity=is_velocity,
|
||||||
|
vmin=spin_min.value(), vmax=spin_max.value(),
|
||||||
|
label=label, unit=unit, tag=tag))
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Export Fused ROI dialog
|
# Export Fused ROI dialog
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
+439
-58
@@ -1,6 +1,7 @@
|
|||||||
"""The SrasViewerWindow main window and application entry point."""
|
"""The SrasViewerWindow main window and application entry point."""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
|
from collections import Counter
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
@@ -9,8 +10,8 @@ from PyQt6.QtCore import QObject, QSettings, QSignalBlocker, Qt, QThread
|
|||||||
from PyQt6.QtGui import QAction
|
from PyQt6.QtGui import QAction
|
||||||
from PyQt6.QtWidgets import (
|
from PyQt6.QtWidgets import (
|
||||||
QApplication, QCheckBox, QComboBox, QDialog, QFileDialog, QFrame,
|
QApplication, QCheckBox, QComboBox, QDialog, QFileDialog, QFrame,
|
||||||
QHBoxLayout, QLabel, QMainWindow, QProgressDialog, QPushButton,
|
QHBoxLayout, QLabel, QMainWindow, QMessageBox, QProgressDialog,
|
||||||
QSizePolicy, QSpinBox, QSplitter, QVBoxLayout, QWidget,
|
QPushButton, QSizePolicy, QSpinBox, QSplitter, QVBoxLayout, QWidget,
|
||||||
)
|
)
|
||||||
|
|
||||||
import sras_compute as compute
|
import sras_compute as compute
|
||||||
@@ -19,23 +20,25 @@ from sras_compute import (
|
|||||||
load_manual_alignment, save_manual_alignment, sidecar_path,
|
load_manual_alignment, save_manual_alignment, sidecar_path,
|
||||||
)
|
)
|
||||||
from sras_format import (
|
from sras_format import (
|
||||||
CH3_IDX, CH4_IDX, CH_NAMES, SrasFile, mv_to_adc,
|
CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES, SrasFile, mv_to_adc,
|
||||||
_FALLBACK_YMULT_MV, _FALLBACK_YOFF_ADC,
|
_FALLBACK_YMULT_MV, _FALLBACK_YOFF_ADC,
|
||||||
)
|
)
|
||||||
from sras_workers import (
|
from sras_workers import (
|
||||||
BatchCacheWorker, ComputeWorker, DcPrecomputeWorker, LoadWorker,
|
BatchCacheWorker, BatchExportImagesWorker, BatchExportWorker,
|
||||||
|
ComputeWorker, DcPrecomputeWorker, LoadWorker,
|
||||||
)
|
)
|
||||||
|
|
||||||
from .canvases import ImageCanvas, WaveformCanvas
|
from .canvases import ImageCanvas, WaveformCanvas
|
||||||
from .common import (
|
from .common import (
|
||||||
CH1_DERIVED_MODES, CH_LABELS, CMAPS, VELOCITY_MODE_IDX, _CHANNEL_DISPLAY,
|
CH1_DERIVED_MODES, CH_LABELS, CMAPS, VELOCITY_MODE_IDX, _CHANNEL_DISPLAY,
|
||||||
_CSS_BUSY, _axes_extent, _CSS_HINT, _CSS_INFO, _CSS_MUTED, _CSS_WARN, _LEFT_PANEL_W,
|
_CSS_BUSY, _axes_extent, _CSS_HINT, _CSS_INFO, _CSS_MUTED, _CSS_WARN, _LEFT_PANEL_W,
|
||||||
_RIGHT_PANEL_W, Jobs, _combo, _form, _group, _make_dspin, _scroll_panel,
|
_MASKED_HIGHLIGHT_COLOR, _RIGHT_PANEL_W, Jobs, _combo, _form, _group, _make_dspin,
|
||||||
_wrap_label,
|
_scroll_panel, _wrap_label,
|
||||||
)
|
)
|
||||||
from .align_wizard import AlignmentWizard
|
from .align_wizard import AlignmentWizard
|
||||||
from .dialogs import (
|
from .dialogs import (
|
||||||
FftOptionsDialog, FusedRoiExportDialog, RowAverageFftOptionsDialog,
|
BatchExportDialog, FftOptionsDialog, FusedRoiExportDialog,
|
||||||
|
RowAverageFftOptionsDialog,
|
||||||
)
|
)
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -58,6 +61,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._pending_ch: int = 0
|
self._pending_ch: int = 0
|
||||||
self._pending_bg_sub: bool = True
|
self._pending_bg_sub: bool = True
|
||||||
self._pending_threshold: float = 50.0 # mV
|
self._pending_threshold: float = 50.0 # mV
|
||||||
|
self._pending_min_freq_mhz: float = 0.0
|
||||||
self._pending_fft_pad_factor: int = 1
|
self._pending_fft_pad_factor: int = 1
|
||||||
|
|
||||||
# Live background jobs, keyed by role — see _run_worker.
|
# Live background jobs, keyed by role — see _run_worker.
|
||||||
@@ -70,7 +74,6 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._settings = QSettings(QSettings.Format.IniFormat,
|
self._settings = QSettings(QSettings.Format.IniFormat,
|
||||||
QSettings.Scope.UserScope,
|
QSettings.Scope.UserScope,
|
||||||
"sras-viewer", "sras-viewer")
|
"sras-viewer", "sras-viewer")
|
||||||
compute.set_fft_backend(str(self._settings.value("fft/backend", "scipy")))
|
|
||||||
try:
|
try:
|
||||||
pad = int(self._settings.value("fft/pad_factor", 1))
|
pad = int(self._settings.value("fft/pad_factor", 1))
|
||||||
except (TypeError, ValueError):
|
except (TypeError, ValueError):
|
||||||
@@ -85,6 +88,10 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
|
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
|
||||||
self._batch_errors: list[str] = []
|
self._batch_errors: list[str] = []
|
||||||
|
|
||||||
|
# Export menu: batch image export
|
||||||
|
self._export_errors: list[str] = []
|
||||||
|
self._export_ok_count: int = 0
|
||||||
|
|
||||||
# Display-only settings (colormap, grating) never trigger a
|
# Display-only settings (colormap, grating) never trigger a
|
||||||
# recompute — they're applied to cached data on redraw. DC images
|
# recompute — they're applied to cached data on redraw. DC images
|
||||||
# (CH3/CH4) are cheap and precomputed for every angle in the
|
# (CH3/CH4) are cheap and precomputed for every angle in the
|
||||||
@@ -92,13 +99,13 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
# computed lazily (with a progress popup) the first time an
|
# computed lazily (with a progress popup) the first time an
|
||||||
# angle/threshold combination is viewed — using the cached DC4
|
# angle/threshold combination is viewed — using the cached DC4
|
||||||
# image to skip the FFT entirely for masked-out pixels — and
|
# image to skip the FFT entirely for masked-out pixels — and
|
||||||
# cached per (angle, threshold) so revisiting the same combination
|
# cached per (angle, threshold, min peak freq) so revisiting the
|
||||||
# is free. bg-sub/pad are deliberately not part of the key: once an
|
# same combination is free. bg-sub/pad are deliberately not part of
|
||||||
# angle has any FFT image (live or from the file's own stored
|
# the key: once an angle has any FFT image (live or from the file's
|
||||||
# cache), it stays displayed regardless of those controls — see
|
# own stored cache), it stays displayed regardless of those
|
||||||
# _fft_cache_key.
|
# controls — see _fft_cache_key.
|
||||||
self._dc_cache: dict[tuple[int, int], np.ndarray] = {}
|
self._dc_cache: dict[tuple[int, int], np.ndarray] = {}
|
||||||
self._fft_cache: dict[tuple[int, float], np.ndarray] = {}
|
self._fft_cache: dict[tuple[int, float, float], np.ndarray] = {}
|
||||||
self._dc_generation: int = 0
|
self._dc_generation: int = 0
|
||||||
|
|
||||||
# Angle alignment ("Fusion" menu)
|
# Angle alignment ("Fusion" menu)
|
||||||
@@ -272,6 +279,31 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self.lbl_threshold_adc = _wrap_label(
|
self.lbl_threshold_adc = _wrap_label(
|
||||||
f"≈ {mv_to_adc(50.0):.1f} ADC counts", _CSS_MUTED)
|
f"≈ {mv_to_adc(50.0):.1f} ADC counts", _CSS_MUTED)
|
||||||
tl.addWidget(self.lbl_threshold_adc)
|
tl.addWidget(self.lbl_threshold_adc)
|
||||||
|
|
||||||
|
min_freq_form = _form()
|
||||||
|
self.spin_min_freq_mhz = _make_dspin(0.0, 10000.0, 3, suffix=" MHz",
|
||||||
|
value=0.0, step=1.0)
|
||||||
|
self.spin_min_freq_mhz.setEnabled(False)
|
||||||
|
self.spin_min_freq_mhz.setToolTip(
|
||||||
|
"Excludes every FFT bin below this frequency from the peak\n"
|
||||||
|
"search (0 = off, only true DC is excluded). A pixel can pass\n"
|
||||||
|
"the DC threshold above yet still carry only weak real signal —\n"
|
||||||
|
"when that happens, the un-subtracted background's DC-leakage\n"
|
||||||
|
"skirt can have more power than the genuine signal, so the peak\n"
|
||||||
|
"search resolves to a near-zero frequency even though the pixel\n"
|
||||||
|
"is real, valid data. Raising this floor above that skirt forces\n"
|
||||||
|
"the search to report the strongest peak that is plausibly real\n"
|
||||||
|
"signal instead.\n"
|
||||||
|
"Raising the floor also re-masks images already shown or stored\n"
|
||||||
|
"in this file's cache: pixels whose stored peak falls below it\n"
|
||||||
|
"display as invalid. Lowering it below a stored cache's own\n"
|
||||||
|
"floor needs a recompute (bins below that floor were never\n"
|
||||||
|
"searched). Batch Compute FFT records the floor in the file and\n"
|
||||||
|
"re-resolves masked pixels to their true above-floor peak."
|
||||||
|
)
|
||||||
|
self.spin_min_freq_mhz.editingFinished.connect(self._on_min_freq_changed)
|
||||||
|
min_freq_form.addRow("Min peak freq:", self.spin_min_freq_mhz)
|
||||||
|
tl.addLayout(min_freq_form)
|
||||||
vl.addWidget(self.grp_threshold)
|
vl.addWidget(self.grp_threshold)
|
||||||
|
|
||||||
# Background subtraction (v4+ files only)
|
# Background subtraction (v4+ files only)
|
||||||
@@ -417,6 +449,22 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self.chk_auto.toggled.connect(self._on_autoscale_toggled)
|
self.chk_auto.toggled.connect(self._on_autoscale_toggled)
|
||||||
dl.addWidget(self.chk_auto)
|
dl.addWidget(self.chk_auto)
|
||||||
|
|
||||||
|
self.chk_highlight_masked = QCheckBox("Highlight masked (no-data) pixels")
|
||||||
|
self.chk_highlight_masked.setChecked(True)
|
||||||
|
self.chk_highlight_masked.setEnabled(False)
|
||||||
|
self.chk_highlight_masked.setToolTip(
|
||||||
|
"RF/Velocity only. A pixel below the DC threshold has no FFT\n"
|
||||||
|
"result at all and is otherwise shown as 0 on the same grayscale\n"
|
||||||
|
"ramp as a real, valid pixel whose peak just happens to be a low\n"
|
||||||
|
"frequency — the two look identical (both near-black). When\n"
|
||||||
|
"checked, masked pixels are drawn in a distinct highlight color\n"
|
||||||
|
"instead, excluded from the colormap's data range, so real\n"
|
||||||
|
"low-frequency pixels keep their own true shade. Uncheck to\n"
|
||||||
|
"restore the old behavior where both blend together."
|
||||||
|
)
|
||||||
|
self.chk_highlight_masked.toggled.connect(self._on_highlight_masked_toggled)
|
||||||
|
dl.addWidget(self.chk_highlight_masked)
|
||||||
|
|
||||||
range_form = _form()
|
range_form = _form()
|
||||||
for label, attr in (("min:", "spin_vmin"), ("max:", "spin_vmax")):
|
for label, attr in (("min:", "spin_vmin"), ("max:", "spin_vmax")):
|
||||||
spin = _make_dspin(-1e9, 1e9, 4)
|
spin = _make_dspin(-1e9, 1e9, 4)
|
||||||
@@ -440,7 +488,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
|
|
||||||
fft_menu = menubar.addMenu("&FFT")
|
fft_menu = menubar.addMenu("&FFT")
|
||||||
fft_act = QAction("FFT &Options…", self)
|
fft_act = QAction("FFT &Options…", self)
|
||||||
fft_act.setStatusTip("Configure FFT backend and zero-padding")
|
fft_act.setStatusTip("Configure FFT zero-padding")
|
||||||
fft_act.triggered.connect(self._on_fft_options)
|
fft_act.triggered.connect(self._on_fft_options)
|
||||||
fft_menu.addAction(fft_act)
|
fft_menu.addAction(fft_act)
|
||||||
|
|
||||||
@@ -480,6 +528,29 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._batch_fft_rowavg_act.triggered.connect(self._on_batch_compute_row_avg)
|
self._batch_fft_rowavg_act.triggered.connect(self._on_batch_compute_row_avg)
|
||||||
convert_menu.addAction(self._batch_fft_rowavg_act)
|
convert_menu.addAction(self._batch_fft_rowavg_act)
|
||||||
|
|
||||||
|
convert_menu.addSeparator()
|
||||||
|
self._batch_export_images_act = QAction(
|
||||||
|
"Batch Export View as &Images…", self)
|
||||||
|
self._batch_export_images_act.setStatusTip(
|
||||||
|
"Select .sras files and export the currently selected view "
|
||||||
|
"(channel, angle, threshold, colormap, etc.) as one PNG per "
|
||||||
|
"file, rendered the same way it is shown on screen. Read-only "
|
||||||
|
"— never modifies the source files. Aligned View is ignored "
|
||||||
|
"even if enabled, since an alignment result belongs to one "
|
||||||
|
"specific file's geometry.")
|
||||||
|
self._batch_export_images_act.triggered.connect(
|
||||||
|
self._on_batch_export_images)
|
||||||
|
convert_menu.addAction(self._batch_export_images_act)
|
||||||
|
|
||||||
|
export_menu = menubar.addMenu("&Export")
|
||||||
|
self._batch_export_act = QAction("&Batch Export Images…", self)
|
||||||
|
self._batch_export_act.setStatusTip(
|
||||||
|
"Export DC/RF/Velocity map PNGs for every angle of the open "
|
||||||
|
"file, with a fixed colorbar range per channel.")
|
||||||
|
self._batch_export_act.setEnabled(False)
|
||||||
|
self._batch_export_act.triggered.connect(self._on_batch_export)
|
||||||
|
export_menu.addAction(self._batch_export_act)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Drag-and-drop
|
# Drag-and-drop
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -623,10 +694,13 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if s.precomputed_row_avg_n else "")
|
if s.precomputed_row_avg_n else "")
|
||||||
pad_note = (f", pad {s.precomputed_pad_factor}x"
|
pad_note = (f", pad {s.precomputed_pad_factor}x"
|
||||||
if s.precomputed_pad_factor > 1 else "")
|
if s.precomputed_pad_factor > 1 else "")
|
||||||
|
floor_note = (f", floor ≥ {s.precomputed_min_freq_mhz:g} MHz"
|
||||||
|
if s.precomputed_min_freq_mhz > 0 else "")
|
||||||
notes.append(
|
notes.append(
|
||||||
f"Cached images: DC {n_dc}/{s.n_angles} angles, "
|
f"Cached images: DC {n_dc}/{s.n_angles} angles, "
|
||||||
f"FFT {n_fft}/{s.n_angles} angles{bg_note if n_fft else ''}"
|
f"FFT {n_fft}/{s.n_angles} angles{bg_note if n_fft else ''}"
|
||||||
f"{avg_note if n_fft else ''}{pad_note if n_fft else ''}"
|
f"{avg_note if n_fft else ''}{pad_note if n_fft else ''}"
|
||||||
|
f"{floor_note if n_fft else ''} "
|
||||||
"— display is instant for cached angles")
|
"— display is instant for cached angles")
|
||||||
notes += self._cache_mismatch_notes()
|
notes += self._cache_mismatch_notes()
|
||||||
elif s.version == 7:
|
elif s.version == 7:
|
||||||
@@ -635,17 +709,24 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
|
|
||||||
def _cache_mismatch_notes(self) -> list[str]:
|
def _cache_mismatch_notes(self) -> list[str]:
|
||||||
"""Informational only: whether the file's stored FFT cache was
|
"""Informational only: whether the file's stored FFT cache was
|
||||||
computed under different bg-sub/pad settings than these controls
|
computed under different bg-sub/pad/min-freq settings than these
|
||||||
currently say. The display always shows the stored image as-is
|
controls currently say. The display always shows the stored image
|
||||||
regardless (see _stored_fft_image) — these controls only affect a
|
(re-masked for a raised floor) regardless — see _stored_fft_image;
|
||||||
future live compute for an angle with nothing cached yet, or an
|
these controls only affect a future live compute for an angle with
|
||||||
explicit batch recompute, never what's already on screen.
|
nothing cached yet, or an explicit batch recompute, never what's
|
||||||
|
already on screen.
|
||||||
|
|
||||||
row_avg_n is compared against the file's own recorded value (a
|
row_avg_n is compared against the file's own recorded value (a
|
||||||
self-match), so it never contributes a reason here — there's no
|
self-match), so it never contributes a reason here — there's no
|
||||||
live control for it to diverge from, and the "Cached images" line
|
live control for it to diverge from, and the "Cached images" line
|
||||||
above already reports it.
|
above already reports it.
|
||||||
|
|
||||||
|
The min peak frequency floor gets two directions: a live floor
|
||||||
|
*below* the stored one is a genuine mismatch reason (the stored
|
||||||
|
search never looked below its floor), while a live floor *above*
|
||||||
|
the stored one is servable — pixels under it are shown as masked —
|
||||||
|
so that direction gets its own softer note.
|
||||||
|
|
||||||
Asks compute for the reasons rather than restating the accept rule,
|
Asks compute for the reasons rather than restating the accept rule,
|
||||||
so a new provenance field can only be added in one place.
|
so a new provenance field can only be added in one place.
|
||||||
"""
|
"""
|
||||||
@@ -653,15 +734,26 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if s is None or all(x is None for x in s.precomputed_freq_mhz):
|
if s is None or all(x is None for x in s.precomputed_freq_mhz):
|
||||||
return []
|
return []
|
||||||
|
|
||||||
|
live_floor = self.spin_min_freq_mhz.value()
|
||||||
|
notes = []
|
||||||
reasons = compute.cache_mismatch_reasons(
|
reasons = compute.cache_mismatch_reasons(
|
||||||
s, n_fft=self._current_n_fft(),
|
s, n_fft=self._current_n_fft(),
|
||||||
apply_bg_sub=self.chk_bg_sub.isChecked(),
|
apply_bg_sub=self.chk_bg_sub.isChecked(),
|
||||||
row_avg_n=s.precomputed_row_avg_n)
|
row_avg_n=s.precomputed_row_avg_n,
|
||||||
if not reasons:
|
min_freq_mhz=live_floor)
|
||||||
return []
|
if reasons:
|
||||||
return ["Note: current bg-sub/pad controls differ from the stored "
|
notes.append(
|
||||||
"cache — " + "; ".join(reasons) + ". Shown as stored; use "
|
"Note: current bg-sub/pad/min-freq controls differ from the "
|
||||||
"Batch Compute to recompute with these settings."]
|
"stored cache — " + "; ".join(reasons) + ". Shown as stored; "
|
||||||
|
"use Batch Compute to recompute with these settings.")
|
||||||
|
if round(live_floor * 1000) > round(s.precomputed_min_freq_mhz * 1000):
|
||||||
|
notes.append(
|
||||||
|
f"Note: Min peak freq ({live_floor:g} MHz) is above the "
|
||||||
|
f"stored cache's floor ({s.precomputed_min_freq_mhz:g} MHz) — "
|
||||||
|
f"stored pixels whose peak falls below {live_floor:g} MHz are "
|
||||||
|
"shown as masked; Batch Compute FFT re-resolves them above "
|
||||||
|
"the floor instead.")
|
||||||
|
return notes
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Controls
|
# Controls
|
||||||
@@ -684,7 +776,9 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
|
|
||||||
# Threshold and bg-sub apply to all CH1 modes
|
# Threshold and bg-sub apply to all CH1 modes
|
||||||
self.spin_threshold_mv.setEnabled(is_ch1)
|
self.spin_threshold_mv.setEnabled(is_ch1)
|
||||||
|
self.spin_min_freq_mhz.setEnabled(is_ch1)
|
||||||
self.chk_bg_sub.setEnabled(has_file and s.background is not None and is_ch1)
|
self.chk_bg_sub.setEnabled(has_file and s.background is not None and is_ch1)
|
||||||
|
self.chk_highlight_masked.setEnabled(is_ch1)
|
||||||
self.spin_grating_um.setEnabled(is_vel)
|
self.spin_grating_um.setEnabled(is_vel)
|
||||||
self.grp_velocity.setVisible(is_vel)
|
self.grp_velocity.setVisible(is_vel)
|
||||||
|
|
||||||
@@ -702,6 +796,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._wizard_act.setEnabled(
|
self._wizard_act.setEnabled(
|
||||||
has_file and s.n_angles > 1 and self._align_wizard is None)
|
has_file and s.n_angles > 1 and self._align_wizard is None)
|
||||||
self.chk_aligned_view.setEnabled(enabled and self._alignment_result is not None)
|
self.chk_aligned_view.setEnabled(enabled and self._alignment_result is not None)
|
||||||
|
|
||||||
|
self._batch_export_act.setEnabled(has_file and not self._job_running(Jobs.EXPORT))
|
||||||
self._update_roi_ui()
|
self._update_roi_ui()
|
||||||
|
|
||||||
def _on_channel_changed(self):
|
def _on_channel_changed(self):
|
||||||
@@ -733,6 +829,16 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if self._is_fft_mode():
|
if self._is_fft_mode():
|
||||||
self._refresh_display()
|
self._refresh_display()
|
||||||
|
|
||||||
|
def _on_min_freq_changed(self):
|
||||||
|
# Like the DC threshold, this is a genuine cache-key change that can
|
||||||
|
# be cheaply re-applied to a stored image — tighten-only: raising the
|
||||||
|
# floor masks stored pixels below it, while lowering it below the
|
||||||
|
# stored cache's own floor can only be answered by a fresh compute
|
||||||
|
# (see cached_rf_image / cache_mismatch_reasons).
|
||||||
|
if self._is_fft_mode():
|
||||||
|
self._update_scan_info_labels()
|
||||||
|
self._refresh_display()
|
||||||
|
|
||||||
def _on_autoscale_toggled(self, checked: bool):
|
def _on_autoscale_toggled(self, checked: bool):
|
||||||
manual = not checked
|
manual = not checked
|
||||||
self.spin_vmin.setEnabled(manual and self._sras is not None)
|
self.spin_vmin.setEnabled(manual and self._sras is not None)
|
||||||
@@ -740,6 +846,10 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if self._sras is not None and self._current_image is not None:
|
if self._sras is not None and self._current_image is not None:
|
||||||
self._redraw_image(self._current_image)
|
self._redraw_image(self._current_image)
|
||||||
|
|
||||||
|
def _on_highlight_masked_toggled(self, checked: bool):
|
||||||
|
if self._sras is not None and self._current_image is not None:
|
||||||
|
self._redraw_image(self._current_image)
|
||||||
|
|
||||||
def _on_manual_range_changed(self):
|
def _on_manual_range_changed(self):
|
||||||
if not self.chk_auto.isChecked() and self._current_image is not None:
|
if not self.chk_auto.isChecked() and self._current_image is not None:
|
||||||
self._redraw_image(self._current_image)
|
self._redraw_image(self._current_image)
|
||||||
@@ -1014,15 +1124,18 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
return freq_mhz
|
return freq_mhz
|
||||||
|
|
||||||
def _fft_cache_key(self, angle_idx: int) -> tuple:
|
def _fft_cache_key(self, angle_idx: int) -> tuple:
|
||||||
"""Keyed by angle and DC threshold only. Once any FFT image exists
|
"""Keyed by angle, DC threshold, and min peak frequency only. Once
|
||||||
for an angle this session — live-computed or pulled from the file's
|
any FFT image exists for an angle this session — live-computed or
|
||||||
own stored cache — it stays the displayed image for that angle
|
pulled from the file's own stored cache — it stays the displayed
|
||||||
regardless of later bg-sub/pad toggles; those only affect a future
|
image for that angle regardless of later bg-sub/pad toggles; those
|
||||||
live compute for an angle with nothing cached yet, or an explicit
|
only affect a future live compute for an angle with nothing cached
|
||||||
batch recompute (see _stored_fft_image). Threshold stays in the key
|
yet, or an explicit batch recompute (see _stored_fft_image).
|
||||||
because re-masking against it is free and meant to stay interactive
|
Threshold and min-freq are both in the key because both are cheaply
|
||||||
(see _on_threshold_changed)."""
|
re-applied when a stored image is served (_stored_fft_image takes
|
||||||
return (angle_idx, self.spin_threshold_mv.value())
|
them live), so the cached value genuinely reflects every component
|
||||||
|
of its key and revisiting a combination is instant."""
|
||||||
|
return (angle_idx, self.spin_threshold_mv.value(),
|
||||||
|
self.spin_min_freq_mhz.value())
|
||||||
|
|
||||||
def _aligned_cache_key(self, angle_idx: int, ch_idx: int) -> tuple:
|
def _aligned_cache_key(self, angle_idx: int, ch_idx: int) -> tuple:
|
||||||
"""Mirrors _fft_cache's key granularity so a stale aligned image is
|
"""Mirrors _fft_cache's key granularity so a stale aligned image is
|
||||||
@@ -1099,9 +1212,13 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
controls never gate whether it's used, only what a *future* compute
|
controls never gate whether it's used, only what a *future* compute
|
||||||
produces. See _cache_mismatch_notes for the informational (non-
|
produces. See _cache_mismatch_notes for the informational (non-
|
||||||
blocking) note when the live controls diverge from what's shown.
|
blocking) note when the live controls diverge from what's shown.
|
||||||
Only the DC threshold is taken live: re-masking a stored image
|
The DC threshold and the min peak frequency floor are taken live:
|
||||||
against it is free, unlike bg-sub/pad/row-averaging which are baked
|
re-masking a stored image against either is free, unlike
|
||||||
irreversibly into the stored numbers.
|
bg-sub/pad/row-averaging, which are baked irreversibly into the
|
||||||
|
stored numbers. The floor is tighten-only, though — a live floor
|
||||||
|
*below* the stored cache's own floor is the one case where a stored
|
||||||
|
image genuinely can't answer (cached_rf_image refuses it), and the
|
||||||
|
caller falls through to a real compute.
|
||||||
|
|
||||||
allow_dc_recompute=False keeps this off the I/O path: if the mask
|
allow_dc_recompute=False keeps this off the I/O path: if the mask
|
||||||
would mean reading a whole CH4 channel, this declines and the caller
|
would mean reading a whole CH4 channel, this declines and the caller
|
||||||
@@ -1118,7 +1235,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
n_fft=n_fft,
|
n_fft=n_fft,
|
||||||
row_avg_n=s.precomputed_row_avg_n,
|
row_avg_n=s.precomputed_row_avg_n,
|
||||||
dc4_mv=self._dc_cache.get((angle_idx, CH4_IDX)),
|
dc4_mv=self._dc_cache.get((angle_idx, CH4_IDX)),
|
||||||
allow_dc_recompute=False)
|
allow_dc_recompute=False,
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value())
|
||||||
|
|
||||||
def _cached_value_image(self, angle_idx: int, ch_idx: int) -> np.ndarray | None:
|
def _cached_value_image(self, angle_idx: int, ch_idx: int) -> np.ndarray | None:
|
||||||
"""The already-available (no compute) image for (angle, channel):
|
"""The already-available (no compute) image for (angle, channel):
|
||||||
@@ -1194,6 +1312,24 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._redraw_image(img)
|
self._redraw_image(img)
|
||||||
self._update_roi_ui()
|
self._update_roi_ui()
|
||||||
|
|
||||||
|
def _dc_validity_mask(self, angle_idx: int, aligned: bool) -> np.ndarray | None:
|
||||||
|
"""True where a CH1/Velocity pixel passed the DC threshold and so
|
||||||
|
actually has a real FFT result, on the same grid as display_img.
|
||||||
|
None if the CH4 DC image for this angle isn't cached yet — the
|
||||||
|
background DC precompute hasn't reached it, and this is deliberately
|
||||||
|
not worth a synchronous recompute just to redraw."""
|
||||||
|
dc4 = self._dc_cache.get((angle_idx, CH4_IDX))
|
||||||
|
if dc4 is None:
|
||||||
|
return None
|
||||||
|
valid = dc4 >= self.spin_threshold_mv.value()
|
||||||
|
if aligned:
|
||||||
|
# apply_alignment defaults to nearest-neighbor (order=0), so a
|
||||||
|
# 0.0/1.0 float warp stays exactly 0 or 1 - no blending at mask
|
||||||
|
# edges to second-guess with a >= 0.5 cutoff.
|
||||||
|
valid = apply_alignment(self._alignment_result, angle_idx,
|
||||||
|
valid.astype(np.float32)) >= 0.5
|
||||||
|
return valid
|
||||||
|
|
||||||
def _redraw_image(self, img: np.ndarray):
|
def _redraw_image(self, img: np.ndarray):
|
||||||
s = self._sras
|
s = self._sras
|
||||||
angle_idx = self._current_angle
|
angle_idx = self._current_angle
|
||||||
@@ -1214,8 +1350,32 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
dy = float(y_axis[1] - y_axis[0]) if len(y_axis) > 1 else 1.0
|
dy = float(y_axis[1] - y_axis[0]) if len(y_axis) > 1 else 1.0
|
||||||
extent = _axes_extent(x_axis, y_axis, dx, dy)
|
extent = _axes_extent(x_axis, y_axis, dx, dy)
|
||||||
|
|
||||||
|
# Masked-out pixels are stored as a plain 0, the same value near
|
||||||
|
# the bottom of a linear colormap as a real low-frequency pixel -
|
||||||
|
# the two are indistinguishable there. Pull masked pixels out to
|
||||||
|
# NaN (drawn in a distinct highlight color, excluded from the
|
||||||
|
# auto-scale range) so a real pixel keeps its own true shade
|
||||||
|
# instead of disappearing into the same black as "no data". In an
|
||||||
|
# FFT-derived mode a value of exactly 0 is itself the "no valid
|
||||||
|
# peak" sentinel (DC-masked, below the min-freq floor, or an empty
|
||||||
|
# spectrum — bin 0 is always excluded, so no genuine peak is ever
|
||||||
|
# 0), so those pixels are masked by value too; that covers
|
||||||
|
# floor-masked pixels even when no DC4 image is cached yet.
|
||||||
|
highlight_masked = (ch_idx in CH1_DERIVED_MODES
|
||||||
|
and self.chk_highlight_masked.isChecked())
|
||||||
|
if highlight_masked:
|
||||||
|
mask_valid = self._dc_validity_mask(angle_idx, aligned)
|
||||||
|
if mask_valid is not None and mask_valid.shape != display_img.shape:
|
||||||
|
mask_valid = None
|
||||||
|
display_img = display_img.astype(np.float32, copy=True)
|
||||||
|
if mask_valid is not None:
|
||||||
|
display_img[~mask_valid] = np.nan
|
||||||
|
display_img[display_img == 0.0] = np.nan
|
||||||
|
|
||||||
if self.chk_auto.isChecked():
|
if self.chk_auto.isChecked():
|
||||||
vmin, vmax = float(display_img.min()), float(display_img.max())
|
vmin, vmax = float(np.nanmin(display_img)), float(np.nanmax(display_img))
|
||||||
|
if not np.isfinite(vmin):
|
||||||
|
vmin, vmax = 0.0, 0.0 # every pixel masked out
|
||||||
for spin, val in ((self.spin_vmin, vmin), (self.spin_vmax, vmax)):
|
for spin, val in ((self.spin_vmin, vmin), (self.spin_vmax, vmax)):
|
||||||
with QSignalBlocker(spin):
|
with QSignalBlocker(spin):
|
||||||
spin.setValue(val)
|
spin.setValue(val)
|
||||||
@@ -1239,6 +1399,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
vmin=vmin, vmax=vmax,
|
vmin=vmin, vmax=vmax,
|
||||||
xlabel="X (mm)", ylabel="Y (mm)",
|
xlabel="X (mm)", ylabel="Y (mm)",
|
||||||
title=title, colorbar_label=colorbar_label,
|
title=title, colorbar_label=colorbar_label,
|
||||||
|
bad_color=_MASKED_HIGHLIGHT_COLOR if highlight_masked else None,
|
||||||
)
|
)
|
||||||
self.statusBar().showMessage(
|
self.statusBar().showMessage(
|
||||||
f"{s.path.name} | {ch_label} @ {angle_deg:.1f}° "
|
f"{s.path.name} | {ch_label} @ {angle_deg:.1f}° "
|
||||||
@@ -1262,6 +1423,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._pending_ch = ch_idx
|
self._pending_ch = ch_idx
|
||||||
self._pending_bg_sub = self.chk_bg_sub.isChecked()
|
self._pending_bg_sub = self.chk_bg_sub.isChecked()
|
||||||
self._pending_threshold = self.spin_threshold_mv.value()
|
self._pending_threshold = self.spin_threshold_mv.value()
|
||||||
|
self._pending_min_freq_mhz = self.spin_min_freq_mhz.value()
|
||||||
self._pending_fft_pad_factor = self._fft_pad_factor
|
self._pending_fft_pad_factor = self._fft_pad_factor
|
||||||
|
|
||||||
worker = ComputeWorker(
|
worker = ComputeWorker(
|
||||||
@@ -1274,6 +1436,11 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
# need to re-read the CH4 channel from disk.
|
# need to re-read the CH4 channel from disk.
|
||||||
dc4_mv=self._dc_cache.get((angle_idx, CH4_IDX)),
|
dc4_mv=self._dc_cache.get((angle_idx, CH4_IDX)),
|
||||||
is_fft_mode=is_fft,
|
is_fft_mode=is_fft,
|
||||||
|
# Mirror _stored_fft_image: a stored row-averaged cache must stay
|
||||||
|
# eligible even when this fallback compute is the one that ends
|
||||||
|
# up serving it (e.g. before DC precompute reaches this angle).
|
||||||
|
row_avg_n=self._sras.precomputed_row_avg_n,
|
||||||
|
min_freq_mhz=self._pending_min_freq_mhz,
|
||||||
)
|
)
|
||||||
if not self._run_worker(
|
if not self._run_worker(
|
||||||
Jobs.COMPUTE, worker,
|
Jobs.COMPUTE, worker,
|
||||||
@@ -1302,9 +1469,10 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
already be cached."""
|
already be cached."""
|
||||||
if (self.spin_angle.value(), self.combo_channel.currentIndex(),
|
if (self.spin_angle.value(), self.combo_channel.currentIndex(),
|
||||||
self.chk_bg_sub.isChecked(), self.spin_threshold_mv.value(),
|
self.chk_bg_sub.isChecked(), self.spin_threshold_mv.value(),
|
||||||
self._fft_pad_factor) != (
|
self.spin_min_freq_mhz.value(), self._fft_pad_factor) != (
|
||||||
self._pending_angle, self._pending_ch, self._pending_bg_sub,
|
self._pending_angle, self._pending_ch, self._pending_bg_sub,
|
||||||
self._pending_threshold, self._pending_fft_pad_factor):
|
self._pending_threshold, self._pending_min_freq_mhz,
|
||||||
|
self._pending_fft_pad_factor):
|
||||||
self._refresh_display()
|
self._refresh_display()
|
||||||
|
|
||||||
def _on_compute_done(self, result):
|
def _on_compute_done(self, result):
|
||||||
@@ -1315,7 +1483,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
ch_idx = self._pending_ch
|
ch_idx = self._pending_ch
|
||||||
|
|
||||||
if ch_idx in CH1_DERIVED_MODES:
|
if ch_idx in CH1_DERIVED_MODES:
|
||||||
self._fft_cache[(angle_idx, self._pending_threshold)] = result
|
key = (angle_idx, self._pending_threshold, self._pending_min_freq_mhz)
|
||||||
|
self._fft_cache[key] = result
|
||||||
img = self._scale_for_display(result, ch_idx)
|
img = self._scale_for_display(result, ch_idx)
|
||||||
else:
|
else:
|
||||||
img = result
|
img = result
|
||||||
@@ -1394,7 +1563,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if self._current_ch in CH1_DERIVED_MODES:
|
if self._current_ch in CH1_DERIVED_MODES:
|
||||||
self.wave_canvas.show_rf_waveform(
|
self.wave_canvas.show_rf_waveform(
|
||||||
self._sras, angle_idx, row_idx, frame_idx,
|
self._sras, angle_idx, row_idx, frame_idx,
|
||||||
apply_bg_sub=self.chk_bg_sub.isChecked())
|
apply_bg_sub=self.chk_bg_sub.isChecked(),
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value())
|
||||||
else:
|
else:
|
||||||
self.wave_canvas.show_dc_waveform(
|
self.wave_canvas.show_dc_waveform(
|
||||||
self._sras, angle_idx, self._current_ch, row_idx, frame_idx)
|
self._sras, angle_idx, self._current_ch, row_idx, frame_idx)
|
||||||
@@ -1446,7 +1616,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
# Cache the FFT at the pad the viewer is actually displaying at,
|
# Cache the FFT at the pad the viewer is actually displaying at,
|
||||||
# otherwise the batch stores images this window can never use.
|
# otherwise the batch stores images this window can never use.
|
||||||
worker = BatchCacheWorker(paths, mode, self.chk_bg_sub.isChecked(),
|
worker = BatchCacheWorker(paths, mode, self.chk_bg_sub.isChecked(),
|
||||||
pad_factor=self._fft_pad_factor)
|
pad_factor=self._fft_pad_factor,
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value())
|
||||||
started = self._run_worker(
|
started = self._run_worker(
|
||||||
Jobs.BATCH, worker,
|
Jobs.BATCH, worker,
|
||||||
connect=(
|
connect=(
|
||||||
@@ -1459,9 +1630,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if not started:
|
if not started:
|
||||||
return # a second trigger snuck in while the file dialog was open
|
return # a second trigger snuck in while the file dialog was open
|
||||||
|
|
||||||
self._batch_dc_act.setEnabled(False)
|
self._set_batch_actions_enabled(False)
|
||||||
self._batch_fft_act.setEnabled(False)
|
|
||||||
self._batch_fft_rowavg_act.setEnabled(False)
|
|
||||||
self._show_progress(
|
self._show_progress(
|
||||||
Jobs.BATCH, f"Batch computing {label} for {len(paths)} file(s)…",
|
Jobs.BATCH, f"Batch computing {label} for {len(paths)} file(s)…",
|
||||||
maximum=100)
|
maximum=100)
|
||||||
@@ -1488,7 +1657,8 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._batch_errors = []
|
self._batch_errors = []
|
||||||
worker = BatchCacheWorker(paths, "fft_rowavg", self.chk_bg_sub.isChecked(),
|
worker = BatchCacheWorker(paths, "fft_rowavg", self.chk_bg_sub.isChecked(),
|
||||||
dc_threshold_mv=threshold_mv, row_avg_n=n,
|
dc_threshold_mv=threshold_mv, row_avg_n=n,
|
||||||
pad_factor=self._fft_pad_factor)
|
pad_factor=self._fft_pad_factor,
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value())
|
||||||
started = self._run_worker(
|
started = self._run_worker(
|
||||||
Jobs.BATCH, worker,
|
Jobs.BATCH, worker,
|
||||||
connect=(
|
connect=(
|
||||||
@@ -1501,9 +1671,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
if not started:
|
if not started:
|
||||||
return # a second trigger snuck in while a dialog was open
|
return # a second trigger snuck in while a dialog was open
|
||||||
|
|
||||||
self._batch_dc_act.setEnabled(False)
|
self._set_batch_actions_enabled(False)
|
||||||
self._batch_fft_act.setEnabled(False)
|
|
||||||
self._batch_fft_rowavg_act.setEnabled(False)
|
|
||||||
self._show_progress(
|
self._show_progress(
|
||||||
Jobs.BATCH,
|
Jobs.BATCH,
|
||||||
f"Batch computing row-averaged FFT (n={n}, threshold={threshold_mv:.1f} mV) "
|
f"Batch computing row-averaged FFT (n={n}, threshold={threshold_mv:.1f} mV) "
|
||||||
@@ -1534,9 +1702,225 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._load_file(str(self._sras.path))
|
self._load_file(str(self._sras.path))
|
||||||
|
|
||||||
def _after_batch(self):
|
def _after_batch(self):
|
||||||
self._batch_dc_act.setEnabled(True)
|
self._set_batch_actions_enabled(True)
|
||||||
self._batch_fft_act.setEnabled(True)
|
|
||||||
self._batch_fft_rowavg_act.setEnabled(True)
|
def _set_batch_actions_enabled(self, enabled: bool):
|
||||||
|
"""All four Convert-menu batch actions share one Jobs.BATCH slot;
|
||||||
|
grey out every sibling while one runs, rather than leaving one
|
||||||
|
clickable but silently no-op'd by the busy guard."""
|
||||||
|
for act in (self._batch_dc_act, self._batch_fft_act,
|
||||||
|
self._batch_fft_rowavg_act, self._batch_export_images_act):
|
||||||
|
act.setEnabled(enabled)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Batch export view as images
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _view_figsize(self) -> tuple[float, float]:
|
||||||
|
"""The image canvas's current figure size in inches — handed to every
|
||||||
|
image export so the PNG has the shape the view is being read at.
|
||||||
|
|
||||||
|
The canvas draws with aspect="auto" and expands with the window, so
|
||||||
|
the map's proportions are the widget's, not the data's; a renderer
|
||||||
|
with its own hard-coded size necessarily distorts the export
|
||||||
|
relative to the screen.
|
||||||
|
|
||||||
|
Inches rather than pixels because that is what Figure takes, and
|
||||||
|
matplotlib's Qt canvas keeps size_inches in logical units (it scales
|
||||||
|
figure.dpi by the device pixel ratio and divides physical pixels by
|
||||||
|
it), so the same window gives the same export on a HiDPI display as
|
||||||
|
on a standard one. Read at trigger time: a resize mid-batch must not
|
||||||
|
change the shape of images later in the same run.
|
||||||
|
"""
|
||||||
|
w, h = self.image_canvas.figure.get_size_inches()
|
||||||
|
return (float(w), float(h))
|
||||||
|
|
||||||
|
def _on_batch_export_images(self):
|
||||||
|
if self._job_running(Jobs.BATCH):
|
||||||
|
return
|
||||||
|
paths, _ = QFileDialog.getOpenFileNames(
|
||||||
|
self, "Select .sras files to export the current view from", "",
|
||||||
|
"SRAS files (*.sras);;All files (*)")
|
||||||
|
if not paths:
|
||||||
|
return
|
||||||
|
out_dir = QFileDialog.getExistingDirectory(
|
||||||
|
self, "Select output folder for exported images")
|
||||||
|
if not out_dir:
|
||||||
|
return
|
||||||
|
|
||||||
|
angle_idx = self.spin_angle.value()
|
||||||
|
ch_idx = self.combo_channel.currentIndex()
|
||||||
|
is_fft_mode = ch_idx in CH1_DERIVED_MODES
|
||||||
|
mode_str, _unit, colorbar_label = _CHANNEL_DISPLAY[ch_idx]
|
||||||
|
|
||||||
|
self._batch_errors = []
|
||||||
|
self._batch_export_names = []
|
||||||
|
worker = BatchExportImagesWorker(
|
||||||
|
paths,
|
||||||
|
out_dir=out_dir, angle_idx=angle_idx, ch_idx=ch_idx,
|
||||||
|
is_fft_mode=is_fft_mode, is_velocity=(ch_idx == VELOCITY_MODE_IDX),
|
||||||
|
dc_threshold_mv=self.spin_threshold_mv.value(),
|
||||||
|
apply_bg_sub=self.chk_bg_sub.isChecked(),
|
||||||
|
pad_factor=self._fft_pad_factor,
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value(),
|
||||||
|
grating_um=self.spin_grating_um.value(),
|
||||||
|
cmap=self.combo_cmap.currentText(),
|
||||||
|
auto_scale=self.chk_auto.isChecked(),
|
||||||
|
vmin=self.spin_vmin.value(), vmax=self.spin_vmax.value(),
|
||||||
|
highlight_masked=self.chk_highlight_masked.isChecked(),
|
||||||
|
mode_str=mode_str, colorbar_label=colorbar_label,
|
||||||
|
mask_color=_MASKED_HIGHLIGHT_COLOR,
|
||||||
|
figsize=self._view_figsize(),
|
||||||
|
)
|
||||||
|
started = self._run_worker(
|
||||||
|
Jobs.BATCH, worker,
|
||||||
|
connect=(
|
||||||
|
("progress", lambda pct: self._set_progress(Jobs.BATCH, pct)),
|
||||||
|
("file_done", self._on_batch_export_file_done),
|
||||||
|
("finished",
|
||||||
|
lambda p=paths, d=out_dir: self._on_batch_export_finished(p, d)),
|
||||||
|
),
|
||||||
|
on_done=self._after_batch,
|
||||||
|
)
|
||||||
|
if not started:
|
||||||
|
return # a second trigger snuck in while a dialog was open
|
||||||
|
|
||||||
|
self._set_batch_actions_enabled(False)
|
||||||
|
self._show_progress(
|
||||||
|
Jobs.BATCH, f"Exporting images for {len(paths)} file(s)…",
|
||||||
|
maximum=100)
|
||||||
|
|
||||||
|
def _on_batch_export_file_done(self, path: str, err: str, out_name: str):
|
||||||
|
if err:
|
||||||
|
self._batch_errors.append(f"{Path(path).name} — {err}")
|
||||||
|
else:
|
||||||
|
self._batch_export_names.append(out_name)
|
||||||
|
self._show_progress(Jobs.BATCH, f"Exported {Path(path).name}…")
|
||||||
|
|
||||||
|
def _on_batch_export_finished(self, paths: list[str], out_dir: str):
|
||||||
|
self._close_progress(Jobs.BATCH)
|
||||||
|
|
||||||
|
n_total = len(paths)
|
||||||
|
n_failed = len(self._batch_errors)
|
||||||
|
n_ok = n_total - n_failed
|
||||||
|
summary = (f"Batch export: {n_ok}/{n_total} image(s) written to "
|
||||||
|
f"{Path(out_dir).name}")
|
||||||
|
if n_failed:
|
||||||
|
summary += f", {n_failed} failed: {'; '.join(self._batch_errors)}"
|
||||||
|
dupes = sum(1 for _name, count in Counter(self._batch_export_names).items()
|
||||||
|
if count > 1)
|
||||||
|
if dupes:
|
||||||
|
summary += (f" | {dupes} filename collision(s) — later file(s) "
|
||||||
|
"overwrote earlier ones with the same output name")
|
||||||
|
self.statusBar().showMessage(summary)
|
||||||
|
self._batch_errors = []
|
||||||
|
self._batch_export_names = []
|
||||||
|
# No reload: unlike Batch Compute, export never touches the source files.
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Export menu: batch image export
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _export_default_ranges(self) -> dict[tuple[int, bool], tuple[float, float]]:
|
||||||
|
"""Best-effort default min/max per exportable channel, from whatever
|
||||||
|
is already cached — this must never trigger a fresh compute just to
|
||||||
|
seed the dialog (CH1/Velocity in particular can be expensive)."""
|
||||||
|
s = self._sras
|
||||||
|
ranges: dict[tuple[int, bool], tuple[float, float]] = {}
|
||||||
|
|
||||||
|
for ch_idx, precomputed in ((CH3_IDX, s.precomputed_dc3_mv),
|
||||||
|
(CH4_IDX, s.precomputed_dc4_mv)):
|
||||||
|
arrays = [self._dc_cache.get((a, ch_idx)) for a in range(s.n_angles)]
|
||||||
|
arrays = [a for a in arrays if a is not None]
|
||||||
|
if not arrays:
|
||||||
|
arrays = [a for a in precomputed if a is not None]
|
||||||
|
if arrays:
|
||||||
|
ranges[(ch_idx, False)] = (
|
||||||
|
float(min(a.min() for a in arrays)),
|
||||||
|
float(max(a.max() for a in arrays)))
|
||||||
|
|
||||||
|
freq_arrays = list(self._fft_cache.values())
|
||||||
|
if not freq_arrays:
|
||||||
|
freq_arrays = [a for a in s.precomputed_freq_mhz if a is not None]
|
||||||
|
if freq_arrays:
|
||||||
|
fmin = float(min(a.min() for a in freq_arrays))
|
||||||
|
fmax = float(max(a.max() for a in freq_arrays))
|
||||||
|
ranges[(CH1_IDX, False)] = (fmin, fmax)
|
||||||
|
grating = self.spin_grating_um.value()
|
||||||
|
ranges[(CH1_IDX, True)] = (fmin * grating, fmax * grating)
|
||||||
|
|
||||||
|
return ranges
|
||||||
|
|
||||||
|
def _on_batch_export(self):
|
||||||
|
if self._sras is None or self._job_running(Jobs.EXPORT):
|
||||||
|
return
|
||||||
|
s = self._sras
|
||||||
|
|
||||||
|
dlg = BatchExportDialog(
|
||||||
|
self, default_dir=str(s.path.parent), default_prefix=s.path.stem,
|
||||||
|
default_ranges=self._export_default_ranges())
|
||||||
|
if dlg.exec() != QDialog.DialogCode.Accepted:
|
||||||
|
return
|
||||||
|
|
||||||
|
output_dir = dlg.get_output_dir()
|
||||||
|
try:
|
||||||
|
Path(output_dir).mkdir(parents=True, exist_ok=True)
|
||||||
|
except OSError as exc:
|
||||||
|
QMessageBox.warning(
|
||||||
|
self, "Batch Export", f"Could not create output folder:\n{exc}")
|
||||||
|
return
|
||||||
|
|
||||||
|
channels = dlg.get_selected_channels()
|
||||||
|
self._export_errors = []
|
||||||
|
self._export_ok_count = 0
|
||||||
|
worker = BatchExportWorker(
|
||||||
|
s, channels, output_dir, dlg.get_prefix(),
|
||||||
|
cmap=self.combo_cmap.currentText(),
|
||||||
|
apply_bg_sub=self.chk_bg_sub.isChecked(),
|
||||||
|
dc_threshold_mv=self.spin_threshold_mv.value(),
|
||||||
|
n_fft=self._current_n_fft(), grating_um=self.spin_grating_um.value(),
|
||||||
|
# Live, like every other display control here and like
|
||||||
|
# _stored_fft_image: the exported RF/Velocity map has to be the
|
||||||
|
# one on screen, floor included.
|
||||||
|
min_freq_mhz=self.spin_min_freq_mhz.value(),
|
||||||
|
figsize=self._view_figsize())
|
||||||
|
started = self._run_worker(
|
||||||
|
Jobs.EXPORT, worker,
|
||||||
|
connect=(
|
||||||
|
("progress", lambda pct: self._set_progress(Jobs.EXPORT, pct)),
|
||||||
|
("file_done", self._on_export_file_done),
|
||||||
|
("finished", self._on_export_finished),
|
||||||
|
),
|
||||||
|
on_done=self._after_export,
|
||||||
|
)
|
||||||
|
if not started:
|
||||||
|
return # a second trigger snuck in while the dialog was open
|
||||||
|
|
||||||
|
self._batch_export_act.setEnabled(False)
|
||||||
|
self._show_progress(
|
||||||
|
Jobs.EXPORT, f"Exporting images to {output_dir}…", maximum=100)
|
||||||
|
|
||||||
|
def _on_export_file_done(self, path: str, err: str):
|
||||||
|
if err:
|
||||||
|
self._export_errors.append(f"{Path(path).name if path else '?'} — {err}")
|
||||||
|
else:
|
||||||
|
self._export_ok_count += 1
|
||||||
|
self._show_progress(Jobs.EXPORT, f"Wrote {Path(path).name}…")
|
||||||
|
|
||||||
|
def _on_export_finished(self):
|
||||||
|
self._close_progress(Jobs.EXPORT)
|
||||||
|
n_ok = self._export_ok_count
|
||||||
|
n_failed = len(self._export_errors)
|
||||||
|
if n_failed:
|
||||||
|
summary = (f"Batch export: {n_ok} image(s) written, {n_failed} "
|
||||||
|
f"failed: {'; '.join(self._export_errors)}")
|
||||||
|
else:
|
||||||
|
summary = f"Batch export: {n_ok} image(s) written."
|
||||||
|
self.statusBar().showMessage(summary)
|
||||||
|
self._export_errors = []
|
||||||
|
|
||||||
|
def _after_export(self):
|
||||||
|
self._update_controls_enabled(self._sras is not None)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Fusion: angle alignment
|
# Fusion: angle alignment
|
||||||
@@ -1620,7 +2004,6 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
def _on_fft_options(self):
|
def _on_fft_options(self):
|
||||||
dlg = FftOptionsDialog(
|
dlg = FftOptionsDialog(
|
||||||
self,
|
self,
|
||||||
current_backend=compute.get_fft_backend(),
|
|
||||||
current_pad_factor=self._fft_pad_factor,
|
current_pad_factor=self._fft_pad_factor,
|
||||||
samples_per_frame=self._sras.samples_per_frame if self._sras else None,
|
samples_per_frame=self._sras.samples_per_frame if self._sras else None,
|
||||||
sample_rate_hz=self._sras.sample_rate_hz if self._sras else None,
|
sample_rate_hz=self._sras.sample_rate_hz if self._sras else None,
|
||||||
@@ -1628,9 +2011,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
)
|
)
|
||||||
if dlg.exec() != QDialog.DialogCode.Accepted:
|
if dlg.exec() != QDialog.DialogCode.Accepted:
|
||||||
return
|
return
|
||||||
compute.set_fft_backend(dlg.get_backend())
|
|
||||||
self._fft_pad_factor = dlg.get_pad_factor()
|
self._fft_pad_factor = dlg.get_pad_factor()
|
||||||
self._settings.setValue("fft/backend", compute.get_fft_backend())
|
|
||||||
self._settings.setValue("fft/pad_factor", self._fft_pad_factor)
|
self._settings.setValue("fft/pad_factor", self._fft_pad_factor)
|
||||||
# Pad factor no longer gates the display: it only affects a future
|
# Pad factor no longer gates the display: it only affects a future
|
||||||
# live compute for an angle with nothing cached yet, or an explicit
|
# live compute for an angle with nothing cached yet, or an explicit
|
||||||
|
|||||||
+286
-9
@@ -10,14 +10,19 @@ everything they need through their constructor and hand results back by signal.
|
|||||||
import os
|
import os
|
||||||
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
|
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
|
||||||
from concurrent.futures.process import BrokenProcessPool
|
from concurrent.futures.process import BrokenProcessPool
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
from matplotlib.backends.backend_agg import FigureCanvasAgg
|
||||||
|
from matplotlib.figure import Figure
|
||||||
from PyQt6.QtCore import QObject, pyqtSignal
|
from PyQt6.QtCore import QObject, pyqtSignal
|
||||||
|
|
||||||
import sras_compute as compute
|
import sras_compute as compute
|
||||||
from sras_align_export import write_aligned_sras
|
from sras_align_export import write_aligned_sras
|
||||||
from sras_compute import cache_file, compute_rf_image, dc_image_mv
|
from sras_compute import cache_file, compute_rf_image, dc_image_mv
|
||||||
from sras_format import CH3_IDX, CH4_IDX, SrasFile
|
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, SrasFile
|
||||||
|
from sras_render import export_view_image, sanitize_figsize
|
||||||
|
|
||||||
# Concurrency caps. Batch conversion runs one process per file, and each of
|
# Concurrency caps. Batch conversion runs one process per file, and each of
|
||||||
# those processes threads internally, so the two must be divided rather than
|
# those processes threads internally, so the two must be divided rather than
|
||||||
@@ -115,7 +120,9 @@ class ComputeWorker(CancellableWorker):
|
|||||||
apply_bg_sub: bool = True, n_fft: int | None = None,
|
apply_bg_sub: bool = True, n_fft: int | None = None,
|
||||||
dc_threshold_mv: float = 0.0,
|
dc_threshold_mv: float = 0.0,
|
||||||
dc4_mv: np.ndarray | None = None,
|
dc4_mv: np.ndarray | None = None,
|
||||||
is_fft_mode: bool = False):
|
is_fft_mode: bool = False,
|
||||||
|
row_avg_n: int = 0,
|
||||||
|
min_freq_mhz: float = 0.0):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self._sras = sras
|
self._sras = sras
|
||||||
self._angle = angle_idx
|
self._angle = angle_idx
|
||||||
@@ -125,6 +132,8 @@ class ComputeWorker(CancellableWorker):
|
|||||||
self._dc_threshold = dc_threshold_mv
|
self._dc_threshold = dc_threshold_mv
|
||||||
self._dc4_mv = dc4_mv
|
self._dc4_mv = dc4_mv
|
||||||
self._is_fft_mode = is_fft_mode
|
self._is_fft_mode = is_fft_mode
|
||||||
|
self._row_avg_n = row_avg_n
|
||||||
|
self._min_freq_mhz = min_freq_mhz
|
||||||
|
|
||||||
def run(self):
|
def run(self):
|
||||||
try:
|
try:
|
||||||
@@ -132,7 +141,9 @@ class ComputeWorker(CancellableWorker):
|
|||||||
img = compute_rf_image(
|
img = compute_rf_image(
|
||||||
self._sras, self._angle, dc_threshold_mv=self._dc_threshold,
|
self._sras, self._angle, dc_threshold_mv=self._dc_threshold,
|
||||||
apply_bg_sub=self._apply_bg_sub, n_fft=self._n_fft,
|
apply_bg_sub=self._apply_bg_sub, n_fft=self._n_fft,
|
||||||
dc4_mv=self._dc4_mv, should_stop=self._stopped)
|
dc4_mv=self._dc4_mv, should_stop=self._stopped,
|
||||||
|
row_avg_n=self._row_avg_n,
|
||||||
|
min_freq_mhz=self._min_freq_mhz)
|
||||||
else:
|
else:
|
||||||
img = dc_image_mv(self._sras, self._angle, self._ch,
|
img = dc_image_mv(self._sras, self._angle, self._ch,
|
||||||
should_stop=self._stopped)
|
should_stop=self._stopped)
|
||||||
@@ -194,7 +205,9 @@ class BatchCacheWorker(QObject):
|
|||||||
|
|
||||||
Both FFT modes cache at *pad_factor*, which the caller sets from the
|
Both FFT modes cache at *pad_factor*, which the caller sets from the
|
||||||
viewer's own padding — a cache stored at a pad the user is not viewing
|
viewer's own padding — a cache stored at a pad the user is not viewing
|
||||||
at is one the display can never use.
|
at is one the display can never use. *min_freq_mhz* travels the same
|
||||||
|
way: the viewer's live min-peak-freq floor, recorded in the store as
|
||||||
|
provenance so a reader knows which bins the peak search considered.
|
||||||
|
|
||||||
Files are processed one per subprocess: they are fully independent, each
|
Files are processed one per subprocess: they are fully independent, each
|
||||||
opens its own memmap and writes only its own bytes, and only path strings
|
opens its own memmap and writes only its own bytes, and only path strings
|
||||||
@@ -208,7 +221,7 @@ class BatchCacheWorker(QObject):
|
|||||||
|
|
||||||
def __init__(self, paths: list[str], mode: str, apply_bg_sub: bool,
|
def __init__(self, paths: list[str], mode: str, apply_bg_sub: bool,
|
||||||
dc_threshold_mv: float | None = None, row_avg_n: int = 0,
|
dc_threshold_mv: float | None = None, row_avg_n: int = 0,
|
||||||
pad_factor: int = 1):
|
pad_factor: int = 1, min_freq_mhz: float = 0.0):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self._paths = paths
|
self._paths = paths
|
||||||
self._mode = mode
|
self._mode = mode
|
||||||
@@ -216,6 +229,7 @@ class BatchCacheWorker(QObject):
|
|||||||
self._dc_threshold = dc_threshold_mv
|
self._dc_threshold = dc_threshold_mv
|
||||||
self._row_avg_n = row_avg_n
|
self._row_avg_n = row_avg_n
|
||||||
self._pad_factor = pad_factor
|
self._pad_factor = pad_factor
|
||||||
|
self._min_freq_mhz = min_freq_mhz
|
||||||
|
|
||||||
def _report(self, path: str, err: str, done: int, total: int):
|
def _report(self, path: str, err: str, done: int, total: int):
|
||||||
self.file_done.emit(path, err)
|
self.file_done.emit(path, err)
|
||||||
@@ -240,10 +254,11 @@ class BatchCacheWorker(QObject):
|
|||||||
with ProcessPoolExecutor(max_workers=n_procs) as executor:
|
with ProcessPoolExecutor(max_workers=n_procs) as executor:
|
||||||
futures = {
|
futures = {
|
||||||
executor.submit(cache_file, p, self._mode, self._apply_bg_sub,
|
executor.submit(cache_file, p, self._mode, self._apply_bg_sub,
|
||||||
compute.get_fft_backend(), per_proc_workers,
|
per_proc_workers,
|
||||||
pad_factor=self._pad_factor,
|
pad_factor=self._pad_factor,
|
||||||
dc_threshold_mv=self._dc_threshold,
|
dc_threshold_mv=self._dc_threshold,
|
||||||
row_avg_n=self._row_avg_n): p
|
row_avg_n=self._row_avg_n,
|
||||||
|
min_freq_mhz=self._min_freq_mhz): p
|
||||||
for p in paths
|
for p in paths
|
||||||
}
|
}
|
||||||
for fut in as_completed(futures):
|
for fut in as_completed(futures):
|
||||||
@@ -266,11 +281,11 @@ class BatchCacheWorker(QObject):
|
|||||||
for path in paths:
|
for path in paths:
|
||||||
try:
|
try:
|
||||||
err = cache_file(path, self._mode, self._apply_bg_sub,
|
err = cache_file(path, self._mode, self._apply_bg_sub,
|
||||||
compute.get_fft_backend(),
|
|
||||||
compute.default_max_workers(),
|
compute.default_max_workers(),
|
||||||
pad_factor=self._pad_factor,
|
pad_factor=self._pad_factor,
|
||||||
dc_threshold_mv=self._dc_threshold,
|
dc_threshold_mv=self._dc_threshold,
|
||||||
row_avg_n=self._row_avg_n)
|
row_avg_n=self._row_avg_n,
|
||||||
|
min_freq_mhz=self._min_freq_mhz)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
err = str(exc)
|
err = str(exc)
|
||||||
done += 1
|
done += 1
|
||||||
@@ -308,6 +323,268 @@ class BatchCacheWorker(QObject):
|
|||||||
self.finished.emit()
|
self.finished.emit()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExportChannel:
|
||||||
|
"""One row of a batch image export: which raw channel to read, whether
|
||||||
|
to apply the Velocity post-multiply, and the fixed display range/labels
|
||||||
|
to render it with.
|
||||||
|
|
||||||
|
Kept free of sras_viewer's display constants (CH_LABELS, VELOCITY_MODE_IDX,
|
||||||
|
etc.) so BatchExportWorker has no dependency on the GUI module — the
|
||||||
|
caller resolves labels/units/filename tags once, up front.
|
||||||
|
"""
|
||||||
|
ch_idx: int # CH1_IDX, CH3_IDX, or CH4_IDX -- which raw data to read
|
||||||
|
is_velocity: bool # True only for the derived Velocity map (post-multiply of CH1 freq)
|
||||||
|
vmin: float
|
||||||
|
vmax: float
|
||||||
|
label: str # e.g. "CH3 -- Bias A (DC mean)"
|
||||||
|
unit: str # colorbar units, e.g. "mV"
|
||||||
|
tag: str # filename tag: "CH1", "CH3", "CH4", "VEL"
|
||||||
|
|
||||||
|
|
||||||
|
def _render_map_png(img: np.ndarray, extent: list[float], *, cmap: str,
|
||||||
|
vmin: float, vmax: float, title: str, colorbar_label: str,
|
||||||
|
out_path: str, figsize: tuple[float, float] | None = None):
|
||||||
|
"""Render one map image to *out_path* with a fixed vmin/vmax, using a
|
||||||
|
headless Agg canvas so this never touches the GUI thread's interactive
|
||||||
|
matplotlib backend. Layout mirrors ImageCanvas.show_image.
|
||||||
|
|
||||||
|
*figsize* is the live canvas's size in inches, so the PNG comes out at
|
||||||
|
the shape the view was being read at instead of a fixed 7x5 -- with
|
||||||
|
aspect="auto" below, the figure box is what sets the map's proportions.
|
||||||
|
None falls back to sras_render.DEFAULT_FIGSIZE."""
|
||||||
|
fig = Figure(figsize=sanitize_figsize(figsize), tight_layout=True)
|
||||||
|
FigureCanvasAgg(fig)
|
||||||
|
ax = fig.add_subplot(111)
|
||||||
|
im = ax.imshow(img, aspect="auto", origin="upper", extent=extent,
|
||||||
|
cmap=cmap, vmin=vmin, vmax=vmax, interpolation="nearest")
|
||||||
|
cb = fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
|
||||||
|
if colorbar_label:
|
||||||
|
cb.set_label(colorbar_label)
|
||||||
|
ax.set_xlabel("X (mm)")
|
||||||
|
ax.set_ylabel("Y (mm)")
|
||||||
|
ax.set_title(title)
|
||||||
|
fig.savefig(out_path, dpi=150)
|
||||||
|
|
||||||
|
|
||||||
|
class BatchExportWorker(CancellableWorker):
|
||||||
|
"""Renders and saves one PNG per (angle, selected channel) for an
|
||||||
|
already-open SrasFile, with each channel's vmin/vmax held fixed across
|
||||||
|
every angle so the colorbar is directly comparable image to image.
|
||||||
|
|
||||||
|
Takes the SrasFile object directly (like ComputeWorker/DcPrecomputeWorker)
|
||||||
|
rather than a path -- this runs against the file already open in the GUI,
|
||||||
|
not an arbitrary batch of files, so there is no need to reopen it in a
|
||||||
|
subprocess the way BatchCacheWorker does.
|
||||||
|
|
||||||
|
Every setting that decides what an FFT-derived pixel *is* has to arrive
|
||||||
|
here explicitly, because compute_rf_image defaults each one to "off" and
|
||||||
|
an omitted argument is therefore not a no-op -- it silently exports a
|
||||||
|
different image than the one on screen. *min_freq_mhz* is the one with
|
||||||
|
teeth: dropping it does not just skip a mask, it also makes this file's
|
||||||
|
stored cache (written by Batch Compute FFT at some earlier floor) look
|
||||||
|
like a match, so the export hands back the pre-floor peaks the user
|
||||||
|
raised the floor to get rid of. It mirrors
|
||||||
|
SrasViewerWindow._stored_fft_image, which takes the floor live for the
|
||||||
|
same reason -- re-masking a stored image against a *higher* floor is
|
||||||
|
free, so the displayed map always honors the spin box.
|
||||||
|
|
||||||
|
Emits progress(int) (0-100 over all angle x channel pairs), file_done(str,
|
||||||
|
str) (output path, error message or ""), and finished() -- same shape as
|
||||||
|
BatchCacheWorker.
|
||||||
|
"""
|
||||||
|
progress = pyqtSignal(int)
|
||||||
|
file_done = pyqtSignal(str, str)
|
||||||
|
finished = pyqtSignal()
|
||||||
|
|
||||||
|
def __init__(self, sras: SrasFile, channels: list[ExportChannel],
|
||||||
|
output_dir: str, prefix: str, *, cmap: str,
|
||||||
|
apply_bg_sub: bool, dc_threshold_mv: float,
|
||||||
|
n_fft: int | None, grating_um: float,
|
||||||
|
min_freq_mhz: float = 0.0,
|
||||||
|
figsize: tuple[float, float] | None = None):
|
||||||
|
super().__init__()
|
||||||
|
self._sras = sras
|
||||||
|
self._channels = channels
|
||||||
|
self._output_dir = Path(output_dir)
|
||||||
|
self._prefix = prefix
|
||||||
|
self._cmap = cmap
|
||||||
|
self._apply_bg_sub = apply_bg_sub
|
||||||
|
self._dc_threshold_mv = dc_threshold_mv
|
||||||
|
self._n_fft = n_fft
|
||||||
|
self._grating_um = grating_um
|
||||||
|
self._min_freq_mhz = min_freq_mhz
|
||||||
|
self._figsize = figsize
|
||||||
|
|
||||||
|
def _angle_extent(self, angle_idx: int) -> list[float]:
|
||||||
|
s = self._sras
|
||||||
|
x_axis = s.x_axis_mm(angle_idx)
|
||||||
|
y_axis = s.y_positions_mm(angle_idx)
|
||||||
|
dx = x_axis[1] - x_axis[0] if len(x_axis) > 1 else s.pixel_x_mm
|
||||||
|
dy = float(y_axis[1] - y_axis[0]) if len(y_axis) > 1 else 1.0
|
||||||
|
return [x_axis[0] - dx / 2, x_axis[-1] + dx / 2,
|
||||||
|
y_axis[-1] + dy / 2, y_axis[0] - dy / 2]
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
try:
|
||||||
|
s = self._sras
|
||||||
|
n_angles = s.n_angles
|
||||||
|
total = n_angles * len(self._channels)
|
||||||
|
done = 0
|
||||||
|
needs_freq = any(c.ch_idx == CH1_IDX for c in self._channels)
|
||||||
|
|
||||||
|
for angle_idx in range(n_angles):
|
||||||
|
if self._stop:
|
||||||
|
break
|
||||||
|
extent = self._angle_extent(angle_idx)
|
||||||
|
angle_deg = s.angles_deg[angle_idx]
|
||||||
|
|
||||||
|
freq_mhz = None
|
||||||
|
if needs_freq:
|
||||||
|
freq_mhz = compute_rf_image(
|
||||||
|
s, angle_idx, dc_threshold_mv=self._dc_threshold_mv,
|
||||||
|
apply_bg_sub=self._apply_bg_sub, n_fft=self._n_fft,
|
||||||
|
should_stop=self._stopped, row_avg_n=s.precomputed_row_avg_n,
|
||||||
|
min_freq_mhz=self._min_freq_mhz)
|
||||||
|
|
||||||
|
for channel in self._channels:
|
||||||
|
if self._stop:
|
||||||
|
break
|
||||||
|
out_path = (self._output_dir
|
||||||
|
/ f"{self._prefix}_angle{angle_idx:02d}_{channel.tag}.png")
|
||||||
|
try:
|
||||||
|
if channel.ch_idx == CH1_IDX:
|
||||||
|
img = (freq_mhz * self._grating_um if channel.is_velocity
|
||||||
|
else freq_mhz)
|
||||||
|
else:
|
||||||
|
img = dc_image_mv(s, angle_idx, channel.ch_idx,
|
||||||
|
should_stop=self._stopped)
|
||||||
|
title = f"{channel.tag} | {angle_deg:.1f}°"
|
||||||
|
colorbar_label = (f"{channel.label} ({channel.unit})"
|
||||||
|
if channel.unit else channel.label)
|
||||||
|
_render_map_png(
|
||||||
|
img, extent, cmap=self._cmap,
|
||||||
|
vmin=channel.vmin, vmax=channel.vmax,
|
||||||
|
title=title, colorbar_label=colorbar_label,
|
||||||
|
out_path=str(out_path), figsize=self._figsize)
|
||||||
|
self.file_done.emit(str(out_path), "")
|
||||||
|
except Exception as exc:
|
||||||
|
self.file_done.emit(str(out_path), str(exc))
|
||||||
|
done += 1
|
||||||
|
self.progress.emit(int(done / max(1, total) * 100))
|
||||||
|
|
||||||
|
self.finished.emit()
|
||||||
|
except Exception as exc:
|
||||||
|
self.file_done.emit("", str(exc))
|
||||||
|
self.finished.emit()
|
||||||
|
|
||||||
|
|
||||||
|
class BatchExportImagesWorker(QObject):
|
||||||
|
"""Renders the view settings captured by the caller at trigger time
|
||||||
|
(angle/channel/threshold/etc.) to one PNG per file in *paths*, via
|
||||||
|
sras_render.export_view_image.
|
||||||
|
|
||||||
|
Same process-pool-with-inline-fallback strategy as BatchCacheWorker
|
||||||
|
above (same _BATCH_MAX_PROCS / _BATCH_POOL_MIN_BYTES thresholds): an
|
||||||
|
FFT-derived (CH1/Velocity) view is exactly the same expensive per-file
|
||||||
|
compute Batch Compute FFT already parallelizes this way. Never mutates
|
||||||
|
*paths* — each file is opened read-only — so unlike BatchCacheWorker
|
||||||
|
there is no version gate.
|
||||||
|
|
||||||
|
Emits progress(int) (0-100 by files completed), file_done(str, str, str)
|
||||||
|
(path, error message or "", output filename this file targeted — set
|
||||||
|
even on most failures so the caller can flag same-stem collisions across
|
||||||
|
the batch without any cross-process bookkeeping), and finished().
|
||||||
|
"""
|
||||||
|
progress = pyqtSignal(int)
|
||||||
|
file_done = pyqtSignal(str, str, str)
|
||||||
|
finished = pyqtSignal()
|
||||||
|
|
||||||
|
def __init__(self, paths: list[str], **render_kwargs):
|
||||||
|
"""*render_kwargs* is exactly export_view_image's keyword-only
|
||||||
|
settings (out_dir, angle_idx, ch_idx, is_fft_mode, is_velocity,
|
||||||
|
dc_threshold_mv, apply_bg_sub, pad_factor, min_freq_mhz, grating_um,
|
||||||
|
cmap, auto_scale, vmin, vmax, highlight_masked, mode_str,
|
||||||
|
colorbar_label, mask_color) — bundled rather than repeated as
|
||||||
|
positional params across __init__/_run_pooled/_run_inline."""
|
||||||
|
super().__init__()
|
||||||
|
self._paths = paths
|
||||||
|
self._kw = render_kwargs
|
||||||
|
|
||||||
|
def _report(self, path: str, err: str, out_name: str, done: int, total: int):
|
||||||
|
self.file_done.emit(path, err, out_name)
|
||||||
|
self.progress.emit(int(done / max(1, total) * 100))
|
||||||
|
|
||||||
|
def _run_pooled(self, paths: list[str], n_procs: int) -> list[str]:
|
||||||
|
"""Same contract as BatchCacheWorker._run_pooled: returns the paths
|
||||||
|
that never got a real answer because the pool itself died, so the
|
||||||
|
caller can retry them in-process."""
|
||||||
|
per_proc_workers = max(1, (os.cpu_count() or 4) // n_procs)
|
||||||
|
unresolved: list[str] = []
|
||||||
|
done = 0
|
||||||
|
|
||||||
|
with ProcessPoolExecutor(max_workers=n_procs) as executor:
|
||||||
|
futures = {
|
||||||
|
executor.submit(export_view_image, p,
|
||||||
|
max_workers=per_proc_workers, **self._kw): p
|
||||||
|
for p in paths
|
||||||
|
}
|
||||||
|
for fut in as_completed(futures):
|
||||||
|
path = futures[fut]
|
||||||
|
try:
|
||||||
|
err, out_name = fut.result()
|
||||||
|
except BrokenProcessPool:
|
||||||
|
unresolved.append(path)
|
||||||
|
continue
|
||||||
|
except Exception as exc:
|
||||||
|
err, out_name = str(exc), ""
|
||||||
|
done += 1
|
||||||
|
self._report(path, err, out_name, done, len(paths))
|
||||||
|
|
||||||
|
return unresolved
|
||||||
|
|
||||||
|
def _run_inline(self, paths: list[str], done: int, total: int):
|
||||||
|
for path in paths:
|
||||||
|
try:
|
||||||
|
err, out_name = export_view_image(
|
||||||
|
path, max_workers=compute.default_max_workers(), **self._kw)
|
||||||
|
except Exception as exc:
|
||||||
|
err, out_name = str(exc), ""
|
||||||
|
done += 1
|
||||||
|
self._report(path, err, out_name, done, total)
|
||||||
|
|
||||||
|
def _worth_pooling(self, paths: list[str]) -> bool:
|
||||||
|
if len(paths) < 2:
|
||||||
|
return False
|
||||||
|
total = 0
|
||||||
|
for p in paths:
|
||||||
|
try:
|
||||||
|
total += os.path.getsize(p)
|
||||||
|
except OSError:
|
||||||
|
pass # unreadable files are reported by export_view_image
|
||||||
|
return total >= _BATCH_POOL_MIN_BYTES
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
paths = self._paths
|
||||||
|
n_procs = max(1, min(_BATCH_MAX_PROCS, len(paths)))
|
||||||
|
|
||||||
|
if not self._worth_pooling(paths):
|
||||||
|
self._run_inline(paths, 0, len(paths))
|
||||||
|
self.finished.emit()
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
unresolved = self._run_pooled(paths, n_procs)
|
||||||
|
except Exception:
|
||||||
|
# The pool could not be created or collapsed wholesale.
|
||||||
|
unresolved = list(paths)
|
||||||
|
|
||||||
|
if unresolved:
|
||||||
|
self._run_inline(unresolved, len(paths) - len(unresolved), len(paths))
|
||||||
|
|
||||||
|
self.finished.emit()
|
||||||
|
|
||||||
|
|
||||||
class Ch4MaskWorker(_PooledWorker):
|
class Ch4MaskWorker(_PooledWorker):
|
||||||
"""Fetches each requested angle's CH4 (Bias B) DC image in mV, for the
|
"""Fetches each requested angle's CH4 (Bias B) DC image in mV, for the
|
||||||
alignment wizard's initial threshold-mask stack.
|
alignment wizard's initial threshold-mask stack.
|
||||||
|
|||||||
@@ -0,0 +1,719 @@
|
|||||||
|
"""Both batch image exports -- Convert -> Batch Export View as Images (one
|
||||||
|
PNG per *file*, via sras_render.export_view_image) and Export -> Batch
|
||||||
|
Export Images (one PNG per *angle x channel* of the open file, via
|
||||||
|
sras_workers.BatchExportWorker). Does the exported PNG actually match what
|
||||||
|
the live view would show, and does the batch dispatch (menu action ->
|
||||||
|
worker -> per-file render) behave like Batch Compute's proven pattern?
|
||||||
|
|
||||||
|
The recurring hazard both halves guard is a dropped view setting: every
|
||||||
|
parameter that decides what an FFT-derived pixel *is* (bg-sub, pad,
|
||||||
|
row-averaging, the min peak frequency floor) defaults to "off" in
|
||||||
|
compute_rf_image, so an argument the export forgets to forward does not
|
||||||
|
degrade gracefully -- it silently renders a different image than the screen,
|
||||||
|
and worse, makes this file's stored cache look like a match so the export
|
||||||
|
hands back peaks from an earlier compute.
|
||||||
|
|
||||||
|
sras_render.export_view_image is tested directly (no Qt) for the plumbing
|
||||||
|
that decides *what* gets rendered -- pad_factor derived per file, masked-
|
||||||
|
pixel NaN fill, per-file auto-scale, velocity scaling -- via a
|
||||||
|
draw_view_image spy rather than pixel-diffing PNGs, the same "spy on the
|
||||||
|
seam, don't inspect the rendered artifact" approach test_highlight_masked_
|
||||||
|
pixels (tests/test_gui.py) uses for the live canvas.
|
||||||
|
|
||||||
|
The GUI-dispatch half drives SrasViewerWindow._on_batch_export_images()
|
||||||
|
end-to-end with patched file dialogs, the same shape as
|
||||||
|
test_stored_cache.py's test_viewer_batch_row_average_dispatch.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
from PyQt6.QtCore import QEventLoop, QTimer
|
||||||
|
from PyQt6.QtWidgets import QApplication
|
||||||
|
|
||||||
|
import sras_render
|
||||||
|
import sras_workers
|
||||||
|
from sras_compute import cache_file, compute_rf_image, dc_image_mv
|
||||||
|
from sras_format import CH1_IDX, CH3_IDX, CH4_IDX, CH_NAMES, SrasFile
|
||||||
|
from sras_render import DEFAULT_FIGSIZE, export_view_image, sanitize_figsize
|
||||||
|
from sras_viewer import SrasViewerWindow, VELOCITY_MODE_IDX
|
||||||
|
from sras_workers import (
|
||||||
|
BatchExportImagesWorker, BatchExportWorker, ExportChannel,
|
||||||
|
)
|
||||||
|
import tools.make_test_sras as gen
|
||||||
|
|
||||||
|
_THRESHOLD_MV = 50.0
|
||||||
|
|
||||||
|
|
||||||
|
def pump(ms: int = 200):
|
||||||
|
loop = QEventLoop()
|
||||||
|
QTimer.singleShot(ms, loop.quit)
|
||||||
|
loop.exec()
|
||||||
|
|
||||||
|
|
||||||
|
def wait_until(pred, timeout_ms: int = 20000, step: int = 100) -> bool:
|
||||||
|
waited = 0
|
||||||
|
while waited < timeout_ms:
|
||||||
|
if pred():
|
||||||
|
return True
|
||||||
|
pump(step)
|
||||||
|
waited += step
|
||||||
|
return pred()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# sras_render.export_view_image -- pure function, no Qt
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_DEFAULT_KW = dict(
|
||||||
|
is_fft_mode=False, is_velocity=False, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
apply_bg_sub=True, pad_factor=1, min_freq_mhz=0.0, grating_um=1.0,
|
||||||
|
cmap="viridis", auto_scale=True, vmin=0.0, vmax=1.0,
|
||||||
|
highlight_masked=False, mode_str="DC", colorbar_label="mV",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _kw(**overrides):
|
||||||
|
kw = dict(_DEFAULT_KW)
|
||||||
|
kw.update(overrides)
|
||||||
|
return kw
|
||||||
|
|
||||||
|
|
||||||
|
def _png_size(path) -> tuple[int, int]:
|
||||||
|
"""(width, height) in pixels, read straight out of the PNG's IHDR chunk
|
||||||
|
(two big-endian uint32s at byte 16) -- no image library needed just to
|
||||||
|
check the shape of an export."""
|
||||||
|
raw = Path(path).read_bytes()
|
||||||
|
assert raw[:8] == b"\x89PNG\r\n\x1a\n", "not a valid PNG"
|
||||||
|
return (int.from_bytes(raw[16:20], "big"),
|
||||||
|
int.from_bytes(raw[20:24], "big"))
|
||||||
|
|
||||||
|
|
||||||
|
def _spy_draw(monkeypatch):
|
||||||
|
"""Patches sras_render.draw_view_image to record the image array and
|
||||||
|
vmin/vmax/bad_color it was called with, then delegates to the real
|
||||||
|
implementation so the PNG is still written -- lets a test check *what*
|
||||||
|
export_view_image computed without depending on rendered PNG pixels."""
|
||||||
|
orig = sras_render.draw_view_image
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def spy(ax, fig, img, extent, cmap, vmin, vmax, xlabel, ylabel, title,
|
||||||
|
colorbar_label="", cb_ticks=None, norm=None, bad_color=None):
|
||||||
|
captured["img"] = np.array(img, copy=True)
|
||||||
|
captured["vmin"] = vmin
|
||||||
|
captured["vmax"] = vmax
|
||||||
|
captured["bad_color"] = bad_color
|
||||||
|
return orig(ax, fig, img, extent, cmap, vmin, vmax, xlabel, ylabel,
|
||||||
|
title, colorbar_label, cb_ticks, norm, bad_color)
|
||||||
|
|
||||||
|
monkeypatch.setattr(sras_render, "draw_view_image", spy)
|
||||||
|
return captured
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_dc_channel_writes_png(tmp_path):
|
||||||
|
path = tmp_path / "dc.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=1, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
err, out_name = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH4_IDX, **_kw())
|
||||||
|
assert err == ""
|
||||||
|
assert out_name == f"dc_angle0_{CH_NAMES[CH4_IDX]}.png"
|
||||||
|
out_path = out_dir / out_name
|
||||||
|
assert out_path.exists() and out_path.stat().st_size > 0
|
||||||
|
assert out_path.read_bytes()[:8] == b"\x89PNG\r\n\x1a\n", "not a valid PNG"
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_out_of_range_angle(tmp_path):
|
||||||
|
path = tmp_path / "short.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=2, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
err, out_name = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=5, ch_idx=CH4_IDX, **_kw())
|
||||||
|
assert err != "" and "2" in err, "error should mention the file's actual angle count"
|
||||||
|
assert out_name == ""
|
||||||
|
assert list(out_dir.iterdir()) == [], "no file written for a failed export"
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_fft_mode_matches_compute_rf_image(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "fft.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=3, samples_per_frame=128)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
captured = _spy_draw(monkeypatch)
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH1_IDX,
|
||||||
|
**_kw(is_fft_mode=True))
|
||||||
|
assert err == ""
|
||||||
|
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
expected = compute_rf_image(sras, 0, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
apply_bg_sub=True, n_fft=None, min_freq_mhz=0.0)
|
||||||
|
assert np.array_equal(captured["img"], expected)
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_velocity_scales_frequency(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "vel.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=4, samples_per_frame=128)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
captured = _spy_draw(monkeypatch)
|
||||||
|
grating_um = 3.5
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=VELOCITY_MODE_IDX,
|
||||||
|
**_kw(is_fft_mode=True, is_velocity=True, grating_um=grating_um))
|
||||||
|
assert err == ""
|
||||||
|
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
freq = compute_rf_image(sras, 0, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
apply_bg_sub=True, n_fft=None, min_freq_mhz=0.0)
|
||||||
|
assert np.array_equal(captured["img"], freq * grating_um)
|
||||||
|
|
||||||
|
|
||||||
|
def test_pad_factor_uses_each_files_own_samples_per_frame(tmp_path, monkeypatch):
|
||||||
|
"""n_fft must be derived per file from that file's own samples_per_frame,
|
||||||
|
never a value carried over from whichever file the caller had open --
|
||||||
|
otherwise every file but one in a batch gets silently mis-padded."""
|
||||||
|
orig_draw = sras_render.draw_view_image
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
for i, spf in enumerate((64, 256)):
|
||||||
|
path = tmp_path / f"pad_{spf}.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=5 + i, samples_per_frame=spf)
|
||||||
|
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
def spy(ax, fig, img, *a, __c=captured, **kw):
|
||||||
|
__c["img"] = np.array(img, copy=True)
|
||||||
|
return orig_draw(ax, fig, img, *a, **kw)
|
||||||
|
|
||||||
|
monkeypatch.setattr(sras_render, "draw_view_image", spy)
|
||||||
|
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH1_IDX,
|
||||||
|
**_kw(is_fft_mode=True, pad_factor=4))
|
||||||
|
assert err == ""
|
||||||
|
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
expected = compute_rf_image(sras, 0, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
apply_bg_sub=True, n_fft=spf * 4,
|
||||||
|
min_freq_mhz=0.0)
|
||||||
|
assert np.array_equal(captured["img"], expected), \
|
||||||
|
f"samples_per_frame={spf}: n_fft must use this file's own value"
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlight_masked_sets_nan_and_bad_color(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "mask.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=7, samples_per_frame=128)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
dc4 = dc_image_mv(sras, 0, CH4_IDX)
|
||||||
|
threshold = float(np.median(dc4))
|
||||||
|
expect_masked = dc4 < threshold
|
||||||
|
assert expect_masked.any() and not expect_masked.all(), \
|
||||||
|
"fixture threshold should mask some but not all pixels"
|
||||||
|
|
||||||
|
captured = _spy_draw(monkeypatch)
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH1_IDX,
|
||||||
|
**_kw(is_fft_mode=True, dc_threshold_mv=threshold,
|
||||||
|
highlight_masked=True, mask_color="magenta"))
|
||||||
|
assert err == ""
|
||||||
|
assert captured["bad_color"] == "magenta"
|
||||||
|
# The export masks by value too (0 == the "no valid peak" sentinel, same
|
||||||
|
# rule as the viewer's _redraw_image); on this fixture every
|
||||||
|
# above-threshold pixel has a nonzero peak, so the value mask coincides
|
||||||
|
# with the DC mask and the NaN set is exactly expect_masked.
|
||||||
|
assert np.array_equal(np.isnan(captured["img"]), expect_masked)
|
||||||
|
valid_vals = captured["img"][~expect_masked]
|
||||||
|
assert not np.isnan(valid_vals).any() and (valid_vals != 0).all(), \
|
||||||
|
"fixture precondition: every valid pixel has a nonzero peak"
|
||||||
|
|
||||||
|
captured2 = _spy_draw(monkeypatch)
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH1_IDX,
|
||||||
|
**_kw(is_fft_mode=True, dc_threshold_mv=threshold, highlight_masked=False))
|
||||||
|
assert err == ""
|
||||||
|
assert captured2["bad_color"] is None
|
||||||
|
assert not np.isnan(captured2["img"]).any()
|
||||||
|
|
||||||
|
|
||||||
|
def test_auto_scale_uses_per_file_min_max(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "scale.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=8, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
captured = _spy_draw(monkeypatch)
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH4_IDX,
|
||||||
|
**_kw(auto_scale=True))
|
||||||
|
assert err == ""
|
||||||
|
img = captured["img"]
|
||||||
|
assert captured["vmin"] == pytest.approx(float(np.nanmin(img)))
|
||||||
|
assert captured["vmax"] == pytest.approx(float(np.nanmax(img)))
|
||||||
|
|
||||||
|
captured2 = _spy_draw(monkeypatch)
|
||||||
|
err, _ = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH4_IDX,
|
||||||
|
**_kw(auto_scale=False, vmin=-5.0, vmax=5.0))
|
||||||
|
assert err == ""
|
||||||
|
assert captured2["vmin"] == -5.0
|
||||||
|
assert captured2["vmax"] == 5.0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("figsize", [(12.0, 4.0), (4.0, 9.0)])
|
||||||
|
def test_export_matches_requested_figsize(tmp_path, figsize):
|
||||||
|
"""The PNG comes out at the caller's figure size, so a view the user has
|
||||||
|
sized wide (or tall) doesn't get squeezed into a fixed 7x5 -- the image
|
||||||
|
is drawn with aspect="auto", so the figure box *is* the map's shape."""
|
||||||
|
path = tmp_path / "shape.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=70, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
dpi = 100
|
||||||
|
err, out_name = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH4_IDX,
|
||||||
|
figsize=figsize, dpi=dpi, **_kw())
|
||||||
|
assert err == ""
|
||||||
|
|
||||||
|
w_px, h_px = _png_size(out_dir / out_name)
|
||||||
|
# Agg truncates inches*dpi to whole pixels; a pixel of slack, not an
|
||||||
|
# aspect-ratio tolerance, is what's being allowed for here.
|
||||||
|
assert abs(w_px - figsize[0] * dpi) <= 1
|
||||||
|
assert abs(h_px - figsize[1] * dpi) <= 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_default_figsize_when_unspecified(tmp_path):
|
||||||
|
"""No figsize (a caller with no live canvas to match) still renders at
|
||||||
|
the viewer's starting size rather than failing or guessing."""
|
||||||
|
path = tmp_path / "default_shape.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=71, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
err, out_name = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH4_IDX,
|
||||||
|
dpi=100, **_kw())
|
||||||
|
assert err == ""
|
||||||
|
w_px, h_px = _png_size(out_dir / out_name)
|
||||||
|
assert abs(w_px - DEFAULT_FIGSIZE[0] * 100) <= 1
|
||||||
|
assert abs(h_px - DEFAULT_FIGSIZE[1] * 100) <= 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("bad", [None, (0.0, 0.0), (-3.0, 5.0), (np.nan, 5.0),
|
||||||
|
(float("inf"), 5.0), (7.0,), "7x5"])
|
||||||
|
def test_sanitize_figsize_never_yields_an_unrenderable_size(bad):
|
||||||
|
"""A degenerate canvas size (collapsed pane, minimized window) must cost
|
||||||
|
at most a wrong-looking image, never a failed export."""
|
||||||
|
w, h = sanitize_figsize(bad)
|
||||||
|
assert np.isfinite(w) and np.isfinite(h)
|
||||||
|
assert w >= 1.0 and h >= 1.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_sanitize_figsize_passes_through_a_normal_canvas_size():
|
||||||
|
assert sanitize_figsize(np.array([12.8, 6.4])) == pytest.approx((12.8, 6.4))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# GUI dispatch: SrasViewerWindow._on_batch_export_images end-to-end
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _make_window(path) -> SrasViewerWindow:
|
||||||
|
app = QApplication.instance() or QApplication([]) # noqa: F841
|
||||||
|
win = SrasViewerWindow()
|
||||||
|
win.show()
|
||||||
|
win._load_file(str(path))
|
||||||
|
assert wait_until(lambda: win._sras is not None), "file loaded"
|
||||||
|
assert wait_until(lambda: all((a, CH4_IDX) in win._dc_cache
|
||||||
|
for a in range(win._sras.n_angles))), \
|
||||||
|
"DC precompute finished"
|
||||||
|
return win
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_images_writes_one_png_per_file(tmp_path):
|
||||||
|
path = tmp_path / "src.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=10, samples_per_frame=64)
|
||||||
|
paths = [str(path)]
|
||||||
|
for i in range(2):
|
||||||
|
p2 = tmp_path / f"other{i}.sras"
|
||||||
|
gen.write(p2, n_angles=2, seed=20 + i, samples_per_frame=64)
|
||||||
|
paths.append(str(p2))
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=(paths, "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
|
||||||
|
angle = win.spin_angle.value()
|
||||||
|
ch_name = CH_NAMES[win.combo_channel.currentIndex()]
|
||||||
|
expected_names = {f"{Path(p).stem}_angle{angle}_{ch_name}.png" for p in paths}
|
||||||
|
actual_names = {p.name for p in out_dir.iterdir()}
|
||||||
|
assert actual_names == expected_names
|
||||||
|
assert "Batch export: 3/3 image(s)" in win.statusBar().currentMessage()
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_out_of_range_angle_reports_error_continues(tmp_path):
|
||||||
|
good_path = tmp_path / "good.sras"
|
||||||
|
short_path = tmp_path / "short.sras"
|
||||||
|
gen.write(good_path, n_angles=3, seed=30, samples_per_frame=64)
|
||||||
|
gen.write(short_path, n_angles=1, seed=31, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(good_path)
|
||||||
|
try:
|
||||||
|
win.spin_angle.setValue(2) # valid for good_path, out of range for short_path
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(good_path), str(short_path)], "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
|
||||||
|
msg = win.statusBar().currentMessage()
|
||||||
|
assert "Batch export: 1/2 image(s)" in msg, msg
|
||||||
|
assert "1 failed" in msg, msg
|
||||||
|
assert len(list(out_dir.iterdir())) == 1, \
|
||||||
|
"the batch must not abort — the good file still exports"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_busy_guard_skips_dialogs(tmp_path, monkeypatch):
|
||||||
|
path = tmp_path / "busy.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=40, samples_per_frame=64)
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
monkeypatch.setattr(win, "_job_running", lambda key: True)
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames") as mock_dlg:
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert mock_dlg.call_count == 0, \
|
||||||
|
"the Jobs.BATCH busy guard must return before opening any dialog"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_ignores_aligned_view_toggle(tmp_path):
|
||||||
|
"""Aligned View is geometry specific to whichever single file the
|
||||||
|
Alignment Wizard last ran against and cannot be meaningfully applied
|
||||||
|
across a batch of different files -- _on_batch_export_images must not
|
||||||
|
read chk_aligned_view / self._alignment_result at all, regardless of
|
||||||
|
what's checked in the live view."""
|
||||||
|
path = tmp_path / "aligned.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=41, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
captured_kwargs = []
|
||||||
|
orig_init = BatchExportImagesWorker.__init__
|
||||||
|
|
||||||
|
def spy_init(self, paths, **kw):
|
||||||
|
captured_kwargs.append(kw)
|
||||||
|
return orig_init(self, paths, **kw)
|
||||||
|
|
||||||
|
win.chk_aligned_view.setChecked(True)
|
||||||
|
with patch.object(BatchExportImagesWorker, "__init__", spy_init), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(path)], "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
|
||||||
|
assert len(captured_kwargs) == 1
|
||||||
|
assert not any("align" in k.lower() for k in captured_kwargs[0]), \
|
||||||
|
captured_kwargs[0].keys()
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_uses_the_live_canvas_aspect_ratio(tmp_path):
|
||||||
|
"""The exported PNG has the shape of the view on screen, not a fixed
|
||||||
|
7x5 -- resize the window and the export follows it."""
|
||||||
|
path = tmp_path / "aspect.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=45, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
win.resize(1400, 700)
|
||||||
|
pump(300) # let the canvas' resizeEvent reach the figure
|
||||||
|
canvas_w, canvas_h = win.image_canvas.figure.get_size_inches()
|
||||||
|
|
||||||
|
captured_kwargs = []
|
||||||
|
orig_init = BatchExportImagesWorker.__init__
|
||||||
|
|
||||||
|
def spy_init(self, paths, **kw):
|
||||||
|
captured_kwargs.append(kw)
|
||||||
|
return orig_init(self, paths, **kw)
|
||||||
|
|
||||||
|
with patch.object(BatchExportImagesWorker, "__init__", spy_init), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(path)], "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
|
||||||
|
assert captured_kwargs[0]["figsize"] == pytest.approx(
|
||||||
|
(canvas_w, canvas_h)), "the live canvas size must travel to the worker"
|
||||||
|
|
||||||
|
out_files = list(out_dir.iterdir())
|
||||||
|
assert len(out_files) == 1
|
||||||
|
w_px, h_px = _png_size(out_files[0])
|
||||||
|
assert w_px / h_px == pytest.approx(canvas_w / canvas_h, rel=0.01)
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_filename_collision_note(tmp_path):
|
||||||
|
dir_a, dir_b = tmp_path / "dir_a", tmp_path / "dir_b"
|
||||||
|
dir_a.mkdir()
|
||||||
|
dir_b.mkdir()
|
||||||
|
path_a, path_b = dir_a / "dup.sras", dir_b / "dup.sras"
|
||||||
|
gen.write(path_a, n_angles=1, seed=50, samples_per_frame=64)
|
||||||
|
gen.write(path_b, n_angles=1, seed=51, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path_a)
|
||||||
|
try:
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(path_a), str(path_b)], "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
|
||||||
|
msg = win.statusBar().currentMessage()
|
||||||
|
assert "Batch export: 2/2 image(s)" in msg, msg
|
||||||
|
assert "collision" in msg, msg
|
||||||
|
assert len(list(out_dir.iterdir())) == 1, \
|
||||||
|
"same-stem inputs silently overwrite to one output file"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_does_not_modify_source_files(tmp_path):
|
||||||
|
path = tmp_path / "untouched.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=60, samples_per_frame=64)
|
||||||
|
before = path.read_bytes()
|
||||||
|
out_dir = tmp_path / "images"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(path)], "")), \
|
||||||
|
patch("sras_viewer.main_window.QFileDialog.getExistingDirectory",
|
||||||
|
return_value=str(out_dir)):
|
||||||
|
win._on_batch_export_images()
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
assert path.read_bytes() == before, "export must never write to the source file"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Export menu: BatchExportWorker (every angle of the open file)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# High enough to sit above several of the generator's synthetic peak bins
|
||||||
|
# (bin spacing is 6.25 GS/s / 64 = 97.66 MHz, peaks land at bins 3-19), so a
|
||||||
|
# floor at this value genuinely changes which peak each pixel resolves to --
|
||||||
|
# without that, a test that the floor is honored would pass either way.
|
||||||
|
_FLOOR_MHZ = 1000.0
|
||||||
|
_GRATING_UM = 2.0
|
||||||
|
|
||||||
|
|
||||||
|
def _velocity_channel() -> ExportChannel:
|
||||||
|
return ExportChannel(ch_idx=CH1_IDX, is_velocity=True, vmin=0.0, vmax=1e5,
|
||||||
|
label="Velocity", unit="m/s", tag="VEL")
|
||||||
|
|
||||||
|
|
||||||
|
def _run_export_worker(monkeypatch, sras, out_dir, **overrides) -> list[np.ndarray]:
|
||||||
|
"""Runs BatchExportWorker to completion on the calling thread (its run()
|
||||||
|
is a plain loop; the QThread in _run_worker is a GUI concern) and returns
|
||||||
|
the image arrays it rendered, captured at the _render_map_png seam."""
|
||||||
|
captured = []
|
||||||
|
orig = sras_workers._render_map_png
|
||||||
|
|
||||||
|
def spy(img, extent, **kw):
|
||||||
|
captured.append(np.array(img, copy=True))
|
||||||
|
return orig(img, extent, **kw)
|
||||||
|
|
||||||
|
monkeypatch.setattr(sras_workers, "_render_map_png", spy)
|
||||||
|
kw = dict(cmap="viridis", apply_bg_sub=True, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
n_fft=None, grating_um=_GRATING_UM, min_freq_mhz=_FLOOR_MHZ)
|
||||||
|
kw.update(overrides)
|
||||||
|
BatchExportWorker(sras, [_velocity_channel()], str(out_dir), "vel", **kw).run()
|
||||||
|
return captured
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_applies_the_min_peak_freq_floor(tmp_path, monkeypatch):
|
||||||
|
"""The exported Velocity map is the floored one the viewer shows, not the
|
||||||
|
unfloored peaks the floor was raised to reject."""
|
||||||
|
path = tmp_path / "floor.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=80, samples_per_frame=64, geometry=[(6, 9)])
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
|
||||||
|
def expected(floor):
|
||||||
|
return compute_rf_image(sras, 0, dc_threshold_mv=_THRESHOLD_MV,
|
||||||
|
apply_bg_sub=True, n_fft=None,
|
||||||
|
min_freq_mhz=floor) * _GRATING_UM
|
||||||
|
|
||||||
|
floored, unfloored = expected(_FLOOR_MHZ), expected(0.0)
|
||||||
|
assert not np.allclose(floored, unfloored), \
|
||||||
|
"fixture must be one where the floor changes the image"
|
||||||
|
|
||||||
|
exported = _run_export_worker(monkeypatch, sras, out_dir)
|
||||||
|
assert len(exported) == 1
|
||||||
|
assert np.allclose(exported[0], floored)
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_does_not_serve_an_unfloored_stored_cache(tmp_path,
|
||||||
|
monkeypatch):
|
||||||
|
"""A file batch-computed before the floor existed has a stored cache with
|
||||||
|
no floor recorded. Asking for that cache at floor 0 (rather than the live
|
||||||
|
floor) makes it a match, so the export would render the stored pre-floor
|
||||||
|
peaks -- the map the user raised the floor to get rid of."""
|
||||||
|
path = tmp_path / "stored_floor.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=81, samples_per_frame=64, geometry=[(6, 9)])
|
||||||
|
assert cache_file(str(path), "fft", apply_bg_sub=True) == ""
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
assert sras.precomputed_freq_mhz[0] is not None, "stored cache written"
|
||||||
|
assert sras.precomputed_min_freq_mhz == 0.0
|
||||||
|
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
exported = _run_export_worker(monkeypatch, sras, out_dir)[0]
|
||||||
|
|
||||||
|
# 0.0 is the "no valid peak" sentinel; anything else below the floor is a
|
||||||
|
# peak that only an unfloored search could have reported.
|
||||||
|
below = (exported > 0) & (exported < _FLOOR_MHZ * _GRATING_UM)
|
||||||
|
assert not below.any(), \
|
||||||
|
f"{below.sum()} sub-floor pixel(s) survived the export"
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_export_dispatch_passes_the_live_view_settings(tmp_path):
|
||||||
|
"""The Export menu hands the worker what the panel currently says --
|
||||||
|
the floor in particular, which is a display control and so persists
|
||||||
|
across file loads while the window's own FFT cache does not."""
|
||||||
|
path = tmp_path / "dispatch.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=82, samples_per_frame=64)
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
|
||||||
|
win = _make_window(path)
|
||||||
|
try:
|
||||||
|
win.spin_min_freq_mhz.setValue(_FLOOR_MHZ)
|
||||||
|
win.spin_grating_um.setValue(_GRATING_UM)
|
||||||
|
win.spin_threshold_mv.setValue(_THRESHOLD_MV)
|
||||||
|
|
||||||
|
class StubDialog:
|
||||||
|
def __init__(self, parent, **kw):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def exec(self):
|
||||||
|
from PyQt6.QtWidgets import QDialog
|
||||||
|
return QDialog.DialogCode.Accepted
|
||||||
|
|
||||||
|
def get_output_dir(self):
|
||||||
|
return str(out_dir)
|
||||||
|
|
||||||
|
def get_prefix(self):
|
||||||
|
return "vel"
|
||||||
|
|
||||||
|
def get_selected_channels(self):
|
||||||
|
return [_velocity_channel()]
|
||||||
|
|
||||||
|
captured_kwargs = []
|
||||||
|
orig_init = BatchExportWorker.__init__
|
||||||
|
|
||||||
|
def spy_init(self, sras, channels, output_dir, prefix, **kw):
|
||||||
|
captured_kwargs.append(kw)
|
||||||
|
return orig_init(self, sras, channels, output_dir, prefix, **kw)
|
||||||
|
|
||||||
|
with patch.object(BatchExportWorker, "__init__", spy_init), \
|
||||||
|
patch("sras_viewer.main_window.BatchExportDialog", StubDialog):
|
||||||
|
win._on_batch_export()
|
||||||
|
assert wait_until(lambda: not win._job_running("export"), 60000), "export ran"
|
||||||
|
|
||||||
|
kw = captured_kwargs[0]
|
||||||
|
assert kw["min_freq_mhz"] == _FLOOR_MHZ
|
||||||
|
assert kw["grating_um"] == _GRATING_UM
|
||||||
|
assert kw["dc_threshold_mv"] == _THRESHOLD_MV
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(200)
|
||||||
|
|
||||||
|
|
||||||
|
def test_export_view_image_serves_a_row_averaged_stored_cache(tmp_path,
|
||||||
|
monkeypatch):
|
||||||
|
"""Batch Export View as Images must ask this file's cache the question the
|
||||||
|
viewer asks it. _stored_fft_image reads at the file's own
|
||||||
|
precomputed_row_avg_n; requesting raw per-pixel instead makes a
|
||||||
|
row-averaged cache a mismatch, and the export silently renders a full raw
|
||||||
|
recompute where the screen shows the smoothed stored image."""
|
||||||
|
path = tmp_path / "rowavg_cache.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=83, samples_per_frame=64, geometry=[(6, 9)])
|
||||||
|
|
||||||
|
# A planted stored image rather than a real row-averaged compute: the
|
||||||
|
# question here is *which* source the export reads, and a distinctive
|
||||||
|
# array answers it without depending on the synthetic waveforms being
|
||||||
|
# smooth enough for averaging to move the numbers.
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
shape = sras.image_shape(0)
|
||||||
|
stored = (100.0 + 10.0 * np.arange(shape[0] * shape[1], dtype=np.float32)
|
||||||
|
).reshape(shape)
|
||||||
|
sras.write_v7_cache(new_freq_mhz=[stored], new_row_avg_n=5,
|
||||||
|
new_bg_sub=True, new_pad_factor=1)
|
||||||
|
|
||||||
|
reread = SrasFile(str(path))
|
||||||
|
raw = compute_rf_image(reread, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
||||||
|
n_fft=None, row_avg_n=0, use_stored=False)
|
||||||
|
assert not np.allclose(stored, raw), "planted cache must be distinguishable"
|
||||||
|
|
||||||
|
out_dir = tmp_path / "out"
|
||||||
|
out_dir.mkdir()
|
||||||
|
captured = _spy_draw(monkeypatch)
|
||||||
|
err, _out_name = export_view_image(
|
||||||
|
str(path), out_dir=str(out_dir), angle_idx=0, ch_idx=CH1_IDX,
|
||||||
|
**_kw(is_fft_mode=True, dc_threshold_mv=-1e9, colorbar_label="MHz",
|
||||||
|
mode_str="RF"))
|
||||||
|
assert err == ""
|
||||||
|
assert np.allclose(captured["img"], stored), \
|
||||||
|
"export rendered a recompute instead of the stored image on screen"
|
||||||
+176
-54
@@ -12,6 +12,7 @@ from pathlib import Path
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
import sras_average
|
||||||
import sras_compute as compute
|
import sras_compute as compute
|
||||||
from sras_compute import (
|
from sras_compute import (
|
||||||
cache_file, compute_dc_image, compute_rf_image, dc_image_mv,
|
cache_file, compute_dc_image, compute_rf_image, dc_image_mv,
|
||||||
@@ -122,7 +123,7 @@ def test_parallel_identity(tmp_path, monkeypatch):
|
|||||||
# the block size so every chunk splits into many FFT tasks — the worst
|
# the block size so every chunk splits into many FFT tasks — the worst
|
||||||
# case for boundary bugs.
|
# case for boundary bugs.
|
||||||
monkeypatch.setattr(compute, "_TOTAL_BYTES_BUDGET", 8 * n_frames * spf * 4)
|
monkeypatch.setattr(compute, "_TOTAL_BYTES_BUDGET", 8 * n_frames * spf * 4)
|
||||||
monkeypatch.setattr(compute, "_FFT_BLOCK", 4)
|
monkeypatch.setattr(compute, "_FFT_BLOCK_MAX", 4)
|
||||||
fft_rows = compute._plan_fft_rows(n_frames, spf, compute._TOTAL_BYTES_BUDGET)
|
fft_rows = compute._plan_fft_rows(n_frames, spf, compute._TOTAL_BYTES_BUDGET)
|
||||||
assert fft_rows < n_rows, \
|
assert fft_rows < n_rows, \
|
||||||
f"FFT work actually splits into multiple chunks ({fft_rows} of {n_rows})"
|
f"FFT work actually splits into multiple chunks ({fft_rows} of {n_rows})"
|
||||||
@@ -155,43 +156,10 @@ def test_parallel_identity(tmp_path, monkeypatch):
|
|||||||
"rf image identical (masked, padded/zoom)"
|
"rf image identical (masked, padded/zoom)"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("spf,bps", [(64, 2), (37, 1)])
|
def test_peak_bins_fuzz():
|
||||||
def test_zoom_identity(tmp_path, monkeypatch, spf, bps):
|
"""Hammer _peak_bins directly with adversarial spectra: noise,
|
||||||
"""The zoom peak search must reproduce the full padded-rfft argmax
|
|
||||||
bit-for-bit, across pad factors, masking, bg-sub, dtype, and backend."""
|
|
||||||
path = tmp_path / f"zoom_{spf}.sras"
|
|
||||||
gen.write(path, n_angles=2, seed=6, samples_per_frame=spf, bps=bps)
|
|
||||||
sras = SrasFile(str(path))
|
|
||||||
dc4 = dc_image_mv(sras, 0, CH4_IDX)
|
|
||||||
thr = float(np.median(dc4))
|
|
||||||
|
|
||||||
backends = ["scipy"] + (["pyfftw"] if compute.PYFFTW_AVAILABLE else [])
|
|
||||||
for backend in backends:
|
|
||||||
monkeypatch.setattr(compute, "_fft_backend", backend)
|
|
||||||
for pad in (4, 8, 40):
|
|
||||||
n_fft = spf * pad
|
|
||||||
for thr_v in (None, thr):
|
|
||||||
for bg in (False, True):
|
|
||||||
ref = compute_rf_image(sras, 0, dc_threshold_mv=thr_v,
|
|
||||||
apply_bg_sub=bg, n_fft=n_fft,
|
|
||||||
exact=True)
|
|
||||||
zoom = compute_rf_image(sras, 0, dc_threshold_mv=thr_v,
|
|
||||||
apply_bg_sub=bg, n_fft=n_fft)
|
|
||||||
diff = int((ref != zoom).sum())
|
|
||||||
assert diff == 0, \
|
|
||||||
(f"{diff} px differ: backend={backend} pad={pad} "
|
|
||||||
f"thr={thr_v} bg={bg} spf={spf}")
|
|
||||||
|
|
||||||
# A threshold above every pixel masks everything: both paths must agree
|
|
||||||
# on an all-zero image.
|
|
||||||
all_masked = compute_rf_image(sras, 0, dc_threshold_mv=1e9, n_fft=spf * 8)
|
|
||||||
assert not all_masked.any()
|
|
||||||
|
|
||||||
|
|
||||||
def test_zoom_identity_fuzz():
|
|
||||||
"""Hammer _peak_bins_zoom directly with adversarial spectra: noise,
|
|
||||||
un-subtracted DC offsets, on-bin and off-bin tones, near-tie tone pairs,
|
un-subtracted DC offsets, on-bin and off-bin tones, near-tie tone pairs,
|
||||||
and all-zero rows."""
|
and all-zero rows — against an independent scipy.fft reference."""
|
||||||
import scipy.fft as scipy_fft
|
import scipy.fft as scipy_fft
|
||||||
|
|
||||||
rng = np.random.default_rng(42)
|
rng = np.random.default_rng(42)
|
||||||
@@ -222,14 +190,102 @@ def test_zoom_identity_fuzz():
|
|||||||
P[:, 0] = 0.0
|
P[:, 0] = 0.0
|
||||||
ref = np.argmax(P, axis=1)
|
ref = np.argmax(P, axis=1)
|
||||||
|
|
||||||
zp = compute._zoom_plan(spf, n_fft)
|
got = compute._peak_bins(w, n_fft)
|
||||||
got = compute._peak_bins_zoom(w, zp)
|
|
||||||
bad = np.nonzero(ref != got)[0]
|
bad = np.nonzero(ref != got)[0]
|
||||||
assert not len(bad), \
|
assert not len(bad), \
|
||||||
(f"spf={spf} pad={pad}: rows {bad.tolist()} picked "
|
(f"spf={spf} pad={pad}: rows {bad.tolist()} picked "
|
||||||
f"{got[bad].tolist()} instead of {ref[bad].tolist()}")
|
f"{got[bad].tolist()} instead of {ref[bad].tolist()}")
|
||||||
|
|
||||||
|
|
||||||
|
def test_peak_bins_fuzz_min_freq():
|
||||||
|
"""Same adversarial-spectra fuzz as test_peak_bins_fuzz, but with a swept
|
||||||
|
min_bin floor: bins below the floor must be excluded from the argmax
|
||||||
|
exactly as the independent scipy.fft reference is, when zeroed the same
|
||||||
|
way before argmax."""
|
||||||
|
import scipy.fft as scipy_fft
|
||||||
|
|
||||||
|
rng = np.random.default_rng(43)
|
||||||
|
for _ in range(25):
|
||||||
|
spf = int(rng.integers(16, 220))
|
||||||
|
pad = int(rng.choice([4, 5, 8, 16, 40]))
|
||||||
|
n_fft = spf * pad
|
||||||
|
n_wf = 24
|
||||||
|
w = rng.normal(scale=20.0, size=(n_wf, spf))
|
||||||
|
t = np.arange(spf)
|
||||||
|
for r in range(6):
|
||||||
|
f = rng.uniform(1.0, spf / 2 - 1)
|
||||||
|
w[r] = 60 * np.sin(2 * np.pi * f * t / spf) + w[r] * (r % 2)
|
||||||
|
f1, f2 = rng.uniform(2.0, spf / 2 - 2, size=2)
|
||||||
|
w[6] = 50 * np.sin(2 * np.pi * f1 * t / spf) \
|
||||||
|
+ 49.9 * np.sin(2 * np.pi * f2 * t / spf)
|
||||||
|
w[7] = 50 * np.sin(2 * np.pi * f1 * t / spf) \
|
||||||
|
+ 50 * np.cos(2 * np.pi * f2 * t / spf)
|
||||||
|
w[8] = 90 + rng.normal(scale=5.0, size=spf)
|
||||||
|
w[9] = 0.0
|
||||||
|
w = w.astype(np.float32)
|
||||||
|
|
||||||
|
S = scipy_fft.rfft(w, n=n_fft, axis=-1, workers=1)
|
||||||
|
P = S.real ** 2
|
||||||
|
P += S.imag ** 2
|
||||||
|
n_bins_fine = n_fft // 2 + 1
|
||||||
|
min_bin = int(rng.integers(1, max(2, n_bins_fine // 3)))
|
||||||
|
P[:, :min_bin] = 0.0
|
||||||
|
ref = np.argmax(P, axis=1)
|
||||||
|
|
||||||
|
got = compute._peak_bins(w, n_fft, min_bin)
|
||||||
|
bad = np.nonzero(ref != got)[0]
|
||||||
|
assert not len(bad), \
|
||||||
|
(f"spf={spf} pad={pad} min_bin={min_bin}: rows {bad.tolist()} picked "
|
||||||
|
f"{got[bad].tolist()} instead of {ref[bad].tolist()}")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("spf", [64, 500, 2500])
|
||||||
|
def test_fft_block_for(spf):
|
||||||
|
"""Block size == _FFT_BLOCK_MAX at natural resolution, shrinks and stays
|
||||||
|
>= _FFT_BLOCK_MIN as n_len grows, and the implied per-thread byte
|
||||||
|
estimate respects _FFT_PLAN_BYTES_BUDGET except when the floor is
|
||||||
|
engaged."""
|
||||||
|
block_natural = compute._fft_block_for(spf, spf)
|
||||||
|
assert block_natural == compute._FFT_BLOCK_MAX
|
||||||
|
|
||||||
|
prev = compute._FFT_BLOCK_MAX
|
||||||
|
for pad in (2, 4, 8, 40, 500):
|
||||||
|
n_len = spf * pad
|
||||||
|
block = compute._fft_block_for(spf, n_len)
|
||||||
|
assert compute._FFT_BLOCK_MIN <= block <= prev
|
||||||
|
bytes_per_wf = 4 * spf + 8 * (n_len // 2 + 1)
|
||||||
|
if block > compute._FFT_BLOCK_MIN:
|
||||||
|
assert block * bytes_per_wf <= compute._FFT_PLAN_BYTES_BUDGET
|
||||||
|
prev = block
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_rf_image_min_freq_mhz(tmp_path):
|
||||||
|
"""min_freq_mhz threads through compute_rf_image end-to-end, for both
|
||||||
|
the natural-resolution and padded paths: 0.0 (default) must reproduce
|
||||||
|
the pre-existing image exactly, and a floor above every real peak must
|
||||||
|
collapse the image to bin 0 (0 MHz) — the same fallback the low-level
|
||||||
|
search uses when nothing survives the floor."""
|
||||||
|
path = tmp_path / "floor_e2e.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=12, samples_per_frame=64)
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
huge_floor = float(sras.freq_axis_mhz(None)[-1]) + 1.0 # above Nyquist
|
||||||
|
|
||||||
|
for n_fft in (None, 64 * 8):
|
||||||
|
unfiltered = compute_rf_image(sras, 0, dc_threshold_mv=None,
|
||||||
|
apply_bg_sub=False, n_fft=n_fft)
|
||||||
|
same = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=False,
|
||||||
|
n_fft=n_fft, min_freq_mhz=0.0)
|
||||||
|
assert np.array_equal(unfiltered, same), \
|
||||||
|
f"n_fft={n_fft}: min_freq_mhz=0.0 changed the output"
|
||||||
|
assert unfiltered.any(), \
|
||||||
|
f"n_fft={n_fft}: fixture should have real signal"
|
||||||
|
|
||||||
|
collapsed = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=False,
|
||||||
|
n_fft=n_fft, min_freq_mhz=huge_floor)
|
||||||
|
assert not collapsed.any(), \
|
||||||
|
f"n_fft={n_fft}: floor above Nyquist should collapse to 0 MHz"
|
||||||
|
|
||||||
|
|
||||||
def test_nomask_equals_low_threshold(tmp_path):
|
def test_nomask_equals_low_threshold(tmp_path):
|
||||||
"""dc_threshold_mv=None must equal a threshold below every pixel, while
|
"""dc_threshold_mv=None must equal a threshold below every pixel, while
|
||||||
skipping the CH4 read."""
|
skipping the CH4 read."""
|
||||||
@@ -307,45 +363,111 @@ def test_legacy_parse(tmp_path):
|
|||||||
|
|
||||||
|
|
||||||
def test_sras_average(tmp_path):
|
def test_sras_average(tmp_path):
|
||||||
"""The sras_average.py CLI: frame averaging with remainder handling."""
|
"""The sras_average.py CLI: v6 frame averaging with remainder handling,
|
||||||
src = tmp_path / "legacy_v4.sras"
|
per-angle ragged geometry, and the laser_freq_hz X-axis correction."""
|
||||||
meta = gen.write_legacy(src, version=4, n_angles=2, n_rows=4, n_frames=12,
|
src = tmp_path / "v6.sras"
|
||||||
samples_per_frame=32, seed=4)
|
meta = gen.write(src, n_angles=2, seed=4, samples_per_frame=32,
|
||||||
dst = tmp_path / "legacy_v4_avg.sras"
|
geometry=[(4, 12)])
|
||||||
|
src_sras = SrasFile(str(src))
|
||||||
|
|
||||||
|
dst = tmp_path / "v6_avg.sras"
|
||||||
proc = subprocess.run(
|
proc = subprocess.run(
|
||||||
[sys.executable, str(REPO / "sras_average.py"), str(src), str(dst), "--n", "4"],
|
[sys.executable, str(REPO / "sras_average.py"), str(src), str(dst), "--n", "4"],
|
||||||
capture_output=True, text=True, cwd=REPO)
|
capture_output=True, text=True, cwd=REPO)
|
||||||
assert proc.returncode == 0, (proc.stderr or proc.stdout).strip()[-200:]
|
assert proc.returncode == 0, (proc.stderr or proc.stdout).strip()[-200:]
|
||||||
|
|
||||||
avg = SrasFile(str(dst))
|
avg = SrasFile(str(dst))
|
||||||
assert avg.version == 4
|
assert avg.version == 6
|
||||||
assert list(avg.n_frames) == [3, 3], f"{list(avg.n_frames)}"
|
assert list(avg.n_frames) == [3, 3], f"{list(avg.n_frames)}"
|
||||||
assert (avg.n_angles == 2 and list(avg.n_rows) == [4, 4]
|
assert (avg.n_angles == 2 and list(avg.n_rows) == [4, 4]
|
||||||
and avg.n_channels == meta["n_channels"])
|
and avg.n_channels == src_sras.n_channels == 3)
|
||||||
assert np.allclose(avg.ch_ymult_mv, SrasFile(str(src)).ch_ymult_mv), \
|
assert np.allclose(avg.ch_ymult_mv, src_sras.ch_ymult_mv), \
|
||||||
"calibration preserved"
|
"calibration preserved"
|
||||||
assert np.array_equal(avg.background, SrasFile(str(src)).background), \
|
assert np.array_equal(avg.background, src_sras.background), \
|
||||||
"background preserved"
|
"background preserved"
|
||||||
src_data = meta["data"]
|
assert avg.laser_freq_hz == pytest.approx(src_sras.laser_freq_hz / 4), \
|
||||||
expect0 = src_data[0][:, :, 0:4, :].astype(np.float32).mean(axis=2).astype(np.int16)
|
"laser_freq_hz divided by N keeps pixel_x_mm correct after binning"
|
||||||
|
assert np.array_equal(avg.x_start_mm, src_sras.x_start_mm), \
|
||||||
|
"per-angle x_start unchanged"
|
||||||
|
|
||||||
|
src_waves = meta["waveforms"]
|
||||||
|
# int16 (not float32) before .mean(): matches _average_block's own
|
||||||
|
# float64-accumulator behavior for integer input, so this doesn't
|
||||||
|
# drift from what _average_block actually guarantees.
|
||||||
|
expect0 = src_waves[0][:, :, 0:4, :].astype(np.int16).mean(axis=2).astype(np.int16)
|
||||||
assert np.array_equal(np.asarray(avg.data[0])[:, :, 0, :], expect0), \
|
assert np.array_equal(np.asarray(avg.data[0])[:, :, 0, :], expect0), \
|
||||||
"first averaged group equals the mean of its 4 source frames"
|
"first averaged group equals the mean of its 4 source frames"
|
||||||
|
|
||||||
# Remainder handling: 12 frames / 5 -> 2 full groups + 1 partial.
|
# Remainder handling: 12 frames / 5 -> 2 full groups + 1 partial.
|
||||||
dst2 = tmp_path / "legacy_v4_avg5.sras"
|
dst2 = tmp_path / "v6_avg5.sras"
|
||||||
subprocess.run([sys.executable, str(REPO / "sras_average.py"),
|
proc2 = subprocess.run([sys.executable, str(REPO / "sras_average.py"),
|
||||||
str(src), str(dst2), "--n", "5"],
|
str(src), str(dst2), "--n", "5"],
|
||||||
capture_output=True, text=True, cwd=REPO)
|
capture_output=True, text=True, cwd=REPO)
|
||||||
|
assert proc2.returncode == 0, (proc2.stderr or proc2.stdout).strip()[-200:]
|
||||||
assert list(SrasFile(str(dst2)).n_frames) == [3, 3], \
|
assert list(SrasFile(str(dst2)).n_frames) == [3, 3], \
|
||||||
"partial trailing group kept by default"
|
"partial trailing group kept by default"
|
||||||
dst3 = tmp_path / "legacy_v4_avg5d.sras"
|
dst3 = tmp_path / "v6_avg5d.sras"
|
||||||
subprocess.run([sys.executable, str(REPO / "sras_average.py"),
|
proc3 = subprocess.run([sys.executable, str(REPO / "sras_average.py"),
|
||||||
str(src), str(dst3), "--n", "5", "--discard-remainder"],
|
str(src), str(dst3), "--n", "5", "--discard-remainder"],
|
||||||
capture_output=True, text=True, cwd=REPO)
|
capture_output=True, text=True, cwd=REPO)
|
||||||
|
assert proc3.returncode == 0, (proc3.stderr or proc3.stdout).strip()[-200:]
|
||||||
assert list(SrasFile(str(dst3)).n_frames) == [2, 2], \
|
assert list(SrasFile(str(dst3)).n_frames) == [2, 2], \
|
||||||
"--discard-remainder drops the partial group"
|
"--discard-remainder drops the partial group"
|
||||||
|
|
||||||
|
|
||||||
|
def test_sras_average_v7_cache_dropped(tmp_path):
|
||||||
|
"""A v7 input's cache tail is indexed by frame count, so it's invalid
|
||||||
|
after averaging changes that count -- the output must always be plain
|
||||||
|
v6, never a v7 carrying a stale cache."""
|
||||||
|
src = tmp_path / "v7.sras"
|
||||||
|
gen.write(src, n_angles=2, seed=1, samples_per_frame=32, geometry=[(3, 8)])
|
||||||
|
src_sras = SrasFile(str(src))
|
||||||
|
src_sras.write_v7_cache(
|
||||||
|
new_dc3_mv=[compute_dc_image(src_sras, a, CH3_IDX) for a in range(src_sras.n_angles)],
|
||||||
|
new_dc4_mv=[compute_dc_image(src_sras, a, CH4_IDX) for a in range(src_sras.n_angles)])
|
||||||
|
assert SrasFile(str(src)).version == 7
|
||||||
|
|
||||||
|
dst = tmp_path / "v7_avg.sras"
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, str(REPO / "sras_average.py"), str(src), str(dst), "--n", "2"],
|
||||||
|
capture_output=True, text=True, cwd=REPO)
|
||||||
|
assert proc.returncode == 0, (proc.stderr or proc.stdout).strip()[-200:]
|
||||||
|
assert SrasFile(str(dst)).version == 6, "cache-bearing input still writes plain v6"
|
||||||
|
|
||||||
|
|
||||||
|
def test_sras_average_rejects_legacy(tmp_path):
|
||||||
|
"""This tool only speaks v6/v7 now; a legacy file must fail clearly
|
||||||
|
rather than being silently misparsed."""
|
||||||
|
src = tmp_path / "legacy_v4.sras"
|
||||||
|
gen.write_legacy(src, version=4, n_angles=1, n_rows=2, n_frames=8,
|
||||||
|
samples_per_frame=16, seed=0)
|
||||||
|
dst = tmp_path / "legacy_v4_avg.sras"
|
||||||
|
proc = subprocess.run(
|
||||||
|
[sys.executable, str(REPO / "sras_average.py"), str(src), str(dst), "--n", "2"],
|
||||||
|
capture_output=True, text=True, cwd=REPO)
|
||||||
|
assert proc.returncode != 0
|
||||||
|
assert "v6" in proc.stderr and "v7" in proc.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_sras_average_chunking_matches_unchunked(tmp_path):
|
||||||
|
"""A tiny memory budget (forcing one row per chunk) must produce
|
||||||
|
byte-identical output to a huge budget (everything in one chunk) -- the
|
||||||
|
load-bearing correctness claim of the memory-bounded rewrite: chunk
|
||||||
|
boundaries must never affect the averaged result."""
|
||||||
|
src = tmp_path / "v6.sras"
|
||||||
|
gen.write(src, n_angles=2, seed=7, samples_per_frame=48,
|
||||||
|
geometry=[(6, 10), (5, 13)])
|
||||||
|
sras = SrasFile(str(src))
|
||||||
|
|
||||||
|
dst_tiny = tmp_path / "avg_tiny.sras"
|
||||||
|
dst_big = tmp_path / "avg_big.sras"
|
||||||
|
sras_average.write_v6_averaged(sras, dst_tiny, 3, False, budget=1)
|
||||||
|
sras_average.write_v6_averaged(sras, dst_big, 3, False, budget=1 << 30)
|
||||||
|
|
||||||
|
assert dst_tiny.read_bytes() == dst_big.read_bytes(), \
|
||||||
|
"chunk size must not affect the averaged output"
|
||||||
|
|
||||||
|
|
||||||
def test_unsupported_version_reported(tmp_path):
|
def test_unsupported_version_reported(tmp_path):
|
||||||
"""cache_file must report, not raise, for a file it can't handle."""
|
"""cache_file must report, not raise, for a file it can't handle."""
|
||||||
bogus = tmp_path / "bogus.sras"
|
bogus = tmp_path / "bogus.sras"
|
||||||
|
|||||||
@@ -223,6 +223,54 @@ def test_threshold_change_recomputes(ctx):
|
|||||||
f"masking zeroed some pixels ({n_zero} of {win._current_image.size})"
|
f"masking zeroed some pixels ({n_zero} of {win._current_image.size})"
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlight_masked_pixels(ctx):
|
||||||
|
"""Masked (below-threshold) pixels are drawn as NaN - filled with a
|
||||||
|
highlight color, separate from the normal colormap - so they can't be
|
||||||
|
mistaken for a real, possibly-low-frequency pixel; unchecking restores
|
||||||
|
the old behavior where both blend into the same plain 0."""
|
||||||
|
win = ctx.win
|
||||||
|
assert win.chk_highlight_masked.isChecked(), "on by default"
|
||||||
|
dc4 = win._dc_cache[(0, CH4_IDX)]
|
||||||
|
expect_masked = dc4 < win.spin_threshold_mv.value()
|
||||||
|
assert expect_masked.any() and not expect_masked.all(), \
|
||||||
|
"fixture threshold should mask some but not all pixels"
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
orig = win.image_canvas.show_image
|
||||||
|
|
||||||
|
def spy(img, *a, **kw):
|
||||||
|
calls.append((np.array(img, copy=True), kw.get("bad_color")))
|
||||||
|
return orig(img, *a, **kw)
|
||||||
|
|
||||||
|
with patch.object(win.image_canvas, "show_image", side_effect=spy):
|
||||||
|
win._redraw_image(win._current_image)
|
||||||
|
shown, bad_color = calls[-1]
|
||||||
|
assert bad_color is not None, "highlight color set while checkbox is on"
|
||||||
|
# The highlight masks by value too: in an FFT mode, exactly 0 is the
|
||||||
|
# "no valid peak" sentinel (DC-masked, below the min-freq floor, or an
|
||||||
|
# empty spectrum), so the NaN set is the union of the DC mask and the
|
||||||
|
# zero-valued pixels. On this fixture every above-threshold pixel has a
|
||||||
|
# nonzero peak, so the union equals the DC mask alone.
|
||||||
|
expect_nan = expect_masked | (win._current_image == 0)
|
||||||
|
assert np.array_equal(np.isnan(shown), expect_nan), \
|
||||||
|
"NaN where DC4 is below threshold or the value-0 sentinel, nowhere else"
|
||||||
|
assert np.array_equal(expect_nan, expect_masked), \
|
||||||
|
"fixture precondition: every valid pixel has a nonzero peak"
|
||||||
|
|
||||||
|
win.chk_highlight_masked.setChecked(False)
|
||||||
|
calls.clear()
|
||||||
|
with patch.object(win.image_canvas, "show_image", side_effect=spy):
|
||||||
|
win._redraw_image(win._current_image)
|
||||||
|
shown2, bad_color2 = calls[-1]
|
||||||
|
assert bad_color2 is None, "no highlight color once unchecked"
|
||||||
|
assert not np.isnan(shown2).any(), "unchecked: no pixel pulled out to NaN"
|
||||||
|
assert np.array_equal(shown2, win._current_image), \
|
||||||
|
"unchecked: displayed array is the raw, unmodified image"
|
||||||
|
|
||||||
|
win.chk_highlight_masked.setChecked(True)
|
||||||
|
pump(60)
|
||||||
|
|
||||||
|
|
||||||
def test_bg_sub_toggle(ctx):
|
def test_bg_sub_toggle(ctx):
|
||||||
"""bg-sub no longer gates the display: it only affects a future live
|
"""bg-sub no longer gates the display: it only affects a future live
|
||||||
compute for an angle with nothing cached yet, or an explicit batch
|
compute for an angle with nothing cached yet, or an explicit batch
|
||||||
|
|||||||
+34
-15
@@ -14,7 +14,7 @@ import pytest
|
|||||||
|
|
||||||
import sras_compute as compute
|
import sras_compute as compute
|
||||||
from sras_compute import compute_rf_image, dc_image_mv
|
from sras_compute import compute_rf_image, dc_image_mv
|
||||||
from sras_format import CH4_IDX, SrasFile
|
from sras_format import CH1_IDX, CH4_IDX, SrasFile
|
||||||
import tools.make_test_sras as gen
|
import tools.make_test_sras as gen
|
||||||
|
|
||||||
|
|
||||||
@@ -193,26 +193,45 @@ def test_row_average_respects_own_center_mask(tmp_path):
|
|||||||
|
|
||||||
def test_row_average_composes_with_padding(tmp_path):
|
def test_row_average_composes_with_padding(tmp_path):
|
||||||
"""row_avg_n and n_fft (zero-padding) are independent knobs: using them
|
"""row_avg_n and n_fft (zero-padding) are independent knobs: using them
|
||||||
together must not raise, and must still agree with the exact (non-zoom)
|
together must not raise, and must agree bit-for-bit with an independent
|
||||||
reference path at that pad factor -- i.e. row-averaging composes with
|
reference that row-averages the raw waveforms and background-subtracts
|
||||||
the zoom peak search correctly, not just with the direct one."""
|
them, then runs a plain scipy rfft + argmax at the same pad factor --
|
||||||
|
i.e. row-averaging composes correctly with the padded peak search."""
|
||||||
|
import scipy.fft as scipy_fft
|
||||||
|
|
||||||
path = tmp_path / "padded_rowavg.sras"
|
path = tmp_path / "padded_rowavg.sras"
|
||||||
gen.write(path, n_angles=1, seed=12, samples_per_frame=64)
|
gen.write(path, n_angles=1, seed=12, samples_per_frame=64)
|
||||||
sras = SrasFile(str(path))
|
sras = SrasFile(str(path))
|
||||||
spf = sras.samples_per_frame
|
spf = sras.samples_per_frame
|
||||||
|
n_rows, n_frames = sras.image_shape(0)
|
||||||
|
n_fft = spf * 40
|
||||||
|
|
||||||
raw_natural = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True)
|
raw_natural = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True)
|
||||||
avg_natural = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
avg_natural = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
||||||
row_avg_n=3)
|
row_avg_n=3)
|
||||||
avg_padded_zoom = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
avg_padded = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
||||||
row_avg_n=3, n_fft=spf * 40)
|
row_avg_n=3, n_fft=n_fft)
|
||||||
avg_padded_exact = compute_rf_image(sras, 0, dc_threshold_mv=None, apply_bg_sub=True,
|
|
||||||
row_avg_n=3, n_fft=spf * 40, exact=True)
|
|
||||||
|
|
||||||
assert avg_natural.shape == raw_natural.shape == avg_padded_zoom.shape
|
assert avg_natural.shape == raw_natural.shape == avg_padded.shape
|
||||||
assert np.all(np.isfinite(avg_padded_zoom))
|
assert np.all(np.isfinite(avg_padded))
|
||||||
assert np.array_equal(avg_padded_zoom, avg_padded_exact), \
|
|
||||||
"row-averaged waveforms feed the zoom and exact FFT paths identically"
|
weights = compute._row_average_weights(3)
|
||||||
|
freq32 = sras.freq_axis_mhz(n_fft).astype(np.float32)
|
||||||
|
data = sras.data[0]
|
||||||
|
expected = np.zeros((n_rows, n_frames), dtype=np.float32)
|
||||||
|
for r in range(n_rows):
|
||||||
|
v = np.ones(n_frames, dtype=bool)
|
||||||
|
avg = compute._row_average_waveforms(
|
||||||
|
data[r, CH1_IDX].astype(np.float32), v, weights)
|
||||||
|
avg = avg - sras.background
|
||||||
|
S = scipy_fft.rfft(avg, n=n_fft, axis=-1, workers=1)
|
||||||
|
power = S.real ** 2 + S.imag ** 2
|
||||||
|
power[:, 0] = 0.0
|
||||||
|
expected[r] = freq32[np.argmax(power, axis=1)]
|
||||||
|
|
||||||
|
assert np.array_equal(avg_padded, expected), \
|
||||||
|
"row-averaged waveforms feed the padded peak search identically " \
|
||||||
|
"to an independent reference"
|
||||||
|
|
||||||
|
|
||||||
def test_row_average_improves_snr_recovery():
|
def test_row_average_improves_snr_recovery():
|
||||||
@@ -233,8 +252,8 @@ def test_row_average_improves_snr_recovery():
|
|||||||
weights = compute._row_average_weights(8) # wide window: lots of averaging
|
weights = compute._row_average_weights(8) # wide window: lots of averaging
|
||||||
averaged = compute._row_average_waveforms(raw, valid, weights)
|
averaged = compute._row_average_waveforms(raw, valid, weights)
|
||||||
|
|
||||||
raw_bins = compute._peak_bins_direct(raw, spf)
|
raw_bins = compute._peak_bins(raw, spf)
|
||||||
avg_bins = compute._peak_bins_direct(averaged, spf)
|
avg_bins = compute._peak_bins(averaged, spf)
|
||||||
|
|
||||||
raw_hits = int(np.sum(raw_bins == true_bin))
|
raw_hits = int(np.sum(raw_bins == true_bin))
|
||||||
avg_hits = int(np.sum(avg_bins == true_bin))
|
avg_hits = int(np.sum(avg_bins == true_bin))
|
||||||
@@ -257,7 +276,7 @@ def test_row_average_parallel_identity(tmp_path, monkeypatch):
|
|||||||
sras = SrasFile(str(path))
|
sras = SrasFile(str(path))
|
||||||
|
|
||||||
monkeypatch.setattr(compute, "_TOTAL_BYTES_BUDGET", 8 * n_frames * spf * 4)
|
monkeypatch.setattr(compute, "_TOTAL_BYTES_BUDGET", 8 * n_frames * spf * 4)
|
||||||
monkeypatch.setattr(compute, "_FFT_BLOCK", 4)
|
monkeypatch.setattr(compute, "_FFT_BLOCK_MAX", 4)
|
||||||
# row_avg_n > 0 halves the effective budget before chunk planning.
|
# row_avg_n > 0 halves the effective budget before chunk planning.
|
||||||
fft_rows = compute._plan_fft_rows(n_frames, spf, compute._TOTAL_BYTES_BUDGET // 2)
|
fft_rows = compute._plan_fft_rows(n_frames, spf, compute._TOTAL_BYTES_BUDGET // 2)
|
||||||
assert fft_rows < n_rows, \
|
assert fft_rows < n_rows, \
|
||||||
|
|||||||
+360
-4
@@ -90,7 +90,7 @@ def no_fft(monkeypatch):
|
|||||||
"""Make any real FFT work loud: returns a list that stays empty unless a
|
"""Make any real FFT work loud: returns a list that stays empty unless a
|
||||||
peak search actually runs."""
|
peak search actually runs."""
|
||||||
calls = []
|
calls = []
|
||||||
for name in ("_peak_bins_direct", "_peak_bins_zoom"):
|
for name in ("_peak_bins",):
|
||||||
original = getattr(compute, name)
|
original = getattr(compute, name)
|
||||||
|
|
||||||
def spy(*args, _f=original, **kwargs):
|
def spy(*args, _f=original, **kwargs):
|
||||||
@@ -147,7 +147,7 @@ def test_cache_is_stored_at_the_configured_pad(tmp_path, pad):
|
|||||||
gen.write(path, n_angles=2, seed=13, samples_per_frame=256)
|
gen.write(path, n_angles=2, seed=13, samples_per_frame=256)
|
||||||
spf = SrasFile(str(path)).samples_per_frame
|
spf = SrasFile(str(path)).samples_per_frame
|
||||||
|
|
||||||
assert cache_file(str(path), "fft", True, "scipy", 0, pad) == ""
|
assert cache_file(str(path), "fft", True, 0, pad) == ""
|
||||||
sras = SrasFile(str(path))
|
sras = SrasFile(str(path))
|
||||||
assert sras.precomputed_pad_factor == pad, "pad factor survives the round trip"
|
assert sras.precomputed_pad_factor == pad, "pad factor survives the round trip"
|
||||||
|
|
||||||
@@ -180,12 +180,18 @@ def test_cach_v1_reads_as_natural_resolution(tmp_path):
|
|||||||
# Rewrite the tail as a genuine CACH v1 block (old header, no pad field).
|
# Rewrite the tail as a genuine CACH v1 block (old header, no pad field).
|
||||||
v2 = SrasFile(str(path))
|
v2 = SrasFile(str(path))
|
||||||
freq, entries = v2.precomputed_freq_mhz, list(range(v2.n_angles))
|
freq, entries = v2.precomputed_freq_mhz, list(range(v2.n_angles))
|
||||||
|
tail_offset = v2._cache_tail_offset()
|
||||||
payload = struct.pack(fmt.CACH_HDR_FMT, fmt.CACH_MAGIC, 1, fmt.CACH_FLAG_FFT)
|
payload = struct.pack(fmt.CACH_HDR_FMT, fmt.CACH_MAGIC, 1, fmt.CACH_FLAG_FFT)
|
||||||
payload += struct.pack(fmt.SFFT_HDR_FMT_V1, fmt.SFFT_MAGIC,
|
payload += struct.pack(fmt.SFFT_HDR_FMT_V1, fmt.SFFT_MAGIC,
|
||||||
fmt.SFFT_FLAG_BG_SUB, len(entries))
|
fmt.SFFT_FLAG_BG_SUB, len(entries))
|
||||||
for a in entries:
|
for a in entries:
|
||||||
payload += struct.pack(">H", a) + freq[a].astype(">f4").tobytes()
|
payload += struct.pack(">H", a) + freq[a].astype(">f4").tobytes()
|
||||||
head = path.read_bytes()[:v2._cache_tail_offset()]
|
# Windows refuses to truncate a file with a live mapping (write_bytes
|
||||||
|
# opens 'wb'), and every SrasFile holds its waveform memmaps for life —
|
||||||
|
# drop the instance first. The parsed freq arrays are plain copies and
|
||||||
|
# stay usable.
|
||||||
|
del v2
|
||||||
|
head = path.read_bytes()[:tail_offset]
|
||||||
path.write_bytes(head + payload)
|
path.write_bytes(head + payload)
|
||||||
|
|
||||||
v1 = SrasFile(str(path))
|
v1 = SrasFile(str(path))
|
||||||
@@ -493,12 +499,16 @@ def test_cach_v1_backward_compat_defaults_row_avg_n_zero(tmp_path):
|
|||||||
|
|
||||||
v2 = SrasFile(str(path))
|
v2 = SrasFile(str(path))
|
||||||
freq, entries = v2.precomputed_freq_mhz, list(range(v2.n_angles))
|
freq, entries = v2.precomputed_freq_mhz, list(range(v2.n_angles))
|
||||||
|
tail_offset = v2._cache_tail_offset()
|
||||||
payload = struct.pack(fmt.CACH_HDR_FMT, fmt.CACH_MAGIC, 1, fmt.CACH_FLAG_FFT)
|
payload = struct.pack(fmt.CACH_HDR_FMT, fmt.CACH_MAGIC, 1, fmt.CACH_FLAG_FFT)
|
||||||
payload += struct.pack(fmt.SFFT_HDR_FMT_V1, fmt.SFFT_MAGIC,
|
payload += struct.pack(fmt.SFFT_HDR_FMT_V1, fmt.SFFT_MAGIC,
|
||||||
fmt.SFFT_FLAG_BG_SUB, len(entries))
|
fmt.SFFT_FLAG_BG_SUB, len(entries))
|
||||||
for a in entries:
|
for a in entries:
|
||||||
payload += struct.pack(">H", a) + freq[a].astype(">f4").tobytes()
|
payload += struct.pack(">H", a) + freq[a].astype(">f4").tobytes()
|
||||||
head = path.read_bytes()[:v2._cache_tail_offset()]
|
# See test_cach_v1_reads_as_natural_resolution: release the memmaps
|
||||||
|
# before write_bytes truncates, or Windows raises EINVAL.
|
||||||
|
del v2
|
||||||
|
head = path.read_bytes()[:tail_offset]
|
||||||
path.write_bytes(head + payload)
|
path.write_bytes(head + payload)
|
||||||
|
|
||||||
v1 = SrasFile(str(path))
|
v1 = SrasFile(str(path))
|
||||||
@@ -510,6 +520,288 @@ def test_cach_v1_backward_compat_defaults_row_avg_n_zero(tmp_path):
|
|||||||
"a v1 tail (predating this feature) can never satisfy a row-averaged request"
|
"a v1 tail (predating this feature) can never satisfy a row-averaged request"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Min peak frequency floor: serve-time masking, on-disk provenance, batch
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _biting_floor(stored: np.ndarray) -> float:
|
||||||
|
"""A floor that zeroes some-but-not-all of *stored*'s positive peaks:
|
||||||
|
the median distinct positive value, so pixels below it get masked and
|
||||||
|
pixels at/above it survive (the mask is a strict <)."""
|
||||||
|
positive = np.unique(stored[stored > 0])
|
||||||
|
assert len(positive) >= 2, "fixture must have varied peak frequencies"
|
||||||
|
return float(positive[len(positive) // 2])
|
||||||
|
|
||||||
|
|
||||||
|
def test_stored_image_served_with_raised_floor_masked(rig, no_fft):
|
||||||
|
"""The core of the 5 m/s bug fix: a floor above the stored one (here 0)
|
||||||
|
is re-applied when the stored image is served — pixels whose stored
|
||||||
|
peak falls below it come back as the 0.0 invalid sentinel, everything
|
||||||
|
else passes through, and no FFT runs. Both directly through
|
||||||
|
cached_rf_image and through compute_rf_image's fast path."""
|
||||||
|
assert rig.sras.precomputed_min_freq_mhz == 0.0, "batched without a floor"
|
||||||
|
stored = rig.sras.precomputed_freq_mhz[0]
|
||||||
|
floor = _biting_floor(stored)
|
||||||
|
expect = np.where(stored < floor, np.float32(0.0), stored)
|
||||||
|
|
||||||
|
img = cached_rf_image(rig.sras, 0, None, apply_bg_sub=True,
|
||||||
|
min_freq_mhz=floor)
|
||||||
|
assert img is not None, "an equal-or-higher floor is servable"
|
||||||
|
assert np.array_equal(img, expect)
|
||||||
|
assert (img == 0).any() and (img > 0).any(), \
|
||||||
|
"the floor bites some pixels but not all"
|
||||||
|
|
||||||
|
via_compute = compute_rf_image(rig.sras, 0, dc_threshold_mv=None,
|
||||||
|
apply_bg_sub=True, min_freq_mhz=floor)
|
||||||
|
assert np.array_equal(via_compute, expect), \
|
||||||
|
"compute_rf_image's fast path applies the same serve-time mask"
|
||||||
|
assert not no_fft, f"serving + masking must not run an FFT: {no_fft}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_min_freq_floor_round_trips_and_gates_serving(tmp_path):
|
||||||
|
"""cache_file records the floor in the SFFT header and the accept rule
|
||||||
|
is asymmetric: an equal-or-higher request is servable, a lower one is
|
||||||
|
refused (the stored search never looked below its floor). 20.1 pins the
|
||||||
|
fixed-point kHz encoding — a float32 header field would read back as
|
||||||
|
20.10000038…, above the requested 20.1, and mismatch forever."""
|
||||||
|
path = tmp_path / "floor_roundtrip.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=31, samples_per_frame=128)
|
||||||
|
floor = 20.1
|
||||||
|
assert cache_file(str(path), "fft", True, min_freq_mhz=floor) == ""
|
||||||
|
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
assert sras.precomputed_min_freq_mhz == floor, "exact fixed-point round-trip"
|
||||||
|
assert all(((img == 0) | (img >= floor)).all()
|
||||||
|
for img in sras.precomputed_freq_mhz), \
|
||||||
|
"no stored peak below the floor"
|
||||||
|
|
||||||
|
def reasons(f):
|
||||||
|
return compute.cache_mismatch_reasons(
|
||||||
|
sras, n_fft=None, apply_bg_sub=True, row_avg_n=0, min_freq_mhz=f)
|
||||||
|
|
||||||
|
assert reasons(floor) == []
|
||||||
|
assert reasons(floor + 5.0) == [], "a higher request is servable (masked)"
|
||||||
|
low = reasons(0.0)
|
||||||
|
assert low and "min-peak-freq floor" in low[0], \
|
||||||
|
"a lower request cannot be answered by the stored search"
|
||||||
|
assert cached_rf_image(sras, 0, None, apply_bg_sub=True) is None, \
|
||||||
|
"default floor-0 request refused against a floored store"
|
||||||
|
assert cached_rf_image(sras, 0, None, apply_bg_sub=True,
|
||||||
|
min_freq_mhz=floor) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_batch_recompute_resolves_not_masks(tmp_path, no_fft):
|
||||||
|
"""Re-batching an already-cached file at a raised floor must run the
|
||||||
|
real FFT and store re-resolved peaks — never let compute_rf_image's
|
||||||
|
fast path serve the file's own stale cache back to it and bake the
|
||||||
|
masked copy in as if it were a recompute (silent, permanent data
|
||||||
|
loss: a masked pixel's true above-floor peak is unrecoverable)."""
|
||||||
|
path = tmp_path / "rebatch.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=32, samples_per_frame=256)
|
||||||
|
assert cache_file(str(path), "fft", True) == ""
|
||||||
|
|
||||||
|
first = SrasFile(str(path))
|
||||||
|
n_angles = first.n_angles
|
||||||
|
# On the header's kHz grid up front (as the spinbox value would be), so
|
||||||
|
# the recorded floor reads back equal — cache_file quantizes whatever it
|
||||||
|
# is given, and this test wants that to be the identity.
|
||||||
|
floor = round(_biting_floor(first.precomputed_freq_mhz[0]) * 1000) / 1000.0
|
||||||
|
bites = [img < floor for img in first.precomputed_freq_mhz]
|
||||||
|
assert bites[0].any(), "the floor must actually bite this fixture"
|
||||||
|
# What a real floored compute gives, from a view blinded to the cache.
|
||||||
|
first.precomputed_freq_mhz = [None] * n_angles
|
||||||
|
expected = [compute_rf_image(first, a, dc_threshold_mv=None,
|
||||||
|
apply_bg_sub=True, min_freq_mhz=floor)
|
||||||
|
for a in range(n_angles)]
|
||||||
|
del first # release memmaps before cache_file rewrites the tail
|
||||||
|
|
||||||
|
no_fft.clear()
|
||||||
|
assert cache_file(str(path), "fft", True, min_freq_mhz=floor) == ""
|
||||||
|
assert no_fft, "the re-batch ran a real FFT"
|
||||||
|
|
||||||
|
after = SrasFile(str(path))
|
||||||
|
assert after.precomputed_min_freq_mhz == floor
|
||||||
|
for a in range(n_angles):
|
||||||
|
assert np.array_equal(after.precomputed_freq_mhz[a], expected[a]), \
|
||||||
|
f"angle {a}: stored image is a real floored recompute"
|
||||||
|
assert (after.precomputed_freq_mhz[a][bites[a]] >= floor).all(), \
|
||||||
|
f"angle {a}: bitten pixels re-resolved above the floor, not zeroed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_min_freq_carries_forward_through_dc_write(tmp_path):
|
||||||
|
"""A later DC-only write must leave the FFT block's recorded floor
|
||||||
|
untouched, like row_avg_n and pad_factor."""
|
||||||
|
path = tmp_path / "floor_carry.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=33, samples_per_frame=128)
|
||||||
|
assert cache_file(str(path), "fft", True, min_freq_mhz=75.0) == ""
|
||||||
|
assert cache_file(str(path), "dc", True) == ""
|
||||||
|
after = SrasFile(str(path))
|
||||||
|
assert after.precomputed_min_freq_mhz == 75.0, \
|
||||||
|
"floor survives a DC-only write"
|
||||||
|
|
||||||
|
|
||||||
|
def test_cach_v3_backward_compat_defaults_floor_zero(tmp_path):
|
||||||
|
"""A v3 CACH tail predates the min peak frequency floor entirely (no
|
||||||
|
min_freq_khz field) — readers must still parse it in full, treating it
|
||||||
|
as floor 0: servable as-is at floor 0, and serve-maskable at any higher
|
||||||
|
one. This is what protects existing real-world v7 caches from silently
|
||||||
|
becoming unusable after the v4 bump ships."""
|
||||||
|
path = tmp_path / "v3_floor.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=34, samples_per_frame=128)
|
||||||
|
assert cache_file(str(path), "fft", True) == ""
|
||||||
|
|
||||||
|
# Rewrite the tail as a genuine CACH v3 block (no min_freq field).
|
||||||
|
v4 = SrasFile(str(path))
|
||||||
|
freq, entries = v4.precomputed_freq_mhz, list(range(v4.n_angles))
|
||||||
|
tail_offset = v4._cache_tail_offset()
|
||||||
|
payload = struct.pack(fmt.CACH_HDR_FMT, fmt.CACH_MAGIC, 3, fmt.CACH_FLAG_FFT)
|
||||||
|
payload += struct.pack(fmt.SFFT_HDR_FMT_V3, fmt.SFFT_MAGIC,
|
||||||
|
fmt.SFFT_FLAG_BG_SUB, len(entries), 0, 1)
|
||||||
|
for a in entries:
|
||||||
|
payload += struct.pack(">H", a) + freq[a].astype(">f4").tobytes()
|
||||||
|
# See test_cach_v1_reads_as_natural_resolution: release the memmaps
|
||||||
|
# before write_bytes truncates, or Windows raises EINVAL.
|
||||||
|
del v4
|
||||||
|
head = path.read_bytes()[:tail_offset]
|
||||||
|
path.write_bytes(head + payload)
|
||||||
|
|
||||||
|
v3 = SrasFile(str(path))
|
||||||
|
assert v3.precomputed_min_freq_mhz == 0.0
|
||||||
|
assert v3.precomputed_pad_factor == 1 and v3.precomputed_bg_sub is True
|
||||||
|
assert all(np.array_equal(v3.precomputed_freq_mhz[a], freq[a])
|
||||||
|
for a in entries), "v3 images read back unchanged"
|
||||||
|
assert cached_rf_image(v3, 0, None, apply_bg_sub=True) is not None
|
||||||
|
floor = _biting_floor(freq[0])
|
||||||
|
served = cached_rf_image(v3, 0, None, apply_bg_sub=True,
|
||||||
|
min_freq_mhz=floor)
|
||||||
|
assert served is not None
|
||||||
|
assert np.array_equal(served,
|
||||||
|
np.where(freq[0] < floor, np.float32(0.0), freq[0]))
|
||||||
|
|
||||||
|
|
||||||
|
def test_min_freq_validation(tmp_path):
|
||||||
|
path = tmp_path / "floor_bad.sras"
|
||||||
|
gen.write(path, n_angles=1, seed=35, samples_per_frame=64)
|
||||||
|
assert cache_file(str(path), "fft", True, min_freq_mhz=-1.0), \
|
||||||
|
"negative floor must be an error, not a write"
|
||||||
|
assert cache_file(str(path), "fft", True, min_freq_mhz=float("nan")), \
|
||||||
|
"NaN floor must be an error, not a write"
|
||||||
|
sras = SrasFile(str(path))
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
sras.write_v7_cache(new_min_freq_mhz=-0.5)
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewer_reapplies_floor_to_stored_images_without_computing(
|
||||||
|
rig, no_fft, monkeypatch):
|
||||||
|
"""The session-cache poisoning bug behind the '95 MHz peak but 5 m/s'
|
||||||
|
report: changing 'Min peak freq' against a stored cache used to re-file
|
||||||
|
the identical un-floored image under a key claiming the new floor — the
|
||||||
|
UI looked updated, the pixels weren't. Now the floor really is
|
||||||
|
re-applied on serve (masked, still no compute), and clearing it
|
||||||
|
restores the unmasked image, still without computing."""
|
||||||
|
app = QApplication.instance() or QApplication([]) # noqa: F841
|
||||||
|
win = SrasViewerWindow()
|
||||||
|
win.show()
|
||||||
|
|
||||||
|
dispatched = []
|
||||||
|
original_start = type(win)._start_compute
|
||||||
|
monkeypatch.setattr(type(win), "_start_compute",
|
||||||
|
lambda self: (dispatched.append(self.spin_angle.value()),
|
||||||
|
original_start(self))[1])
|
||||||
|
try:
|
||||||
|
win._load_file(str(rig.path))
|
||||||
|
assert wait_until(lambda: win._sras is not None), "file loaded"
|
||||||
|
assert wait_until(lambda: all((a, CH4_IDX) in win._dc_cache
|
||||||
|
for a in range(rig.n_angles))), \
|
||||||
|
"DC precompute finished"
|
||||||
|
|
||||||
|
win.combo_channel.setCurrentIndex(CH1_IDX)
|
||||||
|
assert wait_until(lambda: win._current_ch == CH1_IDX), "CH1 displayed"
|
||||||
|
assert np.allclose(win._current_image, rig.fresh[0], atol=1e-3)
|
||||||
|
|
||||||
|
# Round to the spinbox's 3-decimal granularity; the chosen bin value
|
||||||
|
# still survives its own (strict-<) floor after rounding down.
|
||||||
|
floor = round(_biting_floor(rig.fresh[0]), 3)
|
||||||
|
expect = np.where(rig.fresh[0] < floor, np.float32(0.0), rig.fresh[0])
|
||||||
|
|
||||||
|
dispatched.clear()
|
||||||
|
no_fft.clear()
|
||||||
|
win.spin_min_freq_mhz.setValue(floor)
|
||||||
|
win._on_min_freq_changed()
|
||||||
|
pump(120)
|
||||||
|
assert np.allclose(win._current_image, expect, atol=1e-3), \
|
||||||
|
"raised floor re-masks the stored image on serve"
|
||||||
|
assert (win._current_image == 0).any() and (win._current_image > 0).any()
|
||||||
|
assert dispatched == [] and not no_fft, \
|
||||||
|
f"re-masked serve needs no compute (jobs={dispatched}, fft={no_fft})"
|
||||||
|
key = (0, win.spin_threshold_mv.value(), win.spin_min_freq_mhz.value())
|
||||||
|
assert key in win._fft_cache
|
||||||
|
assert np.allclose(win._fft_cache[key], expect, atol=1e-3), \
|
||||||
|
"the session cache holds the value its key claims"
|
||||||
|
|
||||||
|
win.spin_min_freq_mhz.setValue(0.0)
|
||||||
|
win._on_min_freq_changed()
|
||||||
|
pump(120)
|
||||||
|
assert np.allclose(win._current_image, rig.fresh[0], atol=1e-3), \
|
||||||
|
"clearing the floor restores the unmasked stored image"
|
||||||
|
assert dispatched == [] and not no_fft
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(300)
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewer_batch_fft_records_the_floor(tmp_path, no_fft, monkeypatch):
|
||||||
|
"""Convert → Batch Compute FFT with a floor set: the live spinbox value
|
||||||
|
reaches cache_file, lands in the reloaded file's provenance and the
|
||||||
|
info panel, and the viewer then serves the floored cache without
|
||||||
|
recomputing — the tooltip's promised remedy, end to end."""
|
||||||
|
path = tmp_path / "floor_gui.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=36, samples_per_frame=256)
|
||||||
|
|
||||||
|
app = QApplication.instance() or QApplication([]) # noqa: F841
|
||||||
|
win = SrasViewerWindow()
|
||||||
|
win.show()
|
||||||
|
|
||||||
|
dispatched = []
|
||||||
|
original_start = type(win)._start_compute
|
||||||
|
monkeypatch.setattr(type(win), "_start_compute",
|
||||||
|
lambda self: (dispatched.append(self.spin_angle.value()),
|
||||||
|
original_start(self))[1])
|
||||||
|
try:
|
||||||
|
win._load_file(str(path))
|
||||||
|
assert wait_until(lambda: win._sras is not None), "file loaded"
|
||||||
|
assert wait_until(lambda: all((a, CH4_IDX) in win._dc_cache
|
||||||
|
for a in range(win._sras.n_angles))), \
|
||||||
|
"DC precompute finished"
|
||||||
|
|
||||||
|
floor = 100.0
|
||||||
|
win.spin_min_freq_mhz.setValue(floor)
|
||||||
|
with patch("sras_viewer.main_window.QFileDialog.getOpenFileNames",
|
||||||
|
return_value=([str(path)], "")):
|
||||||
|
win._on_batch_compute("fft")
|
||||||
|
assert wait_until(lambda: not win._job_running("batch"), 60000), "batch ran"
|
||||||
|
assert wait_until(lambda: win._sras is not None
|
||||||
|
and win._sras.version == 7), "file reloaded as v7"
|
||||||
|
pump(200)
|
||||||
|
assert win._sras.precomputed_min_freq_mhz == floor, \
|
||||||
|
"the viewer's floor reached the stored provenance"
|
||||||
|
assert "floor ≥ 100 MHz" in win.lbl_frame_warn.text()
|
||||||
|
|
||||||
|
no_fft.clear()
|
||||||
|
dispatched.clear()
|
||||||
|
win.combo_channel.setCurrentIndex(CH1_IDX)
|
||||||
|
assert wait_until(lambda: win._current_ch == CH1_IDX), "CH1 displayed"
|
||||||
|
pump(120)
|
||||||
|
assert dispatched == [] and not no_fft, \
|
||||||
|
f"floored cache serves the view directly (jobs={dispatched}, fft={no_fft})"
|
||||||
|
img = win._current_image
|
||||||
|
assert ((img == 0) | (img >= floor)).all(), \
|
||||||
|
"no displayed peak below the floor"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(300)
|
||||||
|
|
||||||
|
|
||||||
def test_viewer_batch_row_average_dispatch(tmp_path, monkeypatch, no_fft):
|
def test_viewer_batch_row_average_dispatch(tmp_path, monkeypatch, no_fft):
|
||||||
"""Driving the new 'Batch Compute Row-Averaged FFT and Store' action
|
"""Driving the new 'Batch Compute Row-Averaged FFT and Store' action
|
||||||
end-to-end through the real menu handler: dialog values reach the
|
end-to-end through the real menu handler: dialog values reach the
|
||||||
@@ -593,3 +885,67 @@ def test_viewer_batch_row_average_dispatch(tmp_path, monkeypatch, no_fft):
|
|||||||
finally:
|
finally:
|
||||||
win.close()
|
win.close()
|
||||||
pump(300)
|
pump(300)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_worker_fallback_honors_row_avg_n(tmp_path, monkeypatch, no_fft):
|
||||||
|
"""The other half of the row-averaged recompute bug: _stored_fft_image
|
||||||
|
isn't the only path that can serve a view. When the DC4 mask isn't known
|
||||||
|
yet without I/O (e.g. background DC precompute hasn't reached this angle),
|
||||||
|
_stored_fft_image's allow_dc_recompute=False guard bails and
|
||||||
|
_refresh_display falls through to _start_compute's ComputeWorker instead.
|
||||||
|
That worker must still ask compute_rf_image for the file's own
|
||||||
|
row_avg_n -- not silently default to 0 -- so its own internal cache
|
||||||
|
fast path also serves the stored row-averaged image rather than running
|
||||||
|
a real, non-averaged FFT."""
|
||||||
|
path = tmp_path / "rowavg_fallback.sras"
|
||||||
|
gen.write(path, n_angles=2, seed=33, samples_per_frame=128)
|
||||||
|
# Only fft_rowavg -- deliberately no DC block, exactly what "Batch
|
||||||
|
# Compute Row-Averaged FFT and Store" leaves on disk by itself.
|
||||||
|
err = cache_file(str(path), "fft_rowavg", True, dc_threshold_mv=-1e9, row_avg_n=5)
|
||||||
|
assert err == "", err
|
||||||
|
|
||||||
|
app = QApplication.instance() or QApplication([]) # noqa: F841
|
||||||
|
win = SrasViewerWindow()
|
||||||
|
win.show()
|
||||||
|
|
||||||
|
# Keep _dc_cache empty so _stored_fft_image can't resolve a mask without
|
||||||
|
# I/O and _refresh_display must fall through to _start_compute.
|
||||||
|
monkeypatch.setattr(type(win), "_start_dc_precompute", lambda self: None)
|
||||||
|
|
||||||
|
dispatched = []
|
||||||
|
original_start = type(win)._start_compute
|
||||||
|
monkeypatch.setattr(type(win), "_start_compute",
|
||||||
|
lambda self: (dispatched.append(self.spin_angle.value()),
|
||||||
|
original_start(self))[1])
|
||||||
|
try:
|
||||||
|
win._load_file(str(path))
|
||||||
|
assert wait_until(lambda: win._sras is not None), "file loaded"
|
||||||
|
assert win._sras.precomputed_row_avg_n == 5
|
||||||
|
# The initial default view (CH4, DC) has nothing cached either, so
|
||||||
|
# it dispatches its own one-off DC compute for angle 0 on load --
|
||||||
|
# unrelated to this bug. Let that settle, then clear it so no DC4 is
|
||||||
|
# available for any angle, simulating "background DC precompute
|
||||||
|
# hasn't reached this angle yet".
|
||||||
|
assert wait_until(lambda: not win._job_running("compute")), "initial DC view settled"
|
||||||
|
dispatched.clear()
|
||||||
|
win._dc_cache.clear()
|
||||||
|
|
||||||
|
no_fft.clear()
|
||||||
|
win.combo_channel.setCurrentIndex(CH1_IDX)
|
||||||
|
assert wait_until(lambda: win._current_ch == CH1_IDX
|
||||||
|
and not win._job_running("compute")), "CH1 displayed"
|
||||||
|
|
||||||
|
assert dispatched == [0], \
|
||||||
|
"with no DC cache available yet, the fallback compute must run"
|
||||||
|
assert not no_fft, \
|
||||||
|
f"the fallback's own compute_rf_image call must still hit the " \
|
||||||
|
f"stored row-averaged cache internally: {no_fft}"
|
||||||
|
expected = compute.cached_rf_image(
|
||||||
|
win._sras, 0, dc_threshold_mv=win.spin_threshold_mv.value(),
|
||||||
|
apply_bg_sub=win.chk_bg_sub.isChecked(), row_avg_n=5)
|
||||||
|
assert expected is not None
|
||||||
|
assert np.allclose(win._current_image, expected, atol=1e-3), \
|
||||||
|
"the displayed image is the stored row-averaged one, not a raw recompute"
|
||||||
|
finally:
|
||||||
|
win.close()
|
||||||
|
pump(300)
|
||||||
|
|||||||
+29
-26
@@ -1,8 +1,10 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Benchmark the FFT peak-search path: exact vs zoom, serial vs pooled.
|
"""Benchmark the FFT peak-search path: serial vs pooled.
|
||||||
|
|
||||||
Reports wall time, waveforms/s, CPU utilization (utime+stime over wall, in
|
Reports wall time, waveforms/s, CPU utilization (utime+stime over wall, in
|
||||||
cores), and verifies every variant against the exact reference image.
|
cores), the estimated per-thread resident pyFFTW plan footprint at each pad
|
||||||
|
factor (see sras_compute._fft_block_for), and verifies pooled output against
|
||||||
|
the serial reference.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
python tools/bench_fft.py # synthetic, pads 1/8/40
|
python tools/bench_fft.py # synthetic, pads 1/8/40
|
||||||
@@ -22,7 +24,7 @@ import numpy as np
|
|||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||||
|
|
||||||
import sras_compute as compute # noqa: E402
|
import sras_compute as compute # noqa: E402
|
||||||
from sras_compute import compute_rf_image, set_fft_backend # noqa: E402
|
from sras_compute import compute_rf_image # noqa: E402
|
||||||
from sras_format import SrasFile # noqa: E402
|
from sras_format import SrasFile # noqa: E402
|
||||||
import tools.make_test_sras as gen # noqa: E402
|
import tools.make_test_sras as gen # noqa: E402
|
||||||
from tools.check_equivalence import row_slice # noqa: E402
|
from tools.check_equivalence import row_slice # noqa: E402
|
||||||
@@ -38,18 +40,28 @@ def _timed(fn):
|
|||||||
return out, wall, cpu / max(wall, 1e-9)
|
return out, wall, cpu / max(wall, 1e-9)
|
||||||
|
|
||||||
|
|
||||||
def bench(sras, pads, backends):
|
def _plan_mb(spf: int, n_len: int, n_workers: int) -> float:
|
||||||
|
"""Estimated resident pyFFTW plan-buffer footprint across the whole
|
||||||
|
pool at this transform length (see sras_compute._fft_block_for)."""
|
||||||
|
block = compute._fft_block_for(spf, n_len)
|
||||||
|
bytes_per_wf = 4 * spf + 8 * (n_len // 2 + 1)
|
||||||
|
return block * bytes_per_wf * n_workers / (1024 * 1024)
|
||||||
|
|
||||||
|
|
||||||
|
def bench(sras, pads):
|
||||||
n_wf = sum(int(sras.n_rows[a]) * int(sras.n_frames[a])
|
n_wf = sum(int(sras.n_rows[a]) * int(sras.n_frames[a])
|
||||||
for a in range(sras.n_angles))
|
for a in range(sras.n_angles))
|
||||||
spf = sras.samples_per_frame
|
spf = sras.samples_per_frame
|
||||||
print(f"{n_wf} waveforms x {spf} samples, {sras.n_angles} angle(s)")
|
n_workers = compute._MAX_WORKERS
|
||||||
print(f"{'pad':>4} {'backend':>8} {'variant':>16} {'wall':>9} "
|
print(f"{n_wf} waveforms x {spf} samples, {sras.n_angles} angle(s), "
|
||||||
f"{'wf/s':>10} {'util':>6} match")
|
f"{n_workers} workers")
|
||||||
|
print(f"{'pad':>4} {'variant':>8} {'wall':>9} {'wf/s':>10} {'util':>6} "
|
||||||
|
f"{'plan MB':>9} match")
|
||||||
|
|
||||||
for pad in pads:
|
for pad in pads:
|
||||||
n_fft = spf * pad if pad > 1 else None
|
n_fft = spf * pad if pad > 1 else None
|
||||||
for backend in backends:
|
n_len = n_fft if n_fft is not None else spf
|
||||||
set_fft_backend(backend)
|
plan_mb = _plan_mb(spf, n_len, n_workers)
|
||||||
|
|
||||||
def run(**kw):
|
def run(**kw):
|
||||||
imgs = [compute_rf_image(sras, a, dc_threshold_mv=None,
|
imgs = [compute_rf_image(sras, a, dc_threshold_mv=None,
|
||||||
@@ -57,16 +69,13 @@ def bench(sras, pads, backends):
|
|||||||
for a in range(sras.n_angles)]
|
for a in range(sras.n_angles)]
|
||||||
return np.concatenate([i.ravel() for i in imgs])
|
return np.concatenate([i.ravel() for i in imgs])
|
||||||
|
|
||||||
ref, wall, util = _timed(lambda: run(exact=True))
|
ref, wall, util = _timed(lambda: run(max_workers=1))
|
||||||
rows = [("exact(serial)", ref, wall, util, True)]
|
rows = [("serial", ref, wall, util, True)]
|
||||||
for label, kw in (("zoom(serial)", dict(max_workers=1)),
|
img, wall, util = _timed(lambda: run())
|
||||||
("zoom(pool)", {})):
|
rows.append(("pooled", img, wall, util, bool(np.array_equal(img, ref))))
|
||||||
img, wall, util = _timed(lambda: run(**kw))
|
|
||||||
rows.append((label, img, wall, util, bool(np.array_equal(img, ref))))
|
|
||||||
for label, img, wall, util, ok in rows:
|
for label, img, wall, util, ok in rows:
|
||||||
print(f"{pad:>4} {backend:>8} {label:>16} {wall:>8.2f}s "
|
print(f"{pad:>4} {label:>8} {wall:>8.2f}s {n_wf / wall:>10.0f} "
|
||||||
f"{n_wf / wall:>10.0f} {util:>5.1f}x "
|
f"{util:>5.1f}x {plan_mb:>8.1f} {'OK' if ok else 'MISMATCH'}")
|
||||||
f"{'OK' if ok else 'MISMATCH'}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
@@ -76,30 +85,24 @@ def main():
|
|||||||
p.add_argument("--spf", type=int, default=2500)
|
p.add_argument("--spf", type=int, default=2500)
|
||||||
p.add_argument("--rows", type=int, default=8)
|
p.add_argument("--rows", type=int, default=8)
|
||||||
p.add_argument("--frames", type=int, default=1024)
|
p.add_argument("--frames", type=int, default=1024)
|
||||||
p.add_argument("--backends", default=None,
|
|
||||||
help="comma-separated (default: scipy,pyfftw if available)")
|
|
||||||
p.add_argument("--real", help="path to a real .sras file")
|
p.add_argument("--real", help="path to a real .sras file")
|
||||||
p.add_argument("--real-rows", type=int, default=32,
|
p.add_argument("--real-rows", type=int, default=32,
|
||||||
help="rows of angle 0 to use from the real file")
|
help="rows of angle 0 to use from the real file")
|
||||||
args = p.parse_args()
|
args = p.parse_args()
|
||||||
|
|
||||||
pads = [int(x) for x in args.pads.split(",")]
|
pads = [int(x) for x in args.pads.split(",")]
|
||||||
if args.backends:
|
|
||||||
backends = args.backends.split(",")
|
|
||||||
else:
|
|
||||||
backends = ["scipy"] + (["pyfftw"] if compute.PYFFTW_AVAILABLE else [])
|
|
||||||
|
|
||||||
if args.real:
|
if args.real:
|
||||||
sras = row_slice(SrasFile(args.real), 0, args.real_rows)
|
sras = row_slice(SrasFile(args.real), 0, args.real_rows)
|
||||||
sras.data = [sras.data[0]]
|
sras.data = [sras.data[0]]
|
||||||
sras.n_angles = 1
|
sras.n_angles = 1
|
||||||
bench(sras, pads, backends)
|
bench(sras, pads)
|
||||||
else:
|
else:
|
||||||
with tempfile.TemporaryDirectory(prefix="sras_bench_") as tmp:
|
with tempfile.TemporaryDirectory(prefix="sras_bench_") as tmp:
|
||||||
path = Path(tmp) / "bench.sras"
|
path = Path(tmp) / "bench.sras"
|
||||||
gen.write(path, n_angles=1, seed=0, samples_per_frame=args.spf,
|
gen.write(path, n_angles=1, seed=0, samples_per_frame=args.spf,
|
||||||
geometry=[(args.rows, args.frames)])
|
geometry=[(args.rows, args.frames)])
|
||||||
bench(SrasFile(str(path)), pads, backends)
|
bench(SrasFile(str(path)), pads)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -261,15 +261,10 @@ def write_rotating(path: Path, n_angles: int = 5, samples_per_frame: int = 4,
|
|||||||
"y_starts": y_starts, "dx_mm": _ROT_DX_MM, "dy_mm": _ROT_DY_MM}
|
"y_starts": y_starts, "dx_mm": _ROT_DX_MM, "dy_mm": _ROT_DY_MM}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def write_legacy(path: Path, version: int = 4, n_angles: int = 2,
|
def write_legacy(path: Path, version: int = 4, n_angles: int = 2,
|
||||||
n_rows: int = 4, n_frames: int = 10,
|
n_rows: int = 4, n_frames: int = 10,
|
||||||
samples_per_frame: int = 32, seed: int = 0) -> dict:
|
samples_per_frame: int = 32, seed: int = 0) -> dict:
|
||||||
"""Write a v2/v3/v4 file: uniform geometry, one flat waveform block.
|
"""Write a v2/v3/v4 file: uniform geometry, one flat waveform block."""
|
||||||
|
|
||||||
Used to exercise sras_average.py, which only handles the legacy formats.
|
|
||||||
"""
|
|
||||||
rng = np.random.default_rng(seed)
|
rng = np.random.default_rng(seed)
|
||||||
n_ch, bps = 3, 1
|
n_ch, bps = 3, 1
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user