Implement row-averaged FFT feature for same-row SNR cleanup

Add row-averaged FFT feature with configurable window size (row_avg_n
parameter) for improved signal-to-noise ratio on noisy scans. Includes:

- Gaussian-weighted same-row neighbor averaging (never crosses rows)
- Masked/renormalized convolution handling for edge cases and masked samples
- Cache format v2 with row_avg_n tracking to prevent silent cache mismatches
- GUI dialog option for row-average window configuration
- Comprehensive tests validating kernel properties, background subtraction
  invariance, and cache dispatch

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
Thomas Ales
2026-08-07 21:50:05 -05:00
parent eecb0e3a82
commit 3989c2a1b8
10 changed files with 1363 additions and 51 deletions
+39 -8
View File
@@ -220,7 +220,7 @@ actions.
| Offset | Size | Type | Field | Description |
|--------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `1`. Readers must treat the file as uncached if this is not a version they understand (unlike v5's `PREC` section, which read but never validated its version byte). |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `2`; readers also accept `1` (a `1` tail predates row-averaged FFT caching — see the `SFFT` block below and [CACH tail version history](#cach-tail-version-history)). Any other value → treat the file as uncached (unlike v5's `PREC` section, which read but never validated its version byte). |
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |
### DC block `SDCB` (present iff `block_flags & 0x01`)
@@ -249,13 +249,20 @@ value, same as v5's `PREC` section.
### FFT block `SFFT` (present iff `block_flags & 0x02`)
7-byte block header, format `">4sBH"`:
Block header layout depends on `cach_version`:
- **`cach_version` 1**: 7 bytes, format `">4sBH"` — magic, flags, n_stored.
- **`cach_version` 2**: 8 bytes, format `">4sBHB"` — magic, flags, n_stored,
`row_avg_n`. Always written by current code; a `cach_version` 1 tail (no
trailing byte) is still read, with `row_avg_n` taken as `0` for every
entry it stores.
| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bits 1–7 reserved. |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bit 1 = `row_averaged` — `peak_freq_mhz` came from same-row, distance-weighted averaged CH1 waveforms rather than raw per-pixel ones; `row_avg_n` (below) is the neighbor half-width used. Bits 2–7 reserved. |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
| 7 | 1 | `u8` | `row_avg_n` | *`cach_version` 2 only.* Same-row neighbor half-width, in pixels, that `peak_freq_mhz` was averaged over before its FFT; `0` = raw (unaveraged). Meaningful only when `flags` bit 1 is set — a `cach_version` 1 tail has no such byte and is always `row_avg_n = 0`. |
followed by `n_stored` entries, each:
@@ -264,9 +271,10 @@ u16 angle_idx — index into the angle table (0-ba
f32[n_rows[angle_idx] × n_frames[angle_idx]] peak_freq_mhz — CH1 FFT peak frequency, MHz, row-major
```
**`peak_freq_mhz`** is computed without any DC-threshold masking (i.e. the
FFT is run on every pixel unconditionally, same as v5's `PREC` convention).
Readers apply the DC4 threshold at display time:
**`peak_freq_mhz`** for a raw store (`row_avg_n == 0`) is computed without
any DC-threshold masking (i.e. the FFT is run on every pixel
unconditionally, same as v5's `PREC` convention). Readers apply the DC4
threshold at display time:
```
pixel is valid ⟺ dc4_mv[r][f] ≥ threshold_mv
@@ -276,10 +284,22 @@ display_value = peak_freq_mhz[r][f] if valid, else 0
using the DC4 image from the DC block if that angle is also cached there,
else computed on demand.
For a row-averaged store (`row_avg_n > 0`), the DC4 threshold is applied
*during* the store — a pixel below threshold is left at `0` and never
contributes to any neighbor's average — since neighbor validity can't be
deferred to display time the way plain masking can. The threshold value
itself is not recorded, only that averaging happened and at what window
size. Readers still apply their own live DC4 threshold at display time
exactly as for a raw store, using whatever mask they currently have.
Readers must fall back to real-time FFT computation (ignoring stored
`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested, or
the reader's background-subtraction setting doesn't match `flags.bg_sub_applied`.
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested,
the reader's background-subtraction setting doesn't match
`flags.bg_sub_applied`, or the reader's requested `row_avg_n` doesn't match
the stored value exactly — a raw request must never be served a
row-averaged store, or vice versa, and a request at one window size must
never be served a store at another.
### In-place write ordering
@@ -294,6 +314,17 @@ interrupted write leaves harmless trailing bytes rather than a corrupt file,
and the next successful write overwrites them via the same deterministic
`cache_offset`.
### CACH tail version history
Distinct from the outer `.sras` file `version` byte (top of this document),
which has stayed `7` since the Cache Tail was introduced — this is the inner
`cach_version` byte inside the `CACH` header itself.
| cach_version | Change |
|--------------|--------|
| 1 | Initial Cache Tail: `SDCB` (DC) and `SFFT` (FFT, 7-byte header) blocks. |
| 2 | `SFFT` header grows one byte, `row_avg_n` — the same-row neighbor half-width the stored `peak_freq_mhz` was averaged over before its FFT, `0` = raw. Readers still accept a `cach_version` 1 tail, treated as `row_avg_n = 0` for every angle it stores, so files cached before this change keep working without a recompute. |
---
## Acquisition Settings (fixed by sc3_aui_app.py)