diff --git a/scan_format.md b/scan_format.md index c864ff4..3c90e16 100644 --- a/scan_format.md +++ b/scan_format.md @@ -1,4 +1,4 @@ -# SRAS Scan Binary Format — Version 6 +# SRAS Scan Binary Format — Version 6 / 7 Each `.sras` file contains **one complete scan**: all GR rotation angles and all Y rows. Files are named `{prefix}.sras`. @@ -10,6 +10,14 @@ rotated by that specific angle** — not the worst case across all angles — so fewer rows than a 45° scan of the same ROI, and the file format reflects that instead of forcing every angle to the largest bounding box. +v7 is byte-identical to v6 (same header, angle table, geometry table, row +table, preamble blocks, background block, waveform data) plus an optional +trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images +(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version` +field and the presence of that trailing section differ. Scans come off the +scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file +to v7 **in place** the first time either cache block is computed and stored. + --- ## File Layout @@ -22,6 +30,7 @@ instead of forcing every angle to the largest bounding box. [Preamble Blocks — n_channels × (uint16 length + UTF-8 WFMOutpre string)] [Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes] [Waveform Data (ragged) — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes] +[Cache Tail (optional) — "CACH" + DC block (optional) + FFT block (optional); v7 only] ``` All multi-byte integers and floats use **big-endian** byte order @@ -182,6 +191,111 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not --- +## Cache Tail (v7) + +Present iff `version == 7` and `file_size > cache_offset`, where: + +``` +cache_offset = data_offset + Σ over angles a of: + n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample +``` + +i.e. exactly `data_offset + waveform_bytes` — the same "Total data size" +formula as Waveform Data above. This offset is derivable from the header and +Per-Angle Geometry Table alone and does **not** depend on which cache +block(s) are present, so a writer can always seek straight there without +reading or touching any waveform byte before it. + +Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle +entry (FFT + both DC channels together) in a dense, uniform-geometry array, +the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks — +each sized per-angle from the Per-Angle Geometry Table, each independently +present, and each independently updatable in any order, any number of +times, without disturbing the other. This matches `sras_viewer.py`'s +"Convert" menu, which exposes DC and FFT store as two separate batch +actions. + +### CACH outer header (6 bytes, `">4sBB"`) + +| Offset | Size | Type | Field | Description | +|--------|------|------|-------|-------------| +| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. | +| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `1`. Readers must treat the file as uncached if this is not a version they understand (unlike v5's `PREC` section, which read but never validated its version byte). | +| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. | + +### DC block `SDCB` (present iff `block_flags & 0x01`) + +7-byte block header, format `">4sBH"`: + +| Offset (rel) | Size | Type | Field | Description | +|--------------|------|------|-------|-------------| +| 0 | 4 | `char[4]` | `magic` | `SDCB` | +| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use | +| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` | + +followed by `n_stored` entries, each: + +``` +u16 angle_idx — index into the angle table (0-based) +f32[n_rows[angle_idx] × n_frames[angle_idx]] dc3_mv — CH3 waveform mean, mV, row-major +f32[n_rows[angle_idx] × n_frames[angle_idx]] dc4_mv — CH4 waveform mean, mV, row-major +``` + +Entries may appear in any order and need not be contiguous from angle 0 — +this supports storing (or re-storing) a subset of angles, or an +interrupted batch run leaving only some angles cached. Readers bounds-check +`angle_idx < n_angles` on each entry and stop parsing on an out-of-range +value, same as v5's `PREC` section. + +### FFT block `SFFT` (present iff `block_flags & 0x02`) + +7-byte block header, format `">4sBH"`: + +| Offset (rel) | Size | Type | Field | Description | +|--------------|------|------|-------|-------------| +| 0 | 4 | `char[4]` | `magic` | `SFFT` | +| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bits 1–7 reserved. | +| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow | + +followed by `n_stored` entries, each: + +``` +u16 angle_idx — index into the angle table (0-based) +f32[n_rows[angle_idx] × n_frames[angle_idx]] peak_freq_mhz — CH1 FFT peak frequency, MHz, row-major +``` + +**`peak_freq_mhz`** is computed without any DC-threshold masking (i.e. the +FFT is run on every pixel unconditionally, same as v5's `PREC` convention). +Readers apply the DC4 threshold at display time: + +``` +pixel is valid ⟺ dc4_mv[r][f] ≥ threshold_mv +display_value = peak_freq_mhz[r][f] if valid, else 0 +``` + +using the DC4 image from the DC block if that angle is also cached there, +else computed on demand. + +Readers must fall back to real-time FFT computation (ignoring stored +`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain +gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested, or +the reader's background-subtraction setting doesn't match `flags.bg_sub_applied`. + +### In-place write ordering + +A writer updating a file's Cache Tail must write the payload (CACH header + +whichever block(s) are present) **before** flipping the header's `version` +byte to `7`, and `truncate()` the file to the new payload's end immediately +after writing it. If the process is interrupted between the payload write +and the version-byte flip, the file is still valid v6 — v6 parsing only +bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts +exactly how many bytes follow the last angle's waveform block — so the +interrupted write leaves harmless trailing bytes rather than a corrupt file, +and the next successful write overwrites them via the same deterministic +`cache_offset`. + +--- + ## Acquisition Settings (fixed by sc3_aui_app.py) | Parameter | Value | @@ -206,5 +320,6 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not | 3 | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. | | 4 | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. | | 5 | (skipped) | -| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`); the "Pre-process and Save as v5" fast-path export is not offered for v6 files since the flat v5 layout cannot represent per-angle geometry. | +| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). | +| 7 | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. | diff --git a/sras_viewer.py b/sras_viewer.py index 9b7e3ca..13c657d 100644 --- a/sras_viewer.py +++ b/sras_viewer.py @@ -15,11 +15,16 @@ header before acquisition, but the scope may acquire fewer frames. The actual count is computed from the file size and used for the reshape so channels are correctly aligned. -Scan geometry: v6 files scan a different bounding box per angle (x_start, +Scan geometry: v6/v7 files scan a different bounding box per angle (x_start, x_delta, n_frames, n_rows all vary by angle), so geometry is exposed per-angle via SrasFile.n_rows / n_frames / x_start_mm arrays and the x_axis_mm() / y_positions_mm() methods. v2–v5 files have uniform geometry across angles, so those arrays simply repeat the same value n_angles times. + +v7 files are v6 files with an optional trailing cache section holding +precomputed per-angle DC and/or FFT images (see scan_format.md), so display +never has to recompute them after the "Convert" menu's batch actions have +stored them once. """ import re @@ -90,6 +95,29 @@ HDR_SIZE_V6 = struct.calcsize(HDR_FMT_V6) # 49 bytes GEO_FMT_V6 = ">ffIH" GEO_SIZE_V6 = struct.calcsize(GEO_FMT_V6) # 14 bytes +# v7: identical to v6 (same header/geometry/waveform layout, version byte +# is the only header difference) plus an optional trailing cache section +# ("CACH") holding precomputed per-angle DC and/or FFT images so a v7 file +# never needs to recompute them on open. See scan_format.md for the full +# spec. Sub-block sizes are per-angle (n_rows[a] * n_frames[a]), taken from +# the Per-Angle Geometry Table already parsed for v6 — no new geometry +# fields are needed. +CACH_MAGIC = b"CACH" +CACH_HDR_FMT = ">4sBB" # magic, cach_version, block_flags +CACH_HDR_SIZE = struct.calcsize(CACH_HDR_FMT) # 6 bytes +CACH_VERSION = 1 +CACH_FLAG_DC = 0x01 +CACH_FLAG_FFT = 0x02 + +SDCB_MAGIC = b"SDCB" +SDCB_HDR_FMT = ">4sBH" # magic, reserved, n_stored +SDCB_HDR_SIZE = struct.calcsize(SDCB_HDR_FMT) # 7 bytes + +SFFT_MAGIC = b"SFFT" +SFFT_HDR_FMT = ">4sBH" # magic, flags, n_stored +SFFT_HDR_SIZE = struct.calcsize(SFFT_HDR_FMT) # 7 bytes +SFFT_FLAG_BG_SUB = 0x01 + # Fixed-order channels in the file: index 0=CH1, 1=CH3, 2=CH4 # Fixed channel indices into the .sras data array (CH1=RF, CH3/CH4=Bias DC) CH1_IDX, CH3_IDX, CH4_IDX = 0, 1, 2 @@ -149,15 +177,21 @@ def adc_to_mv(adc: float, ymult_mv: float = _FALLBACK_YMULT_MV, # --------------------------------------------------------------------------- class SrasFile: - """Parsed in-memory representation of a v2–v6 .sras file. + """Parsed in-memory representation of a v2–v7 .sras file. Scan geometry (rows, frames, x_start) is exposed per-angle via the ``n_rows`` / ``n_frames`` / ``x_start_mm`` arrays and the ``x_axis_mm()`` - / ``y_positions_mm()`` methods, since v6 files scan a different bounding - box per angle. v2–v5 files have uniform geometry, so these arrays just - repeat the same value ``n_angles`` times. Waveform data is likewise - exposed as ``data[angle_idx]``, an array of shape + / ``y_positions_mm()`` methods, since v6/v7 files scan a different + bounding box per angle. v2–v5 files have uniform geometry, so these + arrays just repeat the same value ``n_angles`` times. Waveform data is + likewise exposed as ``data[angle_idx]``, an array of shape ``(n_rows[a], n_channels, n_frames[a], samples_per_frame)``. + + v7 files (v6 plus an optional trailing cache section) expose any stored + precomputed images as ``precomputed_dc3_mv`` / ``precomputed_dc4_mv`` / + ``precomputed_freq_mhz`` — for v6/v7 these are ragged per-angle lists + (``list[np.ndarray | None]``, one entry per angle); for v5 they are dense + ``(n_angles, n_rows, n_frames)`` arrays, since v5 geometry is uniform. """ def __init__(self, path: str): @@ -174,7 +208,7 @@ class SrasFile: self.version = version if version in (2, 3, 4, 5): self._parse_legacy() - elif version == 6: + elif version in (6, 7): self._parse_v6() else: raise ValueError(f"Unsupported version: {version}") @@ -368,11 +402,6 @@ class SrasFile: self.frame_count_mismatch = False self.n_frames_remainder = 0 - self.precomputed_freq_mhz: np.ndarray | None = None - self.precomputed_dc4_mv: np.ndarray | None = None - self.precomputed_dc3_mv: np.ndarray | None = None - self.precomputed_bg_sub: bool = False - angles = np.frombuffer(f.read(n_angles * 4), dtype=">f4").astype(np.float32) x_start = np.empty(n_angles, dtype=np.float64) @@ -409,6 +438,8 @@ class SrasFile: data_offset = f.tell() + self._data_offset = data_offset + # ---- Memory-map each angle's ragged waveform block ------------- # v6 gives each angle its own row/frame count, so waveform data is # no longer one uniform (n_angles, n_rows, ...) block — each angle's @@ -450,108 +481,157 @@ class SrasFile: self.n_rows = n_rows[:n_complete] self._y_pos_per_angle = y_pos_per_angle[:n_complete] + # Precomputed-image cache (v7 only). Ragged per-angle lists — unlike + # v5's dense (n_angles, n_rows, n_frames) arrays, v6/v7 geometry + # varies per angle, so each entry is its own (n_rows[a], n_frames[a]) + # array or None if that angle isn't cached yet. + self.precomputed_freq_mhz: list[np.ndarray | None] = [None] * n_complete + self.precomputed_dc4_mv: list[np.ndarray | None] = [None] * n_complete + self.precomputed_dc3_mv: list[np.ndarray | None] = [None] * n_complete + self.precomputed_bg_sub: bool = False + + if self.version == 7 and offset < file_size: + self._parse_cach_section(offset) + # ------------------------------------------------------------------ - # v5 writer + # v7 cache tail (CACH section: precomputed DC / FFT images) # ------------------------------------------------------------------ - def write_v5(self, dest_path: str, - freq_images: np.ndarray, - dc4_images: np.ndarray, - dc3_images: np.ndarray, - bg_sub_applied: bool, - progress_cb=None): - """Write a v5 .sras file to *dest_path*. + def _cache_tail_offset(self) -> int: + """Deterministic file offset where the CACH tail starts (or would + start), derived purely from the header + Per-Angle Geometry Table — + independent of whether a cache tail is actually present. Used by + both the parser and the in-place writer.""" + n_ch = self.n_channels + spf = self.samples_per_frame + bps = self.bytes_per_sample + waveform_bytes = int(sum( + int(self.n_rows[a]) * n_ch * int(self.n_frames[a]) * spf * bps + for a in range(self.n_angles) + )) + return self._data_offset + waveform_bytes - Copies the raw waveform bytes verbatim from the current file, - bumps the version byte to 5, patches n_frames_hdr to the actual - frame count, then appends the PREC section. + def _parse_cach_section(self, offset: int): + """Parse the v7 CACH tail that holds precomputed DC/FFT images.""" + with open(self.path, "rb") as f: + f.seek(offset) + header_raw = f.read(CACH_HDR_SIZE) + if len(header_raw) < CACH_HDR_SIZE: + return + magic, cach_version, block_flags = struct.unpack(CACH_HDR_FMT, header_raw) + if magic != CACH_MAGIC or cach_version != CACH_VERSION: + return - *freq_images* / *dc4_images* / *dc3_images*: - shape (n_angles, n_rows, n_frames) float32. + if block_flags & CACH_FLAG_DC: + sdcb_raw = f.read(SDCB_HDR_SIZE) + if len(sdcb_raw) < SDCB_HDR_SIZE: + return + sdcb_magic, _reserved, n_stored = struct.unpack(SDCB_HDR_FMT, sdcb_raw) + if sdcb_magic != SDCB_MAGIC: + return + for _ in range(n_stored): + (angle_idx,) = struct.unpack(">H", f.read(2)) + if angle_idx >= self.n_angles: + break + px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx]) + shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx])) + self.precomputed_dc3_mv[angle_idx] = np.frombuffer( + f.read(px * 4), dtype=">f4").reshape(shape) + self.precomputed_dc4_mv[angle_idx] = np.frombuffer( + f.read(px * 4), dtype=">f4").reshape(shape) - *progress_cb*: optional callable(fraction: float) for UI updates. + if block_flags & CACH_FLAG_FFT: + sfft_raw = f.read(SFFT_HDR_SIZE) + if len(sfft_raw) < SFFT_HDR_SIZE: + return + sfft_magic, flags, n_stored = struct.unpack(SFFT_HDR_FMT, sfft_raw) + if sfft_magic != SFFT_MAGIC: + return + self.precomputed_bg_sub = bool(flags & SFFT_FLAG_BG_SUB) + for _ in range(n_stored): + (angle_idx,) = struct.unpack(">H", f.read(2)) + if angle_idx >= self.n_angles: + break + px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx]) + shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx])) + self.precomputed_freq_mhz[angle_idx] = np.frombuffer( + f.read(px * 4), dtype=">f4").reshape(shape) - Only valid for v2–v5 source files, which have uniform per-angle - geometry. v6 files scan a different bounding box per angle and - cannot be losslessly represented in the flat v5 layout. + def write_v7_cache(self, *, + new_dc3_mv: list[np.ndarray | None] | None = None, + new_dc4_mv: list[np.ndarray | None] | None = None, + new_freq_mhz: list[np.ndarray | None] | None = None, + new_bg_sub: bool | None = None): + """Store computed DC and/or FFT images into this file's CACH tail, + in place, converting a v6 source to v7 (or updating an existing v7 + file). Only the block(s) passed in are recomputed; whichever block + isn't passed is carried forward unchanged from whatever this + ``SrasFile`` already has in memory (from parsing, or a prior write + in this same session) — its bytes are never re-read from disk. + + The waveform data itself is never touched: the cache tail always + starts at ``_cache_tail_offset()``, a fixed offset derived from the + header and geometry table alone. """ - if self.version == 6: - raise NotImplementedError( - "Pre-process to v5 is not supported for v6 source files " - "(per-angle geometry does not fit the flat v5 layout).") + if self.version not in (6, 7): + raise ValueError( + f"write_v7_cache only supports v6/v7 source files, got v{self.version}") - import shutil + final_dc3 = new_dc3_mv if new_dc3_mv is not None else self.precomputed_dc3_mv + final_dc4 = new_dc4_mv if new_dc4_mv is not None else self.precomputed_dc4_mv + final_freq = new_freq_mhz if new_freq_mhz is not None else self.precomputed_freq_mhz + final_bg_sub = new_bg_sub if new_bg_sub is not None else self.precomputed_bg_sub - dest = Path(dest_path) - src = self.path - n_rows0 = int(self.n_rows[0]) - n_frames0 = int(self.n_frames[0]) + dc_entries = [a for a in range(self.n_angles) if final_dc3[a] is not None] + fft_entries = [a for a in range(self.n_angles) if final_freq[a] is not None] - # --- Copy the source file verbatim, then patch the header ------- - shutil.copy2(str(src), str(dest)) + block_flags = 0 + if dc_entries: + block_flags |= CACH_FLAG_DC + if fft_entries: + block_flags |= CACH_FLAG_FFT - waveform_bytes = (self.n_angles * n_rows0 * self.n_channels - * n_frames0 * self.samples_per_frame - * self.bytes_per_sample) + payload = bytearray() + payload += struct.pack(CACH_HDR_FMT, CACH_MAGIC, CACH_VERSION, block_flags) - with open(str(dest), "r+b") as f: - # Patch version byte (offset 4 in the header struct) - f.seek(4) - f.write(struct.pack("B", 5)) + if dc_entries: + payload += struct.pack(SDCB_HDR_FMT, SDCB_MAGIC, 0, len(dc_entries)) + for a in dc_entries: + payload += struct.pack(">H", a) + payload += final_dc3[a].astype(">f4").tobytes() + payload += final_dc4[a].astype(">f4").tobytes() - # Patch n_frames_hdr (uint32, big-endian) with the actual count. - # Locate its offset: magic(4) + ver(1) + n_angles(2) + n_rows(2) = 9 - # then x_start(4)+x_delta(4)+vel(4)+freq(4) = 16, total = 25 - # then n_frames_hdr is at offset 25 as ">I" (4 bytes) - f.seek(25) - f.write(struct.pack(">I", n_frames0)) + if fft_entries: + fft_flags = SFFT_FLAG_BG_SUB if final_bg_sub else 0 + payload += struct.pack(SFFT_HDR_FMT, SFFT_MAGIC, fft_flags, len(fft_entries)) + for a in fft_entries: + payload += struct.pack(">H", a) + payload += final_freq[a].astype(">f4").tobytes() - # Truncate anything after the waveform data (e.g. old PREC tail) - # and seek to the append position. - waveform_end = self._data_offset_for_write() - f.seek(waveform_end + waveform_bytes) + cache_offset = self._cache_tail_offset() + with open(self.path, "r+b") as f: + f.seek(cache_offset) + f.write(payload) f.truncate() + f.flush() + os.fsync(f.fileno()) + # Version-byte flip last: if the process dies before this point, + # the file is still readable as plain v6 (v6 parsing only + # bounds-checks per-angle offset+nbytes <= file_size, it never + # asserts exactly how many bytes follow the last angle) — so an + # interrupted write can never corrupt the file, only leave + # harmless trailing bytes that the next successful write + # overwrites via this same deterministic cache_offset. + f.seek(4) + f.write(struct.pack("B", 7)) + f.flush() + os.fsync(f.fileno()) - # --- Write PREC section ------------------------------------- - n_stored = self.n_angles - flags = 0x01 if bg_sub_applied else 0x00 - f.write(b"PREC") - f.write(struct.pack("BB", 1, flags)) - f.write(struct.pack(">H", n_stored)) - - for aidx in range(n_stored): - if progress_cb is not None: - progress_cb(aidx / n_stored) - f.write(struct.pack(">H", aidx)) - f.write(freq_images[aidx].astype(">f4").tobytes()) - f.write(dc4_images[aidx].astype(">f4").tobytes()) - f.write(dc3_images[aidx].astype(">f4").tobytes()) - - if progress_cb is not None: - progress_cb(1.0) - - def _data_offset_for_write(self) -> int: - """Return the file offset where waveform data starts (used by write_v5).""" - # Re-derive the offset by walking the header fields, since we do not - # persist data_offset as an attribute from _parse. - with open(self.path, "rb") as f: - fields = struct.unpack(HDR_FMT, f.read(HDR_SIZE)) - ver = fields[1] - n_ch = fields[12] - n_rows0 = int(self.n_rows[0]) - - with open(self.path, "rb") as f: - f.seek(HDR_SIZE) - f.read(self.n_angles * 4) # angles - f.read(n_rows0 * 4) # y_pos - if ver >= 3: - for _ in range(n_ch): - (length,) = struct.unpack(">H", f.read(2)) - f.read(length) - if ver >= 4: - (n_bg,) = struct.unpack(">I", f.read(4)) - f.read(n_bg) - return f.tell() + self.version = 7 + self.precomputed_dc3_mv = final_dc3 + self.precomputed_dc4_mv = final_dc4 + self.precomputed_freq_mhz = final_freq + self.precomputed_bg_sub = final_bg_sub # ------------------------------------------------------------------ # Axes helpers @@ -621,6 +701,21 @@ def compute_dc_image(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray: return img +def _cached_dc_mv(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray | None: + """Return a v7-cached DC image (mV, already converted) for + (angle_idx, ch_idx) if the file's CACH section has it, else None. + + Only applies to the ragged v6/v7 list representation of + ``precomputed_dc3_mv``/``precomputed_dc4_mv`` — v5's dense PREC arrays + don't track per-angle validity (unstored angles are left at 0.0 rather + than a sentinel), so they're intentionally not consulted here. + """ + store = sras.precomputed_dc3_mv if ch_idx == CH3_IDX else sras.precomputed_dc4_mv + if isinstance(store, list) and angle_idx < len(store): + return store[angle_idx] + return None + + def compute_rf_image(sras: SrasFile, angle_idx: int, dc_threshold_mv: float, apply_bg_sub: bool = True, @@ -640,9 +735,10 @@ def compute_rf_image(sras: SrasFile, angle_idx: int, computed chunk-by-chunk internally (which does need every pixel's CH4 data, since that's what determines validity in the first place). - Fast path: if the file contains v5 precomputed peak-frequency images, - and zero-padding is not active, and the bg-sub flag matches, the stored - images are used directly — no FFT is run. + Fast path: if the file contains precomputed peak-frequency images for + this angle (v5's dense per-file PREC section, or v7's ragged per-angle + CACH section), and zero-padding is not active, and the bg-sub flag + matches, the stored image is used directly — no FFT is run. Otherwise, data is processed in row chunks sized to a fixed memory budget (see ``_chunk_rows_for``) to bound peak RAM regardless of scan @@ -652,15 +748,39 @@ def compute_rf_image(sras: SrasFile, angle_idx: int, n_frames = int(sras.n_frames[angle_idx]) data = sras.data[angle_idx] - # ---- Fast path: v5 precomputed images ---------------------------------- + # ---- Fast path: precomputed images (v5 PREC or v7 CACH) ---------------- + # v5 stores a dense (n_angles, n_rows, n_frames) ndarray (uniform + # geometry); v6/v7 store a ragged list, one entry per angle (None where + # that angle hasn't been cached), since geometry varies per angle. + freq_store = sras.precomputed_freq_mhz + if isinstance(freq_store, list): + angle_cached = angle_idx < len(freq_store) and freq_store[angle_idx] is not None + else: + angle_cached = freq_store is not None + can_use_precomputed = ( - sras.precomputed_freq_mhz is not None + angle_cached and n_fft is None # no custom zero-padding and sras.precomputed_bg_sub == (apply_bg_sub and sras.background is not None) ) if can_use_precomputed: - freq_img = sras.precomputed_freq_mhz[angle_idx].copy() - dc4_img = sras.precomputed_dc4_mv[angle_idx] + freq_img = freq_store[angle_idx].copy() + + # DC4 mask, in priority order: already-cached DC block, caller- + # supplied image, or a fresh (cheap — no FFT) recompute. + dc4_store = sras.precomputed_dc4_mv + if isinstance(dc4_store, list): + dc4_img = _cached_dc_mv(sras, angle_idx, CH4_IDX) + else: + dc4_img = dc4_store[angle_idx] if dc4_store is not None else None + if dc4_img is None: + if dc4_mv is not None: + dc4_img = dc4_mv + else: + dc4_img = adc_to_mv(compute_dc_image(sras, angle_idx, CH4_IDX), + sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX], + sras.ch_yzero_mv[CH4_IDX]) + freq_img[dc4_img < dc_threshold_mv] = 0.0 return freq_img @@ -897,7 +1017,8 @@ def _compute_angle_alignment(sras: SrasFile, ref_angle_idx: int, """Top-level alignment driver. Runs on a background thread (see AngleAlignmentWorker) — deliberately recomputes CH4 DC images from scratch via compute_dc_image rather than reading the GUI-thread - _dc_cache dict, mirroring PreprocessWorker's existing precedent.""" + _dc_cache dict, since background-thread workers must not touch + GUI-thread-owned caches (BatchCacheWorker follows the same rule).""" n = sras.n_angles # already the *complete*-angle count for aborted v6 scans dx_ref, dy_ref = _pixel_pitch_mm(sras, ref_angle_idx) @@ -1037,7 +1158,12 @@ class ComputeWorker(QObject): dc4_mv=self._dc4_mv) self.finished.emit(img) else: - # DC channels: convert ADC counts → mV + # DC channels: use the v7 cache if this angle is already + # stored (already mV), else compute fresh and convert. + cached = _cached_dc_mv(self._sras, self._angle, self._ch) + if cached is not None: + self.finished.emit(cached) + return adc_img = compute_dc_image(self._sras, self._angle, self._ch) img = adc_to_mv(adc_img, self._sras.ch_ymult_mv[self._ch], @@ -1076,6 +1202,11 @@ class DcPrecomputeWorker(QObject): for a in range(self._sras.n_angles): if self._stop: break + cached_dc3 = _cached_dc_mv(self._sras, a, CH3_IDX) + cached_dc4 = _cached_dc_mv(self._sras, a, CH4_IDX) + if cached_dc3 is not None and cached_dc4 is not None: + self.angle_done.emit(a, cached_dc3, cached_dc4) + continue dc3_mv = adc_to_mv( compute_dc_image(self._sras, a, CH3_IDX), self._sras.ch_ymult_mv[CH3_IDX], @@ -1092,69 +1223,93 @@ class DcPrecomputeWorker(QObject): self.error.emit(str(exc)) -class PreprocessWorker(QObject): - """Compute all-angle FFT and DC images and write a v5 file. +class BatchCacheWorker(QObject): + """Batch-computes and stores DC or FFT images into each of *paths*'s + v7 CACH tail, in place — converting v6 sources to v7 on first use, or + updating an existing v7 file's cache blocks without disturbing whatever + the other block already holds. - Emits ``progress(int)`` (0–100) as each angle completes and - ``finished(str)`` with an empty string on success or an error message - on failure. Only used for v2–v5 source files (uniform geometry). + *mode* is ``"dc"`` (CH3/CH4 mean images) or ``"fft"`` (CH1 peak-frequency + images, unmasked — masking is applied at display time, same as v5's + PREC convention). + + Emits ``progress(int)`` (0–100, weighted by total angle count across the + whole batch), ``file_done(str, str)`` (path, error message or "" on + success) after each file so one file's failure doesn't abort the batch, + and ``finished()`` once every file has been attempted. """ - progress = pyqtSignal(int) # 0–100 - finished = pyqtSignal(str) # empty = success, else error message + progress = pyqtSignal(int) + file_done = pyqtSignal(str, str) + finished = pyqtSignal() - def __init__(self, sras: SrasFile, dest_path: str, apply_bg_sub: bool): + def __init__(self, paths: list[str], mode: str, apply_bg_sub: bool): super().__init__() - self._sras = sras - self._dest_path = dest_path + self._paths = paths + self._mode = mode self._apply_bg_sub = apply_bg_sub def run(self): - try: - sras = self._sras - n = sras.n_angles - n_rows0 = int(sras.n_rows[0]) - n_frames0 = int(sras.n_frames[0]) - freq_images = np.empty((n, n_rows0, n_frames0), dtype=np.float32) - dc4_images = np.empty_like(freq_images) - dc3_images = np.empty_like(freq_images) + # Pass 1: quick open of each file just to weight progress by total + # angle count. Don't hold every file's memmap open at once — reopen + # fresh per file in pass 2 below. + total_angles = 0 + for path in self._paths: + try: + total_angles += SrasFile(path).n_angles + except Exception: + pass # unreadable files are reported properly in pass 2 + total_angles = max(total_angles, 1) - for aidx in range(n): - # Compute raw peak-frequency (no DC-threshold masking yet) - freq_images[aidx] = compute_rf_image( - sras, aidx, dc_threshold_mv=-1e9, # mask nothing - apply_bg_sub=self._apply_bg_sub) + done_angles = 0 + for path in self._paths: + try: + sras = SrasFile(path) + if sras.version not in (6, 7): + self.file_done.emit( + path, f"unsupported version {sras.version} — only " + "v6/v7 files can be batch-cached") + continue - # DC images (ADC counts → mV) - dc4_images[aidx] = adc_to_mv( - compute_dc_image(sras, aidx, CH4_IDX), - sras.ch_ymult_mv[CH4_IDX], - sras.ch_yoff_adc[CH4_IDX], - sras.ch_yzero_mv[CH4_IDX]) - dc3_images[aidx] = adc_to_mv( - compute_dc_image(sras, aidx, CH3_IDX), - sras.ch_ymult_mv[CH3_IDX], - sras.ch_yoff_adc[CH3_IDX], - sras.ch_yzero_mv[CH3_IDX]) + n = sras.n_angles + if self._mode == "dc": + new_dc3 = [None] * n + new_dc4 = [None] * n + for a in range(n): + new_dc3[a] = adc_to_mv( + compute_dc_image(sras, a, CH3_IDX), + sras.ch_ymult_mv[CH3_IDX], sras.ch_yoff_adc[CH3_IDX], + sras.ch_yzero_mv[CH3_IDX]) + new_dc4[a] = adc_to_mv( + compute_dc_image(sras, a, CH4_IDX), + sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX], + sras.ch_yzero_mv[CH4_IDX]) + done_angles += 1 + self.progress.emit(int(done_angles / total_angles * 100)) + sras.write_v7_cache(new_dc3_mv=new_dc3, new_dc4_mv=new_dc4) + else: + effective_bg = self._apply_bg_sub and sras.background is not None + new_freq = [None] * n + for a in range(n): + new_freq[a] = compute_rf_image( + sras, a, dc_threshold_mv=-1e9, # mask nothing + apply_bg_sub=effective_bg) + done_angles += 1 + self.progress.emit(int(done_angles / total_angles * 100)) + sras.write_v7_cache(new_freq_mhz=new_freq, new_bg_sub=effective_bg) - self.progress.emit(int((aidx + 1) / n * 90)) + self.file_done.emit(path, "") + except Exception as exc: + self.file_done.emit(path, str(exc)) - bg_sub_flag = self._apply_bg_sub and sras.background is not None - sras.write_v5( - self._dest_path, - freq_images, dc4_images, dc3_images, - bg_sub_applied=bg_sub_flag, - progress_cb=lambda frac: self.progress.emit(90 + int(frac * 10)), - ) - self.finished.emit("") - except Exception as exc: - self.finished.emit(str(exc)) + self.finished.emit() class AngleAlignmentWorker(QObject): """Computes rotation+translation alignment for every angle in *sras*, referenced to *ref_angle_idx*, from each angle's binarized CH4 mask. Rotation is analytic (from sras.angles_deg); only translation is found - by phase correlation. Mirrors PreprocessWorker's progress/finished shape. + by phase correlation. Uses the same progress(int)/finished(...) worker + shape as the other background-thread workers in this file. """ progress = pyqtSignal(int) # 0–100 finished = pyqtSignal(object, str) # AlignmentResult|None, error msg ("" = success) @@ -1794,7 +1949,11 @@ class SrasViewerWindow(QMainWindow): self._fft_pad_factor: int = 1 # 1 = no padding self._pending_fft_pad_factor: int = 1 - self._preprocess_thread: QThread | None = None + # Convert menu: batch DC/FFT compute-and-store (v6 -> v7) + self._batch_thread: QThread | None = None + self._batch_worker: BatchCacheWorker | None = None + self._batch_errors: list[str] = [] + self._batch_progress_dlg: QProgressDialog | None = None # Display-only settings (colormap, grating) never trigger a # recompute — they're applied to cached data on redraw. DC images @@ -2110,15 +2269,6 @@ class SrasViewerWindow(QMainWindow): fft_act.triggered.connect(self._on_fft_options) fft_menu.addAction(fft_act) - fft_menu.addSeparator() - self._preprocess_act = QAction("&Pre-process and Save as v5…", self) - self._preprocess_act.setStatusTip( - "Compute FFT and DC images for all angles and save to a v5 file " - "for instant re-opening (no FFT on load). Not available for v6 files.") - self._preprocess_act.setEnabled(False) - self._preprocess_act.triggered.connect(self._on_preprocess) - fft_menu.addAction(self._preprocess_act) - fusion_menu = menubar.addMenu("&Fusion") self._alignment_act = QAction("Angle &Alignment", self) self._alignment_act.setStatusTip( @@ -2128,6 +2278,21 @@ class SrasViewerWindow(QMainWindow): self._alignment_act.triggered.connect(self._on_angle_alignment) fusion_menu.addAction(self._alignment_act) + convert_menu = menubar.addMenu("&Convert") + self._batch_dc_act = QAction("Batch Compute DC and &Store…", self) + self._batch_dc_act.setStatusTip( + "Select .sras files and compute+store DC images (CH3/CH4 mean) " + "for every angle, converting v6 files to v7 in place.") + self._batch_dc_act.triggered.connect(lambda: self._on_batch_compute("dc")) + convert_menu.addAction(self._batch_dc_act) + + self._batch_fft_act = QAction("Batch Compute FFT and Sto&re…", self) + self._batch_fft_act.setStatusTip( + "Select .sras files and compute+store FFT peak-frequency images " + "for every angle, converting v6 files to v7 in place.") + self._batch_fft_act.triggered.connect(lambda: self._on_batch_compute("fft")) + convert_menu.addAction(self._batch_fft_act) + # ------------------------------------------------------------------ # Drag-and-drop # ------------------------------------------------------------------ @@ -2265,11 +2430,22 @@ class SrasViewerWindow(QMainWindow): ) if s.background is not None: notes.append(f"Background waveform: {len(s.background)} samples") - if s.precomputed_freq_mhz is not None: + if isinstance(s.precomputed_freq_mhz, np.ndarray): bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)" notes.append(f"v5: precomputed images present{bg_note} — display is instant") - if s.version == 6: - notes.append("v6 format: rows / frames / x_start are per-angle") + if s.version in (6, 7): + notes.append("v6/v7 format: rows / frames / x_start are per-angle") + if s.version == 7: + n_dc = sum(1 for x in s.precomputed_dc4_mv if x is not None) + n_fft = sum(1 for x in s.precomputed_freq_mhz if x is not None) + if n_dc or n_fft: + bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)" + notes.append( + f"v7: cached DC ({n_dc}/{s.n_angles} angles), " + f"FFT ({n_fft}/{s.n_angles} angles{bg_note if n_fft else ''}) " + "— display is instant for cached angles") + else: + notes.append("v7 format: no cache blocks stored yet") self.lbl_frame_warn.setText("\n".join(notes)) # ------------------------------------------------------------------ @@ -2299,10 +2475,11 @@ class SrasViewerWindow(QMainWindow): self.btn_export_csv.setEnabled(is_ch1 and self._current_image is not None) # ROI: always usable once a file is loaded (independent of channel) self.btn_draw_roi.setEnabled(enabled and s is not None) - # Pre-process: available for v2–v5 files when loaded and not already running - can_preprocess = (enabled and s is not None and s.version != 6 - and self._preprocess_thread is None) - self._preprocess_act.setEnabled(can_preprocess) + # Batch Convert actions: pick their own files, independent of + # whatever's currently open — only gated on no batch already running + can_batch = self._batch_thread is None + self._batch_dc_act.setEnabled(can_batch) + self._batch_fft_act.setEnabled(can_batch) # Angle alignment: needs >1 angle and no alignment already running can_align = (enabled and s is not None and s.n_angles > 1 and self._alignment_thread is None) @@ -2897,75 +3074,90 @@ class SrasViewerWindow(QMainWindow): self._progress_dlg = None # ------------------------------------------------------------------ - # Pre-process → save v5 + # Convert menu: batch DC/FFT compute-and-store (v6 -> v7) # ------------------------------------------------------------------ - def _on_preprocess(self): - if self._sras is None: + def _on_batch_compute(self, mode: str): + if self._batch_thread is not None: return - if self._preprocess_thread is not None: - return - s = self._sras - if s.version == 6: - self.statusBar().showMessage( - "Pre-process to v5 is not supported for v6 files (per-angle geometry).") - return - default_name = s.path.stem + "_v5" + s.path.suffix - dest, _ = QFileDialog.getSaveFileName( - self, "Save Pre-processed v5 File", - str(s.path.parent / default_name), + label = "DC" if mode == "dc" else "FFT" + paths, _ = QFileDialog.getOpenFileNames( + self, f"Select .sras files to batch-compute {label}", "", "SRAS files (*.sras);;All files (*)", ) - if not dest: + if not paths: return - if self._preprocess_thread is not None: - return # a second trigger snuck in while the save dialog was open + if self._batch_thread is not None: + return # a second trigger snuck in while the file dialog was open - apply_bg = self.chk_bg_sub.isChecked() and s.background is not None + apply_bg = self.chk_bg_sub.isChecked() + self._batch_errors = [] - n_total = s.n_angles - n_px = int(s.n_rows[0]) * int(s.n_frames[0]) - approx_mb = n_total * n_px * 3 * 4 / 1e6 + # Claim self._batch_thread before anything below that can pump the + # Qt event loop — see the comment in _start_compute for why. + self._batch_worker = BatchCacheWorker(paths, mode, apply_bg) + self._batch_thread = QThread() + self._batch_worker.moveToThread(self._batch_thread) + self._batch_thread.started.connect(self._batch_worker.run) + self._batch_worker.progress.connect(self._on_batch_progress) + self._batch_worker.file_done.connect(self._on_batch_file_done) + self._batch_worker.finished.connect(lambda paths=paths: self._on_batch_finished(paths)) + self._batch_worker.finished.connect(self._batch_thread.quit) + self._batch_thread.finished.connect(self._on_batch_thread_finished) - # Claim self._preprocess_thread before any call below that can pump - # the Qt event loop — see the comment in _start_compute for why. - self._preprocess_worker = PreprocessWorker(s, dest, apply_bg) - self._preprocess_thread = QThread() - self._preprocess_worker.moveToThread(self._preprocess_thread) - self._preprocess_thread.started.connect(self._preprocess_worker.run) - self._preprocess_worker.progress.connect(self._on_preprocess_progress) - self._preprocess_worker.finished.connect(self._on_preprocess_done) - self._preprocess_worker.finished.connect(self._preprocess_thread.quit) - self._preprocess_thread.finished.connect(self._on_preprocess_thread_finished) + self._batch_dc_act.setEnabled(False) + self._batch_fft_act.setEnabled(False) - self._preprocess_act.setEnabled(False) - self._show_progress( - f"Pre-processing {n_total} angle(s) " - f"({int(s.n_rows[0])}×{int(s.n_frames[0])} px each, ~{approx_mb:.0f} MB output)…" - ) + self._batch_progress_dlg = QProgressDialog( + f"Batch computing {label} for {len(paths)} file(s)…", "", 0, 100, self) + self._batch_progress_dlg.setWindowTitle("Please wait…") + self._batch_progress_dlg.setCancelButton(None) + self._batch_progress_dlg.setWindowModality(Qt.WindowModality.WindowModal) + self._batch_progress_dlg.setMinimumDuration(300) + self._batch_progress_dlg.show() - self._preprocess_thread.start() + self._batch_thread.start() - def _on_preprocess_progress(self, pct: int): - if self._progress_dlg is not None: - self._progress_dlg.setValue(pct) + def _on_batch_progress(self, pct: int): + if self._batch_progress_dlg is not None: + self._batch_progress_dlg.setValue(pct) - def _on_preprocess_done(self, error_msg: str): - self._close_progress() - if error_msg: - self.statusBar().showMessage(f"Pre-process failed: {error_msg}") + def _on_batch_file_done(self, path: str, err: str): + if err: + self._batch_errors.append(f"{Path(path).name} — {err}") + if self._batch_progress_dlg is not None: + self._batch_progress_dlg.setLabelText(f"Processed {Path(path).name}…") + + def _on_batch_finished(self, paths: list[str]): + if self._batch_progress_dlg is not None: + self._batch_progress_dlg.close() + self._batch_progress_dlg = None + + n_total = len(paths) + n_failed = len(self._batch_errors) + n_ok = n_total - n_failed + if n_failed: + summary = (f"Batch store: {n_ok}/{n_total} file(s) updated, " + f"{n_failed} failed: {'; '.join(self._batch_errors)}") else: - self.statusBar().showMessage("v5 file written — re-open it for instant display.") + summary = f"Batch store: {n_ok}/{n_total} file(s) updated." + self.statusBar().showMessage(summary) + self._batch_errors = [] - def _on_preprocess_thread_finished(self): + # If the currently-open file was in this batch, reload it so the + # GUI picks up the newly-written v7 cache instead of stale state. + if self._sras is not None and str(self._sras.path) in paths: + self._load_file(str(self._sras.path)) + + def _on_batch_thread_finished(self): # See the comment in _on_compute_thread_finished: wait() before # releasing our reference to avoid destroying a QThread whose OS # thread hasn't fully joined yet. - if self._preprocess_thread is not None: - self._preprocess_thread.wait() - self._preprocess_thread = None - self._preprocess_act.setEnabled( - self._sras is not None and self._sras.version != 6) + if self._batch_thread is not None: + self._batch_thread.wait() + self._batch_thread = None + self._batch_dc_act.setEnabled(True) + self._batch_fft_act.setEnabled(True) # ------------------------------------------------------------------ # Fusion: angle alignment @@ -3066,7 +3258,7 @@ class SrasViewerWindow(QMainWindow): def closeEvent(self, event): if self._dc_precompute_worker is not None: self._dc_precompute_worker.stop() - for attr in ("_load_thread", "_compute_thread", "_preprocess_thread", + for attr in ("_load_thread", "_compute_thread", "_batch_thread", "_dc_precompute_thread", "_alignment_thread"): t = getattr(self, attr, None) if t is not None: