tka/sras-viewer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor/parallel and dedup #1

Merged
tka merged 10 commits from refactor/parallel-and-dedup into main 2026-07-31 15:20:33 -05:00
Conversation 0 Commits 10 Files Changed 11
+530 -223
2 changed files with 530 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit ed0eba4bae - Show all commits
scan_format.md
+117 -2
View File
@@ -1,4 +1,4 @@
# SRAS Scan Binary Format — Version 6
# SRAS Scan Binary Format — Version 6 / 7
Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows. Files are named `{prefix}.sras`.
@@ -10,6 +10,14 @@ rotated by that specific angle** — not the worst case across all angles — so
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.
v7 is byte-identical to v6 (same header, angle table, geometry table, row
table, preamble blocks, background block, waveform data) plus an optional
trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images
(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version`
field and the presence of that trailing section differ. Scans come off the
scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file
to v7 **in place** the first time either cache block is computed and stored.
---
## File Layout
@@ -22,6 +30,7 @@ instead of forcing every angle to the largest bounding box.
[Preamble Blocks — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged) — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
[Cache Tail (optional) — "CACH" + DC block (optional) + FFT block (optional); v7 only]
```
All multi-byte integers and floats use **big-endian** byte order
@@ -182,6 +191,111 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not
---
## Cache Tail (v7)
Present iff `version == 7` and `file_size > cache_offset`, where:
```
cache_offset = data_offset + Σ over angles a of:
n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample
```
i.e. exactly `data_offset + waveform_bytes` — the same "Total data size"
formula as Waveform Data above. This offset is derivable from the header and
Per-Angle Geometry Table alone and does **not** depend on which cache
block(s) are present, so a writer can always seek straight there without
reading or touching any waveform byte before it.
Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle
entry (FFT + both DC channels together) in a dense, uniform-geometry array,
the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks —
each sized per-angle from the Per-Angle Geometry Table, each independently
present, and each independently updatable in any order, any number of
times, without disturbing the other. This matches `sras_viewer.py`'s
"Convert" menu, which exposes DC and FFT store as two separate batch
actions.
### CACH outer header (6 bytes, `">4sBB"`)
| Offset | Size | Type | Field | Description |
|--------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `1`. Readers must treat the file as uncached if this is not a version they understand (unlike v5's `PREC` section, which read but never validated its version byte). |
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |
### DC block `SDCB` (present iff `block_flags & 0x01`)
7-byte block header, format `">4sBH"`:
| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SDCB` |
| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` |
followed by `n_stored` entries, each:
```
u16 angle_idx — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]] dc3_mv — CH3 waveform mean, mV, row-major
f32[n_rows[angle_idx] × n_frames[angle_idx]] dc4_mv — CH4 waveform mean, mV, row-major
```
Entries may appear in any order and need not be contiguous from angle 0 —
this supports storing (or re-storing) a subset of angles, or an
interrupted batch run leaving only some angles cached. Readers bounds-check
`angle_idx < n_angles` on each entry and stop parsing on an out-of-range
value, same as v5's `PREC` section.
### FFT block `SFFT` (present iff `block_flags & 0x02`)
7-byte block header, format `">4sBH"`:
| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bits 1–7 reserved. |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
followed by `n_stored` entries, each:
```
u16 angle_idx — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]] peak_freq_mhz — CH1 FFT peak frequency, MHz, row-major
```
**`peak_freq_mhz`** is computed without any DC-threshold masking (i.e. the
FFT is run on every pixel unconditionally, same as v5's `PREC` convention).
Readers apply the DC4 threshold at display time:
```
pixel is valid ⟺ dc4_mv[r][f] ≥ threshold_mv
display_value = peak_freq_mhz[r][f] if valid, else 0
```
using the DC4 image from the DC block if that angle is also cached there,
else computed on demand.
Readers must fall back to real-time FFT computation (ignoring stored
`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested, or
the reader's background-subtraction setting doesn't match `flags.bg_sub_applied`.
### In-place write ordering
A writer updating a file's Cache Tail must write the payload (CACH header +
whichever block(s) are present) **before** flipping the header's `version`
byte to `7`, and `truncate()` the file to the new payload's end immediately
after writing it. If the process is interrupted between the payload write
and the version-byte flip, the file is still valid v6 — v6 parsing only
bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts
exactly how many bytes follow the last angle's waveform block — so the
interrupted write leaves harmless trailing bytes rather than a corrupt file,
and the next successful write overwrites them via the same deterministic
`cache_offset`.
---
## Acquisition Settings (fixed by sc3_aui_app.py)
| Parameter | Value |
@@ -206,5 +320,6 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not
| 3 | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4 | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5 | (skipped) |
| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`); the "Pre-process and Save as v5" fast-path export is not offered for v6 files since the flat v5 layout cannot represent per-angle geometry. |
| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). |
| 7 | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. |
sras_viewer.py
+412 -220
View File
@@ -15,11 +15,16 @@ header before acquisition, but the scope may acquire fewer frames. The actual
count is computed from the file size and used for the reshape so channels are
correctly aligned.
Scan geometry: v6 files scan a different bounding box per angle (x_start,
Scan geometry: v6/v7 files scan a different bounding box per angle (x_start,
x_delta, n_frames, n_rows all vary by angle), so geometry is exposed per-angle
via SrasFile.n_rows / n_frames / x_start_mm arrays and the x_axis_mm() /
y_positions_mm() methods. v2–v5 files have uniform geometry across angles, so
those arrays simply repeat the same value n_angles times.
v7 files are v6 files with an optional trailing cache section holding
precomputed per-angle DC and/or FFT images (see scan_format.md), so display
never has to recompute them after the "Convert" menu's batch actions have
stored them once.
"""
import re
@@ -90,6 +95,29 @@ HDR_SIZE_V6 = struct.calcsize(HDR_FMT_V6) # 49 bytes
GEO_FMT_V6 = ">ffIH"
GEO_SIZE_V6 = struct.calcsize(GEO_FMT_V6) # 14 bytes
# v7: identical to v6 (same header/geometry/waveform layout, version byte
# is the only header difference) plus an optional trailing cache section
# ("CACH") holding precomputed per-angle DC and/or FFT images so a v7 file
# never needs to recompute them on open. See scan_format.md for the full
# spec. Sub-block sizes are per-angle (n_rows[a] * n_frames[a]), taken from
# the Per-Angle Geometry Table already parsed for v6 — no new geometry
# fields are needed.
CACH_MAGIC = b"CACH"
CACH_HDR_FMT = ">4sBB" # magic, cach_version, block_flags
CACH_HDR_SIZE = struct.calcsize(CACH_HDR_FMT) # 6 bytes
CACH_VERSION = 1
CACH_FLAG_DC = 0x01
CACH_FLAG_FFT = 0x02
SDCB_MAGIC = b"SDCB"
SDCB_HDR_FMT = ">4sBH" # magic, reserved, n_stored
SDCB_HDR_SIZE = struct.calcsize(SDCB_HDR_FMT) # 7 bytes
SFFT_MAGIC = b"SFFT"
SFFT_HDR_FMT = ">4sBH" # magic, flags, n_stored
SFFT_HDR_SIZE = struct.calcsize(SFFT_HDR_FMT) # 7 bytes
SFFT_FLAG_BG_SUB = 0x01
# Fixed-order channels in the file: index 0=CH1, 1=CH3, 2=CH4
# Fixed channel indices into the .sras data array (CH1=RF, CH3/CH4=Bias DC)
CH1_IDX, CH3_IDX, CH4_IDX = 0, 1, 2
@@ -149,15 +177,21 @@ def adc_to_mv(adc: float, ymult_mv: float = _FALLBACK_YMULT_MV,
# ---------------------------------------------------------------------------
class SrasFile:
"""Parsed in-memory representation of a v2–v6 .sras file.
"""Parsed in-memory representation of a v2–v7 .sras file.
Scan geometry (rows, frames, x_start) is exposed per-angle via the
``n_rows`` / ``n_frames`` / ``x_start_mm`` arrays and the ``x_axis_mm()``
/ ``y_positions_mm()`` methods, since v6 files scan a different bounding
box per angle. v2–v5 files have uniform geometry, so these arrays just
repeat the same value ``n_angles`` times. Waveform data is likewise
exposed as ``data[angle_idx]``, an array of shape
/ ``y_positions_mm()`` methods, since v6/v7 files scan a different
bounding box per angle. v2–v5 files have uniform geometry, so these
arrays just repeat the same value ``n_angles`` times. Waveform data is
likewise exposed as ``data[angle_idx]``, an array of shape
``(n_rows[a], n_channels, n_frames[a], samples_per_frame)``.
v7 files (v6 plus an optional trailing cache section) expose any stored
precomputed images as ``precomputed_dc3_mv`` / ``precomputed_dc4_mv`` /
``precomputed_freq_mhz`` — for v6/v7 these are ragged per-angle lists
(``list[np.ndarray | None]``, one entry per angle); for v5 they are dense
``(n_angles, n_rows, n_frames)`` arrays, since v5 geometry is uniform.
"""
def __init__(self, path: str):
@@ -174,7 +208,7 @@ class SrasFile:
self.version = version
if version in (2, 3, 4, 5):
self._parse_legacy()
elif version == 6:
elif version in (6, 7):
self._parse_v6()
else:
raise ValueError(f"Unsupported version: {version}")
@@ -368,11 +402,6 @@ class SrasFile:
self.frame_count_mismatch = False
self.n_frames_remainder = 0
self.precomputed_freq_mhz: np.ndarray | None = None
self.precomputed_dc4_mv: np.ndarray | None = None
self.precomputed_dc3_mv: np.ndarray | None = None
self.precomputed_bg_sub: bool = False
angles = np.frombuffer(f.read(n_angles * 4), dtype=">f4").astype(np.float32)
x_start = np.empty(n_angles, dtype=np.float64)
@@ -409,6 +438,8 @@ class SrasFile:
data_offset = f.tell()
self._data_offset = data_offset
# ---- Memory-map each angle's ragged waveform block -------------
# v6 gives each angle its own row/frame count, so waveform data is
# no longer one uniform (n_angles, n_rows, ...) block — each angle's
@@ -450,108 +481,157 @@ class SrasFile:
self.n_rows = n_rows[:n_complete]
self._y_pos_per_angle = y_pos_per_angle[:n_complete]
# Precomputed-image cache (v7 only). Ragged per-angle lists — unlike
# v5's dense (n_angles, n_rows, n_frames) arrays, v6/v7 geometry
# varies per angle, so each entry is its own (n_rows[a], n_frames[a])
# array or None if that angle isn't cached yet.
self.precomputed_freq_mhz: list[np.ndarray | None] = [None] * n_complete
self.precomputed_dc4_mv: list[np.ndarray | None] = [None] * n_complete
self.precomputed_dc3_mv: list[np.ndarray | None] = [None] * n_complete
self.precomputed_bg_sub: bool = False
if self.version == 7 and offset < file_size:
self._parse_cach_section(offset)
# ------------------------------------------------------------------
# v5 writer
# v7 cache tail (CACH section: precomputed DC / FFT images)
# ------------------------------------------------------------------
def write_v5(self, dest_path: str,
freq_images: np.ndarray,
dc4_images: np.ndarray,
dc3_images: np.ndarray,
bg_sub_applied: bool,
progress_cb=None):
"""Write a v5 .sras file to *dest_path*.
def _cache_tail_offset(self) -> int:
"""Deterministic file offset where the CACH tail starts (or would
start), derived purely from the header + Per-Angle Geometry Table —
independent of whether a cache tail is actually present. Used by
both the parser and the in-place writer."""
n_ch = self.n_channels
spf = self.samples_per_frame
bps = self.bytes_per_sample
waveform_bytes = int(sum(
int(self.n_rows[a]) * n_ch * int(self.n_frames[a]) * spf * bps
for a in range(self.n_angles)
))
return self._data_offset + waveform_bytes
Copies the raw waveform bytes verbatim from the current file,
bumps the version byte to 5, patches n_frames_hdr to the actual
frame count, then appends the PREC section.
def _parse_cach_section(self, offset: int):
"""Parse the v7 CACH tail that holds precomputed DC/FFT images."""
with open(self.path, "rb") as f:
f.seek(offset)
header_raw = f.read(CACH_HDR_SIZE)
if len(header_raw) < CACH_HDR_SIZE:
return
magic, cach_version, block_flags = struct.unpack(CACH_HDR_FMT, header_raw)
if magic != CACH_MAGIC or cach_version != CACH_VERSION:
return
*freq_images* / *dc4_images* / *dc3_images*:
shape (n_angles, n_rows, n_frames) float32.
if block_flags & CACH_FLAG_DC:
sdcb_raw = f.read(SDCB_HDR_SIZE)
if len(sdcb_raw) < SDCB_HDR_SIZE:
return
sdcb_magic, _reserved, n_stored = struct.unpack(SDCB_HDR_FMT, sdcb_raw)
if sdcb_magic != SDCB_MAGIC:
return
for _ in range(n_stored):
(angle_idx,) = struct.unpack(">H", f.read(2))
if angle_idx >= self.n_angles:
break
px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx])
shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx]))
self.precomputed_dc3_mv[angle_idx] = np.frombuffer(
f.read(px * 4), dtype=">f4").reshape(shape)
self.precomputed_dc4_mv[angle_idx] = np.frombuffer(
f.read(px * 4), dtype=">f4").reshape(shape)
*progress_cb*: optional callable(fraction: float) for UI updates.
if block_flags & CACH_FLAG_FFT:
sfft_raw = f.read(SFFT_HDR_SIZE)
if len(sfft_raw) < SFFT_HDR_SIZE:
return
sfft_magic, flags, n_stored = struct.unpack(SFFT_HDR_FMT, sfft_raw)
if sfft_magic != SFFT_MAGIC:
return
self.precomputed_bg_sub = bool(flags & SFFT_FLAG_BG_SUB)
for _ in range(n_stored):
(angle_idx,) = struct.unpack(">H", f.read(2))
if angle_idx >= self.n_angles:
break
px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx])
shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx]))
self.precomputed_freq_mhz[angle_idx] = np.frombuffer(
f.read(px * 4), dtype=">f4").reshape(shape)
Only valid for v2–v5 source files, which have uniform per-angle
geometry. v6 files scan a different bounding box per angle and
cannot be losslessly represented in the flat v5 layout.
def write_v7_cache(self, *,
new_dc3_mv: list[np.ndarray | None] | None = None,
new_dc4_mv: list[np.ndarray | None] | None = None,
new_freq_mhz: list[np.ndarray | None] | None = None,
new_bg_sub: bool | None = None):
"""Store computed DC and/or FFT images into this file's CACH tail,
in place, converting a v6 source to v7 (or updating an existing v7
file). Only the block(s) passed in are recomputed; whichever block
isn't passed is carried forward unchanged from whatever this
``SrasFile`` already has in memory (from parsing, or a prior write
in this same session) — its bytes are never re-read from disk.
The waveform data itself is never touched: the cache tail always
starts at ``_cache_tail_offset()``, a fixed offset derived from the
header and geometry table alone.
"""
if self.version == 6:
raise NotImplementedError(
"Pre-process to v5 is not supported for v6 source files "
"(per-angle geometry does not fit the flat v5 layout).")
if self.version not in (6, 7):
raise ValueError(
f"write_v7_cache only supports v6/v7 source files, got v{self.version}")
import shutil
final_dc3 = new_dc3_mv if new_dc3_mv is not None else self.precomputed_dc3_mv
final_dc4 = new_dc4_mv if new_dc4_mv is not None else self.precomputed_dc4_mv
final_freq = new_freq_mhz if new_freq_mhz is not None else self.precomputed_freq_mhz
final_bg_sub = new_bg_sub if new_bg_sub is not None else self.precomputed_bg_sub
dest = Path(dest_path)
src = self.path
n_rows0 = int(self.n_rows[0])
n_frames0 = int(self.n_frames[0])
dc_entries = [a for a in range(self.n_angles) if final_dc3[a] is not None]
fft_entries = [a for a in range(self.n_angles) if final_freq[a] is not None]
# --- Copy the source file verbatim, then patch the header -------
shutil.copy2(str(src), str(dest))
block_flags = 0
if dc_entries:
block_flags |= CACH_FLAG_DC
if fft_entries:
block_flags |= CACH_FLAG_FFT
waveform_bytes = (self.n_angles * n_rows0 * self.n_channels
* n_frames0 * self.samples_per_frame
* self.bytes_per_sample)
payload = bytearray()
payload += struct.pack(CACH_HDR_FMT, CACH_MAGIC, CACH_VERSION, block_flags)
with open(str(dest), "r+b") as f:
# Patch version byte (offset 4 in the header struct)
f.seek(4)
f.write(struct.pack("B", 5))
if dc_entries:
payload += struct.pack(SDCB_HDR_FMT, SDCB_MAGIC, 0, len(dc_entries))
for a in dc_entries:
payload += struct.pack(">H", a)
payload += final_dc3[a].astype(">f4").tobytes()
payload += final_dc4[a].astype(">f4").tobytes()
# Patch n_frames_hdr (uint32, big-endian) with the actual count.
# Locate its offset: magic(4) + ver(1) + n_angles(2) + n_rows(2) = 9
# then x_start(4)+x_delta(4)+vel(4)+freq(4) = 16, total = 25
# then n_frames_hdr is at offset 25 as ">I" (4 bytes)
f.seek(25)
f.write(struct.pack(">I", n_frames0))
if fft_entries:
fft_flags = SFFT_FLAG_BG_SUB if final_bg_sub else 0
payload += struct.pack(SFFT_HDR_FMT, SFFT_MAGIC, fft_flags, len(fft_entries))
for a in fft_entries:
payload += struct.pack(">H", a)
payload += final_freq[a].astype(">f4").tobytes()
# Truncate anything after the waveform data (e.g. old PREC tail)
# and seek to the append position.
waveform_end = self._data_offset_for_write()
f.seek(waveform_end + waveform_bytes)
cache_offset = self._cache_tail_offset()
with open(self.path, "r+b") as f:
f.seek(cache_offset)
f.write(payload)
f.truncate()
f.flush()
os.fsync(f.fileno())
# Version-byte flip last: if the process dies before this point,
# the file is still readable as plain v6 (v6 parsing only
# bounds-checks per-angle offset+nbytes <= file_size, it never
# asserts exactly how many bytes follow the last angle) — so an
# interrupted write can never corrupt the file, only leave
# harmless trailing bytes that the next successful write
# overwrites via this same deterministic cache_offset.
f.seek(4)
f.write(struct.pack("B", 7))
f.flush()
os.fsync(f.fileno())
# --- Write PREC section -------------------------------------
n_stored = self.n_angles
flags = 0x01 if bg_sub_applied else 0x00
f.write(b"PREC")
f.write(struct.pack("BB", 1, flags))
f.write(struct.pack(">H", n_stored))
for aidx in range(n_stored):
if progress_cb is not None:
progress_cb(aidx / n_stored)
f.write(struct.pack(">H", aidx))
f.write(freq_images[aidx].astype(">f4").tobytes())
f.write(dc4_images[aidx].astype(">f4").tobytes())
f.write(dc3_images[aidx].astype(">f4").tobytes())
if progress_cb is not None:
progress_cb(1.0)
def _data_offset_for_write(self) -> int:
"""Return the file offset where waveform data starts (used by write_v5)."""
# Re-derive the offset by walking the header fields, since we do not
# persist data_offset as an attribute from _parse.
with open(self.path, "rb") as f:
fields = struct.unpack(HDR_FMT, f.read(HDR_SIZE))
ver = fields[1]
n_ch = fields[12]
n_rows0 = int(self.n_rows[0])
with open(self.path, "rb") as f:
f.seek(HDR_SIZE)
f.read(self.n_angles * 4) # angles
f.read(n_rows0 * 4) # y_pos
if ver >= 3:
for _ in range(n_ch):
(length,) = struct.unpack(">H", f.read(2))
f.read(length)
if ver >= 4:
(n_bg,) = struct.unpack(">I", f.read(4))
f.read(n_bg)
return f.tell()
self.version = 7
self.precomputed_dc3_mv = final_dc3
self.precomputed_dc4_mv = final_dc4
self.precomputed_freq_mhz = final_freq
self.precomputed_bg_sub = final_bg_sub
# ------------------------------------------------------------------
# Axes helpers
@@ -621,6 +701,21 @@ def compute_dc_image(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray:
return img
def _cached_dc_mv(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray | None:
"""Return a v7-cached DC image (mV, already converted) for
(angle_idx, ch_idx) if the file's CACH section has it, else None.
Only applies to the ragged v6/v7 list representation of
``precomputed_dc3_mv``/``precomputed_dc4_mv`` — v5's dense PREC arrays
don't track per-angle validity (unstored angles are left at 0.0 rather
than a sentinel), so they're intentionally not consulted here.
"""
store = sras.precomputed_dc3_mv if ch_idx == CH3_IDX else sras.precomputed_dc4_mv
if isinstance(store, list) and angle_idx < len(store):
return store[angle_idx]
return None
def compute_rf_image(sras: SrasFile, angle_idx: int,
dc_threshold_mv: float,
apply_bg_sub: bool = True,
@@ -640,9 +735,10 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
computed chunk-by-chunk internally (which does need every pixel's
CH4 data, since that's what determines validity in the first place).
Fast path: if the file contains v5 precomputed peak-frequency images,
and zero-padding is not active, and the bg-sub flag matches, the stored
images are used directly — no FFT is run.
Fast path: if the file contains precomputed peak-frequency images for
this angle (v5's dense per-file PREC section, or v7's ragged per-angle
CACH section), and zero-padding is not active, and the bg-sub flag
matches, the stored image is used directly — no FFT is run.
Otherwise, data is processed in row chunks sized to a fixed memory
budget (see ``_chunk_rows_for``) to bound peak RAM regardless of scan
@@ -652,15 +748,39 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
n_frames = int(sras.n_frames[angle_idx])
data = sras.data[angle_idx]
# ---- Fast path: v5 precomputed images ----------------------------------
# ---- Fast path: precomputed images (v5 PREC or v7 CACH) ----------------
# v5 stores a dense (n_angles, n_rows, n_frames) ndarray (uniform
# geometry); v6/v7 store a ragged list, one entry per angle (None where
# that angle hasn't been cached), since geometry varies per angle.
freq_store = sras.precomputed_freq_mhz
if isinstance(freq_store, list):
angle_cached = angle_idx < len(freq_store) and freq_store[angle_idx] is not None
else:
angle_cached = freq_store is not None
can_use_precomputed = (
sras.precomputed_freq_mhz is not None
angle_cached
and n_fft is None # no custom zero-padding
and sras.precomputed_bg_sub == (apply_bg_sub and sras.background is not None)
)
if can_use_precomputed:
freq_img = sras.precomputed_freq_mhz[angle_idx].copy()
dc4_img = sras.precomputed_dc4_mv[angle_idx]
freq_img = freq_store[angle_idx].copy()
# DC4 mask, in priority order: already-cached DC block, caller-
# supplied image, or a fresh (cheap — no FFT) recompute.
dc4_store = sras.precomputed_dc4_mv
if isinstance(dc4_store, list):
dc4_img = _cached_dc_mv(sras, angle_idx, CH4_IDX)
else:
dc4_img = dc4_store[angle_idx] if dc4_store is not None else None
if dc4_img is None:
if dc4_mv is not None:
dc4_img = dc4_mv
else:
dc4_img = adc_to_mv(compute_dc_image(sras, angle_idx, CH4_IDX),
sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX],
sras.ch_yzero_mv[CH4_IDX])
freq_img[dc4_img < dc_threshold_mv] = 0.0
return freq_img
@@ -897,7 +1017,8 @@ def _compute_angle_alignment(sras: SrasFile, ref_angle_idx: int,
"""Top-level alignment driver. Runs on a background thread (see
AngleAlignmentWorker) — deliberately recomputes CH4 DC images from
scratch via compute_dc_image rather than reading the GUI-thread
_dc_cache dict, mirroring PreprocessWorker's existing precedent."""
_dc_cache dict, since background-thread workers must not touch
GUI-thread-owned caches (BatchCacheWorker follows the same rule)."""
n = sras.n_angles # already the *complete*-angle count for aborted v6 scans
dx_ref, dy_ref = _pixel_pitch_mm(sras, ref_angle_idx)
@@ -1037,7 +1158,12 @@ class ComputeWorker(QObject):
dc4_mv=self._dc4_mv)
self.finished.emit(img)
else:
# DC channels: convert ADC counts → mV
# DC channels: use the v7 cache if this angle is already
# stored (already mV), else compute fresh and convert.
cached = _cached_dc_mv(self._sras, self._angle, self._ch)
if cached is not None:
self.finished.emit(cached)
return
adc_img = compute_dc_image(self._sras, self._angle, self._ch)
img = adc_to_mv(adc_img,
self._sras.ch_ymult_mv[self._ch],
@@ -1076,6 +1202,11 @@ class DcPrecomputeWorker(QObject):
for a in range(self._sras.n_angles):
if self._stop:
break
cached_dc3 = _cached_dc_mv(self._sras, a, CH3_IDX)
cached_dc4 = _cached_dc_mv(self._sras, a, CH4_IDX)
if cached_dc3 is not None and cached_dc4 is not None:
self.angle_done.emit(a, cached_dc3, cached_dc4)
continue
dc3_mv = adc_to_mv(
compute_dc_image(self._sras, a, CH3_IDX),
self._sras.ch_ymult_mv[CH3_IDX],
@@ -1092,69 +1223,93 @@ class DcPrecomputeWorker(QObject):
self.error.emit(str(exc))
class PreprocessWorker(QObject):
"""Compute all-angle FFT and DC images and write a v5 file.
class BatchCacheWorker(QObject):
"""Batch-computes and stores DC or FFT images into each of *paths*'s
v7 CACH tail, in place — converting v6 sources to v7 on first use, or
updating an existing v7 file's cache blocks without disturbing whatever
the other block already holds.
Emits ``progress(int)`` (0–100) as each angle completes and
``finished(str)`` with an empty string on success or an error message
on failure. Only used for v2–v5 source files (uniform geometry).
*mode* is ``"dc"`` (CH3/CH4 mean images) or ``"fft"`` (CH1 peak-frequency
images, unmasked — masking is applied at display time, same as v5's
PREC convention).
Emits ``progress(int)`` (0–100, weighted by total angle count across the
whole batch), ``file_done(str, str)`` (path, error message or "" on
success) after each file so one file's failure doesn't abort the batch,
and ``finished()`` once every file has been attempted.
"""
progress = pyqtSignal(int) # 0–100
finished = pyqtSignal(str) # empty = success, else error message
progress = pyqtSignal(int)
file_done = pyqtSignal(str, str)
finished = pyqtSignal()
def __init__(self, sras: SrasFile, dest_path: str, apply_bg_sub: bool):
def __init__(self, paths: list[str], mode: str, apply_bg_sub: bool):
super().__init__()
self._sras = sras
self._dest_path = dest_path
self._paths = paths
self._mode = mode
self._apply_bg_sub = apply_bg_sub
def run(self):
# Pass 1: quick open of each file just to weight progress by total
# angle count. Don't hold every file's memmap open at once — reopen
# fresh per file in pass 2 below.
total_angles = 0
for path in self._paths:
try:
sras = self._sras
total_angles += SrasFile(path).n_angles
except Exception:
pass # unreadable files are reported properly in pass 2
total_angles = max(total_angles, 1)
done_angles = 0
for path in self._paths:
try:
sras = SrasFile(path)
if sras.version not in (6, 7):
self.file_done.emit(
path, f"unsupported version {sras.version} — only "
"v6/v7 files can be batch-cached")
continue
n = sras.n_angles
n_rows0 = int(sras.n_rows[0])
n_frames0 = int(sras.n_frames[0])
freq_images = np.empty((n, n_rows0, n_frames0), dtype=np.float32)
dc4_images = np.empty_like(freq_images)
dc3_images = np.empty_like(freq_images)
for aidx in range(n):
# Compute raw peak-frequency (no DC-threshold masking yet)
freq_images[aidx] = compute_rf_image(
sras, aidx, dc_threshold_mv=-1e9, # mask nothing
apply_bg_sub=self._apply_bg_sub)
# DC images (ADC counts → mV)
dc4_images[aidx] = adc_to_mv(
compute_dc_image(sras, aidx, CH4_IDX),
sras.ch_ymult_mv[CH4_IDX],
sras.ch_yoff_adc[CH4_IDX],
sras.ch_yzero_mv[CH4_IDX])
dc3_images[aidx] = adc_to_mv(
compute_dc_image(sras, aidx, CH3_IDX),
sras.ch_ymult_mv[CH3_IDX],
sras.ch_yoff_adc[CH3_IDX],
if self._mode == "dc":
new_dc3 = [None] * n
new_dc4 = [None] * n
for a in range(n):
new_dc3[a] = adc_to_mv(
compute_dc_image(sras, a, CH3_IDX),
sras.ch_ymult_mv[CH3_IDX], sras.ch_yoff_adc[CH3_IDX],
sras.ch_yzero_mv[CH3_IDX])
new_dc4[a] = adc_to_mv(
compute_dc_image(sras, a, CH4_IDX),
sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX],
sras.ch_yzero_mv[CH4_IDX])
done_angles += 1
self.progress.emit(int(done_angles / total_angles * 100))
sras.write_v7_cache(new_dc3_mv=new_dc3, new_dc4_mv=new_dc4)
else:
effective_bg = self._apply_bg_sub and sras.background is not None
new_freq = [None] * n
for a in range(n):
new_freq[a] = compute_rf_image(
sras, a, dc_threshold_mv=-1e9, # mask nothing
apply_bg_sub=effective_bg)
done_angles += 1
self.progress.emit(int(done_angles / total_angles * 100))
sras.write_v7_cache(new_freq_mhz=new_freq, new_bg_sub=effective_bg)
self.progress.emit(int((aidx + 1) / n * 90))
bg_sub_flag = self._apply_bg_sub and sras.background is not None
sras.write_v5(
self._dest_path,
freq_images, dc4_images, dc3_images,
bg_sub_applied=bg_sub_flag,
progress_cb=lambda frac: self.progress.emit(90 + int(frac * 10)),
)
self.finished.emit("")
self.file_done.emit(path, "")
except Exception as exc:
self.finished.emit(str(exc))
self.file_done.emit(path, str(exc))
self.finished.emit()
class AngleAlignmentWorker(QObject):
"""Computes rotation+translation alignment for every angle in *sras*,
referenced to *ref_angle_idx*, from each angle's binarized CH4 mask.
Rotation is analytic (from sras.angles_deg); only translation is found
by phase correlation. Mirrors PreprocessWorker's progress/finished shape.
by phase correlation. Uses the same progress(int)/finished(...) worker
shape as the other background-thread workers in this file.
"""
progress = pyqtSignal(int) # 0–100
finished = pyqtSignal(object, str) # AlignmentResult|None, error msg ("" = success)
@@ -1794,7 +1949,11 @@ class SrasViewerWindow(QMainWindow):
self._fft_pad_factor: int = 1 # 1 = no padding
self._pending_fft_pad_factor: int = 1
self._preprocess_thread: QThread | None = None
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
self._batch_thread: QThread | None = None
self._batch_worker: BatchCacheWorker | None = None
self._batch_errors: list[str] = []
self._batch_progress_dlg: QProgressDialog | None = None
# Display-only settings (colormap, grating) never trigger a
# recompute — they're applied to cached data on redraw. DC images
@@ -2110,15 +2269,6 @@ class SrasViewerWindow(QMainWindow):
fft_act.triggered.connect(self._on_fft_options)
fft_menu.addAction(fft_act)
fft_menu.addSeparator()
self._preprocess_act = QAction("&Pre-process and Save as v5…", self)
self._preprocess_act.setStatusTip(
"Compute FFT and DC images for all angles and save to a v5 file "
"for instant re-opening (no FFT on load). Not available for v6 files.")
self._preprocess_act.setEnabled(False)
self._preprocess_act.triggered.connect(self._on_preprocess)
fft_menu.addAction(self._preprocess_act)
fusion_menu = menubar.addMenu("&Fusion")
self._alignment_act = QAction("Angle &Alignment", self)
self._alignment_act.setStatusTip(
@@ -2128,6 +2278,21 @@ class SrasViewerWindow(QMainWindow):
self._alignment_act.triggered.connect(self._on_angle_alignment)
fusion_menu.addAction(self._alignment_act)
convert_menu = menubar.addMenu("&Convert")
self._batch_dc_act = QAction("Batch Compute DC and &Store…", self)
self._batch_dc_act.setStatusTip(
"Select .sras files and compute+store DC images (CH3/CH4 mean) "
"for every angle, converting v6 files to v7 in place.")
self._batch_dc_act.triggered.connect(lambda: self._on_batch_compute("dc"))
convert_menu.addAction(self._batch_dc_act)
self._batch_fft_act = QAction("Batch Compute FFT and Sto&re…", self)
self._batch_fft_act.setStatusTip(
"Select .sras files and compute+store FFT peak-frequency images "
"for every angle, converting v6 files to v7 in place.")
self._batch_fft_act.triggered.connect(lambda: self._on_batch_compute("fft"))
convert_menu.addAction(self._batch_fft_act)
# ------------------------------------------------------------------
# Drag-and-drop
# ------------------------------------------------------------------
@@ -2265,11 +2430,22 @@ class SrasViewerWindow(QMainWindow):
)
if s.background is not None:
notes.append(f"Background waveform: {len(s.background)} samples")
if s.precomputed_freq_mhz is not None:
if isinstance(s.precomputed_freq_mhz, np.ndarray):
bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)"
notes.append(f"v5: precomputed images present{bg_note} — display is instant")
if s.version == 6:
notes.append("v6 format: rows / frames / x_start are per-angle")
if s.version in (6, 7):
notes.append("v6/v7 format: rows / frames / x_start are per-angle")
if s.version == 7:
n_dc = sum(1 for x in s.precomputed_dc4_mv if x is not None)
n_fft = sum(1 for x in s.precomputed_freq_mhz if x is not None)
if n_dc or n_fft:
bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)"
notes.append(
f"v7: cached DC ({n_dc}/{s.n_angles} angles), "
f"FFT ({n_fft}/{s.n_angles} angles{bg_note if n_fft else ''}) "
"— display is instant for cached angles")
else:
notes.append("v7 format: no cache blocks stored yet")
self.lbl_frame_warn.setText("\n".join(notes))
# ------------------------------------------------------------------
@@ -2299,10 +2475,11 @@ class SrasViewerWindow(QMainWindow):
self.btn_export_csv.setEnabled(is_ch1 and self._current_image is not None)
# ROI: always usable once a file is loaded (independent of channel)
self.btn_draw_roi.setEnabled(enabled and s is not None)
# Pre-process: available for v2–v5 files when loaded and not already running
can_preprocess = (enabled and s is not None and s.version != 6
and self._preprocess_thread is None)
self._preprocess_act.setEnabled(can_preprocess)
# Batch Convert actions: pick their own files, independent of
# whatever's currently open — only gated on no batch already running
can_batch = self._batch_thread is None
self._batch_dc_act.setEnabled(can_batch)
self._batch_fft_act.setEnabled(can_batch)
# Angle alignment: needs >1 angle and no alignment already running
can_align = (enabled and s is not None and s.n_angles > 1
and self._alignment_thread is None)
@@ -2897,75 +3074,90 @@ class SrasViewerWindow(QMainWindow):
self._progress_dlg = None
# ------------------------------------------------------------------
# Pre-process → save v5
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
# ------------------------------------------------------------------
def _on_preprocess(self):
if self._sras is None:
def _on_batch_compute(self, mode: str):
if self._batch_thread is not None:
return
if self._preprocess_thread is not None:
return
s = self._sras
if s.version == 6:
self.statusBar().showMessage(
"Pre-process to v5 is not supported for v6 files (per-angle geometry).")
return
default_name = s.path.stem + "_v5" + s.path.suffix
dest, _ = QFileDialog.getSaveFileName(
self, "Save Pre-processed v5 File",
str(s.path.parent / default_name),
label = "DC" if mode == "dc" else "FFT"
paths, _ = QFileDialog.getOpenFileNames(
self, f"Select .sras files to batch-compute {label}", "",
"SRAS files (*.sras);;All files (*)",
)
if not dest:
if not paths:
return
if self._preprocess_thread is not None:
return # a second trigger snuck in while the save dialog was open
if self._batch_thread is not None:
return # a second trigger snuck in while the file dialog was open
apply_bg = self.chk_bg_sub.isChecked() and s.background is not None
apply_bg = self.chk_bg_sub.isChecked()
self._batch_errors = []
n_total = s.n_angles
n_px = int(s.n_rows[0]) * int(s.n_frames[0])
approx_mb = n_total * n_px * 3 * 4 / 1e6
# Claim self._batch_thread before anything below that can pump the
# Qt event loop — see the comment in _start_compute for why.
self._batch_worker = BatchCacheWorker(paths, mode, apply_bg)
self._batch_thread = QThread()
self._batch_worker.moveToThread(self._batch_thread)
self._batch_thread.started.connect(self._batch_worker.run)
self._batch_worker.progress.connect(self._on_batch_progress)
self._batch_worker.file_done.connect(self._on_batch_file_done)
self._batch_worker.finished.connect(lambda paths=paths: self._on_batch_finished(paths))
self._batch_worker.finished.connect(self._batch_thread.quit)
self._batch_thread.finished.connect(self._on_batch_thread_finished)
# Claim self._preprocess_thread before any call below that can pump
# the Qt event loop — see the comment in _start_compute for why.
self._preprocess_worker = PreprocessWorker(s, dest, apply_bg)
self._preprocess_thread = QThread()
self._preprocess_worker.moveToThread(self._preprocess_thread)
self._preprocess_thread.started.connect(self._preprocess_worker.run)
self._preprocess_worker.progress.connect(self._on_preprocess_progress)
self._preprocess_worker.finished.connect(self._on_preprocess_done)
self._preprocess_worker.finished.connect(self._preprocess_thread.quit)
self._preprocess_thread.finished.connect(self._on_preprocess_thread_finished)
self._batch_dc_act.setEnabled(False)
self._batch_fft_act.setEnabled(False)
self._preprocess_act.setEnabled(False)
self._show_progress(
f"Pre-processing {n_total} angle(s) "
f"({int(s.n_rows[0])}×{int(s.n_frames[0])} px each, ~{approx_mb:.0f} MB output)…"
)
self._batch_progress_dlg = QProgressDialog(
f"Batch computing {label} for {len(paths)} file(s)…", "", 0, 100, self)
self._batch_progress_dlg.setWindowTitle("Please wait…")
self._batch_progress_dlg.setCancelButton(None)
self._batch_progress_dlg.setWindowModality(Qt.WindowModality.WindowModal)
self._batch_progress_dlg.setMinimumDuration(300)
self._batch_progress_dlg.show()
self._preprocess_thread.start()
self._batch_thread.start()
def _on_preprocess_progress(self, pct: int):
if self._progress_dlg is not None:
self._progress_dlg.setValue(pct)
def _on_batch_progress(self, pct: int):
if self._batch_progress_dlg is not None:
self._batch_progress_dlg.setValue(pct)
def _on_preprocess_done(self, error_msg: str):
self._close_progress()
if error_msg:
self.statusBar().showMessage(f"Pre-process failed: {error_msg}")
def _on_batch_file_done(self, path: str, err: str):
if err:
self._batch_errors.append(f"{Path(path).name} — {err}")
if self._batch_progress_dlg is not None:
self._batch_progress_dlg.setLabelText(f"Processed {Path(path).name}…")
def _on_batch_finished(self, paths: list[str]):
if self._batch_progress_dlg is not None:
self._batch_progress_dlg.close()
self._batch_progress_dlg = None
n_total = len(paths)
n_failed = len(self._batch_errors)
n_ok = n_total - n_failed
if n_failed:
summary = (f"Batch store: {n_ok}/{n_total} file(s) updated, "
f"{n_failed} failed: {'; '.join(self._batch_errors)}")
else:
self.statusBar().showMessage("v5 file written — re-open it for instant display.")
summary = f"Batch store: {n_ok}/{n_total} file(s) updated."
self.statusBar().showMessage(summary)
self._batch_errors = []
def _on_preprocess_thread_finished(self):
# If the currently-open file was in this batch, reload it so the
# GUI picks up the newly-written v7 cache instead of stale state.
if self._sras is not None and str(self._sras.path) in paths:
self._load_file(str(self._sras.path))
def _on_batch_thread_finished(self):
# See the comment in _on_compute_thread_finished: wait() before
# releasing our reference to avoid destroying a QThread whose OS
# thread hasn't fully joined yet.
if self._preprocess_thread is not None:
self._preprocess_thread.wait()
self._preprocess_thread = None
self._preprocess_act.setEnabled(
self._sras is not None and self._sras.version != 6)
if self._batch_thread is not None:
self._batch_thread.wait()
self._batch_thread = None
self._batch_dc_act.setEnabled(True)
self._batch_fft_act.setEnabled(True)
# ------------------------------------------------------------------
# Fusion: angle alignment
@@ -3066,7 +3258,7 @@ class SrasViewerWindow(QMainWindow):
def closeEvent(self, event):
if self._dc_precompute_worker is not None:
self._dc_precompute_worker.stop()
for attr in ("_load_thread", "_compute_thread", "_preprocess_thread",
for attr in ("_load_thread", "_compute_thread", "_batch_thread",
"_dc_precompute_thread", "_alignment_thread"):
t = getattr(self, attr, None)
if t is not None: