Refactor/parallel and dedup #1
+117
-2
@@ -1,4 +1,4 @@
|
|||||||
# SRAS Scan Binary Format — Version 6
|
# SRAS Scan Binary Format — Version 6 / 7
|
||||||
|
|
||||||
Each `.sras` file contains **one complete scan**: all GR rotation angles and all
|
Each `.sras` file contains **one complete scan**: all GR rotation angles and all
|
||||||
Y rows. Files are named `{prefix}.sras`.
|
Y rows. Files are named `{prefix}.sras`.
|
||||||
@@ -10,6 +10,14 @@ rotated by that specific angle** — not the worst case across all angles — so
|
|||||||
fewer rows than a 45° scan of the same ROI, and the file format reflects that
|
fewer rows than a 45° scan of the same ROI, and the file format reflects that
|
||||||
instead of forcing every angle to the largest bounding box.
|
instead of forcing every angle to the largest bounding box.
|
||||||
|
|
||||||
|
v7 is byte-identical to v6 (same header, angle table, geometry table, row
|
||||||
|
table, preamble blocks, background block, waveform data) plus an optional
|
||||||
|
trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images
|
||||||
|
(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version`
|
||||||
|
field and the presence of that trailing section differ. Scans come off the
|
||||||
|
scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file
|
||||||
|
to v7 **in place** the first time either cache block is computed and stored.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## File Layout
|
## File Layout
|
||||||
@@ -22,6 +30,7 @@ instead of forcing every angle to the largest bounding box.
|
|||||||
[Preamble Blocks — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
|
[Preamble Blocks — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
|
||||||
[Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes]
|
[Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes]
|
||||||
[Waveform Data (ragged) — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
|
[Waveform Data (ragged) — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
|
||||||
|
[Cache Tail (optional) — "CACH" + DC block (optional) + FFT block (optional); v7 only]
|
||||||
```
|
```
|
||||||
|
|
||||||
All multi-byte integers and floats use **big-endian** byte order
|
All multi-byte integers and floats use **big-endian** byte order
|
||||||
@@ -182,6 +191,111 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Cache Tail (v7)
|
||||||
|
|
||||||
|
Present iff `version == 7` and `file_size > cache_offset`, where:
|
||||||
|
|
||||||
|
```
|
||||||
|
cache_offset = data_offset + Σ over angles a of:
|
||||||
|
n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample
|
||||||
|
```
|
||||||
|
|
||||||
|
i.e. exactly `data_offset + waveform_bytes` — the same "Total data size"
|
||||||
|
formula as Waveform Data above. This offset is derivable from the header and
|
||||||
|
Per-Angle Geometry Table alone and does **not** depend on which cache
|
||||||
|
block(s) are present, so a writer can always seek straight there without
|
||||||
|
reading or touching any waveform byte before it.
|
||||||
|
|
||||||
|
Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle
|
||||||
|
entry (FFT + both DC channels together) in a dense, uniform-geometry array,
|
||||||
|
the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks —
|
||||||
|
each sized per-angle from the Per-Angle Geometry Table, each independently
|
||||||
|
present, and each independently updatable in any order, any number of
|
||||||
|
times, without disturbing the other. This matches `sras_viewer.py`'s
|
||||||
|
"Convert" menu, which exposes DC and FFT store as two separate batch
|
||||||
|
actions.
|
||||||
|
|
||||||
|
### CACH outer header (6 bytes, `">4sBB"`)
|
||||||
|
|
||||||
|
| Offset | Size | Type | Field | Description |
|
||||||
|
|--------|------|------|-------|-------------|
|
||||||
|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
|
||||||
|
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `1`. Readers must treat the file as uncached if this is not a version they understand (unlike v5's `PREC` section, which read but never validated its version byte). |
|
||||||
|
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |
|
||||||
|
|
||||||
|
### DC block `SDCB` (present iff `block_flags & 0x01`)
|
||||||
|
|
||||||
|
7-byte block header, format `">4sBH"`:
|
||||||
|
|
||||||
|
| Offset (rel) | Size | Type | Field | Description |
|
||||||
|
|--------------|------|------|-------|-------------|
|
||||||
|
| 0 | 4 | `char[4]` | `magic` | `SDCB` |
|
||||||
|
| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use |
|
||||||
|
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` |
|
||||||
|
|
||||||
|
followed by `n_stored` entries, each:
|
||||||
|
|
||||||
|
```
|
||||||
|
u16 angle_idx — index into the angle table (0-based)
|
||||||
|
f32[n_rows[angle_idx] × n_frames[angle_idx]] dc3_mv — CH3 waveform mean, mV, row-major
|
||||||
|
f32[n_rows[angle_idx] × n_frames[angle_idx]] dc4_mv — CH4 waveform mean, mV, row-major
|
||||||
|
```
|
||||||
|
|
||||||
|
Entries may appear in any order and need not be contiguous from angle 0 —
|
||||||
|
this supports storing (or re-storing) a subset of angles, or an
|
||||||
|
interrupted batch run leaving only some angles cached. Readers bounds-check
|
||||||
|
`angle_idx < n_angles` on each entry and stop parsing on an out-of-range
|
||||||
|
value, same as v5's `PREC` section.
|
||||||
|
|
||||||
|
### FFT block `SFFT` (present iff `block_flags & 0x02`)
|
||||||
|
|
||||||
|
7-byte block header, format `">4sBH"`:
|
||||||
|
|
||||||
|
| Offset (rel) | Size | Type | Field | Description |
|
||||||
|
|--------------|------|------|-------|-------------|
|
||||||
|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
|
||||||
|
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bits 1–7 reserved. |
|
||||||
|
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
|
||||||
|
|
||||||
|
followed by `n_stored` entries, each:
|
||||||
|
|
||||||
|
```
|
||||||
|
u16 angle_idx — index into the angle table (0-based)
|
||||||
|
f32[n_rows[angle_idx] × n_frames[angle_idx]] peak_freq_mhz — CH1 FFT peak frequency, MHz, row-major
|
||||||
|
```
|
||||||
|
|
||||||
|
**`peak_freq_mhz`** is computed without any DC-threshold masking (i.e. the
|
||||||
|
FFT is run on every pixel unconditionally, same as v5's `PREC` convention).
|
||||||
|
Readers apply the DC4 threshold at display time:
|
||||||
|
|
||||||
|
```
|
||||||
|
pixel is valid ⟺ dc4_mv[r][f] ≥ threshold_mv
|
||||||
|
display_value = peak_freq_mhz[r][f] if valid, else 0
|
||||||
|
```
|
||||||
|
|
||||||
|
using the DC4 image from the DC block if that angle is also cached there,
|
||||||
|
else computed on demand.
|
||||||
|
|
||||||
|
Readers must fall back to real-time FFT computation (ignoring stored
|
||||||
|
`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain
|
||||||
|
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested, or
|
||||||
|
the reader's background-subtraction setting doesn't match `flags.bg_sub_applied`.
|
||||||
|
|
||||||
|
### In-place write ordering
|
||||||
|
|
||||||
|
A writer updating a file's Cache Tail must write the payload (CACH header +
|
||||||
|
whichever block(s) are present) **before** flipping the header's `version`
|
||||||
|
byte to `7`, and `truncate()` the file to the new payload's end immediately
|
||||||
|
after writing it. If the process is interrupted between the payload write
|
||||||
|
and the version-byte flip, the file is still valid v6 — v6 parsing only
|
||||||
|
bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts
|
||||||
|
exactly how many bytes follow the last angle's waveform block — so the
|
||||||
|
interrupted write leaves harmless trailing bytes rather than a corrupt file,
|
||||||
|
and the next successful write overwrites them via the same deterministic
|
||||||
|
`cache_offset`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Acquisition Settings (fixed by sc3_aui_app.py)
|
## Acquisition Settings (fixed by sc3_aui_app.py)
|
||||||
|
|
||||||
| Parameter | Value |
|
| Parameter | Value |
|
||||||
@@ -206,5 +320,6 @@ using that angle's `x_start` from the Per-Angle Geometry Table (not
|
|||||||
| 3 | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
|
| 3 | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
|
||||||
| 4 | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
|
| 4 | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
|
||||||
| 5 | (skipped) |
|
| 5 | (skipped) |
|
||||||
| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`); the "Pre-process and Save as v5" fast-path export is not offered for v6 files since the flat v5 layout cannot represent per-angle geometry. |
|
| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). |
|
||||||
|
| 7 | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. |
|
||||||
|
|
||||||
|
|||||||
+413
-221
@@ -15,11 +15,16 @@ header before acquisition, but the scope may acquire fewer frames. The actual
|
|||||||
count is computed from the file size and used for the reshape so channels are
|
count is computed from the file size and used for the reshape so channels are
|
||||||
correctly aligned.
|
correctly aligned.
|
||||||
|
|
||||||
Scan geometry: v6 files scan a different bounding box per angle (x_start,
|
Scan geometry: v6/v7 files scan a different bounding box per angle (x_start,
|
||||||
x_delta, n_frames, n_rows all vary by angle), so geometry is exposed per-angle
|
x_delta, n_frames, n_rows all vary by angle), so geometry is exposed per-angle
|
||||||
via SrasFile.n_rows / n_frames / x_start_mm arrays and the x_axis_mm() /
|
via SrasFile.n_rows / n_frames / x_start_mm arrays and the x_axis_mm() /
|
||||||
y_positions_mm() methods. v2–v5 files have uniform geometry across angles, so
|
y_positions_mm() methods. v2–v5 files have uniform geometry across angles, so
|
||||||
those arrays simply repeat the same value n_angles times.
|
those arrays simply repeat the same value n_angles times.
|
||||||
|
|
||||||
|
v7 files are v6 files with an optional trailing cache section holding
|
||||||
|
precomputed per-angle DC and/or FFT images (see scan_format.md), so display
|
||||||
|
never has to recompute them after the "Convert" menu's batch actions have
|
||||||
|
stored them once.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
@@ -90,6 +95,29 @@ HDR_SIZE_V6 = struct.calcsize(HDR_FMT_V6) # 49 bytes
|
|||||||
GEO_FMT_V6 = ">ffIH"
|
GEO_FMT_V6 = ">ffIH"
|
||||||
GEO_SIZE_V6 = struct.calcsize(GEO_FMT_V6) # 14 bytes
|
GEO_SIZE_V6 = struct.calcsize(GEO_FMT_V6) # 14 bytes
|
||||||
|
|
||||||
|
# v7: identical to v6 (same header/geometry/waveform layout, version byte
|
||||||
|
# is the only header difference) plus an optional trailing cache section
|
||||||
|
# ("CACH") holding precomputed per-angle DC and/or FFT images so a v7 file
|
||||||
|
# never needs to recompute them on open. See scan_format.md for the full
|
||||||
|
# spec. Sub-block sizes are per-angle (n_rows[a] * n_frames[a]), taken from
|
||||||
|
# the Per-Angle Geometry Table already parsed for v6 — no new geometry
|
||||||
|
# fields are needed.
|
||||||
|
CACH_MAGIC = b"CACH"
|
||||||
|
CACH_HDR_FMT = ">4sBB" # magic, cach_version, block_flags
|
||||||
|
CACH_HDR_SIZE = struct.calcsize(CACH_HDR_FMT) # 6 bytes
|
||||||
|
CACH_VERSION = 1
|
||||||
|
CACH_FLAG_DC = 0x01
|
||||||
|
CACH_FLAG_FFT = 0x02
|
||||||
|
|
||||||
|
SDCB_MAGIC = b"SDCB"
|
||||||
|
SDCB_HDR_FMT = ">4sBH" # magic, reserved, n_stored
|
||||||
|
SDCB_HDR_SIZE = struct.calcsize(SDCB_HDR_FMT) # 7 bytes
|
||||||
|
|
||||||
|
SFFT_MAGIC = b"SFFT"
|
||||||
|
SFFT_HDR_FMT = ">4sBH" # magic, flags, n_stored
|
||||||
|
SFFT_HDR_SIZE = struct.calcsize(SFFT_HDR_FMT) # 7 bytes
|
||||||
|
SFFT_FLAG_BG_SUB = 0x01
|
||||||
|
|
||||||
# Fixed-order channels in the file: index 0=CH1, 1=CH3, 2=CH4
|
# Fixed-order channels in the file: index 0=CH1, 1=CH3, 2=CH4
|
||||||
# Fixed channel indices into the .sras data array (CH1=RF, CH3/CH4=Bias DC)
|
# Fixed channel indices into the .sras data array (CH1=RF, CH3/CH4=Bias DC)
|
||||||
CH1_IDX, CH3_IDX, CH4_IDX = 0, 1, 2
|
CH1_IDX, CH3_IDX, CH4_IDX = 0, 1, 2
|
||||||
@@ -149,15 +177,21 @@ def adc_to_mv(adc: float, ymult_mv: float = _FALLBACK_YMULT_MV,
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
class SrasFile:
|
class SrasFile:
|
||||||
"""Parsed in-memory representation of a v2–v6 .sras file.
|
"""Parsed in-memory representation of a v2–v7 .sras file.
|
||||||
|
|
||||||
Scan geometry (rows, frames, x_start) is exposed per-angle via the
|
Scan geometry (rows, frames, x_start) is exposed per-angle via the
|
||||||
``n_rows`` / ``n_frames`` / ``x_start_mm`` arrays and the ``x_axis_mm()``
|
``n_rows`` / ``n_frames`` / ``x_start_mm`` arrays and the ``x_axis_mm()``
|
||||||
/ ``y_positions_mm()`` methods, since v6 files scan a different bounding
|
/ ``y_positions_mm()`` methods, since v6/v7 files scan a different
|
||||||
box per angle. v2–v5 files have uniform geometry, so these arrays just
|
bounding box per angle. v2–v5 files have uniform geometry, so these
|
||||||
repeat the same value ``n_angles`` times. Waveform data is likewise
|
arrays just repeat the same value ``n_angles`` times. Waveform data is
|
||||||
exposed as ``data[angle_idx]``, an array of shape
|
likewise exposed as ``data[angle_idx]``, an array of shape
|
||||||
``(n_rows[a], n_channels, n_frames[a], samples_per_frame)``.
|
``(n_rows[a], n_channels, n_frames[a], samples_per_frame)``.
|
||||||
|
|
||||||
|
v7 files (v6 plus an optional trailing cache section) expose any stored
|
||||||
|
precomputed images as ``precomputed_dc3_mv`` / ``precomputed_dc4_mv`` /
|
||||||
|
``precomputed_freq_mhz`` — for v6/v7 these are ragged per-angle lists
|
||||||
|
(``list[np.ndarray | None]``, one entry per angle); for v5 they are dense
|
||||||
|
``(n_angles, n_rows, n_frames)`` arrays, since v5 geometry is uniform.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, path: str):
|
def __init__(self, path: str):
|
||||||
@@ -174,7 +208,7 @@ class SrasFile:
|
|||||||
self.version = version
|
self.version = version
|
||||||
if version in (2, 3, 4, 5):
|
if version in (2, 3, 4, 5):
|
||||||
self._parse_legacy()
|
self._parse_legacy()
|
||||||
elif version == 6:
|
elif version in (6, 7):
|
||||||
self._parse_v6()
|
self._parse_v6()
|
||||||
else:
|
else:
|
||||||
raise ValueError(f"Unsupported version: {version}")
|
raise ValueError(f"Unsupported version: {version}")
|
||||||
@@ -368,11 +402,6 @@ class SrasFile:
|
|||||||
self.frame_count_mismatch = False
|
self.frame_count_mismatch = False
|
||||||
self.n_frames_remainder = 0
|
self.n_frames_remainder = 0
|
||||||
|
|
||||||
self.precomputed_freq_mhz: np.ndarray | None = None
|
|
||||||
self.precomputed_dc4_mv: np.ndarray | None = None
|
|
||||||
self.precomputed_dc3_mv: np.ndarray | None = None
|
|
||||||
self.precomputed_bg_sub: bool = False
|
|
||||||
|
|
||||||
angles = np.frombuffer(f.read(n_angles * 4), dtype=">f4").astype(np.float32)
|
angles = np.frombuffer(f.read(n_angles * 4), dtype=">f4").astype(np.float32)
|
||||||
|
|
||||||
x_start = np.empty(n_angles, dtype=np.float64)
|
x_start = np.empty(n_angles, dtype=np.float64)
|
||||||
@@ -409,6 +438,8 @@ class SrasFile:
|
|||||||
|
|
||||||
data_offset = f.tell()
|
data_offset = f.tell()
|
||||||
|
|
||||||
|
self._data_offset = data_offset
|
||||||
|
|
||||||
# ---- Memory-map each angle's ragged waveform block -------------
|
# ---- Memory-map each angle's ragged waveform block -------------
|
||||||
# v6 gives each angle its own row/frame count, so waveform data is
|
# v6 gives each angle its own row/frame count, so waveform data is
|
||||||
# no longer one uniform (n_angles, n_rows, ...) block — each angle's
|
# no longer one uniform (n_angles, n_rows, ...) block — each angle's
|
||||||
@@ -450,108 +481,157 @@ class SrasFile:
|
|||||||
self.n_rows = n_rows[:n_complete]
|
self.n_rows = n_rows[:n_complete]
|
||||||
self._y_pos_per_angle = y_pos_per_angle[:n_complete]
|
self._y_pos_per_angle = y_pos_per_angle[:n_complete]
|
||||||
|
|
||||||
|
# Precomputed-image cache (v7 only). Ragged per-angle lists — unlike
|
||||||
|
# v5's dense (n_angles, n_rows, n_frames) arrays, v6/v7 geometry
|
||||||
|
# varies per angle, so each entry is its own (n_rows[a], n_frames[a])
|
||||||
|
# array or None if that angle isn't cached yet.
|
||||||
|
self.precomputed_freq_mhz: list[np.ndarray | None] = [None] * n_complete
|
||||||
|
self.precomputed_dc4_mv: list[np.ndarray | None] = [None] * n_complete
|
||||||
|
self.precomputed_dc3_mv: list[np.ndarray | None] = [None] * n_complete
|
||||||
|
self.precomputed_bg_sub: bool = False
|
||||||
|
|
||||||
|
if self.version == 7 and offset < file_size:
|
||||||
|
self._parse_cach_section(offset)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# v5 writer
|
# v7 cache tail (CACH section: precomputed DC / FFT images)
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def write_v5(self, dest_path: str,
|
def _cache_tail_offset(self) -> int:
|
||||||
freq_images: np.ndarray,
|
"""Deterministic file offset where the CACH tail starts (or would
|
||||||
dc4_images: np.ndarray,
|
start), derived purely from the header + Per-Angle Geometry Table —
|
||||||
dc3_images: np.ndarray,
|
independent of whether a cache tail is actually present. Used by
|
||||||
bg_sub_applied: bool,
|
both the parser and the in-place writer."""
|
||||||
progress_cb=None):
|
n_ch = self.n_channels
|
||||||
"""Write a v5 .sras file to *dest_path*.
|
spf = self.samples_per_frame
|
||||||
|
bps = self.bytes_per_sample
|
||||||
|
waveform_bytes = int(sum(
|
||||||
|
int(self.n_rows[a]) * n_ch * int(self.n_frames[a]) * spf * bps
|
||||||
|
for a in range(self.n_angles)
|
||||||
|
))
|
||||||
|
return self._data_offset + waveform_bytes
|
||||||
|
|
||||||
Copies the raw waveform bytes verbatim from the current file,
|
def _parse_cach_section(self, offset: int):
|
||||||
bumps the version byte to 5, patches n_frames_hdr to the actual
|
"""Parse the v7 CACH tail that holds precomputed DC/FFT images."""
|
||||||
frame count, then appends the PREC section.
|
with open(self.path, "rb") as f:
|
||||||
|
f.seek(offset)
|
||||||
|
header_raw = f.read(CACH_HDR_SIZE)
|
||||||
|
if len(header_raw) < CACH_HDR_SIZE:
|
||||||
|
return
|
||||||
|
magic, cach_version, block_flags = struct.unpack(CACH_HDR_FMT, header_raw)
|
||||||
|
if magic != CACH_MAGIC or cach_version != CACH_VERSION:
|
||||||
|
return
|
||||||
|
|
||||||
*freq_images* / *dc4_images* / *dc3_images*:
|
if block_flags & CACH_FLAG_DC:
|
||||||
shape (n_angles, n_rows, n_frames) float32.
|
sdcb_raw = f.read(SDCB_HDR_SIZE)
|
||||||
|
if len(sdcb_raw) < SDCB_HDR_SIZE:
|
||||||
|
return
|
||||||
|
sdcb_magic, _reserved, n_stored = struct.unpack(SDCB_HDR_FMT, sdcb_raw)
|
||||||
|
if sdcb_magic != SDCB_MAGIC:
|
||||||
|
return
|
||||||
|
for _ in range(n_stored):
|
||||||
|
(angle_idx,) = struct.unpack(">H", f.read(2))
|
||||||
|
if angle_idx >= self.n_angles:
|
||||||
|
break
|
||||||
|
px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx])
|
||||||
|
shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx]))
|
||||||
|
self.precomputed_dc3_mv[angle_idx] = np.frombuffer(
|
||||||
|
f.read(px * 4), dtype=">f4").reshape(shape)
|
||||||
|
self.precomputed_dc4_mv[angle_idx] = np.frombuffer(
|
||||||
|
f.read(px * 4), dtype=">f4").reshape(shape)
|
||||||
|
|
||||||
*progress_cb*: optional callable(fraction: float) for UI updates.
|
if block_flags & CACH_FLAG_FFT:
|
||||||
|
sfft_raw = f.read(SFFT_HDR_SIZE)
|
||||||
|
if len(sfft_raw) < SFFT_HDR_SIZE:
|
||||||
|
return
|
||||||
|
sfft_magic, flags, n_stored = struct.unpack(SFFT_HDR_FMT, sfft_raw)
|
||||||
|
if sfft_magic != SFFT_MAGIC:
|
||||||
|
return
|
||||||
|
self.precomputed_bg_sub = bool(flags & SFFT_FLAG_BG_SUB)
|
||||||
|
for _ in range(n_stored):
|
||||||
|
(angle_idx,) = struct.unpack(">H", f.read(2))
|
||||||
|
if angle_idx >= self.n_angles:
|
||||||
|
break
|
||||||
|
px = int(self.n_rows[angle_idx]) * int(self.n_frames[angle_idx])
|
||||||
|
shape = (int(self.n_rows[angle_idx]), int(self.n_frames[angle_idx]))
|
||||||
|
self.precomputed_freq_mhz[angle_idx] = np.frombuffer(
|
||||||
|
f.read(px * 4), dtype=">f4").reshape(shape)
|
||||||
|
|
||||||
Only valid for v2–v5 source files, which have uniform per-angle
|
def write_v7_cache(self, *,
|
||||||
geometry. v6 files scan a different bounding box per angle and
|
new_dc3_mv: list[np.ndarray | None] | None = None,
|
||||||
cannot be losslessly represented in the flat v5 layout.
|
new_dc4_mv: list[np.ndarray | None] | None = None,
|
||||||
|
new_freq_mhz: list[np.ndarray | None] | None = None,
|
||||||
|
new_bg_sub: bool | None = None):
|
||||||
|
"""Store computed DC and/or FFT images into this file's CACH tail,
|
||||||
|
in place, converting a v6 source to v7 (or updating an existing v7
|
||||||
|
file). Only the block(s) passed in are recomputed; whichever block
|
||||||
|
isn't passed is carried forward unchanged from whatever this
|
||||||
|
``SrasFile`` already has in memory (from parsing, or a prior write
|
||||||
|
in this same session) — its bytes are never re-read from disk.
|
||||||
|
|
||||||
|
The waveform data itself is never touched: the cache tail always
|
||||||
|
starts at ``_cache_tail_offset()``, a fixed offset derived from the
|
||||||
|
header and geometry table alone.
|
||||||
"""
|
"""
|
||||||
if self.version == 6:
|
if self.version not in (6, 7):
|
||||||
raise NotImplementedError(
|
raise ValueError(
|
||||||
"Pre-process to v5 is not supported for v6 source files "
|
f"write_v7_cache only supports v6/v7 source files, got v{self.version}")
|
||||||
"(per-angle geometry does not fit the flat v5 layout).")
|
|
||||||
|
|
||||||
import shutil
|
final_dc3 = new_dc3_mv if new_dc3_mv is not None else self.precomputed_dc3_mv
|
||||||
|
final_dc4 = new_dc4_mv if new_dc4_mv is not None else self.precomputed_dc4_mv
|
||||||
|
final_freq = new_freq_mhz if new_freq_mhz is not None else self.precomputed_freq_mhz
|
||||||
|
final_bg_sub = new_bg_sub if new_bg_sub is not None else self.precomputed_bg_sub
|
||||||
|
|
||||||
dest = Path(dest_path)
|
dc_entries = [a for a in range(self.n_angles) if final_dc3[a] is not None]
|
||||||
src = self.path
|
fft_entries = [a for a in range(self.n_angles) if final_freq[a] is not None]
|
||||||
n_rows0 = int(self.n_rows[0])
|
|
||||||
n_frames0 = int(self.n_frames[0])
|
|
||||||
|
|
||||||
# --- Copy the source file verbatim, then patch the header -------
|
block_flags = 0
|
||||||
shutil.copy2(str(src), str(dest))
|
if dc_entries:
|
||||||
|
block_flags |= CACH_FLAG_DC
|
||||||
|
if fft_entries:
|
||||||
|
block_flags |= CACH_FLAG_FFT
|
||||||
|
|
||||||
waveform_bytes = (self.n_angles * n_rows0 * self.n_channels
|
payload = bytearray()
|
||||||
* n_frames0 * self.samples_per_frame
|
payload += struct.pack(CACH_HDR_FMT, CACH_MAGIC, CACH_VERSION, block_flags)
|
||||||
* self.bytes_per_sample)
|
|
||||||
|
|
||||||
with open(str(dest), "r+b") as f:
|
if dc_entries:
|
||||||
# Patch version byte (offset 4 in the header struct)
|
payload += struct.pack(SDCB_HDR_FMT, SDCB_MAGIC, 0, len(dc_entries))
|
||||||
f.seek(4)
|
for a in dc_entries:
|
||||||
f.write(struct.pack("B", 5))
|
payload += struct.pack(">H", a)
|
||||||
|
payload += final_dc3[a].astype(">f4").tobytes()
|
||||||
|
payload += final_dc4[a].astype(">f4").tobytes()
|
||||||
|
|
||||||
# Patch n_frames_hdr (uint32, big-endian) with the actual count.
|
if fft_entries:
|
||||||
# Locate its offset: magic(4) + ver(1) + n_angles(2) + n_rows(2) = 9
|
fft_flags = SFFT_FLAG_BG_SUB if final_bg_sub else 0
|
||||||
# then x_start(4)+x_delta(4)+vel(4)+freq(4) = 16, total = 25
|
payload += struct.pack(SFFT_HDR_FMT, SFFT_MAGIC, fft_flags, len(fft_entries))
|
||||||
# then n_frames_hdr is at offset 25 as ">I" (4 bytes)
|
for a in fft_entries:
|
||||||
f.seek(25)
|
payload += struct.pack(">H", a)
|
||||||
f.write(struct.pack(">I", n_frames0))
|
payload += final_freq[a].astype(">f4").tobytes()
|
||||||
|
|
||||||
# Truncate anything after the waveform data (e.g. old PREC tail)
|
cache_offset = self._cache_tail_offset()
|
||||||
# and seek to the append position.
|
with open(self.path, "r+b") as f:
|
||||||
waveform_end = self._data_offset_for_write()
|
f.seek(cache_offset)
|
||||||
f.seek(waveform_end + waveform_bytes)
|
f.write(payload)
|
||||||
f.truncate()
|
f.truncate()
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
# Version-byte flip last: if the process dies before this point,
|
||||||
|
# the file is still readable as plain v6 (v6 parsing only
|
||||||
|
# bounds-checks per-angle offset+nbytes <= file_size, it never
|
||||||
|
# asserts exactly how many bytes follow the last angle) — so an
|
||||||
|
# interrupted write can never corrupt the file, only leave
|
||||||
|
# harmless trailing bytes that the next successful write
|
||||||
|
# overwrites via this same deterministic cache_offset.
|
||||||
|
f.seek(4)
|
||||||
|
f.write(struct.pack("B", 7))
|
||||||
|
f.flush()
|
||||||
|
os.fsync(f.fileno())
|
||||||
|
|
||||||
# --- Write PREC section -------------------------------------
|
self.version = 7
|
||||||
n_stored = self.n_angles
|
self.precomputed_dc3_mv = final_dc3
|
||||||
flags = 0x01 if bg_sub_applied else 0x00
|
self.precomputed_dc4_mv = final_dc4
|
||||||
f.write(b"PREC")
|
self.precomputed_freq_mhz = final_freq
|
||||||
f.write(struct.pack("BB", 1, flags))
|
self.precomputed_bg_sub = final_bg_sub
|
||||||
f.write(struct.pack(">H", n_stored))
|
|
||||||
|
|
||||||
for aidx in range(n_stored):
|
|
||||||
if progress_cb is not None:
|
|
||||||
progress_cb(aidx / n_stored)
|
|
||||||
f.write(struct.pack(">H", aidx))
|
|
||||||
f.write(freq_images[aidx].astype(">f4").tobytes())
|
|
||||||
f.write(dc4_images[aidx].astype(">f4").tobytes())
|
|
||||||
f.write(dc3_images[aidx].astype(">f4").tobytes())
|
|
||||||
|
|
||||||
if progress_cb is not None:
|
|
||||||
progress_cb(1.0)
|
|
||||||
|
|
||||||
def _data_offset_for_write(self) -> int:
|
|
||||||
"""Return the file offset where waveform data starts (used by write_v5)."""
|
|
||||||
# Re-derive the offset by walking the header fields, since we do not
|
|
||||||
# persist data_offset as an attribute from _parse.
|
|
||||||
with open(self.path, "rb") as f:
|
|
||||||
fields = struct.unpack(HDR_FMT, f.read(HDR_SIZE))
|
|
||||||
ver = fields[1]
|
|
||||||
n_ch = fields[12]
|
|
||||||
n_rows0 = int(self.n_rows[0])
|
|
||||||
|
|
||||||
with open(self.path, "rb") as f:
|
|
||||||
f.seek(HDR_SIZE)
|
|
||||||
f.read(self.n_angles * 4) # angles
|
|
||||||
f.read(n_rows0 * 4) # y_pos
|
|
||||||
if ver >= 3:
|
|
||||||
for _ in range(n_ch):
|
|
||||||
(length,) = struct.unpack(">H", f.read(2))
|
|
||||||
f.read(length)
|
|
||||||
if ver >= 4:
|
|
||||||
(n_bg,) = struct.unpack(">I", f.read(4))
|
|
||||||
f.read(n_bg)
|
|
||||||
return f.tell()
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Axes helpers
|
# Axes helpers
|
||||||
@@ -621,6 +701,21 @@ def compute_dc_image(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray:
|
|||||||
return img
|
return img
|
||||||
|
|
||||||
|
|
||||||
|
def _cached_dc_mv(sras: SrasFile, angle_idx: int, ch_idx: int) -> np.ndarray | None:
|
||||||
|
"""Return a v7-cached DC image (mV, already converted) for
|
||||||
|
(angle_idx, ch_idx) if the file's CACH section has it, else None.
|
||||||
|
|
||||||
|
Only applies to the ragged v6/v7 list representation of
|
||||||
|
``precomputed_dc3_mv``/``precomputed_dc4_mv`` — v5's dense PREC arrays
|
||||||
|
don't track per-angle validity (unstored angles are left at 0.0 rather
|
||||||
|
than a sentinel), so they're intentionally not consulted here.
|
||||||
|
"""
|
||||||
|
store = sras.precomputed_dc3_mv if ch_idx == CH3_IDX else sras.precomputed_dc4_mv
|
||||||
|
if isinstance(store, list) and angle_idx < len(store):
|
||||||
|
return store[angle_idx]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def compute_rf_image(sras: SrasFile, angle_idx: int,
|
def compute_rf_image(sras: SrasFile, angle_idx: int,
|
||||||
dc_threshold_mv: float,
|
dc_threshold_mv: float,
|
||||||
apply_bg_sub: bool = True,
|
apply_bg_sub: bool = True,
|
||||||
@@ -640,9 +735,10 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
computed chunk-by-chunk internally (which does need every pixel's
|
computed chunk-by-chunk internally (which does need every pixel's
|
||||||
CH4 data, since that's what determines validity in the first place).
|
CH4 data, since that's what determines validity in the first place).
|
||||||
|
|
||||||
Fast path: if the file contains v5 precomputed peak-frequency images,
|
Fast path: if the file contains precomputed peak-frequency images for
|
||||||
and zero-padding is not active, and the bg-sub flag matches, the stored
|
this angle (v5's dense per-file PREC section, or v7's ragged per-angle
|
||||||
images are used directly — no FFT is run.
|
CACH section), and zero-padding is not active, and the bg-sub flag
|
||||||
|
matches, the stored image is used directly — no FFT is run.
|
||||||
|
|
||||||
Otherwise, data is processed in row chunks sized to a fixed memory
|
Otherwise, data is processed in row chunks sized to a fixed memory
|
||||||
budget (see ``_chunk_rows_for``) to bound peak RAM regardless of scan
|
budget (see ``_chunk_rows_for``) to bound peak RAM regardless of scan
|
||||||
@@ -652,15 +748,39 @@ def compute_rf_image(sras: SrasFile, angle_idx: int,
|
|||||||
n_frames = int(sras.n_frames[angle_idx])
|
n_frames = int(sras.n_frames[angle_idx])
|
||||||
data = sras.data[angle_idx]
|
data = sras.data[angle_idx]
|
||||||
|
|
||||||
# ---- Fast path: v5 precomputed images ----------------------------------
|
# ---- Fast path: precomputed images (v5 PREC or v7 CACH) ----------------
|
||||||
|
# v5 stores a dense (n_angles, n_rows, n_frames) ndarray (uniform
|
||||||
|
# geometry); v6/v7 store a ragged list, one entry per angle (None where
|
||||||
|
# that angle hasn't been cached), since geometry varies per angle.
|
||||||
|
freq_store = sras.precomputed_freq_mhz
|
||||||
|
if isinstance(freq_store, list):
|
||||||
|
angle_cached = angle_idx < len(freq_store) and freq_store[angle_idx] is not None
|
||||||
|
else:
|
||||||
|
angle_cached = freq_store is not None
|
||||||
|
|
||||||
can_use_precomputed = (
|
can_use_precomputed = (
|
||||||
sras.precomputed_freq_mhz is not None
|
angle_cached
|
||||||
and n_fft is None # no custom zero-padding
|
and n_fft is None # no custom zero-padding
|
||||||
and sras.precomputed_bg_sub == (apply_bg_sub and sras.background is not None)
|
and sras.precomputed_bg_sub == (apply_bg_sub and sras.background is not None)
|
||||||
)
|
)
|
||||||
if can_use_precomputed:
|
if can_use_precomputed:
|
||||||
freq_img = sras.precomputed_freq_mhz[angle_idx].copy()
|
freq_img = freq_store[angle_idx].copy()
|
||||||
dc4_img = sras.precomputed_dc4_mv[angle_idx]
|
|
||||||
|
# DC4 mask, in priority order: already-cached DC block, caller-
|
||||||
|
# supplied image, or a fresh (cheap — no FFT) recompute.
|
||||||
|
dc4_store = sras.precomputed_dc4_mv
|
||||||
|
if isinstance(dc4_store, list):
|
||||||
|
dc4_img = _cached_dc_mv(sras, angle_idx, CH4_IDX)
|
||||||
|
else:
|
||||||
|
dc4_img = dc4_store[angle_idx] if dc4_store is not None else None
|
||||||
|
if dc4_img is None:
|
||||||
|
if dc4_mv is not None:
|
||||||
|
dc4_img = dc4_mv
|
||||||
|
else:
|
||||||
|
dc4_img = adc_to_mv(compute_dc_image(sras, angle_idx, CH4_IDX),
|
||||||
|
sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX],
|
||||||
|
sras.ch_yzero_mv[CH4_IDX])
|
||||||
|
|
||||||
freq_img[dc4_img < dc_threshold_mv] = 0.0
|
freq_img[dc4_img < dc_threshold_mv] = 0.0
|
||||||
return freq_img
|
return freq_img
|
||||||
|
|
||||||
@@ -897,7 +1017,8 @@ def _compute_angle_alignment(sras: SrasFile, ref_angle_idx: int,
|
|||||||
"""Top-level alignment driver. Runs on a background thread (see
|
"""Top-level alignment driver. Runs on a background thread (see
|
||||||
AngleAlignmentWorker) — deliberately recomputes CH4 DC images from
|
AngleAlignmentWorker) — deliberately recomputes CH4 DC images from
|
||||||
scratch via compute_dc_image rather than reading the GUI-thread
|
scratch via compute_dc_image rather than reading the GUI-thread
|
||||||
_dc_cache dict, mirroring PreprocessWorker's existing precedent."""
|
_dc_cache dict, since background-thread workers must not touch
|
||||||
|
GUI-thread-owned caches (BatchCacheWorker follows the same rule)."""
|
||||||
n = sras.n_angles # already the *complete*-angle count for aborted v6 scans
|
n = sras.n_angles # already the *complete*-angle count for aborted v6 scans
|
||||||
dx_ref, dy_ref = _pixel_pitch_mm(sras, ref_angle_idx)
|
dx_ref, dy_ref = _pixel_pitch_mm(sras, ref_angle_idx)
|
||||||
|
|
||||||
@@ -1037,7 +1158,12 @@ class ComputeWorker(QObject):
|
|||||||
dc4_mv=self._dc4_mv)
|
dc4_mv=self._dc4_mv)
|
||||||
self.finished.emit(img)
|
self.finished.emit(img)
|
||||||
else:
|
else:
|
||||||
# DC channels: convert ADC counts → mV
|
# DC channels: use the v7 cache if this angle is already
|
||||||
|
# stored (already mV), else compute fresh and convert.
|
||||||
|
cached = _cached_dc_mv(self._sras, self._angle, self._ch)
|
||||||
|
if cached is not None:
|
||||||
|
self.finished.emit(cached)
|
||||||
|
return
|
||||||
adc_img = compute_dc_image(self._sras, self._angle, self._ch)
|
adc_img = compute_dc_image(self._sras, self._angle, self._ch)
|
||||||
img = adc_to_mv(adc_img,
|
img = adc_to_mv(adc_img,
|
||||||
self._sras.ch_ymult_mv[self._ch],
|
self._sras.ch_ymult_mv[self._ch],
|
||||||
@@ -1076,6 +1202,11 @@ class DcPrecomputeWorker(QObject):
|
|||||||
for a in range(self._sras.n_angles):
|
for a in range(self._sras.n_angles):
|
||||||
if self._stop:
|
if self._stop:
|
||||||
break
|
break
|
||||||
|
cached_dc3 = _cached_dc_mv(self._sras, a, CH3_IDX)
|
||||||
|
cached_dc4 = _cached_dc_mv(self._sras, a, CH4_IDX)
|
||||||
|
if cached_dc3 is not None and cached_dc4 is not None:
|
||||||
|
self.angle_done.emit(a, cached_dc3, cached_dc4)
|
||||||
|
continue
|
||||||
dc3_mv = adc_to_mv(
|
dc3_mv = adc_to_mv(
|
||||||
compute_dc_image(self._sras, a, CH3_IDX),
|
compute_dc_image(self._sras, a, CH3_IDX),
|
||||||
self._sras.ch_ymult_mv[CH3_IDX],
|
self._sras.ch_ymult_mv[CH3_IDX],
|
||||||
@@ -1092,69 +1223,93 @@ class DcPrecomputeWorker(QObject):
|
|||||||
self.error.emit(str(exc))
|
self.error.emit(str(exc))
|
||||||
|
|
||||||
|
|
||||||
class PreprocessWorker(QObject):
|
class BatchCacheWorker(QObject):
|
||||||
"""Compute all-angle FFT and DC images and write a v5 file.
|
"""Batch-computes and stores DC or FFT images into each of *paths*'s
|
||||||
|
v7 CACH tail, in place — converting v6 sources to v7 on first use, or
|
||||||
|
updating an existing v7 file's cache blocks without disturbing whatever
|
||||||
|
the other block already holds.
|
||||||
|
|
||||||
Emits ``progress(int)`` (0–100) as each angle completes and
|
*mode* is ``"dc"`` (CH3/CH4 mean images) or ``"fft"`` (CH1 peak-frequency
|
||||||
``finished(str)`` with an empty string on success or an error message
|
images, unmasked — masking is applied at display time, same as v5's
|
||||||
on failure. Only used for v2–v5 source files (uniform geometry).
|
PREC convention).
|
||||||
|
|
||||||
|
Emits ``progress(int)`` (0–100, weighted by total angle count across the
|
||||||
|
whole batch), ``file_done(str, str)`` (path, error message or "" on
|
||||||
|
success) after each file so one file's failure doesn't abort the batch,
|
||||||
|
and ``finished()`` once every file has been attempted.
|
||||||
"""
|
"""
|
||||||
progress = pyqtSignal(int) # 0–100
|
progress = pyqtSignal(int)
|
||||||
finished = pyqtSignal(str) # empty = success, else error message
|
file_done = pyqtSignal(str, str)
|
||||||
|
finished = pyqtSignal()
|
||||||
|
|
||||||
def __init__(self, sras: SrasFile, dest_path: str, apply_bg_sub: bool):
|
def __init__(self, paths: list[str], mode: str, apply_bg_sub: bool):
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self._sras = sras
|
self._paths = paths
|
||||||
self._dest_path = dest_path
|
self._mode = mode
|
||||||
self._apply_bg_sub = apply_bg_sub
|
self._apply_bg_sub = apply_bg_sub
|
||||||
|
|
||||||
def run(self):
|
def run(self):
|
||||||
try:
|
# Pass 1: quick open of each file just to weight progress by total
|
||||||
sras = self._sras
|
# angle count. Don't hold every file's memmap open at once — reopen
|
||||||
n = sras.n_angles
|
# fresh per file in pass 2 below.
|
||||||
n_rows0 = int(sras.n_rows[0])
|
total_angles = 0
|
||||||
n_frames0 = int(sras.n_frames[0])
|
for path in self._paths:
|
||||||
freq_images = np.empty((n, n_rows0, n_frames0), dtype=np.float32)
|
try:
|
||||||
dc4_images = np.empty_like(freq_images)
|
total_angles += SrasFile(path).n_angles
|
||||||
dc3_images = np.empty_like(freq_images)
|
except Exception:
|
||||||
|
pass # unreadable files are reported properly in pass 2
|
||||||
|
total_angles = max(total_angles, 1)
|
||||||
|
|
||||||
for aidx in range(n):
|
done_angles = 0
|
||||||
# Compute raw peak-frequency (no DC-threshold masking yet)
|
for path in self._paths:
|
||||||
freq_images[aidx] = compute_rf_image(
|
try:
|
||||||
sras, aidx, dc_threshold_mv=-1e9, # mask nothing
|
sras = SrasFile(path)
|
||||||
apply_bg_sub=self._apply_bg_sub)
|
if sras.version not in (6, 7):
|
||||||
|
self.file_done.emit(
|
||||||
|
path, f"unsupported version {sras.version} — only "
|
||||||
|
"v6/v7 files can be batch-cached")
|
||||||
|
continue
|
||||||
|
|
||||||
# DC images (ADC counts → mV)
|
n = sras.n_angles
|
||||||
dc4_images[aidx] = adc_to_mv(
|
if self._mode == "dc":
|
||||||
compute_dc_image(sras, aidx, CH4_IDX),
|
new_dc3 = [None] * n
|
||||||
sras.ch_ymult_mv[CH4_IDX],
|
new_dc4 = [None] * n
|
||||||
sras.ch_yoff_adc[CH4_IDX],
|
for a in range(n):
|
||||||
sras.ch_yzero_mv[CH4_IDX])
|
new_dc3[a] = adc_to_mv(
|
||||||
dc3_images[aidx] = adc_to_mv(
|
compute_dc_image(sras, a, CH3_IDX),
|
||||||
compute_dc_image(sras, aidx, CH3_IDX),
|
sras.ch_ymult_mv[CH3_IDX], sras.ch_yoff_adc[CH3_IDX],
|
||||||
sras.ch_ymult_mv[CH3_IDX],
|
sras.ch_yzero_mv[CH3_IDX])
|
||||||
sras.ch_yoff_adc[CH3_IDX],
|
new_dc4[a] = adc_to_mv(
|
||||||
sras.ch_yzero_mv[CH3_IDX])
|
compute_dc_image(sras, a, CH4_IDX),
|
||||||
|
sras.ch_ymult_mv[CH4_IDX], sras.ch_yoff_adc[CH4_IDX],
|
||||||
|
sras.ch_yzero_mv[CH4_IDX])
|
||||||
|
done_angles += 1
|
||||||
|
self.progress.emit(int(done_angles / total_angles * 100))
|
||||||
|
sras.write_v7_cache(new_dc3_mv=new_dc3, new_dc4_mv=new_dc4)
|
||||||
|
else:
|
||||||
|
effective_bg = self._apply_bg_sub and sras.background is not None
|
||||||
|
new_freq = [None] * n
|
||||||
|
for a in range(n):
|
||||||
|
new_freq[a] = compute_rf_image(
|
||||||
|
sras, a, dc_threshold_mv=-1e9, # mask nothing
|
||||||
|
apply_bg_sub=effective_bg)
|
||||||
|
done_angles += 1
|
||||||
|
self.progress.emit(int(done_angles / total_angles * 100))
|
||||||
|
sras.write_v7_cache(new_freq_mhz=new_freq, new_bg_sub=effective_bg)
|
||||||
|
|
||||||
self.progress.emit(int((aidx + 1) / n * 90))
|
self.file_done.emit(path, "")
|
||||||
|
except Exception as exc:
|
||||||
|
self.file_done.emit(path, str(exc))
|
||||||
|
|
||||||
bg_sub_flag = self._apply_bg_sub and sras.background is not None
|
self.finished.emit()
|
||||||
sras.write_v5(
|
|
||||||
self._dest_path,
|
|
||||||
freq_images, dc4_images, dc3_images,
|
|
||||||
bg_sub_applied=bg_sub_flag,
|
|
||||||
progress_cb=lambda frac: self.progress.emit(90 + int(frac * 10)),
|
|
||||||
)
|
|
||||||
self.finished.emit("")
|
|
||||||
except Exception as exc:
|
|
||||||
self.finished.emit(str(exc))
|
|
||||||
|
|
||||||
|
|
||||||
class AngleAlignmentWorker(QObject):
|
class AngleAlignmentWorker(QObject):
|
||||||
"""Computes rotation+translation alignment for every angle in *sras*,
|
"""Computes rotation+translation alignment for every angle in *sras*,
|
||||||
referenced to *ref_angle_idx*, from each angle's binarized CH4 mask.
|
referenced to *ref_angle_idx*, from each angle's binarized CH4 mask.
|
||||||
Rotation is analytic (from sras.angles_deg); only translation is found
|
Rotation is analytic (from sras.angles_deg); only translation is found
|
||||||
by phase correlation. Mirrors PreprocessWorker's progress/finished shape.
|
by phase correlation. Uses the same progress(int)/finished(...) worker
|
||||||
|
shape as the other background-thread workers in this file.
|
||||||
"""
|
"""
|
||||||
progress = pyqtSignal(int) # 0–100
|
progress = pyqtSignal(int) # 0–100
|
||||||
finished = pyqtSignal(object, str) # AlignmentResult|None, error msg ("" = success)
|
finished = pyqtSignal(object, str) # AlignmentResult|None, error msg ("" = success)
|
||||||
@@ -1794,7 +1949,11 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._fft_pad_factor: int = 1 # 1 = no padding
|
self._fft_pad_factor: int = 1 # 1 = no padding
|
||||||
self._pending_fft_pad_factor: int = 1
|
self._pending_fft_pad_factor: int = 1
|
||||||
|
|
||||||
self._preprocess_thread: QThread | None = None
|
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
|
||||||
|
self._batch_thread: QThread | None = None
|
||||||
|
self._batch_worker: BatchCacheWorker | None = None
|
||||||
|
self._batch_errors: list[str] = []
|
||||||
|
self._batch_progress_dlg: QProgressDialog | None = None
|
||||||
|
|
||||||
# Display-only settings (colormap, grating) never trigger a
|
# Display-only settings (colormap, grating) never trigger a
|
||||||
# recompute — they're applied to cached data on redraw. DC images
|
# recompute — they're applied to cached data on redraw. DC images
|
||||||
@@ -2110,15 +2269,6 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
fft_act.triggered.connect(self._on_fft_options)
|
fft_act.triggered.connect(self._on_fft_options)
|
||||||
fft_menu.addAction(fft_act)
|
fft_menu.addAction(fft_act)
|
||||||
|
|
||||||
fft_menu.addSeparator()
|
|
||||||
self._preprocess_act = QAction("&Pre-process and Save as v5…", self)
|
|
||||||
self._preprocess_act.setStatusTip(
|
|
||||||
"Compute FFT and DC images for all angles and save to a v5 file "
|
|
||||||
"for instant re-opening (no FFT on load). Not available for v6 files.")
|
|
||||||
self._preprocess_act.setEnabled(False)
|
|
||||||
self._preprocess_act.triggered.connect(self._on_preprocess)
|
|
||||||
fft_menu.addAction(self._preprocess_act)
|
|
||||||
|
|
||||||
fusion_menu = menubar.addMenu("&Fusion")
|
fusion_menu = menubar.addMenu("&Fusion")
|
||||||
self._alignment_act = QAction("Angle &Alignment", self)
|
self._alignment_act = QAction("Angle &Alignment", self)
|
||||||
self._alignment_act.setStatusTip(
|
self._alignment_act.setStatusTip(
|
||||||
@@ -2128,6 +2278,21 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._alignment_act.triggered.connect(self._on_angle_alignment)
|
self._alignment_act.triggered.connect(self._on_angle_alignment)
|
||||||
fusion_menu.addAction(self._alignment_act)
|
fusion_menu.addAction(self._alignment_act)
|
||||||
|
|
||||||
|
convert_menu = menubar.addMenu("&Convert")
|
||||||
|
self._batch_dc_act = QAction("Batch Compute DC and &Store…", self)
|
||||||
|
self._batch_dc_act.setStatusTip(
|
||||||
|
"Select .sras files and compute+store DC images (CH3/CH4 mean) "
|
||||||
|
"for every angle, converting v6 files to v7 in place.")
|
||||||
|
self._batch_dc_act.triggered.connect(lambda: self._on_batch_compute("dc"))
|
||||||
|
convert_menu.addAction(self._batch_dc_act)
|
||||||
|
|
||||||
|
self._batch_fft_act = QAction("Batch Compute FFT and Sto&re…", self)
|
||||||
|
self._batch_fft_act.setStatusTip(
|
||||||
|
"Select .sras files and compute+store FFT peak-frequency images "
|
||||||
|
"for every angle, converting v6 files to v7 in place.")
|
||||||
|
self._batch_fft_act.triggered.connect(lambda: self._on_batch_compute("fft"))
|
||||||
|
convert_menu.addAction(self._batch_fft_act)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Drag-and-drop
|
# Drag-and-drop
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -2265,11 +2430,22 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
)
|
)
|
||||||
if s.background is not None:
|
if s.background is not None:
|
||||||
notes.append(f"Background waveform: {len(s.background)} samples")
|
notes.append(f"Background waveform: {len(s.background)} samples")
|
||||||
if s.precomputed_freq_mhz is not None:
|
if isinstance(s.precomputed_freq_mhz, np.ndarray):
|
||||||
bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)"
|
bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)"
|
||||||
notes.append(f"v5: precomputed images present{bg_note} — display is instant")
|
notes.append(f"v5: precomputed images present{bg_note} — display is instant")
|
||||||
if s.version == 6:
|
if s.version in (6, 7):
|
||||||
notes.append("v6 format: rows / frames / x_start are per-angle")
|
notes.append("v6/v7 format: rows / frames / x_start are per-angle")
|
||||||
|
if s.version == 7:
|
||||||
|
n_dc = sum(1 for x in s.precomputed_dc4_mv if x is not None)
|
||||||
|
n_fft = sum(1 for x in s.precomputed_freq_mhz if x is not None)
|
||||||
|
if n_dc or n_fft:
|
||||||
|
bg_note = " (bg-sub)" if s.precomputed_bg_sub else " (no bg-sub)"
|
||||||
|
notes.append(
|
||||||
|
f"v7: cached DC ({n_dc}/{s.n_angles} angles), "
|
||||||
|
f"FFT ({n_fft}/{s.n_angles} angles{bg_note if n_fft else ''}) "
|
||||||
|
"— display is instant for cached angles")
|
||||||
|
else:
|
||||||
|
notes.append("v7 format: no cache blocks stored yet")
|
||||||
self.lbl_frame_warn.setText("\n".join(notes))
|
self.lbl_frame_warn.setText("\n".join(notes))
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -2299,10 +2475,11 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self.btn_export_csv.setEnabled(is_ch1 and self._current_image is not None)
|
self.btn_export_csv.setEnabled(is_ch1 and self._current_image is not None)
|
||||||
# ROI: always usable once a file is loaded (independent of channel)
|
# ROI: always usable once a file is loaded (independent of channel)
|
||||||
self.btn_draw_roi.setEnabled(enabled and s is not None)
|
self.btn_draw_roi.setEnabled(enabled and s is not None)
|
||||||
# Pre-process: available for v2–v5 files when loaded and not already running
|
# Batch Convert actions: pick their own files, independent of
|
||||||
can_preprocess = (enabled and s is not None and s.version != 6
|
# whatever's currently open — only gated on no batch already running
|
||||||
and self._preprocess_thread is None)
|
can_batch = self._batch_thread is None
|
||||||
self._preprocess_act.setEnabled(can_preprocess)
|
self._batch_dc_act.setEnabled(can_batch)
|
||||||
|
self._batch_fft_act.setEnabled(can_batch)
|
||||||
# Angle alignment: needs >1 angle and no alignment already running
|
# Angle alignment: needs >1 angle and no alignment already running
|
||||||
can_align = (enabled and s is not None and s.n_angles > 1
|
can_align = (enabled and s is not None and s.n_angles > 1
|
||||||
and self._alignment_thread is None)
|
and self._alignment_thread is None)
|
||||||
@@ -2897,75 +3074,90 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
self._progress_dlg = None
|
self._progress_dlg = None
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Pre-process → save v5
|
# Convert menu: batch DC/FFT compute-and-store (v6 -> v7)
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def _on_preprocess(self):
|
def _on_batch_compute(self, mode: str):
|
||||||
if self._sras is None:
|
if self._batch_thread is not None:
|
||||||
return
|
return
|
||||||
if self._preprocess_thread is not None:
|
label = "DC" if mode == "dc" else "FFT"
|
||||||
return
|
paths, _ = QFileDialog.getOpenFileNames(
|
||||||
s = self._sras
|
self, f"Select .sras files to batch-compute {label}", "",
|
||||||
if s.version == 6:
|
|
||||||
self.statusBar().showMessage(
|
|
||||||
"Pre-process to v5 is not supported for v6 files (per-angle geometry).")
|
|
||||||
return
|
|
||||||
default_name = s.path.stem + "_v5" + s.path.suffix
|
|
||||||
dest, _ = QFileDialog.getSaveFileName(
|
|
||||||
self, "Save Pre-processed v5 File",
|
|
||||||
str(s.path.parent / default_name),
|
|
||||||
"SRAS files (*.sras);;All files (*)",
|
"SRAS files (*.sras);;All files (*)",
|
||||||
)
|
)
|
||||||
if not dest:
|
if not paths:
|
||||||
return
|
return
|
||||||
if self._preprocess_thread is not None:
|
if self._batch_thread is not None:
|
||||||
return # a second trigger snuck in while the save dialog was open
|
return # a second trigger snuck in while the file dialog was open
|
||||||
|
|
||||||
apply_bg = self.chk_bg_sub.isChecked() and s.background is not None
|
apply_bg = self.chk_bg_sub.isChecked()
|
||||||
|
self._batch_errors = []
|
||||||
|
|
||||||
n_total = s.n_angles
|
# Claim self._batch_thread before anything below that can pump the
|
||||||
n_px = int(s.n_rows[0]) * int(s.n_frames[0])
|
# Qt event loop — see the comment in _start_compute for why.
|
||||||
approx_mb = n_total * n_px * 3 * 4 / 1e6
|
self._batch_worker = BatchCacheWorker(paths, mode, apply_bg)
|
||||||
|
self._batch_thread = QThread()
|
||||||
|
self._batch_worker.moveToThread(self._batch_thread)
|
||||||
|
self._batch_thread.started.connect(self._batch_worker.run)
|
||||||
|
self._batch_worker.progress.connect(self._on_batch_progress)
|
||||||
|
self._batch_worker.file_done.connect(self._on_batch_file_done)
|
||||||
|
self._batch_worker.finished.connect(lambda paths=paths: self._on_batch_finished(paths))
|
||||||
|
self._batch_worker.finished.connect(self._batch_thread.quit)
|
||||||
|
self._batch_thread.finished.connect(self._on_batch_thread_finished)
|
||||||
|
|
||||||
# Claim self._preprocess_thread before any call below that can pump
|
self._batch_dc_act.setEnabled(False)
|
||||||
# the Qt event loop — see the comment in _start_compute for why.
|
self._batch_fft_act.setEnabled(False)
|
||||||
self._preprocess_worker = PreprocessWorker(s, dest, apply_bg)
|
|
||||||
self._preprocess_thread = QThread()
|
|
||||||
self._preprocess_worker.moveToThread(self._preprocess_thread)
|
|
||||||
self._preprocess_thread.started.connect(self._preprocess_worker.run)
|
|
||||||
self._preprocess_worker.progress.connect(self._on_preprocess_progress)
|
|
||||||
self._preprocess_worker.finished.connect(self._on_preprocess_done)
|
|
||||||
self._preprocess_worker.finished.connect(self._preprocess_thread.quit)
|
|
||||||
self._preprocess_thread.finished.connect(self._on_preprocess_thread_finished)
|
|
||||||
|
|
||||||
self._preprocess_act.setEnabled(False)
|
self._batch_progress_dlg = QProgressDialog(
|
||||||
self._show_progress(
|
f"Batch computing {label} for {len(paths)} file(s)…", "", 0, 100, self)
|
||||||
f"Pre-processing {n_total} angle(s) "
|
self._batch_progress_dlg.setWindowTitle("Please wait…")
|
||||||
f"({int(s.n_rows[0])}×{int(s.n_frames[0])} px each, ~{approx_mb:.0f} MB output)…"
|
self._batch_progress_dlg.setCancelButton(None)
|
||||||
)
|
self._batch_progress_dlg.setWindowModality(Qt.WindowModality.WindowModal)
|
||||||
|
self._batch_progress_dlg.setMinimumDuration(300)
|
||||||
|
self._batch_progress_dlg.show()
|
||||||
|
|
||||||
self._preprocess_thread.start()
|
self._batch_thread.start()
|
||||||
|
|
||||||
def _on_preprocess_progress(self, pct: int):
|
def _on_batch_progress(self, pct: int):
|
||||||
if self._progress_dlg is not None:
|
if self._batch_progress_dlg is not None:
|
||||||
self._progress_dlg.setValue(pct)
|
self._batch_progress_dlg.setValue(pct)
|
||||||
|
|
||||||
def _on_preprocess_done(self, error_msg: str):
|
def _on_batch_file_done(self, path: str, err: str):
|
||||||
self._close_progress()
|
if err:
|
||||||
if error_msg:
|
self._batch_errors.append(f"{Path(path).name} — {err}")
|
||||||
self.statusBar().showMessage(f"Pre-process failed: {error_msg}")
|
if self._batch_progress_dlg is not None:
|
||||||
|
self._batch_progress_dlg.setLabelText(f"Processed {Path(path).name}…")
|
||||||
|
|
||||||
|
def _on_batch_finished(self, paths: list[str]):
|
||||||
|
if self._batch_progress_dlg is not None:
|
||||||
|
self._batch_progress_dlg.close()
|
||||||
|
self._batch_progress_dlg = None
|
||||||
|
|
||||||
|
n_total = len(paths)
|
||||||
|
n_failed = len(self._batch_errors)
|
||||||
|
n_ok = n_total - n_failed
|
||||||
|
if n_failed:
|
||||||
|
summary = (f"Batch store: {n_ok}/{n_total} file(s) updated, "
|
||||||
|
f"{n_failed} failed: {'; '.join(self._batch_errors)}")
|
||||||
else:
|
else:
|
||||||
self.statusBar().showMessage("v5 file written — re-open it for instant display.")
|
summary = f"Batch store: {n_ok}/{n_total} file(s) updated."
|
||||||
|
self.statusBar().showMessage(summary)
|
||||||
|
self._batch_errors = []
|
||||||
|
|
||||||
def _on_preprocess_thread_finished(self):
|
# If the currently-open file was in this batch, reload it so the
|
||||||
|
# GUI picks up the newly-written v7 cache instead of stale state.
|
||||||
|
if self._sras is not None and str(self._sras.path) in paths:
|
||||||
|
self._load_file(str(self._sras.path))
|
||||||
|
|
||||||
|
def _on_batch_thread_finished(self):
|
||||||
# See the comment in _on_compute_thread_finished: wait() before
|
# See the comment in _on_compute_thread_finished: wait() before
|
||||||
# releasing our reference to avoid destroying a QThread whose OS
|
# releasing our reference to avoid destroying a QThread whose OS
|
||||||
# thread hasn't fully joined yet.
|
# thread hasn't fully joined yet.
|
||||||
if self._preprocess_thread is not None:
|
if self._batch_thread is not None:
|
||||||
self._preprocess_thread.wait()
|
self._batch_thread.wait()
|
||||||
self._preprocess_thread = None
|
self._batch_thread = None
|
||||||
self._preprocess_act.setEnabled(
|
self._batch_dc_act.setEnabled(True)
|
||||||
self._sras is not None and self._sras.version != 6)
|
self._batch_fft_act.setEnabled(True)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Fusion: angle alignment
|
# Fusion: angle alignment
|
||||||
@@ -3066,7 +3258,7 @@ class SrasViewerWindow(QMainWindow):
|
|||||||
def closeEvent(self, event):
|
def closeEvent(self, event):
|
||||||
if self._dc_precompute_worker is not None:
|
if self._dc_precompute_worker is not None:
|
||||||
self._dc_precompute_worker.stop()
|
self._dc_precompute_worker.stop()
|
||||||
for attr in ("_load_thread", "_compute_thread", "_preprocess_thread",
|
for attr in ("_load_thread", "_compute_thread", "_batch_thread",
|
||||||
"_dc_precompute_thread", "_alignment_thread"):
|
"_dc_precompute_thread", "_alignment_thread"):
|
||||||
t = getattr(self, attr, None)
|
t = getattr(self, attr, None)
|
||||||
if t is not None:
|
if t is not None:
|
||||||
|
|||||||
Reference in New Issue
Block a user