Per-angle background capture: v7/v11 .sras layout
A multi-angle scan runs for hours, but every angle was referenced against
one background captured before the first row of the first angle. That
reference has drifted by the last angle, and comparing angles — the whole
point of a multi-angle scan — was comparing each one against a noise floor
measured at whichever angle came first.
Every angle now captures its own. Before each angle's rows, the operator is
prompted to switch the Genesis laser off, the engine averages a fresh CH1
record, and the operator switches it back on. The data block therefore reads
[background][scan][background][scan] …, one pair per angle.
Format v7 (scan) and v11 (SAW check) carry the background inside the data
block, one length-prefixed block ahead of each angle's rows; the single
block that sat between the preambles and the data is gone. Per-angle offsets
now come from a walk of the data block at parse time rather than arithmetic
over the geometry table, and an angle whose background is not fully on disk
is the frontier — nothing of it was written yet.
v6/v10 files still read: SrasFile hands their one background to every angle,
so readers never branch on the version. Nothing writes them, and a resume
refuses them, since a re-acquired angle writes a block the old layout has no
room for. A resumed v7 angle rewrites its background in place, and the
engine checks the new block fits the room the file has before writing it —
anything else would shift every row behind it.
Two fixes made along the way:
* QtScanController never accepted file_version, so every scan launched
from the app raised TypeError at construction.
* angle_status() left its cursor parked at the frontier, so every angle
past it reported the frontier's own data_offset — which handed a resumed
scan the same write position for several angles. Two recorded offsets in
tests/golden/sras_expected.json are corrected accordingly.
The v6 goldens stay as parser fixtures; the writer is now locked against
bytes the test lays out from scan_format.md itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Thomas Ales
+147
-43
@@ -1,4 +1,5 @@
|
||||
"""SRAS binary scan-file format (v6 and v10) — the single implementation.
|
||||
"""SRAS binary scan-file format (v7/v11, reading v6/v10 too) — the single
|
||||
implementation.
|
||||
|
||||
Full byte-level spec: scan_format.md. Summary:
|
||||
|
||||
@@ -9,20 +10,33 @@ Full byte-level spec: scan_format.md. Summary:
|
||||
geometry table n_angles × >ffIH (x_start x_delta n_frames n_rows)
|
||||
row tables (ragged) per angle: n_rows × >f (y positions)
|
||||
preambles n_channels × (>H length + utf-8 WFMOutpre string)
|
||||
background block >I length + raw int8 CH1 average
|
||||
waveform data angle-major, row-minor, channel-inner:
|
||||
for each angle, for each row, for each channel,
|
||||
data block per angle: [background][rows]
|
||||
background = >I length + raw int8 CH1 average
|
||||
rows = for each row, for each channel,
|
||||
n_frames × samples_per_frame × bytes_per_sample
|
||||
|
||||
Every angle carries its own background: the operator switches the Genesis
|
||||
laser off before each angle and the engine averages a fresh CH1 record, so
|
||||
the reference a reader subtracts was taken minutes — not hours — from the
|
||||
data it is subtracted from. That is the whole difference between v7 and v6,
|
||||
which held a single background for the entire file, ahead of the data block.
|
||||
|
||||
Incomplete files are valid: the data block is one contiguous append-only
|
||||
stream, so the readable prefix defines a single frontier past which nothing
|
||||
has been written yet (see ``SrasFile.angle_status``).
|
||||
has been written yet (see ``SrasFile.angle_status``). Because the background
|
||||
blocks are length-prefixed, the per-angle offsets come from a walk of that
|
||||
stream at parse time rather than from arithmetic on the geometry table.
|
||||
|
||||
Version 10 is the SAW quality check (core.saw_check): byte layout identical
|
||||
to v6, but every angle declares exactly one row — the row-wise middle of the
|
||||
Version 11 is the SAW quality check (core.saw_check): byte layout identical
|
||||
to v7, but every angle declares exactly one row — the row-wise middle of the
|
||||
ROI. The version byte is the whole difference, and it exists so a reader can
|
||||
tell a one-row-per-angle check from a full scan that was aborted after its
|
||||
first row. ``create_scan_file`` enforces the one-row rule at write time.
|
||||
|
||||
v6 and v10 are the pre-per-angle-background versions of the same two files.
|
||||
They are still read (every scan taken before this change is one); nothing
|
||||
writes them any more, and a v6 file cannot be resumed into, since appending
|
||||
v7 blocks to it would shift its data.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -37,14 +51,21 @@ import numpy as np
|
||||
from core.scan_geometry import AngleGeometry, ScanPlan
|
||||
|
||||
MAGIC = b"SRAS"
|
||||
VERSION = 6
|
||||
VERSION = 7
|
||||
# One row per angle, taken from the middle of the ROI — see core.saw_check.
|
||||
VERSION_SAW_CHECK = 10
|
||||
SUPPORTED_VERSIONS = (VERSION, VERSION_SAW_CHECK)
|
||||
VERSION_SAW_CHECK = 11
|
||||
# v6/v10: the same two files with one background for the whole scan, written
|
||||
# ahead of the data block instead of once per angle. Read-only.
|
||||
LEGACY_VERSIONS = (6, 10)
|
||||
WRITABLE_VERSIONS = (VERSION, VERSION_SAW_CHECK)
|
||||
SUPPORTED_VERSIONS = tuple(sorted(WRITABLE_VERSIONS + LEGACY_VERSIONS))
|
||||
SAW_CHECK_VERSIONS = (10, VERSION_SAW_CHECK)
|
||||
HDR_FMT = ">4sBHfffffffIdBB"
|
||||
HDR_SIZE = struct.calcsize(HDR_FMT) # 49 bytes
|
||||
GEOM_FMT = ">ffIH"
|
||||
GEOM_SIZE = struct.calcsize(GEOM_FMT) # 14 bytes
|
||||
BG_LEN_FMT = ">I"
|
||||
BG_LEN_SIZE = struct.calcsize(BG_LEN_FMT) # 4 bytes
|
||||
|
||||
# Oscilloscope channels recorded, in on-disk order.
|
||||
SCAN_CHANNELS = [1, 3, 4]
|
||||
@@ -78,7 +99,8 @@ class AngleStatus:
|
||||
angle_deg: float
|
||||
n_rows: int # declared
|
||||
row_bytes: int
|
||||
data_offset: int
|
||||
bg_offset: int # start of this angle's background block
|
||||
data_offset: int # start of its rows, i.e. just past that block
|
||||
n_rows_available: int
|
||||
status: str # STATUS_OK / STATUS_TRUNCATED / STATUS_MISSING
|
||||
|
||||
@@ -86,33 +108,38 @@ class AngleStatus:
|
||||
def complete(self) -> bool:
|
||||
return self.status == STATUS_OK
|
||||
|
||||
@property
|
||||
def bg_bytes(self) -> int:
|
||||
"""Size of the background block ahead of the rows (0 on v6/v10)."""
|
||||
return self.data_offset - self.bg_offset
|
||||
|
||||
|
||||
def create_scan_file(path: Path, plan: ScanPlan, samples_per_frame: int,
|
||||
sample_rate: float, preambles: list[str],
|
||||
background_waveform: bytes,
|
||||
version: int = VERSION) -> BinaryIO:
|
||||
"""Create a new .sras file and write the header + tables.
|
||||
|
||||
``version`` selects which kind of file this is — VERSION for a full scan,
|
||||
VERSION_SAW_CHECK for a middle-row quality check. The layout is the same
|
||||
either way; the one-row-per-angle rule that gives v10 its meaning is
|
||||
checked here, since nothing downstream can recover from a v10 file that
|
||||
either way; the one-row-per-angle rule that gives v11 its meaning is
|
||||
checked here, since nothing downstream can recover from a v11 file that
|
||||
breaks it.
|
||||
|
||||
Returns an open binary file positioned at the start of the data block;
|
||||
the caller appends waveform rows and must close it (try/finally).
|
||||
Returns an open binary file positioned at the start of the data block.
|
||||
The caller writes each angle as ``write_background_block()`` followed by
|
||||
that angle's rows, and must close the file (try/finally).
|
||||
"""
|
||||
if version not in SUPPORTED_VERSIONS:
|
||||
if version not in WRITABLE_VERSIONS:
|
||||
raise ValueError(
|
||||
f"Cannot write SRAS format version {version} "
|
||||
f"(supported: {', '.join(str(v) for v in SUPPORTED_VERSIONS)})"
|
||||
f"(writable: {', '.join(str(v) for v in WRITABLE_VERSIONS)})"
|
||||
)
|
||||
if version == VERSION_SAW_CHECK:
|
||||
bad = [f"{pa.angle_deg:.1f}° has {pa.n_rows}"
|
||||
for pa in plan.per_angle if pa.n_rows != 1]
|
||||
if bad:
|
||||
raise ValueError(
|
||||
"A v10 SAW-check file holds exactly one row per angle, but "
|
||||
"A v11 SAW-check file holds exactly one row per angle, but "
|
||||
+ ", ".join(bad) + " — build the plan with "
|
||||
"core.saw_check.middle_row_plan()."
|
||||
)
|
||||
@@ -140,19 +167,33 @@ def create_scan_file(path: Path, plan: ScanPlan, samples_per_frame: int,
|
||||
enc = p.encode("utf-8")
|
||||
f.write(struct.pack(">H", len(enc)))
|
||||
f.write(enc)
|
||||
f.write(struct.pack(">I", len(background_waveform)))
|
||||
f.write(background_waveform)
|
||||
return f
|
||||
|
||||
|
||||
def write_background_block(f: BinaryIO, waveform: bytes) -> int:
|
||||
"""Write one angle's background block; returns the bytes written.
|
||||
|
||||
Every angle's rows are preceded by one of these, so a reader walking the
|
||||
data block knows where that angle's frames start.
|
||||
"""
|
||||
f.write(struct.pack(BG_LEN_FMT, len(waveform)))
|
||||
f.write(waveform)
|
||||
return BG_LEN_SIZE + len(waveform)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SrasFile:
|
||||
"""Parsed .sras file (v6 or v10): header, tables, and lazy (memmap) access.
|
||||
"""Parsed .sras file (v7/v11, or legacy v6/v10): header, tables, and lazy
|
||||
(memmap) access.
|
||||
|
||||
Parsing reads only the header/tables — never the waveform block — so
|
||||
opening a multi-GB file is cheap. ``load_angle``/``load_row`` return
|
||||
read-only numpy views backed by a shared mmap; no data is copied until
|
||||
the caller computes on it.
|
||||
Parsing reads only the header/tables and the per-angle background blocks
|
||||
— never the waveform block — so opening a multi-GB file is cheap.
|
||||
``load_angle``/``load_row`` return read-only numpy views backed by a
|
||||
shared mmap; no data is copied until the caller computes on it.
|
||||
|
||||
``backgrounds[i]`` is angle *i*'s own background (v7/v11) or the file's
|
||||
single background repeated for every angle (v6/v10), so a reader never
|
||||
has to branch on the version to subtract the right one.
|
||||
"""
|
||||
path: Path
|
||||
version: int = field(init=False)
|
||||
@@ -160,7 +201,7 @@ class SrasFile:
|
||||
per_angle: list[AngleGeometry] = field(init=False)
|
||||
preambles: list[str] = field(init=False)
|
||||
preambles_raw: list[bytes] = field(init=False)
|
||||
background: bytes = field(init=False)
|
||||
backgrounds: list[bytes] = field(init=False)
|
||||
data_start_offset: int = field(init=False)
|
||||
file_size: int = field(init=False)
|
||||
|
||||
@@ -215,15 +256,65 @@ class SrasFile:
|
||||
self.preambles_raw.append(f.read(plen))
|
||||
self.preambles = [p.decode("utf-8", errors="replace") for p in self.preambles_raw]
|
||||
|
||||
(n_bg,) = struct.unpack(">I", f.read(4))
|
||||
self.background = f.read(n_bg)
|
||||
shared_bg = None
|
||||
if self.is_legacy_layout:
|
||||
# v6/v10: one background for the whole file, ahead of the data.
|
||||
(n_bg,) = struct.unpack(BG_LEN_FMT, f.read(BG_LEN_SIZE))
|
||||
shared_bg = f.read(n_bg)
|
||||
|
||||
self.data_start_offset = f.tell()
|
||||
self._walk_data_block(f, shared_bg)
|
||||
|
||||
def _walk_data_block(self, f, shared_bg: bytes | None):
|
||||
"""Locate every angle's background block and the rows behind it.
|
||||
|
||||
v7 interleaves a length-prefixed background ahead of each angle's
|
||||
rows, so the offsets are no longer pure arithmetic over the geometry
|
||||
table — the walk reads each prefix as it goes. Past the frontier of
|
||||
a partial file there is nothing to read, so the remaining offsets are
|
||||
predicted from the block a writer would have produced (a full record),
|
||||
which is exactly where a resumed scan writes.
|
||||
"""
|
||||
expected_bg = BG_LEN_SIZE + self.header.samples_per_frame
|
||||
self.backgrounds, self._bg_present = [], []
|
||||
self._bg_offsets, self._data_offsets = [], []
|
||||
cursor = self.data_start_offset
|
||||
for ai, pa in enumerate(self.per_angle):
|
||||
if shared_bg is not None:
|
||||
bg, bg_bytes, present = shared_bg, 0, True
|
||||
else:
|
||||
bg, bg_bytes, present = self._read_background(f, cursor, expected_bg)
|
||||
self.backgrounds.append(bg)
|
||||
self._bg_present.append(present)
|
||||
self._bg_offsets.append(cursor)
|
||||
cursor += bg_bytes
|
||||
self._data_offsets.append(cursor)
|
||||
cursor += self.row_bytes(ai) * pa.n_rows
|
||||
|
||||
def _read_background(self, f, offset: int, expected_bytes: int):
|
||||
"""One angle's background block as (waveform, block_bytes, present).
|
||||
|
||||
A block that runs past the end of the file was never written: the
|
||||
walk keeps going with the size a writer would have used, and the
|
||||
angle is reported MISSING.
|
||||
"""
|
||||
if offset + BG_LEN_SIZE > self.file_size:
|
||||
return b"", expected_bytes, False
|
||||
f.seek(offset)
|
||||
(n_bg,) = struct.unpack(BG_LEN_FMT, f.read(BG_LEN_SIZE))
|
||||
if offset + BG_LEN_SIZE + n_bg > self.file_size:
|
||||
return b"", expected_bytes, False
|
||||
return f.read(n_bg), BG_LEN_SIZE + n_bg, True
|
||||
|
||||
@property
|
||||
def is_saw_check(self) -> bool:
|
||||
"""True for a v10 middle-row SAW quality check rather than a scan."""
|
||||
return self.version == VERSION_SAW_CHECK
|
||||
"""True for a middle-row SAW quality check rather than a scan."""
|
||||
return self.version in SAW_CHECK_VERSIONS
|
||||
|
||||
@property
|
||||
def is_legacy_layout(self) -> bool:
|
||||
"""True for v6/v10: one background for the file, not one per angle."""
|
||||
return self.version in LEGACY_VERSIONS
|
||||
|
||||
# ── Frontier / truncation analysis ───────────────────────────────────────
|
||||
|
||||
@@ -237,40 +328,48 @@ class SrasFile:
|
||||
|
||||
Because the data is one contiguous append-only stream, once an angle
|
||||
is found short every later angle is necessarily absent too — there is
|
||||
a single frontier past which nothing has been written yet.
|
||||
a single frontier past which nothing has been written yet. An angle
|
||||
whose background block never made it to disk is short by definition,
|
||||
even though no row of it was due yet.
|
||||
"""
|
||||
statuses = []
|
||||
cursor = self.data_start_offset
|
||||
frontier_seen = False
|
||||
for ai, pa in enumerate(self.per_angle):
|
||||
row_bytes = self.row_bytes(ai)
|
||||
data_offset = cursor
|
||||
if frontier_seen:
|
||||
data_offset = self._data_offsets[ai]
|
||||
if frontier_seen or not self._bg_present[ai]:
|
||||
n_rows_available = 0
|
||||
status = STATUS_MISSING
|
||||
frontier_seen = True
|
||||
else:
|
||||
declared_bytes = row_bytes * pa.n_rows
|
||||
if row_bytes > 0 and cursor + declared_bytes <= self.file_size:
|
||||
if row_bytes > 0 and data_offset + declared_bytes <= self.file_size:
|
||||
n_rows_available = pa.n_rows
|
||||
status = STATUS_OK
|
||||
cursor += declared_bytes
|
||||
else:
|
||||
remaining = max(0, self.file_size - cursor)
|
||||
remaining = max(0, self.file_size - data_offset)
|
||||
n_rows_available = remaining // row_bytes if row_bytes > 0 else 0
|
||||
status = STATUS_MISSING if n_rows_available == 0 else STATUS_TRUNCATED
|
||||
frontier_seen = True
|
||||
statuses.append(AngleStatus(
|
||||
index=ai, angle_deg=pa.angle_deg, n_rows=pa.n_rows,
|
||||
row_bytes=row_bytes, data_offset=data_offset,
|
||||
row_bytes=row_bytes, bg_offset=self._bg_offsets[ai],
|
||||
data_offset=data_offset,
|
||||
n_rows_available=n_rows_available, status=status,
|
||||
))
|
||||
return statuses
|
||||
|
||||
def angle_data_offset(self, angle_idx: int) -> int:
|
||||
offset = self.data_start_offset
|
||||
for ai in range(angle_idx):
|
||||
offset += self.row_bytes(ai) * self.per_angle[ai].n_rows
|
||||
return offset
|
||||
"""Where angle ``angle_idx``'s rows start (past its background)."""
|
||||
return self._data_offsets[angle_idx]
|
||||
|
||||
def angle_block_offset(self, angle_idx: int) -> int:
|
||||
"""Where angle ``angle_idx``'s block starts, background included.
|
||||
|
||||
Equal to ``angle_data_offset`` on v6/v10, which have no per-angle
|
||||
background block.
|
||||
"""
|
||||
return self._bg_offsets[angle_idx]
|
||||
|
||||
# ── Lazy data access ─────────────────────────────────────────────────────
|
||||
|
||||
@@ -286,6 +385,11 @@ class SrasFile:
|
||||
def _dtype(self) -> np.dtype:
|
||||
return np.dtype(np.int16 if self.header.bytes_per_sample == 2 else np.int8)
|
||||
|
||||
def background_array(self, angle_idx: int) -> np.ndarray | None:
|
||||
"""One angle's background as float32 ADC counts, or None if absent."""
|
||||
bg = np.frombuffer(self.backgrounds[angle_idx], dtype=self._dtype())
|
||||
return bg.astype(np.float32) if bg.size else None
|
||||
|
||||
def load_angle(self, angle_idx: int, n_rows: int | None = None) -> np.ndarray:
|
||||
"""Read-only view of one angle's data block, shape
|
||||
(n_rows, n_channels, n_frames, samples_per_frame).
|
||||
|
||||
Reference in New Issue
Block a user