tka/scanengine-3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ff68901c5722aefb57d6f5a3f11edd146f747a29
scanengine-3/scan_format.md
T
Thomas Ales aa06fa1460 Per-angle background capture: v7/v11 .sras layout 
A multi-angle scan runs for hours, but every angle was referenced against
one background captured before the first row of the first angle. That
reference has drifted by the last angle, and comparing angles — the whole
point of a multi-angle scan — was comparing each one against a noise floor
measured at whichever angle came first.

Every angle now captures its own. Before each angle's rows, the operator is
prompted to switch the Genesis laser off, the engine averages a fresh CH1
record, and the operator switches it back on. The data block therefore reads
[background][scan][background][scan] …, one pair per angle.

Format v7 (scan) and v11 (SAW check) carry the background inside the data
block, one length-prefixed block ahead of each angle's rows; the single
block that sat between the preambles and the data is gone. Per-angle offsets
now come from a walk of the data block at parse time rather than arithmetic
over the geometry table, and an angle whose background is not fully on disk
is the frontier — nothing of it was written yet.

v6/v10 files still read: SrasFile hands their one background to every angle,
so readers never branch on the version. Nothing writes them, and a resume
refuses them, since a re-acquired angle writes a block the old layout has no
room for. A resumed v7 angle rewrites its background in place, and the
engine checks the new block fits the room the file has before writing it —
anything else would shift every row behind it.

Two fixes made along the way:

  * QtScanController never accepted file_version, so every scan launched
    from the app raised TypeError at construction.
  * angle_status() left its cursor parked at the frontier, so every angle
    past it reported the frontier's own data_offset — which handed a resumed
    scan the same write position for several angles. Two recorded offsets in
    tests/golden/sras_expected.json are corrected accordingly.

The v6 goldens stay as parser fixtures; the writer is now locked against
bytes the test lays out from scan_format.md itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 12:44:25 -05:00

362 lines
18 KiB
Markdown
Executable File
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# SRAS Scan Binary Format — Versions 7 and 11 (reading 6 and 10)
Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows. Files are named `{prefix}.sras`.
Two versions are written, sharing this layout byte for byte — only the version
field differs, and with it what the file means:
| Version | Meaning | Rows per angle |
|---------|---------|----------------|
| 7 | A full scan. | Whatever the ROI needs. |
| 11 | A middle-row SAW quality check (`{prefix}-sawcheck.sras`). | Exactly 1. |
See [SAW Quality Check (v11)](#saw-quality-check-v11) below.
**Versions 6 and 10** are the same two files as they were written before each
angle carried its own background: a single background block sat between the
preambles and the data block, and the data block held nothing but rows. They
are still read — see [Legacy layout (v6/v10)](#legacy-layout-v6v10) — but
nothing writes them any more, and a v6 file cannot be resumed into, since the
background block a resumed angle writes has no room in it.
Starting in v6, each angle only scans the **bounding box of the nominal ROI
rotated by that specific angle** — not the worst case across all angles — so
`x_start`, `x_delta` (and therefore `n_frames`, the points/row count) and
`n_rows` all vary per angle. A 0°/180° scan of a wide, short ROI needs far
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.
---
## File Layout
```
[Global Header — 49 bytes]
[Angle Table — n_angles × 4 bytes (float32 per angle, degrees)]
[Per-Angle Geometry Table— n_angles × 14 bytes (x_start f32, x_delta f32, n_frames u32, n_rows u16)]
[Row Table (ragged) — sum(n_rows) × 4 bytes (float32 per row, angle-major)]
[Preamble Blocks — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Data Block (ragged) — per angle: [Background Block][Waveform Data]]
[Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data — n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
```
So the data block reads `[background][scan][background][scan] …`, one pair per
angle, in angle-table order.
All multi-byte integers and floats use **big-endian** byte order
(`>` in Python's `struct` module).
---
## Global Header (49 bytes)
| Offset | Size | Type | Field | Description |
|--------|------|-----------|--------------------|--------------------------------------------------|
| 0 | 4 | `4s` | `magic` | Always `SRAS` (0x53 0x52 0x41 0x53) |
| 4 | 1 | `uint8` | `version` | Format version — `7` (scan) or `11` (SAW check) |
| 5 | 2 | `uint16` | `n_angles` | Number of GR rotation angles |
| 7 | 4 | `float32` | `x_start_nominal` | Nominal (pre-rotation) X scan start, mm |
| 11 | 4 | `float32` | `y_start_nominal` | Nominal (pre-rotation) Y scan start, mm |
| 15 | 4 | `float32` | `x_delta_nominal` | Nominal (pre-rotation) X scan width, mm |
| 19 | 4 | `float32` | `y_delta_nominal` | Nominal (pre-rotation) Y scan height, mm |
| 23 | 4 | `float32` | `row_spacing_mm` | Y spacing between rows, mm |
| 27 | 4 | `float32` | `velocity_mm_s` | Stage scan velocity in mm/s |
| 31 | 4 | `float32` | `laser_freq_hz` | Laser repetition rate in Hz |
| 35 | 4 | `uint32` | `samples_per_frame`| Time samples per waveform |
| 39 | 8 | `float64` | `sample_rate_hz` | Oscilloscope sample rate in Hz (e.g. 6.25e9) |
| 47 | 1 | `uint8` | `bytes_per_sample` | Bytes per ADC sample: `1` = int8, `2` = int16 |
| 48 | 1 | `uint8` | `n_channels` | Number of channels recorded (currently `3`) |
**Total header size:** 49 bytes — verified:
`struct.calcsize(">4sBHfffffffIdBB") == 49`.
The `*_nominal` fields describe the ROI as originally entered on the New Scan
page (XS/YS/XD/YD), **before** per-angle bounding-box expansion. They are for
reference/reconstruction only — the actual per-angle scan geometry used for
acquisition is in the Per-Angle Geometry Table below.
---
## Angle Table
Immediately after the header: **n_angles** big-endian float32 values, one per
GR angle (degrees, signed; magnitude 0–180, sign gives physical rotation
direction — negative for the current CW-rotating GR stage).
```
angle[0], angle[1], …, angle[n_angles - 1]
```
---
## Per-Angle Geometry Table
Immediately after the angle table: **n_angles** fixed-size records, one per
angle (same order as the angle table), each 14 bytes:
| Size | Type | Field | Description |
|------|-----------|------------|-------------------------------------------------------|
| 4 | `float32` | `x_start` | X scan start for this angle's bounding box, mm |
| 4 | `float32` | `x_delta` | X scan width for this angle's bounding box, mm |
| 4 | `uint32` | `n_frames` | A-scans per row for this angle (FastFrame count) |
| 2 | `uint16` | `n_rows` | Number of Y rows scanned for this angle |
Format string per record: `">ffIH"`.
---
## Row Table (ragged)
Immediately after the per-angle geometry table: for each angle in order,
that angle's `n_rows` big-endian float32 Y positions (mm), concatenated with
no padding between angles.
```
# angle 0's rows, then angle 1's rows, …
y_mm[0][0], …, y_mm[0][n_rows[0]-1], y_mm[1][0], …, y_mm[n_angles-1][n_rows[-1]-1]
```
Row-table boundaries for angle *a* are derived from the per-angle geometry
table: `sum(n_rows[0:a])` gives the starting index into the flattened array.
---
## Preamble Blocks
Immediately after the row table: **n_channels** length-prefixed UTF-8 strings,
one per channel in `SCAN_CHANNELS` order (CH1, CH3, CH4). Each block is:
```
uint16 length — byte length of the following UTF-8 string
bytes preamble — WFMOutpre response string from the oscilloscope
```
The preamble captures per-channel scaling constants (YMULT, YOFF, YZERO) needed
to convert raw ADC values to volts.
---
## Data Block (ragged)
Immediately after the preamble blocks, and running to the end of the file:
for each angle in angle-table order, that angle's **background block**
followed by that angle's **waveform data**.
```
for angle a in 0 … n_angles-1:
uint32 n_bg_samples # background block
int8[] bg_data
for row in 0 … n_rows[a]-1: # waveform data
for channel in [CH1, CH3, CH4]: # 3 channels, fixed order
for frame in 0 … n_frames[a]-1:
samples[0 … samples_per_frame-1] # bps bytes each
```
### Background block
One CH1 waveform captured with the **Helios (generation) laser enabled** and
the **Genesis (detection) laser disabled**, averaged over 1024 shots
(`core/scope_sras.py`, `BACKGROUND_AVERAGES`). It is a noise/background
reference for subtraction during post-processing.
```
uint32 n_bg_samples — number of samples in the background waveform
int8[] bg_data — raw ADC samples (same encoding as waveform data)
```
`n_bg_samples` equals `samples_per_frame` under normal acquisition settings,
but is **not** assumed to: readers take the per-angle offsets from a walk of
the data block, reading each length prefix as they go, rather than from
arithmetic over the geometry table alone.
Every angle carries its own. The operator is prompted to switch the Genesis
laser off before each angle and back on after the capture, so the reference is
taken minutes from the data it will be subtracted from — a multi-angle scan
runs for hours, and one background captured at the first angle has drifted by
the last. It also makes the angles comparable, which is the entire point of a
multi-angle scan: each is referenced against its own noise floor rather than
against whichever angle happened to be scanned first.
### Waveform data
Stored in **angle-major, row-minor** order, and each angle contributes a
different number of rows (`n_rows[a]`) and a different number of frames per
row (`n_frames[a]`), both taken from that angle's Per-Angle Geometry Table
entry. Within each row, channels are interleaved in ascending channel-index
order, with each channel's FastFrame data written in frame order.
Each sample is a raw signed ADC value. With `bytes_per_sample = 1` this is
**int8** (−128 … +127). With `bytes_per_sample = 2` this is **big-endian
int16**.
Total data-block size:
```
sum over angles a of: 4 + n_bg_samples[a]
+ n_rows[a] × 3 × n_frames[a] × samples_per_frame × bytes_per_sample
```
> **Incomplete files:** If a scan is aborted the file is closed immediately and
> the data block will be shorter than the expected size. Readers should walk
> the data block from its start — background length prefix, then that angle's
> declared row bytes from the Per-Angle Geometry Table — checking `file_size`
> against the running total before reshaping. A fixed
> `(n_angles, n_rows, ...)` reshape (as in pre-v6 readers) will not work since
> row/frame counts are no longer uniform across angles.
>
> An angle whose background block is not fully on disk has nothing of itself
> written yet: it is *missing*, not truncated, and the walk continues past it
> assuming the block a writer would have produced
> (`4 + samples_per_frame` bytes), which is where a resumed scan writes.
---
## Spatial Mapping
The *k*-th waveform (frame) in a row corresponds to the *k*-th laser pulse that
hit the sample. For a row belonging to angle *a*, the physical X position of
that pulse is:
```
x_k = x_start[a] + k * (velocity_mm_s / laser_freq_hz)
```
using that angle's `x_start` from the Per-Angle Geometry Table (not
`x_start_nominal`).
---
## Acquisition Settings (fixed by core/scope_sras.py)
| Parameter | Value |
|-----------------------|------------------------------------------|
| Background trigger | CH2, rising edge, 0.500 V (`TRIG_LEVEL_V`), FastFrame off |
| Scan trigger | Logic AND, CH2 HIGH ∧ CH3 HIGH, 0.500 V |
| Background average | 1024 shots (`BACKGROUND_AVERAGES`), once per angle |
| Horizontal position | 30 (`HORizontal:POSition`) |
| Sample rate | 6.25 GS/s (160 ps/sample) |
| Transfer format | `DATa:ENCdg RIBinary`, `DATa:WIDth 1` |
| Channels recorded | CH1, CH3, CH4 |
| Stage X velocity | 100 mm/s |
| Stage X acceleration | 1500 mm/s² |
| Stage X trigger out | Logic-high at max velocity (`TRIGOUT_MAXV`) |
| Acquisition mode | FastFrame, Normal trigger |
None of these are stored in the file, so they do not affect byte layout — but
they do set where the acoustic packet lands inside each frame. Read them from
`core/scope_sras.py`; earlier revisions of this table drifted from the code.
---
## Acquisition Paths
Two acquisition strategies write **byte-identical** files; the choice is a
runtime flag (`ScanEngine(burst_mode=…)`, exposed as a checkbox in the app) and
is not recorded in the file.
| | Per-row (default) | Burst |
|---|---|---|
| FastFrame acquisitions | one per row | one per `floor(max_frames / n_frames)` rows |
| Curve transfers | one per channel per row | one per channel per burst |
| Stage X trigger out | armed for the whole scan | armed per acquiring pass, dropped for the flyback |
Both paths take the same per-angle background: the scope returns to the
single-record edge trigger for the capture and back to the logic-AND trigger
before the angle's rows, so the two paths still produce byte-identical files.
Burst mode runs a single acquisition across several rows, so the return move
must not trigger: the trigger output is dropped before each flyback and
re-armed for each acquiring pass. Row boundaries inside the burst come from
`ACQuire:NUMFRAMESACQuired?` sampled after each pass — the burst itself carries
no row markers. See `core/scope_burst.py`.
### Row packing
The format has no per-row length field, so a row that over- or under-triggers
cannot be written as it arrived — that would shift every later row. Two
policies are selectable (`ScanEngine(strict_rows=…)`, a checkbox in the app),
and the choice is not recorded in the file:
| | Pad (default) | Strict |
|---|---|---|
| Short row | zero-padded to `n_frames`, warned | scan stops |
| Long row | trailing frames dropped, warned | scan stops |
Pad keeps a scan running through an occasional mis-trigger, at the cost that
the affected row is indistinguishable from a good one afterwards — nothing in
the file records that it was padded. Strict is for data runs where that
ambiguity is worse than a failed scan: it aborts before writing the row, so
the file always ends on a whole-row boundary.
---
## SAW Quality Check (v11)
A full multi-angle scan takes hours, and a rig whose angles disagree produces
all of them before anyone finds out. The SAW quality check acquires **one row
per angle — the row-wise middle of the ROI** — and writes it as a v11 file.
The cost is one row-time per angle instead of `n_rows` of them.
Nothing about the byte layout changes. A v11 file is a v7 file in which every
angle's Per-Angle Geometry Table entry declares `n_rows = 1`, and its Row Table
holds that angle's single middle Y position. Every v7 reader that works from
the geometry table (rather than assuming a uniform shape) reads a v11 file
unchanged. Each angle still carries its own background, so a check costs the
same two operator prompts per angle a scan does.
The version byte earns its keep because the two are otherwise
indistinguishable: **a v7 scan aborted after its first row is not a check**,
even though both hold one row per angle. A reader that guessed from the row
count would treat a failed scan as a deliberate measurement.
Why the middle row in particular: `core/scan_geometry.py` centres every
angle's rotated bounding box on the same nominal ROI centre, so each angle's
middle row crosses that one point on the sample. All the angles therefore
measure the same material, and a spread in their SAW frequencies is a property
of the rig — which is what makes it an alignment check. `saw_check_viewer.py`
plots every angle's frequency on one graph for exactly that comparison.
Writers must honour the one-row rule; `core.sras_format.create_scan_file`
refuses a v11 write for any plan that breaks it. Producing the plan is
`core.saw_check.middle_row_plan(plan)`, and `n_rows // 2` is the middle-row
rule (the upper of the two central rows when the count is even).
---
## Legacy layout (v6/v10)
A v6 or v10 file differs in one place: the background block sits **once**,
between the preamble blocks and the data block, and the data block is
waveform data alone.
```
[Preamble Blocks]
[Background Block — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged) — per angle: rows, as above, with no background between]
```
Everything else — header, tables, row order, spatial mapping — is identical,
which is why `core.sras_format.SrasFile` reads both: it hands the one legacy
background to every angle, so a reader that asks for angle *a*'s background
never has to know which layout it is looking at. Nothing writes v6/v10 any
more, and a resume refuses them, because a re-acquired angle writes a
background block the layout has no room for.
---
## Version History
| Version | Change |
|---------|--------|
| 1 | One file per row; header included `angle_idx`, `row_idx`, `angle_deg`, `y_mm`. |
| 2 | One file per scan; global header with `n_angles`/`n_rows`; separate angle and row tables; three channels (CH1, CH3, CH4) per row. |
| 3 | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4 | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5 | (skipped) |
| 6 | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with v4 readers** (e.g. `sras_viewer.py`, which has not yet been updated for v6). |
| 7 | Background moved into the data block, one per angle: the block now reads `[background][scan][background][scan] …`. Each angle is preceded by its own `uint32` + `int8[]` background, captured (Genesis off, Helios on) just before that angle is scanned, so the reference is contemporary with the data and the angles are comparable to each other. Per-angle offsets therefore come from a walk of the data block rather than arithmetic over the geometry table. **v6 files still read; v6 files cannot be resumed into.** |
| 8–9 | (skipped) |
| 10 | Middle-row SAW quality check on the v6 layout. Byte layout identical to v6, with every angle declaring exactly one row — the row-wise middle of the ROI. Superseded by v11; still read. |
| 11 | Middle-row SAW quality check on the v7 layout: identical to v7 with every angle declaring exactly one row, per-angle backgrounds included. The version byte exists so a check is not confused with a scan aborted after its first row. Written by the main app's *SAW Quality Check*, read by `saw_check_viewer.py`. |
Reference in New Issue View Git Blame Copy Permalink