scanengine-3/scan_format.md

# SRAS Scan Binary Format — Versions 6 and 10

Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows.  Files are named `{prefix}.sras`.

Two versions share this layout byte for byte — only the version field differs,
and with it what the file means:

| Version | Meaning | Rows per angle |
|---------|---------|----------------|
| 6  | A full scan. | Whatever the ROI needs. |
| 10 | A middle-row SAW quality check (`{prefix}-sawcheck.sras`). | Exactly 1. |

See [SAW Quality Check (v10)](#saw-quality-check-v10) below.

Starting in v6, each angle only scans the **bounding box of the nominal ROI
rotated by that specific angle** — not the worst case across all angles — so
`x_start`, `x_delta` (and therefore `n_frames`, the points/row count) and
`n_rows` all vary per angle. A 0°/180° scan of a wide, short ROI needs far
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.

---

## File Layout

```
[Global Header           — 49 bytes]
[Angle Table             — n_angles × 4 bytes  (float32 per angle, degrees)]
[Per-Angle Geometry Table— n_angles × 14 bytes (x_start f32, x_delta f32, n_frames u32, n_rows u16)]
[Row Table (ragged)      — sum(n_rows) × 4 bytes (float32 per row, angle-major)]
[Preamble Blocks         — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Background Block        — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged)  — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
```

All multi-byte integers and floats use **big-endian** byte order
(`>` in Python's `struct` module).

---

## Global Header (49 bytes)

| Offset | Size | Type      | Field              | Description                                      |
|--------|------|-----------|--------------------|--------------------------------------------------|
|  0     |  4   | `4s`      | `magic`            | Always `SRAS` (0x53 0x52 0x41 0x53)              |
|  4     |  1   | `uint8`   | `version`          | Format version — `6` (scan) or `10` (SAW check)  |
|  5     |  2   | `uint16`  | `n_angles`         | Number of GR rotation angles                     |
|  7     |  4   | `float32` | `x_start_nominal`  | Nominal (pre-rotation) X scan start, mm          |
| 11     |  4   | `float32` | `y_start_nominal`  | Nominal (pre-rotation) Y scan start, mm          |
| 15     |  4   | `float32` | `x_delta_nominal`  | Nominal (pre-rotation) X scan width, mm          |
| 19     |  4   | `float32` | `y_delta_nominal`  | Nominal (pre-rotation) Y scan height, mm         |
| 23     |  4   | `float32` | `row_spacing_mm`   | Y spacing between rows, mm                       |
| 27     |  4   | `float32` | `velocity_mm_s`    | Stage scan velocity in mm/s                      |
| 31     |  4   | `float32` | `laser_freq_hz`    | Laser repetition rate in Hz                      |
| 35     |  4   | `uint32`  | `samples_per_frame`| Time samples per waveform                        |
| 39     |  8   | `float64` | `sample_rate_hz`   | Oscilloscope sample rate in Hz (e.g. 6.25e9)     |
| 47     |  1   | `uint8`   | `bytes_per_sample` | Bytes per ADC sample: `1` = int8, `2` = int16    |
| 48     |  1   | `uint8`   | `n_channels`       | Number of channels recorded (currently `3`)       |

**Total header size:** 49 bytes — verified:
`struct.calcsize(">4sBHfffffffIdBB") == 49`.

The `*_nominal` fields describe the ROI as originally entered on the New Scan
page (XS/YS/XD/YD), **before** per-angle bounding-box expansion. They are for
reference/reconstruction only — the actual per-angle scan geometry used for
acquisition is in the Per-Angle Geometry Table below.

---

## Angle Table

Immediately after the header: **n_angles** big-endian float32 values, one per
GR angle (degrees, signed; magnitude 0–180, sign gives physical rotation
direction — negative for the current CW-rotating GR stage).

```
angle[0], angle[1], …, angle[n_angles - 1]
```

---

## Per-Angle Geometry Table

Immediately after the angle table: **n_angles** fixed-size records, one per
angle (same order as the angle table), each 14 bytes:

| Size | Type      | Field      | Description                                          |
|------|-----------|------------|-------------------------------------------------------|
| 4    | `float32` | `x_start`  | X scan start for this angle's bounding box, mm         |
| 4    | `float32` | `x_delta`  | X scan width for this angle's bounding box, mm         |
| 4    | `uint32`  | `n_frames` | A-scans per row for this angle (FastFrame count)       |
| 2    | `uint16`  | `n_rows`   | Number of Y rows scanned for this angle                |

Format string per record: `">ffIH"`.

---

## Row Table (ragged)

Immediately after the per-angle geometry table: for each angle in order,
that angle's `n_rows` big-endian float32 Y positions (mm), concatenated with
no padding between angles.

```
# angle 0's rows, then angle 1's rows, …
y_mm[0][0], …, y_mm[0][n_rows[0]-1], y_mm[1][0], …, y_mm[n_angles-1][n_rows[-1]-1]
```

Row-table boundaries for angle *a* are derived from the per-angle geometry
table: `sum(n_rows[0:a])` gives the starting index into the flattened array.

---

## Preamble Blocks

Immediately after the row table: **n_channels** length-prefixed UTF-8 strings,
one per channel in `SCAN_CHANNELS` order (CH1, CH3, CH4).  Each block is:

```
uint16  length   — byte length of the following UTF-8 string
bytes   preamble — WFMOutpre response string from the oscilloscope
```

The preamble captures per-channel scaling constants (YMULT, YOFF, YZERO) needed
to convert raw ADC values to volts.

---

## Background Block

Immediately after the preamble blocks: a single CH1 waveform captured with the
**Helios (generation) laser enabled** and the **Genesis (detection) laser
disabled**.  This provides a noise/background reference for subtraction during
post-processing.

```
uint32  n_bg_samples   — number of samples in the background waveform
int8[]  bg_data        — raw ADC samples (same encoding as waveform data)
```

`n_bg_samples` equals `samples_per_frame` under normal acquisition settings.

---

## Waveform Data (ragged)

Immediately after the background block.  Data is stored in **angle-major,
row-minor** order, but unlike earlier versions each angle contributes a
different number of rows (`n_rows[a]`) and a different number of frames per
row (`n_frames[a]`), both taken from that angle's Per-Angle Geometry Table
entry.  Within each row, channels are interleaved in ascending channel-index
order, with each channel's FastFrame data written in frame order.

```
for angle a in 0 … n_angles-1:
  for row in 0 … n_rows[a]-1:
    for channel in [CH1, CH3, CH4]:          # 3 channels, fixed order
      for frame in 0 … n_frames[a]-1:
        samples[0 … samples_per_frame-1]     # bps bytes each
```

Each sample is a raw signed ADC value.  With `bytes_per_sample = 1` this is
**int8** (−128 … +127).  With `bytes_per_sample = 2` this is **big-endian
int16**.

Total data size:
```
sum over angles a of: n_rows[a] × 3 × n_frames[a] × samples_per_frame × bytes_per_sample
```

> **Incomplete files:** If a scan is aborted the file is closed immediately and
> the data block will be shorter than the expected size. Readers should
> reconstruct the expected per-angle byte offsets from the Per-Angle Geometry
> Table and check `file_size` against the running total before reshaping —
> a fixed `(n_angles, n_rows, ...)` reshape (as in pre-v6 readers) will not
> work since row/frame counts are no longer uniform across angles.

---

## Spatial Mapping

The *k*-th waveform (frame) in a row corresponds to the *k*-th laser pulse that
hit the sample.  For a row belonging to angle *a*, the physical X position of
that pulse is:

```
x_k = x_start[a] + k * (velocity_mm_s / laser_freq_hz)
```

using that angle's `x_start` from the Per-Angle Geometry Table (not
`x_start_nominal`).

---

## Acquisition Settings (fixed by core/scope_sras.py)

| Parameter            | Value                                    |
|-----------------------|------------------------------------------|
| Setup trigger        | CH2, rising edge, 0.500 V (`TRIG_LEVEL_V`) |
| Scan trigger         | Logic AND, CH2 HIGH ∧ CH3 HIGH, 0.500 V  |
| Horizontal position  | 30 (`HORizontal:POSition`)               |
| Sample rate          | 6.25 GS/s  (160 ps/sample)               |
| Transfer format      | `DATa:ENCdg RIBinary`, `DATa:WIDth 1`    |
| Channels recorded    | CH1, CH3, CH4                            |
| Stage X velocity     | 100 mm/s                                 |
| Stage X acceleration | 1500 mm/s²                               |
| Stage X trigger out  | Logic-high at max velocity (`TRIGOUT_MAXV`) |
| Acquisition mode     | FastFrame, Normal trigger                |

None of these are stored in the file, so they do not affect byte layout — but
they do set where the acoustic packet lands inside each frame.  Read them from
`core/scope_sras.py`; earlier revisions of this table drifted from the code.

---

## Acquisition Paths

Two acquisition strategies write **byte-identical** files; the choice is a
runtime flag (`ScanEngine(burst_mode=…)`, exposed as a checkbox in the app) and
is not recorded in the file.

| | Per-row (default) | Burst |
|---|---|---|
| FastFrame acquisitions | one per row | one per `floor(max_frames / n_frames)` rows |
| Curve transfers | one per channel per row | one per channel per burst |
| Stage X trigger out | armed for the whole scan | armed per acquiring pass, dropped for the flyback |

Burst mode runs a single acquisition across several rows, so the return move
must not trigger: the trigger output is dropped before each flyback and
re-armed for each acquiring pass.  Row boundaries inside the burst come from
`ACQuire:NUMFRAMESACQuired?` sampled after each pass — the burst itself carries
no row markers.  See `core/scope_burst.py`.

### Row packing

The format has no per-row length field, so a row that over- or under-triggers
cannot be written as it arrived — that would shift every later row.  Two
policies are selectable (`ScanEngine(strict_rows=…)`, a checkbox in the app),
and the choice is not recorded in the file:

| | Pad (default) | Strict |
|---|---|---|
| Short row | zero-padded to `n_frames`, warned | scan stops |
| Long row | trailing frames dropped, warned | scan stops |

Pad keeps a scan running through an occasional mis-trigger, at the cost that
the affected row is indistinguishable from a good one afterwards — nothing in
the file records that it was padded.  Strict is for data runs where that
ambiguity is worse than a failed scan: it aborts before writing the row, so
the file always ends on a whole-row boundary.

---

## SAW Quality Check (v10)

A full multi-angle scan takes hours, and a rig whose angles disagree produces
all of them before anyone finds out.  The SAW quality check acquires **one row
per angle — the row-wise middle of the ROI** — and writes it as a v10 file.
The cost is one row-time per angle instead of `n_rows` of them.

Nothing about the byte layout changes.  A v10 file is a v6 file in which every
angle's Per-Angle Geometry Table entry declares `n_rows = 1`, and its Row Table
holds that angle's single middle Y position.  Every v6 reader that works from
the geometry table (rather than assuming a uniform shape) reads a v10 file
unchanged.

The version byte earns its keep because the two are otherwise
indistinguishable: **a v6 scan aborted after its first row is not a check**,
even though both hold one row per angle.  A reader that guessed from the row
count would treat a failed scan as a deliberate measurement.

Why the middle row in particular: `core/scan_geometry.py` centres every
angle's rotated bounding box on the same nominal ROI centre, so each angle's
middle row crosses that one point on the sample.  All the angles therefore
measure the same material, and a spread in their SAW frequencies is a property
of the rig — which is what makes it an alignment check.  `saw_check_viewer.py`
plots every angle's frequency on one graph for exactly that comparison.

Writers must honour the one-row rule; `core.sras_format.create_scan_file`
refuses a v10 write for any plan that breaks it.  Producing the plan is
`core.saw_check.middle_row_plan(plan)`, and `n_rows // 2` is the middle-row
rule (the upper of the two central rows when the count is even).

---

## Version History

| Version | Change |
|---------|--------|
| 1       | One file per row; header included `angle_idx`, `row_idx`, `angle_deg`, `y_mm`. |
| 2       | One file per scan; global header with `n_angles`/`n_rows`; separate angle and row tables; three channels (CH1, CH3, CH4) per row. |
| 3       | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4       | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5       | (skipped) |
| 6       | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with v4 readers** (e.g. `sras_viewer.py`, which has not yet been updated for v6). |
| 7–9     | (skipped) |
| 10      | Middle-row SAW quality check. Byte layout identical to v6, with every angle declaring exactly one row — the row-wise middle of the ROI. A v6 reader that derives its shape from the Per-Angle Geometry Table reads these unchanged; the version byte exists so a check is not confused with a scan aborted after its first row. Written by the main app's *SAW Quality Check*, read by `saw_check_viewer.py`. |