sras-viewer/scan_format.md

# SRAS Scan Binary Format — Version 6 / 7

Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows.  Files are named `{prefix}.sras`.

Starting in v6, each angle only scans the **bounding box of the nominal ROI
rotated by that specific angle** — not the worst case across all angles — so
`x_start`, `x_delta` (and therefore `n_frames`, the points/row count) and
`n_rows` all vary per angle. A 0°/180° scan of a wide, short ROI needs far
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.

v7 is byte-identical to v6 (same header, angle table, geometry table, row
table, preamble blocks, background block, waveform data) plus an optional
trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images
(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version`
field and the presence of that trailing section differ. Scans come off the
scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file
to v7 **in place** the first time either cache block is computed and stored.

---

## File Layout

```
[Global Header           — 49 bytes]
[Angle Table             — n_angles × 4 bytes  (float32 per angle, degrees)]
[Per-Angle Geometry Table— n_angles × 14 bytes (x_start f32, x_delta f32, n_frames u32, n_rows u16)]
[Row Table (ragged)      — sum(n_rows) × 4 bytes (float32 per row, angle-major)]
[Preamble Blocks         — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Background Block        — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged)  — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
[Cache Tail (optional)   — "CACH" + DC block (optional) + FFT block (optional); v7 only]
```

All multi-byte integers and floats use **big-endian** byte order
(`>` in Python's `struct` module).

---

## Global Header (49 bytes)

| Offset | Size | Type      | Field              | Description                                      |
|--------|------|-----------|--------------------|--------------------------------------------------|
|  0     |  4   | `4s`      | `magic`            | Always `SRAS` (0x53 0x52 0x41 0x53)              |
|  4     |  1   | `uint8`   | `version`          | Format version — `6`                             |
|  5     |  2   | `uint16`  | `n_angles`         | Number of GR rotation angles                     |
|  7     |  4   | `float32` | `x_start_nominal`  | Nominal (pre-rotation) X scan start, mm          |
| 11     |  4   | `float32` | `y_start_nominal`  | Nominal (pre-rotation) Y scan start, mm          |
| 15     |  4   | `float32` | `x_delta_nominal`  | Nominal (pre-rotation) X scan width, mm          |
| 19     |  4   | `float32` | `y_delta_nominal`  | Nominal (pre-rotation) Y scan height, mm         |
| 23     |  4   | `float32` | `row_spacing_mm`   | Y spacing between rows, mm                       |
| 27     |  4   | `float32` | `velocity_mm_s`    | Stage scan velocity in mm/s                      |
| 31     |  4   | `float32` | `laser_freq_hz`    | Laser repetition rate in Hz                      |
| 35     |  4   | `uint32`  | `samples_per_frame`| Time samples per waveform                        |
| 39     |  8   | `float64` | `sample_rate_hz`   | Oscilloscope sample rate in Hz (e.g. 6.25e9)     |
| 47     |  1   | `uint8`   | `bytes_per_sample` | Bytes per ADC sample: `1` = int8, `2` = int16    |
| 48     |  1   | `uint8`   | `n_channels`       | Number of channels recorded (currently `3`)       |

**Total header size:** 49 bytes — verified:
`struct.calcsize(">4sBHfffffffIdBB") == 49`.

The `*_nominal` fields describe the ROI as originally entered on the New Scan
page (XS/YS/XD/YD), **before** per-angle bounding-box expansion. They are for
reference/reconstruction only — the actual per-angle scan geometry used for
acquisition is in the Per-Angle Geometry Table below.

---

## Angle Table

Immediately after the header: **n_angles** big-endian float32 values, one per
GR angle (degrees, 0–180).

```
angle[0], angle[1], …, angle[n_angles - 1]
```

---

## Per-Angle Geometry Table

Immediately after the angle table: **n_angles** fixed-size records, one per
angle (same order as the angle table), each 14 bytes:

| Size | Type      | Field      | Description                                          |
|------|-----------|------------|-------------------------------------------------------|
| 4    | `float32` | `x_start`  | X scan start for this angle's bounding box, mm         |
| 4    | `float32` | `x_delta`  | X scan width for this angle's bounding box, mm         |
| 4    | `uint32`  | `n_frames` | A-scans per row for this angle (FastFrame count)       |
| 2    | `uint16`  | `n_rows`   | Number of Y rows scanned for this angle                |

Format string per record: `">ffIH"`.

---

## Row Table (ragged)

Immediately after the per-angle geometry table: for each angle in order,
that angle's `n_rows` big-endian float32 Y positions (mm), concatenated with
no padding between angles.

```
# angle 0's rows, then angle 1's rows, …
y_mm[0][0], …, y_mm[0][n_rows[0]-1], y_mm[1][0], …, y_mm[n_angles-1][n_rows[-1]-1]
```

Row-table boundaries for angle *a* are derived from the per-angle geometry
table: `sum(n_rows[0:a])` gives the starting index into the flattened array.

---

## Preamble Blocks

Immediately after the row table: **n_channels** length-prefixed UTF-8 strings,
one per channel in `SCAN_CHANNELS` order (CH1, CH3, CH4).  Each block is:

```
uint16  length   — byte length of the following UTF-8 string
bytes   preamble — WFMOutpre response string from the oscilloscope
```

The preamble captures per-channel scaling constants (YMULT, YOFF, YZERO) needed
to convert raw ADC values to volts.

---

## Background Block

Immediately after the preamble blocks: a single CH1 waveform captured with the
**Helios (generation) laser enabled** and the **Genesis (detection) laser
disabled**.  This provides a noise/background reference for subtraction during
post-processing.

```
uint32  n_bg_samples   — number of samples in the background waveform
int8[]  bg_data        — raw ADC samples (same encoding as waveform data)
```

`n_bg_samples` equals `samples_per_frame` under normal acquisition settings.

---

## Waveform Data (ragged)

Immediately after the background block.  Data is stored in **angle-major,
row-minor** order, but unlike earlier versions each angle contributes a
different number of rows (`n_rows[a]`) and a different number of frames per
row (`n_frames[a]`), both taken from that angle's Per-Angle Geometry Table
entry.  Within each row, channels are interleaved in ascending channel-index
order, with each channel's FastFrame data written in frame order.

```
for angle a in 0 … n_angles-1:
  for row in 0 … n_rows[a]-1:
    for channel in [CH1, CH3, CH4]:          # 3 channels, fixed order
      for frame in 0 … n_frames[a]-1:
        samples[0 … samples_per_frame-1]     # bps bytes each
```

Each sample is a raw signed ADC value.  With `bytes_per_sample = 1` this is
**int8** (−128 … +127).  With `bytes_per_sample = 2` this is **big-endian
int16**.

Total data size:
```
sum over angles a of: n_rows[a] × 3 × n_frames[a] × samples_per_frame × bytes_per_sample
```

> **Incomplete files:** If a scan is aborted the file is closed immediately and
> the data block will be shorter than the expected size. Readers should
> reconstruct the expected per-angle byte offsets from the Per-Angle Geometry
> Table and check `file_size` against the running total before reshaping —
> a fixed `(n_angles, n_rows, ...)` reshape (as in pre-v6 readers) will not
> work since row/frame counts are no longer uniform across angles.
>
> A consequence worth stating explicitly: because a short file opens
> *successfully* as a scan with fewer angles, a truncated file is not
> detectably broken. Anything that writes a .sras must therefore stage to a
> temporary name and rename on success — the viewer's aligned export writes
> `<name>.part` and `os.replace`s it — or an interrupted write leaves behind
> something that loads without complaint and silently has the wrong angle count.

---

## Files written by the viewer's Alignment Wizard

The acquisition app is not the only producer of this format. The viewer's
`Fusion → Alignment Wizard…` writes a **v6** file holding the aligned, cropped
stack, with these properties:

* Every angle shares one grid — the cropped alignment canvas — so the
  Per-Angle Geometry Table is `n_angles` identical records and the ragged Row
  Table is `n_angles` identical spans. The raggedness v6 exists for is still
  *expressible*, just unused, so any v6 reader works unchanged.
* `x_delta` is the reference angle's own pitch, which is exactly
  `velocity_mm_s / laser_freq_hz`, so the derived X axis stays consistent with
  the header.
* The `*_nominal` header fields describe the crop. Uniquely for these files they
  coincide with the actual per-angle geometry, since after alignment every angle
  really does scan the same box.
* The **Angle Table is unchanged**. Alignment removes the sample's spatial
  rotation, not the acoustic propagation direction each angle measured — that
  direction is the point of a multi-angle scan, so it is preserved.
* Output pixels with no corresponding source pixel (the canvas corners a rotated
  scan cannot reach) hold the per-channel ADC code nearest **0 mV**, not zero.
  Zero ADC decodes to roughly +100 mV on real calibration and would read as
  signal.
* No Cache Tail is written: any cached DC/FFT is indexed by the source's grid
  and would be meaningless on the new one.

---

## Spatial Mapping

The *k*-th waveform (frame) in a row corresponds to the *k*-th laser pulse that
hit the sample.  For a row belonging to angle *a*, the physical X position of
that pulse is:

```
x_k = x_start[a] + k * (velocity_mm_s / laser_freq_hz)
```

using that angle's `x_start` from the Per-Angle Geometry Table (not
`x_start_nominal`).

---

## Cache Tail (v7)

Present iff `version == 7` and `file_size > cache_offset`, where:

```
cache_offset = data_offset + Σ over angles a of:
                 n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample
```

i.e. exactly `data_offset + waveform_bytes` — the same "Total data size"
formula as Waveform Data above. This offset is derivable from the header and
Per-Angle Geometry Table alone and does **not** depend on which cache
block(s) are present, so a writer can always seek straight there without
reading or touching any waveform byte before it.

Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle
entry (FFT + both DC channels together) in a dense, uniform-geometry array,
the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks —
each sized per-angle from the Per-Angle Geometry Table, each independently
present, and each independently updatable in any order, any number of
times, without disturbing the other. This matches `sras_viewer.py`'s
"Convert" menu, which exposes DC and FFT store as two separate batch
actions.

### CACH outer header (6 bytes, `">4sBB"`)

| Offset | Size | Type | Field | Description |
|--------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `4`; readers also accept `1`–`3` (each older tail simply lacks the fields added since — see the `SFFT` block below and [CACH tail version history](#cach-tail-version-history)). Any other value → treat the file as uncached (unlike v5's `PREC` section, which read but never validated its version byte). |
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |

### DC block `SDCB` (present iff `block_flags & 0x01`)

7-byte block header, format `">4sBH"`:

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SDCB` |
| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` |

followed by `n_stored` entries, each:

```
u16                          angle_idx   — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc3_mv  — CH3 waveform mean, mV, row-major
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc4_mv  — CH4 waveform mean, mV, row-major
```

Entries may appear in any order and need not be contiguous from angle 0 —
this supports storing (or re-storing) a subset of angles, or an
interrupted batch run leaving only some angles cached. Readers bounds-check
`angle_idx < n_angles` on each entry and stop parsing on an out-of-range
value, same as v5's `PREC` section.

### FFT block `SFFT` (present iff `block_flags & 0x02`)

Block header layout depends on `cach_version`:

Each `cach_version` appended one trailing field, so the header grows but
never shifts an existing offset:

- **`cach_version` 1**: 7 bytes, format `">4sBH"` — magic, flags, n_stored.
- **`cach_version` 2**: 8 bytes, format `">4sBHB"` — + `row_avg_n`.
- **`cach_version` 3**: 10 bytes, format `">4sBHBH"` — + `pad_factor`.
- **`cach_version` 4**: 14 bytes, format `">4sBHBHI"` — + `min_freq_khz`.
  Always written by current code.

An older tail is read with its absent fields taken as the only value such a
tail can describe: `row_avg_n = 0` for a `cach_version` 1 tail, which
predates row-averaged FFT caching, `pad_factor = 1` for `cach_version`
1 or 2, which predate padded caching and are therefore natural-resolution,
and `min_freq_khz = 0` (no floor) for `cach_version` 1–3, which predate the
min peak frequency floor and therefore searched every bin above DC.
Files cached before any of these changes keep working with no recompute.

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bit 1 = `row_averaged` — `peak_freq_mhz` came from same-row, distance-weighted averaged CH1 waveforms rather than raw per-pixel ones; `row_avg_n` (below) is the neighbor half-width used. Bits 2–7 reserved. |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
| 7 | 1 | `u8` | `row_avg_n` | *`cach_version` ≥ 2 only.* Same-row neighbor half-width, in pixels, that `peak_freq_mhz` was averaged over before its FFT; `0` = raw (unaveraged). Meaningful only when `flags` bit 1 is set — a `cach_version` 1 tail has no such byte and is always `row_avg_n = 0`. |
| 8 | 2 | `u16` | `pad_factor` | *`cach_version` ≥ 3 only.* Zero-padding factor the stored `peak_freq_mhz` was resolved at: `n_fft = pad_factor × samples_per_frame`, so `1` = natural resolution. Never `0`; a `cach_version` 1 or 2 tail has no such field and is always `pad_factor = 1`. |
| 10 | 4 | `u32` | `min_freq_khz` | *`cach_version` ≥ 4 only.* Min peak frequency floor the stored peak search excluded bins below, fixed-point in units of 0.001 MHz (kHz); `0` = no floor. Fixed-point rather than `f32` so a value that round-trips through the file compares exactly against the same value re-requested by a reader (the viewer's floor control has 0.001 MHz granularity). A `cach_version` 1–3 tail has no such field and is always `min_freq_khz = 0`. |

followed by `n_stored` entries, each:

```
u16                          angle_idx      — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  peak_freq_mhz  — CH1 FFT peak frequency, MHz, row-major
```

**`peak_freq_mhz`** for a raw store (`row_avg_n == 0`) is computed without
any DC-threshold masking (i.e. the FFT is run on every pixel
unconditionally, same as v5's `PREC` convention). Readers apply the DC4
threshold at display time:

```
pixel is valid  ⟺  dc4_mv[r][f]  ≥  threshold_mv
display_value   =   peak_freq_mhz[r][f]  if valid, else 0
```

using the DC4 image from the DC block if that angle is also cached there,
else computed on demand.

For a row-averaged store (`row_avg_n > 0`), the DC4 threshold is applied
*during* the store — a pixel below threshold is left at `0` and never
contributes to any neighbor's average — since neighbor validity can't be
deferred to display time the way plain masking can. The threshold value
itself is not recorded, only that averaging happened and at what window
size. Readers still apply their own live DC4 threshold at display time
exactly as for a raw store, using whatever mask they currently have.

Readers must fall back to real-time FFT computation (ignoring stored
`peak_freq_mhz`) whenever the store's recorded provenance doesn't match what
the reader is asking for: time-domain gating is active, the reader's
requested `n_fft` doesn't equal `pad_factor × samples_per_frame`, the
reader's background-subtraction setting doesn't match
`flags.bg_sub_applied`, the reader's requested `row_avg_n` doesn't match
the stored value exactly, or the reader's requested min peak frequency
floor is *below* the stored `min_freq_khz`. A raw request must never be
served a row-averaged store, or vice versa; a request at one row-averaging
window size must never be served a store at another; and a request at one
padding must never be served a store at another, since a padded FFT
interpolates between the natural bins and so resolves genuinely different
peak frequencies. An `n_fft` that is not a whole multiple of
`samples_per_frame` can never match any store, because only an integer
`pad_factor` is representable.

The min peak frequency floor is the one asymmetric provenance field. A
request at a floor *below* the stored one cannot be served: the stored
search never looked at bins below its floor, so the stored numbers cannot
say what a lower-floored search would have found. A request at a floor at
or *above* the stored one **is** servable — the difference is re-applied at
display time by masking every pixel whose stored `peak_freq_mhz` is below
the requested floor to `0` (the same sentinel as the DC threshold mask;
a genuine peak can never be `0`, since bin 0 is always excluded from the
search). Such masked pixels are *invalid*, not re-resolved — only a real
recompute can recover the strongest peak above the floor for them.

### In-place write ordering

A writer updating a file's Cache Tail must write the payload (CACH header +
whichever block(s) are present) **before** flipping the header's `version`
byte to `7`, and `truncate()` the file to the new payload's end immediately
after writing it. If the process is interrupted between the payload write
and the version-byte flip, the file is still valid v6 — v6 parsing only
bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts
exactly how many bytes follow the last angle's waveform block — so the
interrupted write leaves harmless trailing bytes rather than a corrupt file,
and the next successful write overwrites them via the same deterministic
`cache_offset`.

### CACH tail version history

Distinct from the outer `.sras` file `version` byte (top of this document),
which has stayed `7` since the Cache Tail was introduced — this is the inner
`cach_version` byte inside the `CACH` header itself.

| cach_version | Change |
|--------------|--------|
| 1 | Initial Cache Tail: `SDCB` (DC) and `SFFT` (FFT, 7-byte header) blocks. |
| 2 | `SFFT` header grows one byte, `row_avg_n` — the same-row neighbor half-width the stored `peak_freq_mhz` was averaged over before its FFT, `0` = raw. Readers still accept a `cach_version` 1 tail, treated as `row_avg_n = 0` for every angle it stores, so files cached before this change keep working without a recompute. |
| 3 | `SFFT` header grows a `u16` `pad_factor` — the zero-padding factor the stored `peak_freq_mhz` was resolved at, `1` = natural resolution. Before this, a padded view could never use a stored cache at all (the store was pad 1 by definition and readers rejected any `n_fft ≠ samples_per_frame`), so a user working at a pad factor got no benefit from batch-computing a file. Recording the factor lets such a view be served, while still refusing a store resolved at a *different* pad. Readers accept `cach_version` 1 and 2 tails as `pad_factor = 1`. |
| 4 | `SFFT` header grows a `u32` `min_freq_khz` — the min peak frequency floor the stored peak search excluded bins below, in 0.001 MHz units, `0` = no floor. The floor exists because a pixel that passes the DC-bias threshold but carries only weak real signal can otherwise resolve to the un-subtracted background's DC-leakage skirt — an implausibly-near-zero frequency (and so an implausibly slow velocity) for a pixel that has a genuine peak higher up. Recording the floor is what makes it enforceable against a store: without it, a stored image silently bypassed the floor entirely. Unlike the other provenance fields it is asymmetric — a *higher* requested floor is servable by masking stored pixels below it, only a *lower* one forces a recompute (see above). Readers accept `cach_version` 1–3 tails as `min_freq_khz = 0`. |

A reader that does not know a `cach_version` must treat the file as
uncached — not attempt a partial parse — and the file still reads as an
ordinary v7 (byte-identical to v6) scan, so a forward-dated tail costs a
recompute and never correctness.

---

## Acquisition Settings (fixed by sc3_aui_app.py)

| Parameter            | Value                        |
|-----------------------|------------------------------|
| Oscilloscope trigger | CH2, rising edge, 1.24 V     |
| Trigger offset       | 0 % (trigger at left edge)   |
| Sample rate          | 6.25 GS/s  (160 ps/sample)   |
| Channels recorded    | CH1, CH3, CH4                |
| Stage X velocity     | 100 mm/s                     |
| Stage X acceleration | 1500 mm/s²                   |
| Stage X trigger out  | Logic-high at max velocity   |
| Acquisition mode     | FastFrame, Normal trigger    |

---

## Version History

| Version | Change |
|---------|--------|
| 1       | One file per row; header included `angle_idx`, `row_idx`, `angle_deg`, `y_mm`. |
| 2       | One file per scan; global header with `n_angles`/`n_rows`; separate angle and row tables; three channels (CH1, CH3, CH4) per row. |
| 3       | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4       | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5       | (skipped) |
| 6       | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). |
| 7       | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. |