sras-viewer/scan_format.md

# SRAS Scan Binary Format — Version 6 / 7

Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows.  Files are named `{prefix}.sras`.

Starting in v6, each angle only scans the **bounding box of the nominal ROI
rotated by that specific angle** — not the worst case across all angles — so
`x_start`, `x_delta` (and therefore `n_frames`, the points/row count) and
`n_rows` all vary per angle. A 0°/180° scan of a wide, short ROI needs far
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.

v7 is byte-identical to v6 (same header, angle table, geometry table, row
table, preamble blocks, background block, waveform data) plus an optional
trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images
(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version`
field and the presence of that trailing section differ. Scans come off the
scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file
to v7 **in place** the first time either cache block is computed and stored.

---

## File Layout

```
[Global Header           — 49 bytes]
[Angle Table             — n_angles × 4 bytes  (float32 per angle, degrees)]
[Per-Angle Geometry Table— n_angles × 14 bytes (x_start f32, x_delta f32, n_frames u32, n_rows u16)]
[Row Table (ragged)      — sum(n_rows) × 4 bytes (float32 per row, angle-major)]
[Preamble Blocks         — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Background Block        — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged)  — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
[Cache Tail (optional)   — "CACH" + DC block (optional) + FFT block (optional); v7 only]
```

All multi-byte integers and floats use **big-endian** byte order
(`>` in Python's `struct` module).

---

## Global Header (49 bytes)

| Offset | Size | Type      | Field              | Description                                      |
|--------|------|-----------|--------------------|--------------------------------------------------|
|  0     |  4   | `4s`      | `magic`            | Always `SRAS` (0x53 0x52 0x41 0x53)              |
|  4     |  1   | `uint8`   | `version`          | Format version — `6`                             |
|  5     |  2   | `uint16`  | `n_angles`         | Number of GR rotation angles                     |
|  7     |  4   | `float32` | `x_start_nominal`  | Nominal (pre-rotation) X scan start, mm          |
| 11     |  4   | `float32` | `y_start_nominal`  | Nominal (pre-rotation) Y scan start, mm          |
| 15     |  4   | `float32` | `x_delta_nominal`  | Nominal (pre-rotation) X scan width, mm          |
| 19     |  4   | `float32` | `y_delta_nominal`  | Nominal (pre-rotation) Y scan height, mm         |
| 23     |  4   | `float32` | `row_spacing_mm`   | Y spacing between rows, mm                       |
| 27     |  4   | `float32` | `velocity_mm_s`    | Stage scan velocity in mm/s                      |
| 31     |  4   | `float32` | `laser_freq_hz`    | Laser repetition rate in Hz                      |
| 35     |  4   | `uint32`  | `samples_per_frame`| Time samples per waveform                        |
| 39     |  8   | `float64` | `sample_rate_hz`   | Oscilloscope sample rate in Hz (e.g. 6.25e9)     |
| 47     |  1   | `uint8`   | `bytes_per_sample` | Bytes per ADC sample: `1` = int8, `2` = int16    |
| 48     |  1   | `uint8`   | `n_channels`       | Number of channels recorded (currently `3`)       |

**Total header size:** 49 bytes — verified:
`struct.calcsize(">4sBHfffffffIdBB") == 49`.

The `*_nominal` fields describe the ROI as originally entered on the New Scan
page (XS/YS/XD/YD), **before** per-angle bounding-box expansion. They are for
reference/reconstruction only — the actual per-angle scan geometry used for
acquisition is in the Per-Angle Geometry Table below.

---

## Angle Table

Immediately after the header: **n_angles** big-endian float32 values, one per
GR angle (degrees, 0–180).

```
angle[0], angle[1], …, angle[n_angles - 1]
```

---

## Per-Angle Geometry Table

Immediately after the angle table: **n_angles** fixed-size records, one per
angle (same order as the angle table), each 14 bytes:

| Size | Type      | Field      | Description                                          |
|------|-----------|------------|-------------------------------------------------------|
| 4    | `float32` | `x_start`  | X scan start for this angle's bounding box, mm         |
| 4    | `float32` | `x_delta`  | X scan width for this angle's bounding box, mm         |
| 4    | `uint32`  | `n_frames` | A-scans per row for this angle (FastFrame count)       |
| 2    | `uint16`  | `n_rows`   | Number of Y rows scanned for this angle                |

Format string per record: `">ffIH"`.

---

## Row Table (ragged)

Immediately after the per-angle geometry table: for each angle in order,
that angle's `n_rows` big-endian float32 Y positions (mm), concatenated with
no padding between angles.

```
# angle 0's rows, then angle 1's rows, …
y_mm[0][0], …, y_mm[0][n_rows[0]-1], y_mm[1][0], …, y_mm[n_angles-1][n_rows[-1]-1]
```

Row-table boundaries for angle *a* are derived from the per-angle geometry
table: `sum(n_rows[0:a])` gives the starting index into the flattened array.

---

## Preamble Blocks

Immediately after the row table: **n_channels** length-prefixed UTF-8 strings,
one per channel in `SCAN_CHANNELS` order (CH1, CH3, CH4).  Each block is:

```
uint16  length   — byte length of the following UTF-8 string
bytes   preamble — WFMOutpre response string from the oscilloscope
```

The preamble captures per-channel scaling constants (YMULT, YOFF, YZERO) needed
to convert raw ADC values to volts.

---

## Background Block

Immediately after the preamble blocks: a single CH1 waveform captured with the
**Helios (generation) laser enabled** and the **Genesis (detection) laser
disabled**.  This provides a noise/background reference for subtraction during
post-processing.

```
uint32  n_bg_samples   — number of samples in the background waveform
int8[]  bg_data        — raw ADC samples (same encoding as waveform data)
```

`n_bg_samples` equals `samples_per_frame` under normal acquisition settings.

---

## Waveform Data (ragged)

Immediately after the background block.  Data is stored in **angle-major,
row-minor** order, but unlike earlier versions each angle contributes a
different number of rows (`n_rows[a]`) and a different number of frames per
row (`n_frames[a]`), both taken from that angle's Per-Angle Geometry Table
entry.  Within each row, channels are interleaved in ascending channel-index
order, with each channel's FastFrame data written in frame order.

```
for angle a in 0 … n_angles-1:
  for row in 0 … n_rows[a]-1:
    for channel in [CH1, CH3, CH4]:          # 3 channels, fixed order
      for frame in 0 … n_frames[a]-1:
        samples[0 … samples_per_frame-1]     # bps bytes each
```

Each sample is a raw signed ADC value.  With `bytes_per_sample = 1` this is
**int8** (−128 … +127).  With `bytes_per_sample = 2` this is **big-endian
int16**.

Total data size:
```
sum over angles a of: n_rows[a] × 3 × n_frames[a] × samples_per_frame × bytes_per_sample
```

> **Incomplete files:** If a scan is aborted the file is closed immediately and
> the data block will be shorter than the expected size. Readers should
> reconstruct the expected per-angle byte offsets from the Per-Angle Geometry
> Table and check `file_size` against the running total before reshaping —
> a fixed `(n_angles, n_rows, ...)` reshape (as in pre-v6 readers) will not
> work since row/frame counts are no longer uniform across angles.

---

## Spatial Mapping

The *k*-th waveform (frame) in a row corresponds to the *k*-th laser pulse that
hit the sample.  For a row belonging to angle *a*, the physical X position of
that pulse is:

```
x_k = x_start[a] + k * (velocity_mm_s / laser_freq_hz)
```

using that angle's `x_start` from the Per-Angle Geometry Table (not
`x_start_nominal`).

---

## Cache Tail (v7)

Present iff `version == 7` and `file_size > cache_offset`, where:

```
cache_offset = data_offset + Σ over angles a of:
                 n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample
```

i.e. exactly `data_offset + waveform_bytes` — the same "Total data size"
formula as Waveform Data above. This offset is derivable from the header and
Per-Angle Geometry Table alone and does **not** depend on which cache
block(s) are present, so a writer can always seek straight there without
reading or touching any waveform byte before it.

Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle
entry (FFT + both DC channels together) in a dense, uniform-geometry array,
the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks —
each sized per-angle from the Per-Angle Geometry Table, each independently
present, and each independently updatable in any order, any number of
times, without disturbing the other. This matches `sras_viewer.py`'s
"Convert" menu, which exposes DC and FFT store as two separate batch
actions.

### CACH outer header (6 bytes, `">4sBB"`)

| Offset | Size | Type | Field | Description |
|--------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `2`; readers also accept `1` (a `1` tail predates row-averaged FFT caching — see the `SFFT` block below and [CACH tail version history](#cach-tail-version-history)). Any other value → treat the file as uncached (unlike v5's `PREC` section, which read but never validated its version byte). |
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |

### DC block `SDCB` (present iff `block_flags & 0x01`)

7-byte block header, format `">4sBH"`:

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SDCB` |
| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` |

followed by `n_stored` entries, each:

```
u16                          angle_idx   — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc3_mv  — CH3 waveform mean, mV, row-major
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc4_mv  — CH4 waveform mean, mV, row-major
```

Entries may appear in any order and need not be contiguous from angle 0 —
this supports storing (or re-storing) a subset of angles, or an
interrupted batch run leaving only some angles cached. Readers bounds-check
`angle_idx < n_angles` on each entry and stop parsing on an out-of-range
value, same as v5's `PREC` section.

### FFT block `SFFT` (present iff `block_flags & 0x02`)

Block header layout depends on `cach_version`:

- **`cach_version` 1**: 7 bytes, format `">4sBH"` — magic, flags, n_stored.
- **`cach_version` 2**: 8 bytes, format `">4sBHB"` — magic, flags, n_stored,
  `row_avg_n`. Always written by current code; a `cach_version` 1 tail (no
  trailing byte) is still read, with `row_avg_n` taken as `0` for every
  entry it stores.

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bit 1 = `row_averaged` — `peak_freq_mhz` came from same-row, distance-weighted averaged CH1 waveforms rather than raw per-pixel ones; `row_avg_n` (below) is the neighbor half-width used. Bits 2–7 reserved. |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |
| 7 | 1 | `u8` | `row_avg_n` | *`cach_version` 2 only.* Same-row neighbor half-width, in pixels, that `peak_freq_mhz` was averaged over before its FFT; `0` = raw (unaveraged). Meaningful only when `flags` bit 1 is set — a `cach_version` 1 tail has no such byte and is always `row_avg_n = 0`. |

followed by `n_stored` entries, each:

```
u16                          angle_idx      — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  peak_freq_mhz  — CH1 FFT peak frequency, MHz, row-major
```

**`peak_freq_mhz`** for a raw store (`row_avg_n == 0`) is computed without
any DC-threshold masking (i.e. the FFT is run on every pixel
unconditionally, same as v5's `PREC` convention). Readers apply the DC4
threshold at display time:

```
pixel is valid  ⟺  dc4_mv[r][f]  ≥  threshold_mv
display_value   =   peak_freq_mhz[r][f]  if valid, else 0
```

using the DC4 image from the DC block if that angle is also cached there,
else computed on demand.

For a row-averaged store (`row_avg_n > 0`), the DC4 threshold is applied
*during* the store — a pixel below threshold is left at `0` and never
contributes to any neighbor's average — since neighbor validity can't be
deferred to display time the way plain masking can. The threshold value
itself is not recorded, only that averaging happened and at what window
size. Readers still apply their own live DC4 threshold at display time
exactly as for a raw store, using whatever mask they currently have.

Readers must fall back to real-time FFT computation (ignoring stored
`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested,
the reader's background-subtraction setting doesn't match
`flags.bg_sub_applied`, or the reader's requested `row_avg_n` doesn't match
the stored value exactly — a raw request must never be served a
row-averaged store, or vice versa, and a request at one window size must
never be served a store at another.

### In-place write ordering

A writer updating a file's Cache Tail must write the payload (CACH header +
whichever block(s) are present) **before** flipping the header's `version`
byte to `7`, and `truncate()` the file to the new payload's end immediately
after writing it. If the process is interrupted between the payload write
and the version-byte flip, the file is still valid v6 — v6 parsing only
bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts
exactly how many bytes follow the last angle's waveform block — so the
interrupted write leaves harmless trailing bytes rather than a corrupt file,
and the next successful write overwrites them via the same deterministic
`cache_offset`.

### CACH tail version history

Distinct from the outer `.sras` file `version` byte (top of this document),
which has stayed `7` since the Cache Tail was introduced — this is the inner
`cach_version` byte inside the `CACH` header itself.

| cach_version | Change |
|--------------|--------|
| 1 | Initial Cache Tail: `SDCB` (DC) and `SFFT` (FFT, 7-byte header) blocks. |
| 2 | `SFFT` header grows one byte, `row_avg_n` — the same-row neighbor half-width the stored `peak_freq_mhz` was averaged over before its FFT, `0` = raw. Readers still accept a `cach_version` 1 tail, treated as `row_avg_n = 0` for every angle it stores, so files cached before this change keep working without a recompute. |

---

## Acquisition Settings (fixed by sc3_aui_app.py)

| Parameter            | Value                        |
|-----------------------|------------------------------|
| Oscilloscope trigger | CH2, rising edge, 1.24 V     |
| Trigger offset       | 0 % (trigger at left edge)   |
| Sample rate          | 6.25 GS/s  (160 ps/sample)   |
| Channels recorded    | CH1, CH3, CH4                |
| Stage X velocity     | 100 mm/s                     |
| Stage X acceleration | 1500 mm/s²                   |
| Stage X trigger out  | Logic-high at max velocity   |
| Acquisition mode     | FastFrame, Normal trigger    |

---

## Version History

| Version | Change |
|---------|--------|
| 1       | One file per row; header included `angle_idx`, `row_idx`, `angle_deg`, `y_mm`. |
| 2       | One file per scan; global header with `n_angles`/`n_rows`; separate angle and row tables; three channels (CH1, CH3, CH4) per row. |
| 3       | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4       | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5       | (skipped) |
| 6       | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). |
| 7       | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. |