sras-viewer/scan_format.md

# SRAS Scan Binary Format — Version 6 / 7

Each `.sras` file contains **one complete scan**: all GR rotation angles and all
Y rows.  Files are named `{prefix}.sras`.

Starting in v6, each angle only scans the **bounding box of the nominal ROI
rotated by that specific angle** — not the worst case across all angles — so
`x_start`, `x_delta` (and therefore `n_frames`, the points/row count) and
`n_rows` all vary per angle. A 0°/180° scan of a wide, short ROI needs far
fewer rows than a 45° scan of the same ROI, and the file format reflects that
instead of forcing every angle to the largest bounding box.

v7 is byte-identical to v6 (same header, angle table, geometry table, row
table, preamble blocks, background block, waveform data) plus an optional
trailing **Cache Tail** holding precomputed per-angle DC and/or FFT images
(see [Cache Tail (v7)](#cache-tail-v7) below) — only the header's `version`
field and the presence of that trailing section differ. Scans come off the
scope as v6; `sras_viewer.py`'s "Convert" menu batch actions convert a file
to v7 **in place** the first time either cache block is computed and stored.

---

## File Layout

```
[Global Header           — 49 bytes]
[Angle Table             — n_angles × 4 bytes  (float32 per angle, degrees)]
[Per-Angle Geometry Table— n_angles × 14 bytes (x_start f32, x_delta f32, n_frames u32, n_rows u16)]
[Row Table (ragged)      — sum(n_rows) × 4 bytes (float32 per row, angle-major)]
[Preamble Blocks         — n_channels × (uint16 length + UTF-8 WFMOutpre string)]
[Background Block        — uint32 n_bg_samples + n_bg_samples × int8 bytes]
[Waveform Data (ragged)  — per angle: n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bps bytes]
[Cache Tail (optional)   — "CACH" + DC block (optional) + FFT block (optional); v7 only]
```

All multi-byte integers and floats use **big-endian** byte order
(`>` in Python's `struct` module).

---

## Global Header (49 bytes)

| Offset | Size | Type      | Field              | Description                                      |
|--------|------|-----------|--------------------|--------------------------------------------------|
|  0     |  4   | `4s`      | `magic`            | Always `SRAS` (0x53 0x52 0x41 0x53)              |
|  4     |  1   | `uint8`   | `version`          | Format version — `6`                             |
|  5     |  2   | `uint16`  | `n_angles`         | Number of GR rotation angles                     |
|  7     |  4   | `float32` | `x_start_nominal`  | Nominal (pre-rotation) X scan start, mm          |
| 11     |  4   | `float32` | `y_start_nominal`  | Nominal (pre-rotation) Y scan start, mm          |
| 15     |  4   | `float32` | `x_delta_nominal`  | Nominal (pre-rotation) X scan width, mm          |
| 19     |  4   | `float32` | `y_delta_nominal`  | Nominal (pre-rotation) Y scan height, mm         |
| 23     |  4   | `float32` | `row_spacing_mm`   | Y spacing between rows, mm                       |
| 27     |  4   | `float32` | `velocity_mm_s`    | Stage scan velocity in mm/s                      |
| 31     |  4   | `float32` | `laser_freq_hz`    | Laser repetition rate in Hz                      |
| 35     |  4   | `uint32`  | `samples_per_frame`| Time samples per waveform                        |
| 39     |  8   | `float64` | `sample_rate_hz`   | Oscilloscope sample rate in Hz (e.g. 6.25e9)     |
| 47     |  1   | `uint8`   | `bytes_per_sample` | Bytes per ADC sample: `1` = int8, `2` = int16    |
| 48     |  1   | `uint8`   | `n_channels`       | Number of channels recorded (currently `3`)       |

**Total header size:** 49 bytes — verified:
`struct.calcsize(">4sBHfffffffIdBB") == 49`.

The `*_nominal` fields describe the ROI as originally entered on the New Scan
page (XS/YS/XD/YD), **before** per-angle bounding-box expansion. They are for
reference/reconstruction only — the actual per-angle scan geometry used for
acquisition is in the Per-Angle Geometry Table below.

---

## Angle Table

Immediately after the header: **n_angles** big-endian float32 values, one per
GR angle (degrees, 0–180).

```
angle[0], angle[1], …, angle[n_angles - 1]
```

---

## Per-Angle Geometry Table

Immediately after the angle table: **n_angles** fixed-size records, one per
angle (same order as the angle table), each 14 bytes:

| Size | Type      | Field      | Description                                          |
|------|-----------|------------|-------------------------------------------------------|
| 4    | `float32` | `x_start`  | X scan start for this angle's bounding box, mm         |
| 4    | `float32` | `x_delta`  | X scan width for this angle's bounding box, mm         |
| 4    | `uint32`  | `n_frames` | A-scans per row for this angle (FastFrame count)       |
| 2    | `uint16`  | `n_rows`   | Number of Y rows scanned for this angle                |

Format string per record: `">ffIH"`.

---

## Row Table (ragged)

Immediately after the per-angle geometry table: for each angle in order,
that angle's `n_rows` big-endian float32 Y positions (mm), concatenated with
no padding between angles.

```
# angle 0's rows, then angle 1's rows, …
y_mm[0][0], …, y_mm[0][n_rows[0]-1], y_mm[1][0], …, y_mm[n_angles-1][n_rows[-1]-1]
```

Row-table boundaries for angle *a* are derived from the per-angle geometry
table: `sum(n_rows[0:a])` gives the starting index into the flattened array.

---

## Preamble Blocks

Immediately after the row table: **n_channels** length-prefixed UTF-8 strings,
one per channel in `SCAN_CHANNELS` order (CH1, CH3, CH4).  Each block is:

```
uint16  length   — byte length of the following UTF-8 string
bytes   preamble — WFMOutpre response string from the oscilloscope
```

The preamble captures per-channel scaling constants (YMULT, YOFF, YZERO) needed
to convert raw ADC values to volts.

---

## Background Block

Immediately after the preamble blocks: a single CH1 waveform captured with the
**Helios (generation) laser enabled** and the **Genesis (detection) laser
disabled**.  This provides a noise/background reference for subtraction during
post-processing.

```
uint32  n_bg_samples   — number of samples in the background waveform
int8[]  bg_data        — raw ADC samples (same encoding as waveform data)
```

`n_bg_samples` equals `samples_per_frame` under normal acquisition settings.

---

## Waveform Data (ragged)

Immediately after the background block.  Data is stored in **angle-major,
row-minor** order, but unlike earlier versions each angle contributes a
different number of rows (`n_rows[a]`) and a different number of frames per
row (`n_frames[a]`), both taken from that angle's Per-Angle Geometry Table
entry.  Within each row, channels are interleaved in ascending channel-index
order, with each channel's FastFrame data written in frame order.

```
for angle a in 0 … n_angles-1:
  for row in 0 … n_rows[a]-1:
    for channel in [CH1, CH3, CH4]:          # 3 channels, fixed order
      for frame in 0 … n_frames[a]-1:
        samples[0 … samples_per_frame-1]     # bps bytes each
```

Each sample is a raw signed ADC value.  With `bytes_per_sample = 1` this is
**int8** (−128 … +127).  With `bytes_per_sample = 2` this is **big-endian
int16**.

Total data size:
```
sum over angles a of: n_rows[a] × 3 × n_frames[a] × samples_per_frame × bytes_per_sample
```

> **Incomplete files:** If a scan is aborted the file is closed immediately and
> the data block will be shorter than the expected size. Readers should
> reconstruct the expected per-angle byte offsets from the Per-Angle Geometry
> Table and check `file_size` against the running total before reshaping —
> a fixed `(n_angles, n_rows, ...)` reshape (as in pre-v6 readers) will not
> work since row/frame counts are no longer uniform across angles.

---

## Spatial Mapping

The *k*-th waveform (frame) in a row corresponds to the *k*-th laser pulse that
hit the sample.  For a row belonging to angle *a*, the physical X position of
that pulse is:

```
x_k = x_start[a] + k * (velocity_mm_s / laser_freq_hz)
```

using that angle's `x_start` from the Per-Angle Geometry Table (not
`x_start_nominal`).

---

## Cache Tail (v7)

Present iff `version == 7` and `file_size > cache_offset`, where:

```
cache_offset = data_offset + Σ over angles a of:
                 n_rows[a] × n_channels × n_frames[a] × samples_per_frame × bytes_per_sample
```

i.e. exactly `data_offset + waveform_bytes` — the same "Total data size"
formula as Waveform Data above. This offset is derivable from the header and
Per-Angle Geometry Table alone and does **not** depend on which cache
block(s) are present, so a writer can always seek straight there without
reading or touching any waveform byte before it.

Unlike the (removed) v5 `PREC` section, which stored one omnibus per-angle
entry (FFT + both DC channels together) in a dense, uniform-geometry array,
the v7 Cache Tail splits DC and FFT into two **independent** sub-blocks —
each sized per-angle from the Per-Angle Geometry Table, each independently
present, and each independently updatable in any order, any number of
times, without disturbing the other. This matches `sras_viewer.py`'s
"Convert" menu, which exposes DC and FFT store as two separate batch
actions.

### CACH outer header (6 bytes, `">4sBB"`)

| Offset | Size | Type | Field | Description |
|--------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `cach_magic` | `CACH` (ASCII). Missing/wrong magic → treat file as having no cache. |
| 4 | 1 | `u8` | `cach_version` | Cache format version. Currently `1`. Readers must treat the file as uncached if this is not a version they understand (unlike v5's `PREC` section, which read but never validated its version byte). |
| 5 | 1 | `u8` | `block_flags` | Bit 0 = DC block (`SDCB`) follows. Bit 1 = FFT block (`SFFT`) follows, immediately after the DC block if both are present. Bits 2–7 reserved, must be zero on write. |

### DC block `SDCB` (present iff `block_flags & 0x01`)

7-byte block header, format `">4sBH"`:

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SDCB` |
| 4 | 1 | `u8` | `reserved` | `0`, reserved for future use |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow, `0 ≤ n_stored ≤ n_angles` |

followed by `n_stored` entries, each:

```
u16                          angle_idx   — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc3_mv  — CH3 waveform mean, mV, row-major
f32[n_rows[angle_idx] × n_frames[angle_idx]]  dc4_mv  — CH4 waveform mean, mV, row-major
```

Entries may appear in any order and need not be contiguous from angle 0 —
this supports storing (or re-storing) a subset of angles, or an
interrupted batch run leaving only some angles cached. Readers bounds-check
`angle_idx < n_angles` on each entry and stop parsing on an out-of-range
value, same as v5's `PREC` section.

### FFT block `SFFT` (present iff `block_flags & 0x02`)

7-byte block header, format `">4sBH"`:

| Offset (rel) | Size | Type | Field | Description |
|--------------|------|------|-------|-------------|
| 0 | 4 | `char[4]` | `magic` | `SFFT` |
| 4 | 1 | `u8` | `flags` | Bit 0 = `bg_sub_applied` — background waveform was subtracted from CH1 before the FFT when these images were computed. Bits 1–7 reserved. |
| 5 | 2 | `u16` | `n_stored` | Number of angle entries that follow |

followed by `n_stored` entries, each:

```
u16                          angle_idx      — index into the angle table (0-based)
f32[n_rows[angle_idx] × n_frames[angle_idx]]  peak_freq_mhz  — CH1 FFT peak frequency, MHz, row-major
```

**`peak_freq_mhz`** is computed without any DC-threshold masking (i.e. the
FFT is run on every pixel unconditionally, same as v5's `PREC` convention).
Readers apply the DC4 threshold at display time:

```
pixel is valid  ⟺  dc4_mv[r][f]  ≥  threshold_mv
display_value   =   peak_freq_mhz[r][f]  if valid, else 0
```

using the DC4 image from the DC block if that angle is also cached there,
else computed on demand.

Readers must fall back to real-time FFT computation (ignoring stored
`peak_freq_mhz`) under the same conditions as v5's PREC fast path: time-domain
gating is active, zero-padding (`n_fft ≠ samples_per_frame`) is requested, or
the reader's background-subtraction setting doesn't match `flags.bg_sub_applied`.

### In-place write ordering

A writer updating a file's Cache Tail must write the payload (CACH header +
whichever block(s) are present) **before** flipping the header's `version`
byte to `7`, and `truncate()` the file to the new payload's end immediately
after writing it. If the process is interrupted between the payload write
and the version-byte flip, the file is still valid v6 — v6 parsing only
bounds-checks each angle's `offset + nbytes ≤ file_size`, it never asserts
exactly how many bytes follow the last angle's waveform block — so the
interrupted write leaves harmless trailing bytes rather than a corrupt file,
and the next successful write overwrites them via the same deterministic
`cache_offset`.

---

## Acquisition Settings (fixed by sc3_aui_app.py)

| Parameter            | Value                        |
|-----------------------|------------------------------|
| Oscilloscope trigger | CH2, rising edge, 1.24 V     |
| Trigger offset       | 0 % (trigger at left edge)   |
| Sample rate          | 6.25 GS/s  (160 ps/sample)   |
| Channels recorded    | CH1, CH3, CH4                |
| Stage X velocity     | 100 mm/s                     |
| Stage X acceleration | 1500 mm/s²                   |
| Stage X trigger out  | Logic-high at max velocity   |
| Acquisition mode     | FastFrame, Normal trigger    |

---

## Version History

| Version | Change |
|---------|--------|
| 1       | One file per row; header included `angle_idx`, `row_idx`, `angle_deg`, `y_mm`. |
| 2       | One file per scan; global header with `n_angles`/`n_rows`; separate angle and row tables; three channels (CH1, CH3, CH4) per row. |
| 3       | Added preamble blocks (WFMOutpre strings) after the row table, one length-prefixed UTF-8 block per channel. |
| 4       | Added background waveform block (CH1, Helios ON / Genesis OFF) after the preamble blocks; stored as `uint32` sample count followed by raw `int8` ADC bytes. |
| 5       | (skipped) |
| 6       | Each angle now scans only the bounding box of the nominal ROI rotated by that angle instead of the AABB-expanded worst case across all angles. Header no longer carries a single global `x_start`/`x_delta`/`n_rows` — replaced with `*_nominal` reference fields plus a new Per-Angle Geometry Table (`x_start`, `x_delta`, `n_frames`, `n_rows` per angle) and a ragged Row Table / Waveform Data block sized per angle. **Not compatible with pre-v6 readers** that assume uniform geometry. `sras_viewer.py` reads v6 natively (per-angle `n_rows`/`n_frames`/`x_start`). |
| 7       | Adds an optional trailing **Cache Tail** (`CACH` section, see [Cache Tail (v7)](#cache-tail-v7)) after the ragged waveform data, holding independently-present, independently-updatable per-angle DC (`SDCB`: dc3_mv + dc4_mv) and FFT (`SFFT`: peak_freq_mhz) blocks, so previously-computed images redisplay instantly instead of being recomputed. Header / angle table / geometry table / row table / preamble blocks / background block / waveform data are byte-identical to v6 — only the version byte and the optional Cache Tail differ. Scans still come off the scope as v6; `sras_viewer.py`'s "Convert" menu ("Batch Compute DC and Store" / "Batch Compute FFT and Store") converts a file to v7 **in place** on first use, or updates an existing v7 file's cache blocks, without rewriting any waveform bytes. Supersedes the removed "Pre-process and Save as v5" workflow, which was never available for v6 sources since the flat v5 `PREC` layout can't represent per-angle geometry. |