Bound nested parallelism, make compute cancellable, harden batch pool

Three defects found by testing the parallel paths end to end.

Nested parallelism was unbounded. DcPrecomputeWorker and the alignment
driver both parallelise over angles, and each angle's compute_dc_image
then parallelised over rows again: 14 x 14 threads, and — worse — every
angle sized its chunk against the *whole* memory budget, so worst-case
live buffers were ~13.9 GB against a 1 GB budget. Capping the inner
worker count alone does not fix it; the chunk is still sized against the
full budget. plan_angle_level now hands each angle both max_workers=1
and its share of the budget, which bounds the real file at 1002 MB.

Chunk planning could not produce concurrency at all. _plan_chunks sized
chunk_rows first, and _chunk_rows_for spends whatever budget it is given
— so a single chunk consumed the lot and budget//chunk_bytes came back
as one worker. It now picks the worker count first and sizes the chunk to
it, giving 7-16 workers on the real file instead of 1.

Cancellation was too coarse to shut down. stop() was only checked between
angles, and one angle of the real scan is ~40 s, so closing the window
sat through it and returned with threads still running. should_stop is
now polled per row chunk and closeEvent signals every worker before
waiting: close went from 4.0 s with two live threads to 1.04 s with both
finished. A cancelled ComputeWorker emits None so a half-filled image is
never cached.

Also: BatchCacheWorker no longer reports every file as failed when the
process pool itself dies (unguarded __main__, frozen build, sandbox) —
it retries those files in-process. Batches below 512 MB skip the pool
entirely, since interpreter startup would otherwise make small jobs
slower. cache_file rejects an unknown mode instead of silently treating
it as "fft".

Measured on the real 496 GB scan, angle 3, 32 rows:
  DC   2.94s -> 0.75s   RF  6.10s -> 1.99s   peak RSS 1.40 -> 1.93 GB
Batch convert, 24 files / 1 GB: 4.03s -> 2.36s.
Memory budget defaults to 1024 MB (SRAS_MEM_BUDGET_MB), the measured knee.

Testing: tools/test_refactor.py (70 checks: cache round-trip and block
carry-forward, partial-cache handling, parallel-vs-serial identity,
no-mask path, ROI mask vs full-grid, v2/v3/v4 parsing, sras_average
round-trip incl. remainder handling) and tools/test_gui.py (46 checks
driving the real widgets and workers headlessly). check_equivalence.py
still reports all 167 outputs identical to ed0eba4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Thomas Ales
2026-07-31 00:09:46 -05:00
parent 55a4c5e42f
commit 9a7cc396e7
6 changed files with 909 additions and 69 deletions
+12 -4
View File
@@ -1624,6 +1624,8 @@ class SrasViewerWindow(QMainWindow):
def _on_compute_done(self, result):
self._close_progress("main")
if result is None:
return # cancelled mid-compute; the partial image must not cache
angle_idx = self._pending_angle
ch_idx = self._pending_ch
@@ -1883,11 +1885,17 @@ class SrasViewerWindow(QMainWindow):
# ------------------------------------------------------------------
def closeEvent(self, event):
if self._dc_precompute_worker is not None:
self._dc_precompute_worker.stop()
for thread, _worker, _on_done in list(self._jobs.values()):
# Signal every cancellable worker first, then wait. Waiting without
# signalling means sitting out whatever is in flight — on a large
# scan a single angle is ~40 s.
jobs = list(self._jobs.values())
for _thread, worker, _on_done in jobs:
stop = getattr(worker, "stop", None)
if callable(stop):
stop()
for thread, _worker, _on_done in jobs:
thread.quit()
thread.wait(2000)
thread.wait(5000)
super().closeEvent(event)