tka/sras-viewer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bound nested parallelism, make compute cancellable, harden batch pool

Browse Source 
Three defects found by testing the parallel paths end to end.

Nested parallelism was unbounded. DcPrecomputeWorker and the alignment
driver both parallelise over angles, and each angle's compute_dc_image
then parallelised over rows again: 14 x 14 threads, and — worse — every
angle sized its chunk against the *whole* memory budget, so worst-case
live buffers were ~13.9 GB against a 1 GB budget. Capping the inner
worker count alone does not fix it; the chunk is still sized against the
full budget. plan_angle_level now hands each angle both max_workers=1
and its share of the budget, which bounds the real file at 1002 MB.

Chunk planning could not produce concurrency at all. _plan_chunks sized
chunk_rows first, and _chunk_rows_for spends whatever budget it is given
— so a single chunk consumed the lot and budget//chunk_bytes came back
as one worker. It now picks the worker count first and sizes the chunk to
it, giving 7-16 workers on the real file instead of 1.

Cancellation was too coarse to shut down. stop() was only checked between
angles, and one angle of the real scan is ~40 s, so closing the window
sat through it and returned with threads still running. should_stop is
now polled per row chunk and closeEvent signals every worker before
waiting: close went from 4.0 s with two live threads to 1.04 s with both
finished. A cancelled ComputeWorker emits None so a half-filled image is
never cached.

Also: BatchCacheWorker no longer reports every file as failed when the
process pool itself dies (unguarded __main__, frozen build, sandbox) —
it retries those files in-process. Batches below 512 MB skip the pool
entirely, since interpreter startup would otherwise make small jobs
slower. cache_file rejects an unknown mode instead of silently treating
it as "fft".

Measured on the real 496 GB scan, angle 3, 32 rows:
  DC   2.94s -> 0.75s   RF  6.10s -> 1.99s   peak RSS 1.40 -> 1.93 GB
Batch convert, 24 files / 1 GB: 4.03s -> 2.36s.
Memory budget defaults to 1024 MB (SRAS_MEM_BUDGET_MB), the measured knee.

Testing: tools/test_refactor.py (70 checks: cache round-trip and block
carry-forward, partial-cache handling, parallel-vs-serial identity,
no-mask path, ROI mask vs full-grid, v2/v3/v4 parsing, sras_average
round-trip incl. remainder handling) and tools/test_gui.py (46 checks
driving the real widgets and workers headlessly). check_equivalence.py
still reports all 167 outputs identical to ed0eba4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Thomas Ales
2026-07-31 00:09:46 -05:00
parent 55a4c5e42f
commit 9a7cc396e7
6 changed files with 909 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sras_workers.py
+118 -36
View File
@@ -9,6 +9,7 @@ everything they need through their constructor and hand results back by signal.
import os
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
from concurrent.futures.process import BrokenProcessPool
import numpy as np
from PyQt6.QtCore import QObject, pyqtSignal
@@ -27,6 +28,31 @@ from sras_format import CH3_IDX, CH4_IDX, SrasFile
_BATCH_MAX_PROCS = int(os.environ.get("SRAS_BATCH_PROCS", 0)) or min(
4, os.cpu_count() or 2)
# Spawning a pool costs roughly a second of interpreter startup (each child
# re-imports the entry module). That is noise against a multi-GB scan but
# dominates a batch of small files, where it would make the job *slower* —
# so below this total size the batch just runs in the worker thread.
_BATCH_POOL_MIN_BYTES = int(os.environ.get("SRAS_BATCH_POOL_MIN_MB", 512)) * 1024 * 1024
class CancellableWorker(QObject):
"""A worker whose compute polls stop() between row chunks.
Without this a shutdown has to wait out whatever is in flight, and on a
large scan a single angle is ~40 s — far too long to block closing the
window. Chunk-level polling bounds the wait to one chunk instead.
"""
def __init__(self):
super().__init__()
self._stop = False
def stop(self):
self._stop = True
def _stopped(self) -> bool:
return self._stop
class LoadWorker(QObject):
finished = pyqtSignal(object) # SrasFile | None
@@ -44,7 +70,7 @@ class LoadWorker(QObject):
self.finished.emit(None)
class ComputeWorker(QObject):
class ComputeWorker(CancellableWorker):
"""Computes one displayable image for (angle, channel).
For CH1/Velocity (FFT-derived) channels, the FFT is only run for pixels
@@ -79,15 +105,19 @@ class ComputeWorker(QObject):
img = compute_rf_image(
self._sras, self._angle, dc_threshold_mv=self._dc_threshold,
apply_bg_sub=self._apply_bg_sub, n_fft=self._n_fft,
dc4_mv=self._dc4_mv)
dc4_mv=self._dc4_mv, should_stop=self._stopped)
else:
img = dc_image_mv(self._sras, self._angle, self._ch)
self.finished.emit(img)
img = dc_image_mv(self._sras, self._angle, self._ch,
should_stop=self._stopped)
# On cancellation the image is only partly filled, so hand back
# None rather than something that would be cached as real. The
# signal still fires either way — it is what quits the thread.
self.finished.emit(None if self._stop else img)
except Exception as exc:
self.error.emit(str(exc))
class DcPrecomputeWorker(QObject):
class DcPrecomputeWorker(CancellableWorker):
"""Computes CH3/CH4 DC images for every angle in the background.
DC images are cheap (a per-waveform mean, no FFT) compared to the
@@ -108,30 +138,34 @@ class DcPrecomputeWorker(QObject):
def __init__(self, sras: SrasFile):
super().__init__()
self._sras = sras
self._stop = False
def stop(self):
self._stop = True
def _one_angle(self, a: int) -> tuple[int, np.ndarray, np.ndarray]:
return a, dc_image_mv(self._sras, a, CH3_IDX), dc_image_mv(self._sras, a, CH4_IDX)
# max_workers=1 *and* a budget share: this call is one of several
# concurrent angles, and both the thread count and the buffer size
# have to be divided (see compute.plan_angle_level).
kw = dict(max_workers=1, budget=self._angle_budget,
should_stop=self._stopped)
return (a,
dc_image_mv(self._sras, a, CH3_IDX, **kw),
dc_image_mv(self._sras, a, CH4_IDX, **kw))
def run(self):
try:
n = self._sras.n_angles
n_workers = max(1, min(compute._MAX_WORKERS, n))
with ThreadPoolExecutor(max_workers=n_workers) as pool:
n_workers, self._angle_budget = compute.plan_angle_level(self._sras)
pool = ThreadPoolExecutor(max_workers=n_workers)
try:
futures = {pool.submit(self._one_angle, a): a for a in range(n)}
try:
for fut in as_completed(futures):
if self._stop:
break
a, dc3, dc4 = fut.result()
self.angle_done.emit(a, dc3, dc4)
finally:
for fut in as_completed(futures):
if self._stop:
for fut in futures:
fut.cancel()
break
a, dc3, dc4 = fut.result()
self.angle_done.emit(a, dc3, dc4)
finally:
# cancel_futures drops the queued angles; should_stop lets the
# in-flight ones bail within a chunk. Not waiting here is what
# keeps closing the window responsive on a large scan.
pool.shutdown(wait=not self._stop, cancel_futures=True)
self.finished.emit()
except Exception as exc:
self.error.emit(str(exc))
@@ -163,24 +197,27 @@ class BatchCacheWorker(QObject):
self._mode = mode
self._apply_bg_sub = apply_bg_sub
def _pool(self, n_files: int):
"""A process pool, falling back to threads if the platform refuses to
spawn (still a win: the compute releases the GIL)."""
n = max(1, min(_BATCH_MAX_PROCS, n_files))
try:
return ProcessPoolExecutor(max_workers=n), n
except (OSError, ValueError):
return ThreadPoolExecutor(max_workers=n), n
def _report(self, path: str, err: str, done: int, total: int):
self.file_done.emit(path, err)
self.progress.emit(int(done / max(1, total) * 100))
def run(self):
paths = self._paths
executor, n_procs = self._pool(len(paths))
def _run_pooled(self, paths: list[str], n_procs: int) -> list[str]:
"""Process the batch across *n_procs* subprocesses. Returns the paths
that never got a real answer because the pool itself died, so the
caller can retry them in-process.
Under spawn each child re-imports the entry module, so the batch must
survive that going wrong (an unguarded __main__, a frozen build, a
sandbox that forbids subprocesses) rather than reporting every file as
failed — hence the retry list instead of a per-file error.
"""
# Each child threads internally; divide the machine rather than
# letting every process claim every core.
per_proc_workers = max(1, (os.cpu_count() or 4) // n_procs)
unresolved: list[str] = []
done = 0
with executor:
with ProcessPoolExecutor(max_workers=n_procs) as executor:
futures = {
executor.submit(cache_file, p, self._mode, self._apply_bg_sub,
compute.get_fft_backend(), per_proc_workers): p
@@ -190,11 +227,56 @@ class BatchCacheWorker(QObject):
path = futures[fut]
try:
err = fut.result()
except BrokenProcessPool:
unresolved.append(path)
continue
except Exception as exc:
err = str(exc)
done += 1
self.file_done.emit(path, err)
self.progress.emit(int(done / max(1, len(paths)) * 100))
self._report(path, err, done, len(paths))
return unresolved
def _run_inline(self, paths: list[str], done: int, total: int):
"""Fallback / single-file path: compute in this thread. Still uses the
full core count internally, since nothing else is competing."""
for path in paths:
try:
err = cache_file(path, self._mode, self._apply_bg_sub,
compute.get_fft_backend(), compute._MAX_WORKERS)
except Exception as exc:
err = str(exc)
done += 1
self._report(path, err, done, total)
def _worth_pooling(self, paths: list[str]) -> bool:
if len(paths) < 2:
return False
total = 0
for p in paths:
try:
total += os.path.getsize(p)
except OSError:
pass # unreadable files are reported by cache_file
return total >= _BATCH_POOL_MIN_BYTES
def run(self):
paths = self._paths
n_procs = max(1, min(_BATCH_MAX_PROCS, len(paths)))
if not self._worth_pooling(paths):
self._run_inline(paths, 0, len(paths))
self.finished.emit()
return
try:
unresolved = self._run_pooled(paths, n_procs)
except Exception:
# The pool could not be created or collapsed wholesale.
unresolved = list(paths)
if unresolved:
self._run_inline(unresolved, len(paths) - len(unresolved), len(paths))
self.finished.emit()