I finally tracked down the root issues with running Topaz under CryoSPARC, and yesterday I processed about 5,600 micrographs in a single job several times without a stall. Two fixes are needed. First, Topaz itself needs a small code change, adding a gc.collect() call during preprocessing so worker memory doesn’t keep growing. Second, CryoSPARC’s Topaz wrapper needs a fix, because with more than one parallel process every micrograph gets preprocessed three times. Until the wrapper is fixed, use 1 parallel process with multiple CPUs. I’ve tested this on Topaz 0.2.5a and will check 0.3.x when I get a chance.
1. Topaz Train/Extract stall silently after preprocessing (root cause: memory growth in topaz preprocess workers)
Symptom: The job stalls at the transition from preprocessing to training or extraction, with no error in the job log. Rerunning on already-preprocessed micrographs works.
Cause: Each topaz preprocess worker keeps growing in memory with every micrograph it processes. With 2 processes × 4 workers, the SLURM job reached its 32 GB cgroup limit (CONSTRAINT_MEMCG). The kernel then killed one worker (about 8.5 GB including swap). Python’s multiprocessing.Pool started a replacement worker, but the micrograph the killed worker was holding was lost, so imap_unordered waits forever. CryoSPARC shows no error.
Measurements (single worker, /usr/bin/time -v, maximum RSS):
| Micrographs |
Peak RSS |
| 20 |
2.58 GB |
| 100 |
5.45 GB |
That’s about 36 MB retained per micrograph, growing without bound.
-
MALLOC_ARENA_MAX=2, OMP_NUM_THREADS=1, and MKL_NUM_THREADS=1 had no effect, so this is not glibc arena fragmentation.
-
Calling gc.collect() every 10 micrographs brought RSS from about 1.07 GB back down to 0.36 GB each time, while malloc_trim(0) added nothing. The memory is held in reference cycles that Python’s cyclic garbage collector frees too rarely, because large arrays account for few Python objects. It is not a true leak.
Fix (two lines in topaz/commands/normalize.py, 0.2.5a):
diff
import torch
+import gc
...
def __call__(self, path):
...
+ gc.collect()
return name
With the patch, 100 micrographs peak at 0.92 GB (down from 5.45 GB), memory stays flat regardless of dataset size, and the full job runs with about 150–250 MB per worker.
Note on Topaz 0.3.x: I compared the source. The same Normalize class (now in topaz/stats.py) and the same Pool/imap_unordered loop are still used, gmm_fit/norm_fit/downsample are unchanged from 0.2.5, and no gc.collect() was added. The only relevant change is that micrographs are no longer wrapped in a PIL Image. I have not tested 0.3.x for the memory growth itself, so it may or may not still be affected.
2. “Preprocessing over 2 processes” preprocesses every micrograph three times
Observation: The job log shows one topaz preprocess command with 5,071 micrograph paths, followed by “Preprocessing over 2 processes…”. The two topaz preprocess processes actually running contain 15,213 .mrc paths, exactly 3 × 5,071, with only 5,071 unique.
Interpretation (inferred from the counts and argument order): each process seems to receive the full list plus its own half: 2 × 5,071 + 5,071 = 15,213. In the argument list, the full list runs to the end and then restarts partway through, at the beginning of the second half.
Impact: about 3× the preprocessing time. Without the gc fix above, it also means about 3× the memory buildup, which makes the OOM stall in issue 1 much more likely. Near the end, files are simply overwritten with identical results.
Workaround: set Parallel processes = 1 and increase the CPU count instead.
To check on your own job:
bash
for p in $(pgrep -f "bin/topaz preprocess"); do
echo "$p: $(tr '\0' '\n' </proc/$p/cmdline | grep -c '\.mrc$') paths, $(tr '\0' '\n' </proc/$p/cmdline | grep '\.mrc$' | sort -u | wc -l) unique"
done
3. Topaz Extract reuses corrupted preprocessed micrographs from the training job without validating them
-
Topaz Extract reuses <training job>/preprocessed and only preprocesses micrographs that are missing (“Found N processed micrograph(s)… An additional M require preprocessing”).
-
If an earlier run failed mid-write, a truncated or corrupted .mrc stays in that folder. A restarted extract job counts it as processed, and the job then fails later, during picking.
-
Topaz Train, by contrast, deletes the whole folder on restart and redoes all preprocessing.
-
Related: because extract writes into the training job’s folder, an extract run with a different --scale would silently overwrite the training job’s micrographs with differently sized ones.
Workaround: before restarting, check the folder. Every valid file has an identical, predictable size, 1024 + nx·ny·4 bytes, which is 5,893,504 bytes for scale-4 K3 micrographs. Move any file whose size doesn’t match its header out of the folder. I wrote a small script that checks the header against the file size, flags dimensions that differ from the rest, and optionally checks for NaN/Inf, then quarantines the bad files. Happy to share it.
4. Minor: misleading progress bar
The log prints Progress [...] 5071/5071 (100%) before the preprocessing command starts. The job then appears frozen at 100% for the entire preprocessing step, which makes it hard to tell a working job from a stalled one.
Suggestions for the CryoSPARC Topaz wrapper
-
Fix how the micrograph list is split for Parallel processes > 1.
-
Validate reused preprocessed files (header vs. file size at minimum), or have preprocessing write to a temporary name and rename once complete, so interrupted runs never leave files that look valid.
-
Detect a dead or replaced preprocessing worker (or no new output for a long time) and fail the job with a clear message instead of hanging.
-
Show real preprocessing progress, for example by counting output files.
-
Consider a per-job preprocessed folder for extract when its preprocessing settings differ from the training job’s.
Happy to provide logs or test fixes.