Topaz Train FileNotFoundError v5.0.1

kristinay · April 9, 2026, 2:07pm

Hello!

I am writing about the following problem that has been appearing consistently during Topaz Train jobs (both the normal and the sbgrid one), causing them to fail.

The error is this:

2026-03-31 11:26:52.47\] \[CPU: 244.3 MB

FileNotFoundError: [Errno 2] No such file or directory:
‘…/J23/topaz_particles_processed_train.txt’

2026-03-31 11:26:52.59\] \[CPU: 244.3 MB

Training file appeared, beginning monitoring…

2026-03-31 11:26:58.14\] \[CPU: 246.3 MB

Traceback (most recent call last):
File “cli/run.py”, line 105, in cli.run.run_job
File “cli/run.py”, line 210, in cli.run.run_job_function
File “…/cryosparc/cs_cpusrv129_v4.7.0/cryosparc_worker/compute/jobs/topaz/run_topaz.py”, line
627, in run_topaz_wrapper_train
utils.run_process(rc, train_command, topaz_version=topaz_version_for_validation)
File “…/cryosparc/cs_cpusrv129_v4.7.0/cryosparc_worker/compute/jobs/topaz/topaz_utils.py”,
line 143, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Subprocess exited with status 1 (/programs/x86_64-linux/system/sbgrid_bin/topaz train --train-images
/…/J23/image_list_train.txt --train-targets
/…/J23/topaz_particles_processed_tra…)

As far as I understand the topaz train code needs 4 input files: image_list_train.txt, image_list_test.txt, topaz_particles_processed_train.txt and topaz_particles_processed_test.txt. The first two come from the train-test splitting step and are written in the /preprocessed subdirectory of the job directory (here J23/) and this is made clear by the “# writing: …” line of code earlier in the event log. But the other two files have no direct sign of being written beforehand. Another thing I see is that the topaz_particles_processed.txt file (which I assume is the predecessor of the train/test files) is being written in the topaz convert step in the preprocessing and lives directly in the job directory. Furthermore, performing a quick search for the topaz_particles_processed_train.txt and topaz_particles_processed_test.txt files in the job directory (after a job has already failed) returns nothing, meaning these files are not being written anywhere, thus the job fails when it tries to find topaz_particles_processed_train.txt.

Has someone experienced similar problems? This seems like a run_topaz.py code error, which I have tried to overcome with some wrapper scripts, however none were successful.

Some important info: our cryoSPARC version is v.5.0.1 and we are using the sb-grid configuration for Topaz.

I’m open to all kinds of suggestions!

Thanks,

Kristina

wtempel · April 13, 2026, 7:11pm

Welcome to the forum @kristinay . Please can you post the outputs of these commands:

/programs/x86_64-linux/system/sbgrid_bin/topaz --version
project_uid=P99 # replace P99 with actual project UID
cryosparcm job events $project_uid J23 | grep -i -A 10 topaz
ls -al $(cryosparcm cli "api.projects.find_one('$project_uid').project_dir")/J23

kristinay · April 14, 2026, 9:15am

Hello and thank you for the answer! I was wondering, is it possible to send you the outputs directly so not to post internal information on here?

wtempel · April 21, 2026, 6:31pm

Please replace confidential information, such as user identities or target names, when posting on the forum. To facilitate meaningful responses, please use “consistent” replacements (same replacement term for same confidential term).

kristinay · April 28, 2026, 10:55am

Here’s the output of the requested commands:

[CS-user@CS-master]$ /programs/x86_64-linux/system/sbgrid_bin/topaz --version
Traceback (most recent call last):
  File "/programs/x86_64-linux/topaz/0.3.7/bin/topaz", line 8, in <module>
    sys.exit(main())
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/main.py", line 60, in main
    import topaz.commands.train
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/commands/train.py", line 8, in <module>
    from topaz.training import load_data,make_model,train_model
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/training.py", line 11, in <module>
    import pandas as pd
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/__init__.py", line 37, in <module>
    from pandas._config import (
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/_config/__init__.py", line 20, in <module>
    from pandas._config import config
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/_config/config.py", line 68, in <module>
    from pandas._typing import (
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/_typing.py", line 198, in <module>
    np.random.Generator,
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/numpy/__init__.py", line 337, in __getattr__
    import numpy.random as random
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/numpy/random/__init__.py", line 180, in <module>
    from . import _pickle
  File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/numpy/random/_pickle.py", line 3, in <module>
    from .bit_generator import BitGenerator
  File "bit_generator.pyx", line 1, in init numpy.random.bit_generator
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1423, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1395, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1522, in find_spec
  File "<frozen importlib._bootstrap_external>", line 142, in _path_stat
KeyboardInterrupt
 
 
 
[CS-user@CS-master]$ project_uid=P45
 
 
 
[CS-user@CS-master]$ cryosparcm job events $project_uid J84 | grep -i -A 10 topaz
                                 "job_type": "topaz_train",
                                 "worker_bin_path": "/path-to-CS/cryosparc_worker/bin/cryosparcw",
                                 "num_gpu": 1,
                                 "num_cpu": 8,
                                 "ram_gb": 8,
                                 "run_cmd": "/path-to-CS/cryosparc_worker/bin/cryosparcw run --project P45 --job J84 --master CS-master.de --port 39000 --timeout 20000 --auth >> /path-to-data/J84/job.log 2>&1 ",
                                 "run_args": "--project P45 --job J84 --master CS-master.de --port 39000 --timeout 20000 --auth",
                                 "script_path_abs": "/path-to-data/J84/queue_sub_script.sh",
                                 "cluster_job_id": null
                               }
[2026-04-27 05:57:40] [549 MB]
--
[2026-04-27 06:06:29] [136 MB] Importing job module for job type topaz_train...
[2026-04-27 06:07:18] [276 MB] Job ready to run
[2026-04-27 06:07:18] [276 MB] ──────────────────────────────────────────────────────────────
[2026-04-27 06:07:18] [276 MB] Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
                               Citations:
                               - Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
                               - Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920
 
                               Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at:
https://github.com/tbepler/topaz/blob/master/LICENSE.
                               ***************************************************************
[2026-04-27 06:09:40] [278 MB] Starting Topaz process using version 0.3.7...
[2026-04-27 06:09:40] [278 MB] Random seed used is 1732182354
[2026-04-27 06:09:40] [278 MB] --------------------------------------------------------------
[2026-04-27 06:09:40] [278 MB] Starting preprocessing...
[2026-04-27 06:09:40] [278 MB] Using a downsampling factor of 8
[2026-04-27 06:09:40] [278 MB] Progress [▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇] 20/20 (100%)
[2026-04-27 06:09:40] [278 MB] Starting micrograph preprocessing by running command /[..]/software/sbgrid/programs/x86_64-linux/system/sbgrid_bin/topaz preprocess --scale 8 --niters 200 --num-workers 4 -o /path-to-data/J84/preprocessed
@/path-to-data/J84/preprocess_micrograph_list.txt
[2026-04-27 06:09:40] [278 MB] Preprocessing over 2 processes...
[2026-04-27 06:10:09] [278 MB] Inverting negative staining...
[2026-04-27 06:10:09] [278 MB] Inverting negative staining complete.
[2026-04-27 06:10:09] [278 MB] Micrograph preprocessing command complete.
[2026-04-27 06:10:09] [278 MB] Starting particle pick preprocessing by running command /[..]/software/sbgrid/programs/x86_64-linux/system/sbgrid_bin/topaz convert --down-scale 8 --threshold 0.0 -o /path-to-data/J84/topaz_particles_processed.txt
/path-to-data/J84/topaz_particles_raw.txt
[2026-04-27 06:10:21] [278 MB] Particle pick preprocessing command complete.
[2026-04-27 06:10:21] [278 MB] Preprocessing done in 41.121s.
[2026-04-27 06:10:21] [278 MB] --------------------------------------------------------------
[2026-04-27 06:10:21] [278 MB] Starting train-test splitting...
[2026-04-27 06:10:21] [278 MB] Starting dataset splitting by running command /[..]/software/sbgrid/programs/x86_64-linux/system/sbgrid_bin/topaz train_test_split --number 4 --seed 1732182354 --image-dir /path-to-data/J84/preprocessed
/path-to-data/J84/topaz_particles_processed.txt
[2026-04-27 06:10:27] [278 MB] # splitting 20 micrographs with 1766 labeled particles into 16 train and 4 test micrographs
[2026-04-27 06:10:27] [278 MB] # writing: /path-to-data/J84/preprocessed/003474868378360430238_FoilHole_11638812_Data_11633726_10_20250912_072900_EER_patch_aligned_doseweighted_train.txt
[2026-04-27 06:10:27] [278 MB] # writing: /path-to-data/J84/preprocessed/003474868378360430238_FoilHole_11638812_Data_11633726_10_20250912_072900_EER_patch_aligned_doseweighted_test.txt
[2026-04-27 06:10:27] [278 MB] # writing: /path-to-data/J84/preprocessed/image_list_train.txt
[2026-04-27 06:10:27] [278 MB] # writing: /path-to-data/J84/preprocessed/image_list_test.txt
[2026-04-27 06:10:33] [278 MB]
                               Dataset splitting command complete.
[2026-04-27 06:10:33] [278 MB] Train-test splitting done in 11.532s.
[2026-04-27 06:10:33] [278 MB] --------------------------------------------------------------
[2026-04-27 06:10:33] [278 MB] Starting training...
[2026-04-27 06:10:33] [278 MB] Starting training by running command /[..]/software/sbgrid/programs/x86_64-linux/system/sbgrid_bin/topaz train --train-images /path-to-data/J84/image_list_train.txt --train-targets
/path-to-data/J84/topaz_particles_processed_train.txt -s 0 -p 0 --test-images /path-to-data/J84/image_list_test.txt --test-targets /path-to-data/J84/topaz_particles_processed_test.txt --num-particles
125 --learning-rate 0.0002 --minibatch-size 128 --num-epochs 10 --method GE-binomial --slack -1.0 --autoencoder 0.0 --l2 0.0 --minibatch-balance 0.0625 --epoch-size 5000 --model resnet8 --units 32 --dropout 0.0 --bn on --unit-scaling 2 --ngf 32 --num-workers 4 --cross-validation-seed 1732182354 --radius 3 --num-particles 125
--device 0 --no-pretrained --save-prefix=/path-to-data/J84/models/model -o /path-to-data/J84/train_test_curve.txt
[2026-04-27 06:10:33] [278 MB] Starting training progress monitoring for 10 epochs...
[2026-04-27 06:10:33] [278 MB] Monitoring file: /path-to-data/J84/train_test_curve.txt
[2026-04-27 06:10:33] [278 MB] Epoch size: 5000 iterations
[2026-04-27 06:10:39] [278 MB] # Loading model: resnet8
[2026-04-27 06:10:39] [278 MB] # Model parameters: units=32, dropout=0.0, bn=on
[2026-04-27 06:10:39] [278 MB] # Receptive field: 71
[2026-04-27 06:10:39] [278 MB] # Using device=0 with cuda=True
[2026-04-27 06:10:40] [278 MB] # When using GPU to load data, we only load in this process. Setting num_workers = 0.
--
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/bin/topaz", line 8, in <module>
[2026-04-27 06:10:40] [278 MB] sys.exit(main())
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/main.py", line 148, in main
[2026-04-27 06:10:40] [278 MB] args.func(args)
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/commands/train.py", line 140, in main
[2026-04-27 06:10:40] [278 MB] classifier = train_model(classifier, args.train_images, args.train_targets, args.test_images, args.test_targets,
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/training.py", line 607, in train_model
[2026-04-27 06:10:40] [278 MB] num_positive_regions, total_regions, num_images = report_data_stats(train_images_path, train_targets_path, test_images_path, test_targets_path,
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/training.py", line 284, in report_data_stats
[2026-04-27 06:10:40] [278 MB] train_targets = file_utils.read_coordinates(train_targets_path)
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/topaz/utils/files.py", line 201, in read_coordinates
[2026-04-27 06:10:40] [278 MB] particles = pd.read_csv(path, sep='\t', dtype={'image_name':str})
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
[2026-04-27 06:10:40] [278 MB] return _read(filepath_or_buffer, kwds)
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 620, in _read
[2026-04-27 06:10:40] [278 MB] parser = TextFileReader(filepath_or_buffer, **kwds)
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
[2026-04-27 06:10:40] [278 MB] self._engine = self._make_engine(f, self.engine)
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
[2026-04-27 06:10:40] [278 MB] self.handles = get_handle(
[2026-04-27 06:10:40] [278 MB] File "/programs/x86_64-linux/topaz/0.3.7/miniconda/lib/python3.9/site-packages/pandas/io/common.py", line 873, in get_handle
[2026-04-27 06:10:40] [278 MB] handle = open(
[2026-04-27 06:10:40] [278 MB] FileNotFoundError: [Errno 2] No such file or directory: '/path-to-data/J84/topaz_particles_processed_train.txt'
[2026-04-27 06:10:42] [278 MB] Training file appeared, beginning monitoring...
[2026-04-27 06:10:46] [274 MB] Traceback (most recent call last):
                                 File "cli/run.py", line 106, in cli.run.run_job
                                 File "cli/run.py", line 211, in cli.run.run_job_function
                                 File "/path-to-CS/cryosparc_worker/compute/jobs/topaz/run_topaz.py", line 627, in run_topaz_wrapper_train
                                   utils.run_process(rc, train_command, topaz_version=topaz_version_for_validation)
                                 File "/path-to-CS/cryosparc_worker/compute/jobs/topaz/topaz_utils.py", line 143, in run_process
                                   assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
                                          ^^^^^^^^^^^^^^^^^^^^^^^
                               AssertionError: Subprocess exited with status 1 (/[..]/software/sbgrid/programs/x86_64-linux/system/sbgrid_bin/topaz train --train-images /path-to-data/J84/image_list_train.txt --train-targets
/[..]/software/cryosparc/data/C…)
 
 
 
[CS-user@CS-master]$ ls -al $(cryosparcm cli "api.projects.find_one('$project_uid').project_dir")/J84
total 528
drwxr-xr-x+  5 CS-user CS-group   4096 Apr 27 08:10 .
drwxr-xr-x+ 88 CS-user CS-group   4096 Apr 27 07:57 ..
-rw-r--r--+  1 CS-user CS-group  27378 Apr 27 08:10 events.bson
drwxr-xr-x+  2 CS-user CS-group   4096 Apr 27 07:57 gridfs_data
-rw-r--r--+  1 CS-user CS-group  71444 Apr 27 08:10 job.json
-rw-r--r--+  1 CS-user CS-group   9223 Apr 27 08:10 job.log
drwxr-xr-x+  2 CS-user CS-group   4096 Apr 27 08:10 models
-rw-r--r--+  1 CS-user CS-group      0 Apr 27 08:05 P45_J84_slurm.err
-rw-r--r--+  1 CS-user CS-group      0 Apr 27 08:05 P45_J84_slurm.out
drwxr-xr-x+  2 CS-user CS-group   4096 Apr 27 08:10 preprocessed
-rw-r--r--+  1 CS-user CS-group   1367 Apr 27 07:57 queue_sub_script.sh
-rw-r--r--+  1 CS-user CS-group 196321 Apr 27 08:10 topaz_particles_processed.txt
-rw-r--r--+  1 CS-user CS-group 201436 Apr 27 08:10 topaz_particles_raw.txt
-rw-r--r--+  1 CS-user CS-group      0 Apr 27 08:10 train_test_curve.txt

The first command stalls and has to be killed to free the shell. I think it’s a NFS/overcrowded server error.

kstachowski · May 7, 2026, 5:09pm

Hi @kristinay,

I’m sorry for the delayed reply! I tried to replicate this error using Topaz v0.3.10 and Topaz v0.3.18 and I have not observed the behavior in which you report. Have you tried to install Topaz in a standalone environment instead of using the version packaged via sb-grid?

Best,
Kye

kristinay · May 11, 2026, 1:13pm

Hi @kstachowski, I have not tried this yet. I’ll try it and I’ll write here again to share the results. Thank you!

kristinay · August 12, 2026, 12:16pm

Hello again, and I’m sorry for how long it took me to come back with an answer. In the end, installing Topaz in a standalone environment worked and now Topaz is running without any problems. Thank you for the help!

Best,

Kristina