Howdy, I have been trying to use Topaz on a large number of mics (~15000), but it fails every time I run the job on all mics at once. When I break the 15000 roughly in half, the job proceeds as usual.
When Topaz fails, there are zero mics in the preprocessed subdirectory for that respective job, and the error:
Traceback (most recent call last): File "cli/run.py", line 106, in cli.run.run_job File "cli/run.py", line 211, in cli.run.run_job_function File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/compute/jobs/topaz/run_topaz.py", line 544, in run_topaz_wrapper_train utils.run_process(rc, split_command) File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/compute/jobs/topaz/topaz_utils.py", line 143, in run_process assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})" ^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: Subprocess exited with status 1 (/home/cryosparc_user/bin/topaz.sh train_test_split --number 3036 --seed 770498622 --image-dir /scr/cs-austin/CS-ak008/J153/preprocessed /scr/cs-austin/CS-ak008/J153/topaz_particles_processed.txt)
It seems that I’ve found a work around, but I am curious if this is a me-issue or a known characteristic of Topaz. Thanks!