Topaz Extract assertion error

Hi all,

I have been getting an “AssertionError: No particles extracted” error when trying to run Topaz Extract. I am extracting from 2760 micrographs, using a Topaz Train model that appeared to perform well. I did not change any parameters from the default. Full log is below:

License is valid.

Launching job on lane default target [ADDRESS HERE] ...

Running job on master node hostname [ADDRESS HERE]

[CPU: 68.3 MB]   Project P141 Job J265 Started

[CPU: 68.3 MB]   Master running v3.3.1+220315, worker running v3.3.1+220315

[CPU: 68.5 MB]   Working in directory: /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265

[CPU: 68.5 MB]   Running on lane default

[CPU: 68.5 MB]   Resources allocated: 

[CPU: 68.5 MB]     Worker:  [ADDRESS HERE]

[CPU: 68.5 MB]     CPU   :  [4, 5, 6, 7, 8, 9, 10, 11]

[CPU: 68.5 MB]     GPU   :  [2]

[CPU: 68.5 MB]     RAM   :  [2]

[CPU: 68.5 MB]     SSD   :  False

[CPU: 68.5 MB]   --------------------------------------------------------------

[CPU: 68.5 MB]   Importing job module for job type topaz_extract...

[CPU: 194.0 MB]  Job ready to run

[CPU: 194.0 MB]  ***************************************************************

[CPU: 194.0 MB]  Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
Citations:
- Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
- Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920

Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE.
***************************************************************

[CPU: 201.4 MB]  Starting Topaz process using version 0.2.3...

[CPU: 201.4 MB]  Skipping preprocessing.

[CPU: 201.4 MB]  Using preprocessed micrographs from  J251/preprocessed

[CPU: 203.2 MB]  --------------------------------------------------------------

[CPU: 203.2 MB]  Inverting negative staining...

[CPU: 204.1 MB]  Inverting negative staining complete.

[CPU: 204.1 MB]  --------------------------------------------------------------

[CPU: 204.1 MB]  Starting extraction...

[CPU: 204.1 MB]  Starting extraction by running command /home/cryosparc_user/miniconda3/envs/topaz/bin/topaz extract --radius 7 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 2 --model /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J251/models/model_epoch10.sav -o /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265/topaz_particles_prediction.txt [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]

[CPU: 204.1 MB]  UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.

[CPU: 204.1 MB]  Extraction command complete.

[CPU: 204.1 MB]  Starting particle pick thresholding by running command /home/cryosparc_user/miniconda3/envs/topaz/bin/topaz convert -t 0 -o /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265/topaz_particles_prediction_thresholded.txt /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265/topaz_particles_prediction.txt

[CPU: 204.1 MB]  Particle pick thresholding command complete.

[CPU: 204.1 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/scratch/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 1124, in run_topaz_wrapper_extract
    assert particles_extracted, "No particles extracted. Possible causes include \'Particle threshold\' parameter being too high or a version discrepancy between the Topaz or PyTorch instance that trained the model and the one that is being used for extraction."
AssertionError: No particles extracted. Possible causes include 'Particle threshold' parameter being too high or a version discrepancy between the Topaz or PyTorch instance that trained the model and the one that is being used for extraction.

[CPU: 204.1 MB]  UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.

[CPU: 204.1 MB]  Extraction command complete.

[CPU: 204.1 MB]  Starting particle pick thresholding by running command /home/cryosparc_user/miniconda3/envs/topaz/bin/topaz convert -t 0 -o /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265/topaz_particles_prediction_thresholded.txt /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J265/topaz_particles_prediction.txt

[CPU: 204.1 MB]  Particle pick thresholding command complete.

[CPU: 204.1 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/scratch/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 1124, in run_topaz_wrapper_extract
    assert particles_extracted, "No particles extracted. Possible causes include \'Particle threshold\' parameter being too high or a version discrepancy between the Topaz or PyTorch instance that trained the model and the one that is being used for extraction."
AssertionError: No particles extracted. Possible causes include 'Particle threshold' parameter being too high or a version discrepancy between the Topaz or PyTorch instance that trained the model and the one that is being used for extraction.

Hello,

Can you show the topaz command that Cryosparc ran? I would like to see the threshold value you used. Also, when you trained the model how many particles per image did you put?

Best,
-Alex

I set 30 expected particles/mrc for the training job; it seems to have performed well (see attached). This is the command used to start the extraction:

/home/cryosparc_user/miniconda3/envs/topaz/bin/topaz extract --radius 7 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 4 --model /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/[NAME HERE]/P141/J251/models/model_epoch10.sav -o /ddn/gs1/project/cryoemCore/process/cryosparc_callisto/Stanley/P141/J265/topaz_particles_prediction.txt

These are my params for the extract job, none of which are changed from default:

Downsampling factor: 4
Number of iterations: 200
Score threshold: 0
Number of parallel threads: 8
Radius of extracted regions: 7
Extraction threshold: -6
Particle threshold: 0
Max assignment radius: -1
Min extraction radius: 5
Max extraction radius: 100
Step radius: 5
Number of CPUs: 8

Not related to your question, but wonder if you happen to know about this Topaz Train error?

[CPU: 195.0 MB]  training, momentum, eps, torch.backends.cudnn.enabled

[CPU: 195.0 MB]  RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[CPU: 195.0 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/home/groups/brunger/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 346, in run_topaz_wrapper_train
    utils.run_process(train_command)
  File "/home/groups/brunger/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 98, in run_process
    assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/home/groups/brunger/ccw0820/topaz/bin/topaz train --train-images /oak/stanford/groups/brunger/vATPase/SPA/SPA_SupRes/P3/J371/image_list_train.txt --train-targets /oak/stanford/groups/brunger/vATPase/SPA/SPA_SupRes/P3/J371/topaz_particles_processed_train.t…)