Topaz Extract Error: 'no particles_prediction_threshold.txt' found

Troubleshooting
1

While running Topaz Extract (Beta) I ran into the following error:

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/topaz/run_topaz.py”, line 858, in run_topaz_wrapper_extract
with open(particles_out_path_thresholded, ‘rb’) as f:
IOError: [Errno 2] No such file or directory: u’/J83/topaz_particles_prediction_thresholded.txt’

After checking the job directory, I can locate the output from the first command ‘topaz preprocess’. But, I cannot locate the .txt output from second command: ‘topaz extract’, even though ‘topaz extract’ completes, and the job continues. It appears the .txt file was never generated, and now the job cannot proceed.

How is it recommended I resolve this issue?

Best,

2

Hi @010010,

I suspect that the extract command in the job had an issue despite completing.

Could you post the entire job log output of the job? It would help me narrow down the cause of the issue.

Regards,
Jay Yoo

3 
Project P11 Job J85
Master viper.bio.edu Port 39002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 23066
========= monitor process now waiting for main process
MAIN PID 23066
topaz.run_topaz cryosparc2_compute.jobs.jobregister
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
***************************************************************
Running job on hostname %s viper.bio.edu
Allocated Resources :  {u'lane': u'default', u'target': {u'lane': u'default', u'name': u'viper.bio.edu', u'title': u'Worker node viper.bio.edu', u'resource_slots': {u'GPU': [0, 1], u'RAM': [
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]}, u'hostname': u'viper.bio.edu', u'worker_bin_path': u'/net/home/cryosp
arc/cryosparc-software/cryosparc2_worker/bin/cryosparcw', u'cache_path': u'/viper-fast-scratch', u'cache_quota_mb': None, u'resource_fixed': {u'SSD': True}, u'cache_reserve_mb': 10000, u'type': u'node', u
'ssh_str': u'cryosparc@viper.bio.edu', u'desc': None}, u'license': False, u'hostname': u'viper.bio.edu', u'slots': {u'GPU': [0], u'RAM': [0], u'CPU': [0]}, u'fixed': {u'SSD': False}, u'lane_
type': u'default', u'licenses_acquired': 0}
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main
  File "cryosparc2_compute/jobs/topaz/run_topaz.py", line 858, in run_topaz_wrapper_extract
    with open(particles_out_path_thresholded, 'rb') as f:
IOError: [Errno 2] No such file or directory: u'/net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction_thresholded.txt'
========= main process now complete.
========= monitor process now complete.
4

Hi @010010,

Could you post the all streamlogs from the job in the webapp as well? You can copy and paste the output of the “Overview” tab of the job. An example of the streamlog is attached.

event_logs_output
event_logs_output2638×1434 231 KB

Regards,
Jay Yoo

5

License is valid.

Launching job on lane default target viper.bio.purdue.edu …

Running job on master node hostname viper.bio.purdue.edu

Project P11 Job J85 Started

Master running v2.12.4, worker running v2.12.4

Running on lane default

Resources allocated:

Worker: viper.bio.edu

CPU : [0]

GPU : [0]

RAM : [0]

SSD : False

Importing job module for job type topaz_extract…

Job ready to run

Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
Citations:

  • Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
  • Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (209) doi: https://doi.org/10.1101/838920

Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE.

Starting Topaz process using version 0.1.0…

Starting preprocessing…

Starting micrograph preprocessing by running command /apps/miniconda3/envs/topaz/bin/topaz preprocess --scale 16 --niters 200 --num-workers 0 -o /net/home/cryosparc/cryosparc-projects/P11/J85/preprocessed [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]

logp tolerance reached

Micrograph preprocessing command complete.

Preprocessing done in 140.362s.

Starting extraction…

Starting extraction by running command /apps/miniconda3/envs/topaz/bin/topaz extract --radius 7 --threshold -6 --up-scale 16 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 0 --device 0 --model /net/home/cryosparc/cryosparc-projects/P11/J80/models/model_epoch8.sav -o /net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction.txt [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]

Traceback (most recent call last):

File “/apps/miniconda3/envs/topaz/bin/topaz”, line 11, in 

load_entry_point('topaz==0.1.0', 'console_scripts', 'topaz')()

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py”, line 144, in main

args.func(args)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/extract.py”, line 208, in main

threshold = np.percentile(scores_concat, percentile)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/numpy/lib/function_base.py”, line 3705, in percentile

raise ValueError("Percentiles must be in the range [0, 100]")

ValueError: Percentiles must be in the range [0, 100]

Extraction command complete.

Starting particle pick thresholding by running command /apps/miniconda3/envs/topaz/bin/topaz convert -t 0 -o /net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction_thresholded.txt /net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction.txt

Traceback (most recent call last):

File “/apps/miniconda3/envs/topaz/bin/topaz”, line 11, in 

load_entry_point('topaz==0.1.0', 'console_scripts', 'topaz')()

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py”, line 144, in main

args.func(args)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/convert.py”, line 149, in main

coords = file_utils.read_coordinates(path, format=from_forms[i])

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/utils/files.py”, line 140, in read_coordinates

particles = pd.read_csv(path, sep='\t')

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/io/parsers.py”, line 702, in parser_f

return _read(filepath_or_buffer, kwds)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/io/parsers.py”, line 429, in _read

parser = TextFileReader(filepath_or_buffer, **kwds)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/io/parsers.py”, line 895, in init

self._make_engine(self.engine)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/io/parsers.py”, line 1122, in _make_engine

self._engine = CParserWrapper(self.f, **self.options)

File “/apps/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/io/parsers.py”, line 1853, in init

self._reader = parsers.TextReader(src, **kwds)

File “pandas/_libs/parsers.pyx”, line 387, in pandas._libs.parsers.TextReader.cinit

File “pandas/_libs/parsers.pyx”, line 705, in pandas._libs.parsers.TextReader._setup_parser_source

FileNotFoundError: [Errno 2] File b’/net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction.txt’ does not exist: b’/net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction.txt’

Particle pick thresholding command complete.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/topaz/run_topaz.py”, line 858, in run_topaz_wrapper_extract
with open(particles_out_path_thresholded, ‘rb’) as f:
IOError: [Errno 2] No such file or directory: u’/net/home/cryosparc/cryosparc-projects/P11/J85/topaz_particles_prediction_thresholded.txt’

6

Hi @010010,

You seem to be running Topaz on version 0.1.0, could you update to the latest version which is version 0.2.3 as of the creation of this post. Just in case PyTorch is also outdated, try updating PyTorch as well. These can be done by following the installation instructions found here https://github.com/tbepler/topaz under the “Installation” section. It is recommended to use the Anaconda installation if you have not done so already.

Regards,
Jay Yoo

7

This has resolved the issue.

Thank you!