Topaz error: "ValueError: cannot reshape array of size 1792 into shape (1,511,720)"

Particle Picking
1

We started using topaz 0.2.5a with cryosparc V4.7.1-cuda12 on a HPC. Topaz training finished without any issue. However, when we attempted topaz extract job, we got the following error:

[CPU: 223.7 MB]

Starting extraction by running command /blue/chen.zhao/chen.zhao/.conda/envs/topaz025/bin/topaz extract --radius 23 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 5 --device 0 --model /blue/chen.zhao/xxx/XXX/cryosparc/CS-gc15-3/J11/models/model_epoch11.sav -o /blue/chen.zhao/xxx/XXX/cryosparc/CS-gc15-3/J27/topaz_particles_prediction.txt [5895 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]

[CPU: 226.7 MB]

Traceback (most recent call last):

[CPU: 226.7 MB]

File “/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/bin/topaz”, line 33, in

[CPU: 226.7 MB]

sys.exit(load_entry_point(‘topaz-em==0.2.5’, ‘console_scripts’, ‘topaz’)())

[CPU: 226.7 MB]

File “/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/lib/python3.6/site-packages/topaz/main.py”, line 148, in main

[CPU: 226.7 MB]

args.func(args)

[CPU: 226.7 MB]

File “/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/lib/python3.6/site-packages/topaz/commands/extract.py”, line 288, in main

[CPU: 226.7 MB]

for path,score,coords in nms_iterator(stream, radius, threshold, pool=pool):

[CPU: 226.7 MB]

File “/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/lib/python3.6/site-packages/topaz/commands/extract.py”, line 79, in nms_iterator

[CPU: 226.7 MB]

for name,score,coords in pool.imap_unordered(process, scores):

[CPU: 226.7 MB]

File “/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/lib/python3.6/multiprocessing/pool.py”, line 735, in next

[CPU: 226.7 MB]

raise value

[CPU: 226.7 MB]

ValueError: cannot reshape array of size 1792 into shape (1,511,720)

[CPU: 226.7 MB]

Traceback (most recent call last): File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main File “/blue/chen.zhao/apps/cryosparc/prod/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 1182, in run_topaz_wrapper_extract utils.run_process(extract_command) File “/blue/chen.zhao/apps/cryosparc/prod/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 99, in run_process assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})" AssertionError: Subprocess exited with status 1 (/blue/chen.zhao/chen.zhao/.conda/envs/topaz025/bin/topaz extract --radius 23 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 5 --device 0 --model /blue/chen.zhao/xxx/XXX…)

I suspect there may be a mismatch between cryosparc and topaz version. However, we tried topaz v0.3.0 and the training job won’t complete due to mismatch between the file name convention in the new topaz version and the expected file names in the cryosparc wrapper. Any advice is much appreciated!

Best,

Chen

2

@chenzhao May I suggest you initially try running a modified variant of the Topaz Extract job where you create an empty directory that is

  • inside the CryoSPARC project directory, such as
    /blue/chen.zhao/xxx/XXX/cryosparc/CS-gc15-3/topaz_preprocessed_1
  • owned by the Linux account that owns CryoSPARC processes
  • specified, including a trailing slash, like
    /blue/chen.zhao/xxx/XXX/cryosparc/CS-gc15-3/topaz_preprocessed_1/
    for the job’s Absolute path of directory containing preprocessed directory parameter

and let us know whether the modified variant of the job succeeds?

3

@wtempel Thank you very much for your help! I tried this and it seems good so far — definitely lasted longer than before and was able to reach a step further. I will report back if the job can be successfully completed, as the run will take a while.

4

@wtempel It took us a while but I can confirm that this method works! Only a developer can debug as strange as something like this :slight_smile: This is our first time running topaz on the HPC, and we had to tweek the # of processes and request larger memory for the job to complete.