Topaz Extract:ValueError: cannot reshape array of size

Christian · August 1, 2024, 3:33pm

Hi,
we are recently facing issues using the Topaz Extract job in Cryosparc, which reproducibly but inconsistently gives a “ValueError”:

ValueError: cannot reshape array of size 1048320 into shape (1,1023,1440)

Here’s what we’ve done:
We divided our large dataset into 8 subsets of 5k movies, followed by motion correction and CTF estimation.
After blob picking, 2D classification and ab-inito model creation we used the best particles to train a topaz model.
Using this model in Topaz Extract, we could pick (“extract”) on all subsets of micrographs without problems.
Problems arouse after the second iteration of Topaz training with the improved particle set from the first round. The Topaz Train job finished as usual (with no error message) but the Topaz Extract job now crashes with below error message on some of the 8 subsets. The same subsets gave no error in the first iteration. And other subsets show no error at all.

Searching the discussions here hinted at faulty micrograph files, but using the script provided by nfrasser we could not identify any corrupted micrographs.

Thanks a lot for any help or hint or suggestion.

Christian + Timo

wtempel · August 1, 2024, 6:57pm

Welcome to the forum @Christian
Please can you post the outputs of these commands:

cryosparcm cli "get_job('P99', 'J199', 'job_type', 'version', 'params_spec', 'instance_information', 'input_slot_groups', 'status')"
cryosparcm cli "get_job('P99', 'J201', 'job_type', 'version', 'params_spec', 'instance_information', 'input_slot_groups', 'status')"
cryosparcm eventlog P99 J201 | tail -n 40

where you replace:

P99 with the project ID
J199 with the job ID of the Topaz Train job
J201 with the job ID of the failed Topaz Extract job

Christian · August 2, 2024, 1:28pm

Hi @wtempel
Thanks for the prompt reply! Please find below the requested output:

{'_id': '668aea30e37619eedbc83cf4', 'input_slot_groups': [{'connections': [{'group_name': 'split_0', 'job_uid': 'J32', 'slots': [{'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'split_0', 'job_uid': 'J32', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Micrographs for training Topaz', 'name': 'micrographs', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'micrograph_blob', 'optional': False, 'title': 'Raw micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'ctf', 'optional': True, 'title': 'CTF parameters', 'type': 'exposure.ctf'}, {'description': '', 'name': 'micrograph_blob_denoised', 'optional': True, 'title': 'Denoised micrograph data', 'type': 'exposure.micrograph_blob'}], 'title': 'Micrographs', 'type': 'exposure'}, {'connections': [{'group_name': 'particles', 'job_uid': 'J260', 'slots': [{'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'alignments2D', 'result_type': 'particle.alignments2D', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J260', 'result_name': 'ml_properties', 'result_type': 'particle.ml_properties', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Particle locations for training Topaz', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}], 'title': 'Particles', 'type': 'particle'}], 'instance_information': {'available_memory': '345.26GB', 'cpu_model': 'Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz', 'ofd_hard_limit': 51200, 'ofd_soft_limit': 1024, 'physical_cores': 24, 'platform_architecture': 'x86_64', 'platform_node': 'hpcl8023', 'platform_release': '5.14.21-150400.24.97-default', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Fri Oct 27 10:29:06 UTC 2023 (8546fda)', 'total_memory': '377.22GB', 'used_memory': '27.78GB'}, 'job_type': 'topaz_cross_validation', 'params_spec': {'exec_path': {'value': '/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/bin/topaz'}, 'k_fold_interval': {'value': 10}, 'k_fold_num_iterations': {'value': 6}, 'k_fold_start': {'value': 5}, 'num_epochs': {'value': 20}}, 'project_uid': 'P111', 'status': 'completed', 'uid': 'J262', 'version': 'v4.4.1'}


cryosparcyyy@hansolo:~$ /software/cryosparc/server/cryosparc_master/bin/cryosparcm cli "get_job('P111', 'J360', 'job_type', 'version', 'params_spec', 'instance_information', 'input_slot_groups', 'status')"
{'_id': '668ce36ce37619eedb52a82b', 'input_slot_groups': [{'connections': [{'group_name': 'topaz_model', 'job_uid': 'J262', 'slots': [{'group_name': 'topaz_model', 'job_uid': 'J262', 'result_name': 'blob', 'result_type': 'ml_model.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'topaz_model', 'job_uid': 'J262', 'result_name': 'preprocess', 'result_type': 'ml_model.preprocess', 'slot_name': 'preprocess', 'version': 'F'}]}], 'count_max': 1, 'count_min': 0, 'description': 'Topaz model to pick particles with.', 'name': 'model', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'blob', 'optional': False, 'title': 'Topaz model', 'type': 'ml_model.blob'}, {'description': '', 'name': 'preprocess', 'optional': False, 'title': 'Topaz preprocessing parameters', 'type': 'ml_model.preprocess'}], 'title': 'Model', 'type': 'ml_model'}, {'connections': [{'group_name': 'exposures', 'job_uid': 'J18', 'slots': [{'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J18', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Micrographs for picking with Topaz', 'name': 'micrographs', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'micrograph_blob', 'optional': False, 'title': 'Raw micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'ctf', 'optional': True, 'title': '', 'type': 'exposure.ctf'}, {'description': '', 'name': 'micrograph_blob_denoised', 'optional': True, 'title': 'Denoised micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'mscope_params', 'optional': True, 'title': 'Microscope parameters for identifying negatively stained data', 'type': 'exposure.mscope_params'}], 'title': 'Micrographs', 'type': 'exposure'}], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '155.08GB', 'cpu_model': 'Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz', 'driver_version': '12.2', 'gpu_info': [{'id': 0, 'mem': 16900292608, 'name': 'Quadro RTX 5000'}], 'ofd_hard_limit': 51200, 'ofd_soft_limit': 1024, 'physical_cores': 24, 'platform_architecture': 'x86_64', 'platform_node': 'hpcl8045', 'platform_release': '5.14.21-150400.24.97-default', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Fri Oct 27 10:29:06 UTC 2023 (8546fda)', 'total_memory': '377.22GB', 'used_memory': '218.89GB'}, 'job_type': 'topaz_extract', 'params_spec': {'exec_path': {'value': '/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/bin/topaz'}}, 'project_uid': 'P111', 'status': 'failed', 'uid': 'J360', 'version': 'v4.4.1'}




cryosparcyyy@hansolo:~$ /software/cryosparc/server/cryosparc_master/bin/cryosparcm eventlog P111 J360 | tail -n 40
/fs/pool/pool-yyy-em4/cryopsparc/worker/4.0/cryosparc_worker/bin/cryosparcw run --project P111 --job J360 --master_hostname hansolo --master_command_core_port 60002 > /fs/pool/pool-yyy-em9/Peter/cryoEM/2024-05-16_KriosK3_FOCUS/CS-xyz-krios/J360/job.log 2>&1 

==========================================================================
==========================================================================
-------- Submission command: 
ssh hpcl8004 sbatch /fs/pool/pool-yyy-em9/Peter/cryoEM/2024-05-16_KriosK3_FOCUS/CS-xyz-krios/J360/queue_sub_script.sh
-------- Cluster Job ID: 
612562
-------- Queued on cluster at 2024-07-30 14:58:22.755087
-------- Cluster job status at 2024-07-30 14:58:23.340644 (0 retries)
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
            612562   p.hpcl8 yyyspa cryospar PD       0:00      1 (None)
[CPU RAM used: 246 MB] Starting Topaz process using version 0.2.5a...
[CPU RAM used: 247 MB] Skipping preprocessing.
[CPU RAM used: 247 MB] Using preprocessed micrographs from  J262/preprocessed
[CPU RAM used: 254 MB] --------------------------------------------------------------
[CPU RAM used: 254 MB] Inverting negative staining...
[CPU RAM used: 255 MB] Inverting negative staining complete.
[CPU RAM used: 255 MB] --------------------------------------------------------------
[CPU RAM used: 255 MB] Starting extraction...
[CPU RAM used: 255 MB] Starting extraction by running command /fs/pool/pool-xxx-soft/hpcl8/topaz/miniconda/envs/topaz/bin/topaz extract --radius 7 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 0 --model /fs/pool/pool-yyy-em9/Peter/cryoEM/2024-05-16_KriosK3_FOCUS/CS-xyz-krios/J279/models/model_epoch18.sav -o /fs/pool/pool-xyz-em9/Peter/cryoEM/2024-05-16_KriosK3_FOCUS/CS-xyz_krios/J360/topaz_particles_prediction.txt [5235 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]
[CPU RAM used: 255 MB] Traceback (most recent call last):
[CPU RAM used: 255 MB] File "/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/bin/topaz.exe", line 33, in <module>
[CPU RAM used: 255 MB] sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
[CPU RAM used: 255 MB] File "/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main
[CPU RAM used: 255 MB] args.func(args)
[CPU RAM used: 255 MB] File "/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/lib/python3.6/site-packages/topaz/commands/extract.py", line 288, in main
[CPU RAM used: 255 MB] for path,score,coords in nms_iterator(stream, radius, threshold, pool=pool):
[CPU RAM used: 255 MB] File "/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/lib/python3.6/site-packages/topaz/commands/extract.py", line 79, in nms_iterator
[CPU RAM used: 255 MB] for name,score,coords in pool.imap_unordered(process, scores):
[CPU RAM used: 255 MB] File "/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/lib/python3.6/multiprocessing/pool.py", line 735, in next
[CPU RAM used: 255 MB] raise value
[CPU RAM used: 255 MB] ValueError: cannot reshape array of size 1048320 into shape (1,1023,1440)
[CPU RAM used: 256 MB] Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 95, in cryosparc_master.cryosparc_compute.run.main
  File "/fs/gpfs41/lv10/fileset02/pool/pool-yyy-em4/cryopsparc/worker/4.0/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 1128, in run_topaz_wrapper_extract
    utils.run_process(extract_command)
  File "/fs/gpfs41/lv10/fileset02/pool/pool-yyy-em4/cryopsparc/worker/4.0/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process
    assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/fs/pool/pool-host-soft/hpcl8/topaz/miniconda/envs/topaz/bin/topaz extract --radius 7 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 0 --model /fs/pool/pool-yyy-em9/Peter/cry…)

wtempel · August 12, 2024, 4:07pm

@Christian You may want to consider opening an issue about this ValueError at Issues · tbepler/topaz · GitHub.