Hi,
I am combining the new movies and running the template pick using the classes from the previous dataset to get the particles for the Topaz train.
Three datasets collected with the same parameters are used as input micrographs for the template pick and the Topaz train.
I haven’t gotten the error with the same strategy for a different project.
The end of the log is shown below,
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] WARNING: no micrograph found matching image name "010873239182113701823_FoilHole_871125_Data_30843584_30843586_20240701_032843_fractions_patch_aligned_doseweighted". Skipping it.
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] WARNING: no micrograph found matching image name "002751553648898157715_FoilHole_19315514_Data_18215478_18215480_20240619_105754_fractions_patch_aligned_doseweighted". Skipping it.
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] WARNING: no micrograph found matching image name "006433023299928644952_FoilHole_3046490_Data_30843584_30843586_20240702_212158_fractions_patch_aligned_doseweighted". Skipping it.
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] WARNING: no micrograph found matching image name "007571361872567955548_FoilHole_19315590_Data_18215478_18215480_20240619_112001_fractions_patch_aligned_doseweighted". Skipping it.
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] Traceback (most recent call last):
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/topaz/0.2.5/bin/topaz", line 8, in <module>
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] sys.exit(main())
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ^^^^^^
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/topaz/0.2.5/lib/python3.11/site-packages/topaz/main.py", line 148, in main
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] args.func(args)
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/topaz/0.2.5/lib/python3.11/site-packages/topaz/commands/train_test_split.py", line 132, in main
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] image_list_train = pd.DataFrame({'image_name': image_names_train, 'path': paths_train})
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/python/3.11/lib/python3.11/site-packages/pandas/core/frame.py", line 664, in __init__
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/python/3.11/lib/python3.11/site-packages/pandas/core/internals/construction.py", line 493, in dict_to_mgr
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/python/3.11/lib/python3.11/site-packages/pandas/core/internals/construction.py", line 118, in arrays_to_mgr
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] index = _extract_index(arrays)
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ^^^^^^^^^^^^^^^^^^^^^^
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] File "/mnt/nasapps/production/python/3.11/lib/python3.11/site-packages/pandas/core/internals/construction.py", line 666, in _extract_index
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] raise ValueError("All arrays must be of the same length")
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] ValueError: All arrays must be of the same length
[Wed, 24 Jul 2024 17:41:46 GMT] [CPU RAM used: 512 MB] Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
File "/scratch/cluster_scratch/cryosparc/ncif-oreil-cryosparc/cryosparc_worker_oel8/cryosparc_compute/jobs/topaz/run_topaz.py", line 332, in run_topaz_wrapper_train
utils.run_process(split_command)
File "/scratch/cluster_scratch/cryosparc/ncif-oreil-cryosparc/cryosparc_worker_oel8/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/mnt/beegfs/software/topaz.sh train_test_split --number 8144 --seed 1642718247 --image-dir /mnt/beegfs/oreilly_cryo/Kitaik/Arctica/2024/Jun/0618_grid_A_1_1/CS-20240618-grid-a-1-1/J130/preprocessed /mnt/beegfs/oreilly_cryo/Kitaik/Arctica/2024/Jun/0618_grid_…)
This is the output when I run the command;
"cryosparcm cli “get_job(‘P3’, ‘J130’, ‘version’, ‘job_type’, ‘params_spec’, ‘status’, ‘instance_information’, ‘input_slot_groups’)”
{'_id': '669acafd4d4732bfab80f864', 'input_slot_groups': [{'connections': [{'group_name': 'micrographs', 'job_uid': 'J126', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'micrograph_blob_non_dw_AB', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J126', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Micrographs for training Topaz', 'name': 'micrographs', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'micrograph_blob', 'optional': False, 'title': 'Raw micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'micrograph_blob_denoised', 'optional': True, 'title': 'Denoised micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'mscope_params', 'optional': True, 'title': 'Microscope parameters for identifying negatively stained data', 'type': 'exposure.mscope_params'}], 'title': 'Micrographs', 'type': 'exposure'}, {'connections': [{'group_name': 'particles_selected', 'job_uid': 'J128', 'slots': [{'group_name': 'particles_selected', 'job_uid': 'J128', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles_selected', 'job_uid': 'J128', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles_selected', 'job_uid': 'J128', 'result_name': 'alignments2D', 'result_type': 'particle.alignments2D', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles_selected', 'job_uid': 'J128', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles_selected', 'job_uid': 'J128', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Particle locations for training Topaz', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}], 'title': 'Particles', 'type': 'particle'}], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '1.45TB', 'cpu_model': 'Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz', 'driver_version': '12.1', 'gpu_info': [{'id': 0, 'mem': 34079899648, 'name': 'Tesla V100-SXM2-32GB', 'pcie': '0000:15:00'}], 'ofd_hard_limit': 131072, 'ofd_soft_limit': 1024, 'physical_cores': 36, 'platform_architecture': 'x86_64', 'platform_node': 'fsitgl-hpc024p.ncifcrf.gov', 'platform_release': '4.18.0-553.8.1.el8_10.x86_64', 'platform_version': '#1 SMP Tue Jul 2 05:18:08 PDT 2024', 'total_memory': '1.48TB', 'used_memory': '13.02GB'}, 'job_type': 'topaz_train', 'params_spec': {'exec_path': {'value': '/mnt/beegfs/software/topaz.sh'}, 'num_particles': {'value': 400}, 'par_diam': {'value': 500}, 'use_denoised': {'value': False}}, 'project_uid': 'P3', 'status': 'failed', 'uid': 'J130', 'version': 'v4.5.3'}
Any suggestion to fix this error?
Regards,
Kitaik