@DanielAsarnow in your case, is it possible to split the dataset into subsets by frame count, and process them separately?
@wonderful Please can you post the outputs of these commands
csprojectid=P99 # replace with actual project ID
csjobid=J199 # replace with id of the failed job
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'instance_information', 'status', 'params_spec', 'errors_run', 'input_slot_groups')"
cryosparcm joblog $csprojectid $csjobid | tail -n 40
and describe the processing history of your particles. Have particles from different preprocessing workflows been merged?
Hi,@ 
wtempel,Thanks for your reply, here’s my outputs:
(base) amax@amax-SYS-4029GP-TRT:~$ csprojectid=P14
(base) amax@amax-SYS-4029GP-TRT:~$ csjobid=J669
(base) amax@amax-SYS-4029GP-TRT:~$ cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'instance_information', 'status', 'params_spec', 'errors_run', 'input_slot_groups')"
{'_id': '6762c8c28741fc20d764ef96', 'errors_run': [{'message': 'All movies must have the same raw frame size', 'warning': False}], 'input_slot_groups': [{'connections': [{'group_name': 'exposures', 'job_uid': 'J115', 'slots': [{'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': 'movie_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': 'rigid_motion', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': 'spline_motion', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Movies for motion correction', 'name': 'micrograph', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'movie_blob', 'optional': False, 'title': 'Raw movie data', 'type': 'exposure.movie_blob'}, {'description': '', 'name': 'gain_ref_blob', 'optional': True, 'title': 'Raw movie data', 'type': 'exposure.gain_ref_blob'}, {'description': '', 'name': 'mscope_params', 'optional': False, 'title': 'Exposure parameters', 'type': 'exposure.mscope_params'}, {'description': '', 'name': 'rigid_motion', 'optional': False, 'title': 'Full-frame motion estimates', 'type': 'exposure.motion'}, {'description': '', 'name': 'spline_motion', 'optional': True, 'title': 'Spline motion estimates', 'type': 'exposure.motion'}, {'description': '', 'name': 'background_blob', 'optional': False, 'title': 'Background estimates', 'type': 'exposure.stat_blob'}], 'title': 'Exposures', 'type': 'exposure'}, {'connections': [{'group_name': 'hyperparameters', 'job_uid': 'J3', 'slots': [{'group_name': 'hyperparameters', 'job_uid': 'J3', 'result_name': 'refmotion_hyperparameters', 'result_type': 'hyperparameter.refmotion_hyperparameters', 'slot_name': 'refmotion_hyperparameters', 'version': 'F'}, {'group_name': 'hyperparameters', 'job_uid': 'J3', 'result_name': 'refmotion_doseweights', 'result_type': 'hyperparameter.refmotion_doseweights', 'slot_name': 'refmotion_doseweights', 'version': 'F'}]}], 'count_max': 1, 'count_min': 0, 'description': 'Hyperparameters for reference-based motion correction', 'name': 'hyperparameters', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'refmotion_hyperparameters', 'optional': True, 'title': 'Alignment hyperparameters for reference based motion correction', 'type': 'hyperparameter.refmotion_hyperparameters'}, {'description': '', 'name': 'refmotion_doseweights', 'optional': True, 'title': 'Empirical dose weight parameters', 'type': 'hyperparameter.refmotion_doseweights'}], 'title': 'Hyperparameters', 'type': 'hyperparameter'}, {'connections': [{'group_name': 'particles', 'job_uid': 'J650', 'slots': [{'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': 'alignments3D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'ml_properties', 'result_type': 'particle.ml_properties', 'slot_name': None, 'version': 'F'}]}], 'count_max': 1, 'count_min': 1, 'description': '', 'name': 'particles_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'blob', 'optional': False, 'title': 'Particle raster data', 'type': 'particle.blob'}, {'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}, {'description': '', 'name': 'alignments3D', 'optional': False, 'title': 'Computed 3D alignments', 'type': 'particle.alignments3D'}, {'description': '', 'name': 'ctf', 'optional': False, 'title': 'Particle CTFs', 'type': 'particle.ctf'}, {'description': '', 'name': 'motion', 'optional': True, 'title': 'Particle motion data', 'type': 'particle.motion'}], 'title': 'Particles 0', 'type': 'particle'}, {'connections': [{'group_name': 'volume', 'job_uid': 'J650', 'slots': [{'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map', 'result_type': 'volume.blob', 'slot_name': 'map', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_half_A', 'result_type': 'volume.blob', 'slot_name': 'map_half_A', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_half_B', 'result_type': 'volume.blob', 'slot_name': 'map_half_B', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask_refine', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_sharp', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_fsc', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_fsc_auto', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'precision', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': 1, 'count_min': 1, 'description': '', 'name': 'volume_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'map', 'optional': False, 'title': 'Reference volume raw data', 'type': 'volume.blob'}, {'description': '', 'name': 'map_half_A', 'optional': False, 'title': 'Reference volume raw data (half map A)', 'type': 'volume.blob'}, {'description': '', 'name': 'map_half_B', 'optional': False, 'title': 'Reference volume raw data (half map B)', 'type': 'volume.blob'}, {'description': '', 'name': 'mask_refine', 'optional': True, 'title': 'Mask associated with reference volume', 'type': 'volume.blob'}], 'title': 'Reference volume 0', 'type': 'volume'}, {'connections': [{'group_name': 'mask', 'job_uid': 'J650', 'slots': [{'group_name': 'mask', 'job_uid': 'J650', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask', 'version': 'F'}]}], 'count_max': 1, 'count_min': 0, 'description': "Optional mask (overrides the volume's mask_refine)", 'name': 'mask_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'mask', 'optional': False, 'title': 'Static mask', 'type': 'volume.blob'}], 'title': 'Static mask 0', 'type': 'mask'}], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '989.59GB', 'cpu_model': 'Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz', 'driver_version': '12.4', 'gpu_info': [{'id': 0, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:1a:00'}, {'id': 1, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:3d:00'}, {'id': 2, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:3e:00'}, {'id': 3, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:88:00'}, {'id': 4, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:89:00'}, {'id': 5, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:b1:00'}, {'id': 6, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:b2:00'}], 'ofd_hard_limit': 1048576, 'ofd_soft_limit': 1024, 'physical_cores': 48, 'platform_architecture': 'x86_64', 'platform_node': 'amax-SYS-4029GP-TRT', 'platform_release': '5.15.0-125-generic', 'platform_version': '#135~20.04.1-Ubuntu SMP Mon Oct 7 13:56:22 UTC 2024', 'total_memory': '1007.53GB', 'used_memory': '11.97GB'}, 'job_type': 'reference_motion_correction', 'params_spec': {'compute_num_gpus': {'value': 7}, 'fcrop_box_size': {'value': 720}, 'mem_cache_sz': {'value': 0.9}, 'output_f16': {'value': True}, 'skip_mismatching_frames': {'value': True}}, 'project_uid': 'P14', 'status': 'failed', 'uid': 'J669', 'version': 'v4.6.0'}
(base) amax@amax-SYS-4029GP-TRT:~$ cryosparcm joblog $csprojectid $csjobid | tail -n 40
========= sending heartbeat at 2024-12-19 21:59:17.113639
========= sending heartbeat at 2024-12-19 21:59:27.143216
========= sending heartbeat at 2024-12-19 21:59:37.154738
========= sending heartbeat at 2024-12-19 21:59:47.167520
========= sending heartbeat at 2024-12-19 21:59:57.185318
========= sending heartbeat at 2024-12-19 22:00:07.203086
========= sending heartbeat at 2024-12-19 22:00:17.220810
========= sending heartbeat at 2024-12-19 22:00:27.238714
========= sending heartbeat at 2024-12-19 22:00:37.259567
========= sending heartbeat at 2024-12-19 22:00:47.277657
========= sending heartbeat at 2024-12-19 22:00:57.295461
========= sending heartbeat at 2024-12-19 22:01:07.313283
========= sending heartbeat at 2024-12-19 22:01:17.330799
========= sending heartbeat at 2024-12-19 22:01:27.348817
========= sending heartbeat at 2024-12-19 22:01:37.366490
========= sending heartbeat at 2024-12-19 22:01:47.388961
========= sending heartbeat at 2024-12-19 22:01:57.407577
========= sending heartbeat at 2024-12-19 22:02:07.425299
========= sending heartbeat at 2024-12-19 22:02:17.443208
========= sending heartbeat at 2024-12-19 22:02:27.461302
========= sending heartbeat at 2024-12-19 22:02:37.479201
========= sending heartbeat at 2024-12-19 22:02:47.497080
========= sending heartbeat at 2024-12-19 22:02:57.514986
========= sending heartbeat at 2024-12-19 22:03:07.532714
========= sending heartbeat at 2024-12-19 22:03:17.549168
========= sending heartbeat at 2024-12-19 22:03:27.562934
========= sending heartbeat at 2024-12-19 22:03:37.580930
========= sending heartbeat at 2024-12-19 22:03:47.598246
========= sending heartbeat at 2024-12-19 22:03:57.618083
<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
========= sending heartbeat at 2024-12-19 22:04:07.635742
**** handle exception rc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py", line 324, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same raw frame size
set status to failed
========= main process now complete at 2024-12-19 22:04:17.653003.
========= monitor process now complete at 2024-12-19 22:04:17.658119.
All the particles are in the same project, but my movies are collected in several months, so they do different jobs of motioncorect, and then optimize from all particles to 2.22. Then I made Reference based motion correction on the movies of different months respectively. Some jobs could be completed, while others could not.
Maybe in my movies, some of them are 8184 * 11520, some are 11520 * 8184, right? How can I fix this?
Were the jobs configured such that input movies and particles for a given job corresponded to a single data collection session?
Did you Import Movies in separate jobs for each data collection separately and could confirm that some import jobs detected 8184 * 11520, whereas others detected 11520 * 8184?
Hi @wonderful,
We would recommend to examine all previous micrograph processing steps. It is concerning that one of your datasets is rotated 90 degrees. Additionally, we would recommend looking over the failed datasets to understand what the differences are related to the frame size (ie. are some of the movies in superres and others binned? or do some of those movies have a 90 degree rotation while others do not).
All the best,
Kye
I have a similar issue but with a slight variation. My RBMC job was also failing citing the above-mentioned same frame error. I was using exported exposure from live session. I redid import of my movies with skip header disabled but none of them threw any error regarding having different number of frames. I ended up doing RBMC using the skip frames feature which discarded about 11 movies and it got rid of 5700 particles from my dataset. What should be done in this scenario? I don’t want to redo the whole processing of the dataset again from the imported movies. Please advise.
Ordinarily one may use the output of an RBMC job where movies with disparate frame counts were rejected just as one would use the output of an RBMC where the frame counts of all movies matched.
You are correct in that, as of CryoSPARC v4.7.0, a movie import should fail if a movie is encountered whose frame count does not match the first imported movie’s. Even if none of the individual movie import jobs (with header check) failed, it is still possible that not all movie sets had the same frame count.
By the way, I was unable to reconcile the number of exposures shown on the RBMC job card (20708) with the numbers of rejected movies (9720 due to low particle count and 11 disparate frame count) and the sum of the imported movies (9019, 20979, 641), but its possible that the discrepancy might be explained by exposure curation or rejection steps not shown in your post.
@wtempel The discrepency is because the RBMC job was done on exported “Live exposures” which had some micrograph thrown out during the live process while the import job was done separately with header check to check if it throws out movies with inconsistent frames as mentioned earlier in this post.
Small update: Ultimately, I processed the “imported movies with header check” which resulted in 0 movies being failed. Moreover, it didn’t throw any inconsistent frames error when I used those movies in the subsequent RBMC job, which means that the live exposures should also have consistent frames.
While it didn’t complain about the frames, RBMC still threw out 9.86k movies with fewer than 2 particles.
If I turn off the recenter particles option off, then the number of movies thrown out reduces to 9.78k.
As a sanity check, I reextracted the particles (from the new processing) from the 30.6k imported movies to see if the number of micrographs being rejected is the same as the ones rejected by RBMC and found that it is different (4.57k vs 9.78k). So RBMC is rejecting approx 5.21 k movies with my particles for some reason (it says fewer than 2 particles).