Hi @wtempel , I wonder if my issue is something else, but not the in-consistent number of frames as I do not see any failed movies when I import movies again with “skip header check” off. However, this data was collected on falconIV and is in EER format. I saw the post above where they talked about EER data format being an issue and not running in RBMC in Cryosparc.
To confirm, please can you post
- the full error message you observed
- the project and job IDs of the failed job
This is error message, I am getting and Project ID is P29 and job ID is J164:
[CPU: 620.3 MB Avail: 376.90 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 265, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames
@Anamika1 Please can you post
- the error messages from the screenshot as text, to make it easier for other forum users to find relevant discussions
- the version patch level of your CryoSPARC instance
- output of the command
nvidia-smion the computer where the job failed.
hi all, an update from me. I ended up getting the job to run pretty easily as follows.
I used the script below to identify which movies had an abnormal number of frames. I deleted those movies, I re-imported the movies (good ones only this time), re-ran motion correction, then I used the “reassign particles” job to link my particles to the new movies. Then the RBMC ran without issue, yay!
Jesse
1 Like
oops I spoke too soon, I am also getting this error now! Ugh
I am running cryosparc 4.4.0
Note that I received the same error with 1 GPU or 8 GPUs, and there is nothing wrong with the GPU since I have used it perfectly find to process to where I am currently (2.8A).
full error below:
[CPU: 1.37 GB Avail: 508.03 GB]
Resolution cutoffs: alignment 3.995 A, cross-validation 2.825 A
[CPU: 3.36 GB Avail: 505.98 GB]
Removed 103 movies with fewer than 2 particles.
[CPU: 3.40 GB Avail: 505.96 GB]
Removed 43 particles (in micrographs with fewer than 2 total particles).
[CPU: 3.98 GB Avail: 505.40 GB]
--------------------------------------------------------------
STARTING: OPTIMIZE HYPERPARAMETERS
--------------------------------------------------------------
[CPU: 3.98 GB Avail: 505.40 GB]
Working with 35 movies containing 12507 particles
[CPU: 3.98 GB Avail: 505.40 GB]
Computing intended data cache configuration
[CPU: 232.8 MB Avail: 509.19 GB]
====== Job process terminated abnormally.
[CPU: 218.9 MB Avail: 509.40 GB]
DIE: cuCtxCreate: CUDA ERROR: (CUDA_ERROR_DEVICE_UNAVAILABLE) CUDA-capable device(s) is/are busy or unavailable
here is the output from nvidia-smi:
(note that the graphics cards are: NVIDIA GeForce GTX 1080 Ti)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:04:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 46MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:05:00.0 Off | N/A |
| 29% 19C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce ... On | 00000000:06:00.0 Off | N/A |
| 29% 19C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce ... On | 00000000:07:00.0 Off | N/A |
| 29% 13C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA GeForce ... On | 00000000:08:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA GeForce ... On | 00000000:0B:00.0 Off | N/A |
| 29% 17C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA GeForce ... On | 00000000:0C:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA GeForce ... On | 00000000:0D:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 8 NVIDIA GeForce ... On | 00000000:0E:00.0 Off | N/A |
| 29% 13C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 9 NVIDIA GeForce ... On | 00000000:0F:00.0 Off | N/A |
| 29% 14C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3401 G /usr/lib/xorg/Xorg 41MiB |
| 1 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 4 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 5 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 6 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 7 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 8 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 9 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
There is a patch for v4.4.1 (not v4.4.0) that includes a relevant item: Patch 240110 is available for CryoSPARC v4.4.1.
hi,
I just updated to v4.4.1 and the problem persists 
any thoughts what else might be causing this?
thanks
Jesse
Hi , Applying this patch on cryosparc 4.4.1 resolved the issue for me.
Thanks,
Anamika
Hi everyone.
In CryoSPARC v4.5, released today, Ia new parameter was introduced to help alleviate this issue. If the new skip movies with wrong frame count switch is on, then the most common frame count will be assumed to be correct and all movies that don’t have that frame count will be discarded by the job.
– Harris
5 Likes
- When I use the import header check, the “failed movies” are the majority population (about 3k, the first 6 of which have a different no. of frames)
- in RBMC my job can fail with a wrong frame number error even when the skip wrong frame count is on and all micrographs are accepted as having the same frame number by import w/ header check
(re: 1 above Kye’s script works with the “failed” mics swapped for the good ones, just reporting since intended behavior is said to be the most common frame count doesn’t work, failed mics don’t have shapes, etc)
I ended up just moving the movies with different frame counts, reimporting then used Kye’s script that doesn’t work either, the “official” frame count from the first mixed import is used and not the physical one from the selected micrographs.
@kstachowski is there any workaround for if the frame count you want to use is not the one chosen?
@DanielAsarnow in your case, is it possible to split the dataset into subsets by frame count, and process them separately?
@wonderful Please can you post the outputs of these commands
csprojectid=P99 # replace with actual project ID
csjobid=J199 # replace with id of the failed job
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'instance_information', 'status', 'params_spec', 'errors_run', 'input_slot_groups')"
cryosparcm joblog $csprojectid $csjobid | tail -n 40
and describe the processing history of your particles. Have particles from different preprocessing workflows been merged?
Hi,@ 
wtempel,Thanks for your reply, here’s my outputs:
(base) amax@amax-SYS-4029GP-TRT:~$ csprojectid=P14
(base) amax@amax-SYS-4029GP-TRT:~$ csjobid=J669
(base) amax@amax-SYS-4029GP-TRT:~$ cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'instance_information', 'status', 'params_spec', 'errors_run', 'input_slot_groups')"
{'_id': '6762c8c28741fc20d764ef96', 'errors_run': [{'message': 'All movies must have the same raw frame size', 'warning': False}], 'input_slot_groups': [{'connections': [{'group_name': 'exposures', 'job_uid': 'J115', 'slots': [{'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': 'movie_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': 'rigid_motion', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': 'spline_motion', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J115', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Movies for motion correction', 'name': 'micrograph', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'movie_blob', 'optional': False, 'title': 'Raw movie data', 'type': 'exposure.movie_blob'}, {'description': '', 'name': 'gain_ref_blob', 'optional': True, 'title': 'Raw movie data', 'type': 'exposure.gain_ref_blob'}, {'description': '', 'name': 'mscope_params', 'optional': False, 'title': 'Exposure parameters', 'type': 'exposure.mscope_params'}, {'description': '', 'name': 'rigid_motion', 'optional': False, 'title': 'Full-frame motion estimates', 'type': 'exposure.motion'}, {'description': '', 'name': 'spline_motion', 'optional': True, 'title': 'Spline motion estimates', 'type': 'exposure.motion'}, {'description': '', 'name': 'background_blob', 'optional': False, 'title': 'Background estimates', 'type': 'exposure.stat_blob'}], 'title': 'Exposures', 'type': 'exposure'}, {'connections': [{'group_name': 'hyperparameters', 'job_uid': 'J3', 'slots': [{'group_name': 'hyperparameters', 'job_uid': 'J3', 'result_name': 'refmotion_hyperparameters', 'result_type': 'hyperparameter.refmotion_hyperparameters', 'slot_name': 'refmotion_hyperparameters', 'version': 'F'}, {'group_name': 'hyperparameters', 'job_uid': 'J3', 'result_name': 'refmotion_doseweights', 'result_type': 'hyperparameter.refmotion_doseweights', 'slot_name': 'refmotion_doseweights', 'version': 'F'}]}], 'count_max': 1, 'count_min': 0, 'description': 'Hyperparameters for reference-based motion correction', 'name': 'hyperparameters', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'refmotion_hyperparameters', 'optional': True, 'title': 'Alignment hyperparameters for reference based motion correction', 'type': 'hyperparameter.refmotion_hyperparameters'}, {'description': '', 'name': 'refmotion_doseweights', 'optional': True, 'title': 'Empirical dose weight parameters', 'type': 'hyperparameter.refmotion_doseweights'}], 'title': 'Hyperparameters', 'type': 'hyperparameter'}, {'connections': [{'group_name': 'particles', 'job_uid': 'J650', 'slots': [{'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': 'alignments3D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J650', 'result_name': 'ml_properties', 'result_type': 'particle.ml_properties', 'slot_name': None, 'version': 'F'}]}], 'count_max': 1, 'count_min': 1, 'description': '', 'name': 'particles_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'blob', 'optional': False, 'title': 'Particle raster data', 'type': 'particle.blob'}, {'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}, {'description': '', 'name': 'alignments3D', 'optional': False, 'title': 'Computed 3D alignments', 'type': 'particle.alignments3D'}, {'description': '', 'name': 'ctf', 'optional': False, 'title': 'Particle CTFs', 'type': 'particle.ctf'}, {'description': '', 'name': 'motion', 'optional': True, 'title': 'Particle motion data', 'type': 'particle.motion'}], 'title': 'Particles 0', 'type': 'particle'}, {'connections': [{'group_name': 'volume', 'job_uid': 'J650', 'slots': [{'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map', 'result_type': 'volume.blob', 'slot_name': 'map', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_half_A', 'result_type': 'volume.blob', 'slot_name': 'map_half_A', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_half_B', 'result_type': 'volume.blob', 'slot_name': 'map_half_B', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask_refine', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'map_sharp', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_fsc', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'mask_fsc_auto', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J650', 'result_name': 'precision', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': 1, 'count_min': 1, 'description': '', 'name': 'volume_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'map', 'optional': False, 'title': 'Reference volume raw data', 'type': 'volume.blob'}, {'description': '', 'name': 'map_half_A', 'optional': False, 'title': 'Reference volume raw data (half map A)', 'type': 'volume.blob'}, {'description': '', 'name': 'map_half_B', 'optional': False, 'title': 'Reference volume raw data (half map B)', 'type': 'volume.blob'}, {'description': '', 'name': 'mask_refine', 'optional': True, 'title': 'Mask associated with reference volume', 'type': 'volume.blob'}], 'title': 'Reference volume 0', 'type': 'volume'}, {'connections': [{'group_name': 'mask', 'job_uid': 'J650', 'slots': [{'group_name': 'mask', 'job_uid': 'J650', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask', 'version': 'F'}]}], 'count_max': 1, 'count_min': 0, 'description': "Optional mask (overrides the volume's mask_refine)", 'name': 'mask_0', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'mask', 'optional': False, 'title': 'Static mask', 'type': 'volume.blob'}], 'title': 'Static mask 0', 'type': 'mask'}], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '989.59GB', 'cpu_model': 'Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz', 'driver_version': '12.4', 'gpu_info': [{'id': 0, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:1a:00'}, {'id': 1, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:3d:00'}, {'id': 2, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:3e:00'}, {'id': 3, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:88:00'}, {'id': 4, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:89:00'}, {'id': 5, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:b1:00'}, {'id': 6, 'mem': 25430786048, 'name': 'NVIDIA GeForce RTX 3090', 'pcie': '0000:b2:00'}], 'ofd_hard_limit': 1048576, 'ofd_soft_limit': 1024, 'physical_cores': 48, 'platform_architecture': 'x86_64', 'platform_node': 'amax-SYS-4029GP-TRT', 'platform_release': '5.15.0-125-generic', 'platform_version': '#135~20.04.1-Ubuntu SMP Mon Oct 7 13:56:22 UTC 2024', 'total_memory': '1007.53GB', 'used_memory': '11.97GB'}, 'job_type': 'reference_motion_correction', 'params_spec': {'compute_num_gpus': {'value': 7}, 'fcrop_box_size': {'value': 720}, 'mem_cache_sz': {'value': 0.9}, 'output_f16': {'value': True}, 'skip_mismatching_frames': {'value': True}}, 'project_uid': 'P14', 'status': 'failed', 'uid': 'J669', 'version': 'v4.6.0'}
(base) amax@amax-SYS-4029GP-TRT:~$ cryosparcm joblog $csprojectid $csjobid | tail -n 40
========= sending heartbeat at 2024-12-19 21:59:17.113639
========= sending heartbeat at 2024-12-19 21:59:27.143216
========= sending heartbeat at 2024-12-19 21:59:37.154738
========= sending heartbeat at 2024-12-19 21:59:47.167520
========= sending heartbeat at 2024-12-19 21:59:57.185318
========= sending heartbeat at 2024-12-19 22:00:07.203086
========= sending heartbeat at 2024-12-19 22:00:17.220810
========= sending heartbeat at 2024-12-19 22:00:27.238714
========= sending heartbeat at 2024-12-19 22:00:37.259567
========= sending heartbeat at 2024-12-19 22:00:47.277657
========= sending heartbeat at 2024-12-19 22:00:57.295461
========= sending heartbeat at 2024-12-19 22:01:07.313283
========= sending heartbeat at 2024-12-19 22:01:17.330799
========= sending heartbeat at 2024-12-19 22:01:27.348817
========= sending heartbeat at 2024-12-19 22:01:37.366490
========= sending heartbeat at 2024-12-19 22:01:47.388961
========= sending heartbeat at 2024-12-19 22:01:57.407577
========= sending heartbeat at 2024-12-19 22:02:07.425299
========= sending heartbeat at 2024-12-19 22:02:17.443208
========= sending heartbeat at 2024-12-19 22:02:27.461302
========= sending heartbeat at 2024-12-19 22:02:37.479201
========= sending heartbeat at 2024-12-19 22:02:47.497080
========= sending heartbeat at 2024-12-19 22:02:57.514986
========= sending heartbeat at 2024-12-19 22:03:07.532714
========= sending heartbeat at 2024-12-19 22:03:17.549168
========= sending heartbeat at 2024-12-19 22:03:27.562934
========= sending heartbeat at 2024-12-19 22:03:37.580930
========= sending heartbeat at 2024-12-19 22:03:47.598246
========= sending heartbeat at 2024-12-19 22:03:57.618083
<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
========= sending heartbeat at 2024-12-19 22:04:07.635742
**** handle exception rc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py", line 324, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same raw frame size
set status to failed
========= main process now complete at 2024-12-19 22:04:17.653003.
========= monitor process now complete at 2024-12-19 22:04:17.658119.
All the particles are in the same project, but my movies are collected in several months, so they do different jobs of motioncorect, and then optimize from all particles to 2.22. Then I made Reference based motion correction on the movies of different months respectively. Some jobs could be completed, while others could not.
Maybe in my movies, some of them are 8184 * 11520, some are 11520 * 8184, right? How can I fix this?
Were the jobs configured such that input movies and particles for a given job corresponded to a single data collection session?
Did you Import Movies in separate jobs for each data collection separately and could confirm that some import jobs detected 8184 * 11520, whereas others detected 11520 * 8184?
