Reference based motion correction error: All movies must have the same number of frames

On a related note (or I can move this to a new topic), I got AssertionError: All movies must have the same total dose per square angstrom.

I had imported 6 datasets that all had the same parameters except slight dose differences, between 45 and 52 e/A2, and combined these datasets during Patch Motion Correction.

Do I need to re-import with the same dose, maybe rounding all to 50 e/A2, and redo Patch Motion, or do you folks think there’s a cryosparc-tools or similar approach to make changes to what already exists?

You should be able to split them out based on movie location/path, RBMC independently and recombine.

Look in Exposure Group Utilities; Exposure, Split, movie_blob/path and Split Outputs by Exposure Group.

3 Likes

Hi @drichman,

You could also modify the above script (removing the import movies sections, too) to use the total_dose field and make five sets of “good_exposures” and then send those to the external results job.

good_exposures = patch_ctf_exposures.query({'mscope_params/total_dose_e_per_A2': 85.0})

1 Like

hi all,

I did reimport without the “skip header check” and it did indeed find 7 movies without the correct number of frames. Pretty interesting.

However, my pipeline is coming from cryosparc live. I have a particle stack I am happy with. Previously I was linking the “exported exposures” from my live session to run RBMC. But now instead when I try linking the newly imported movies (excluding the 7 bad ones) I get the error below.


    assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ' + missing_inputs + '. Please connect all required inputs.'
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: micrograph.rigid_motion, micrograph.background_blob. Please connect all required inputs.

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

thanks!

Jesse

Hi @orangeboomerang,

You can remove the problematic movies from a live exposure export using a variant of Kye’s script posted above. Something like this:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
)

project_number = "P317"
workspace_number = "W15"
job_number = "J1179" # the live export job
project = cs.find_project(project_number)
movs = job.load_output("accepted_exposures")

num_frames = 40 # set this to the number of frames that the "good" movies have

filtered_movs = movs.mask(movs['movie_blob/shape'][:,0] == num_frames)

cs.save_external_result(
    project_number,
    workspace_number,
    filtered_movs,
    type="exposure",
    name="desired_number_frame_exposures",
)

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

Indeed, we are aware that this currently presents an inconvenience.

– Harris

Hi I am getting the same error and using the script you mentioned, but my script is not running through and getting following error , could you help pointing my mistake :

(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     license="XXXXXX",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     host="localhost",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     base_port=39000,
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     email="XXXXX ",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     password="XXXXXX"
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project_number = "P29"
bash: project_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ workspace_number = "W1"
bash: workspace_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_1_number = "J170"
bash: job_1_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_2_number = "J130"
bash: job_2_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project = cs.find_project(project_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # Movies with bad number of frames
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ imported_movies_job = project.find_job(job_1_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ failed_movies = imported_movies_job.load_output("failed_movies")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # All exposures from patch CTF estimation
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_job = project.find_job(job_2_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_exposures = patch_ctf_job.load_output("exposures")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ cs.save_external_result(
bash: syntax error near unexpected token `newline'
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     project_number,P29
bash: project_number,P29: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     workspace_number,W1
bash: workspace_number,W1: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     good_exposures,
bash: good_exposures,: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     type="exposure",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     name="desired_number_frame_exposures",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ )
bash: syntax error near unexpected token `)'

@Anamika1 The commands you are trying to run are python, not shell commands. To execute the commands on the CryoSPARC master host

  1. Save the relevant commands to a script, say select_exposures.py.
  2. Replace the top portion of the script with
    import os
    from cryosparc_tools.cryosparc.tools import CryoSPARC
    cs = CryoSPARC(host=os.getenv('CRYOSPARC_MASTER_HOSTNAME'),
               base_port=int(os.getenv('CRYOSPARC_BASE_PORT')),
               license=os.getenv('CRYOSPARC_LICENSE_ID'),
               email="your@cryosparc.login.email",
               password="your_cryosparc_password"
               )
    
    Ensure you specify the correct CryoSPARC login email address and password and save the modified script.
  3. Run the modified script
    cryosparcm call python /absolute/path/to/select_exposures.py
    

Hi @wtempel , I wonder if my issue is something else, but not the in-consistent number of frames as I do not see any failed movies when I import movies again with “skip header check” off. However, this data was collected on falconIV and is in EER format. I saw the post above where they talked about EER data format being an issue and not running in RBMC in Cryosparc.

To confirm, please can you post

  1. the full error message you observed
  2. the project and job IDs of the failed job

This is error message, I am getting and Project ID is P29 and job ID is J164:
[CPU: 620.3 MB Avail: 376.90 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 265, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames

Getting a new CUDA error now:

@Anamika1 Please can you post

  • the error messages from the screenshot as text, to make it easier for other forum users to find relevant discussions
  • the version patch level of your CryoSPARC instance
  • output of the command nvidia-smi on the computer where the job failed.

hi all, an update from me. I ended up getting the job to run pretty easily as follows.

I used the script below to identify which movies had an abnormal number of frames. I deleted those movies, I re-imported the movies (good ones only this time), re-ran motion correction, then I used the “reassign particles” job to link my particles to the new movies. Then the RBMC ran without issue, yay!

Jesse

1 Like

oops I spoke too soon, I am also getting this error now! Ugh
I am running cryosparc 4.4.0
Note that I received the same error with 1 GPU or 8 GPUs, and there is nothing wrong with the GPU since I have used it perfectly find to process to where I am currently (2.8A).

full error below:

[CPU:   1.37 GB  Avail: 508.03 GB]
Resolution cutoffs: alignment 3.995 A, cross-validation 2.825 A

[CPU:   3.36 GB  Avail: 505.98 GB]
Removed 103 movies with fewer than 2 particles.

[CPU:   3.40 GB  Avail: 505.96 GB]
Removed 43 particles (in micrographs with fewer than 2 total particles).

[CPU:   3.98 GB  Avail: 505.40 GB]
--------------------------------------------------------------
        STARTING: OPTIMIZE HYPERPARAMETERS
--------------------------------------------------------------

[CPU:   3.98 GB  Avail: 505.40 GB]
Working with 35 movies containing 12507 particles

[CPU:   3.98 GB  Avail: 505.40 GB]
Computing intended data cache configuration

[CPU:  232.8 MB  Avail: 509.19 GB]
====== Job process terminated abnormally.

[CPU:  218.9 MB  Avail: 509.40 GB]
DIE: cuCtxCreate: CUDA ERROR: (CUDA_ERROR_DEVICE_UNAVAILABLE) CUDA-capable device(s) is/are busy or unavailable

here is the output from nvidia-smi:
(note that the graphics cards are: NVIDIA GeForce GTX 1080 Ti)

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |     46MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:05:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  On   | 00000000:06:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  On   | 00000000:07:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  On   | 00000000:08:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  On   | 00000000:0B:00.0 Off |                  N/A |
| 29%   17C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   8  NVIDIA GeForce ...  On   | 00000000:0E:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   9  NVIDIA GeForce ...  On   | 00000000:0F:00.0 Off |                  N/A |
| 29%   14C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                 41MiB |
|    1   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    4   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    5   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    6   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    7   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    8   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    9   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

There is a patch for v4.4.1 (not v4.4.0) that includes a relevant item: Patch 240110 is available for CryoSPARC v4.4.1.

hi,

I just updated to v4.4.1 and the problem persists :expressionless:

any thoughts what else might be causing this?

thanks

Jesse

image

Hi , Applying this patch on cryosparc 4.4.1 resolved the issue for me.

Thanks,

Anamika

You may have to additionally apply the patch Patch 240110 is available for CryoSPARC v4.4.1.