Reference based motion correction error: All movies must have the same number of frames

rbs_sci · January 31, 2024, 1:24am

You should be able to split them out based on movie location/path, RBMC independently and recombine.

Look in Exposure Group Utilities; Exposure, Split, movie_blob/path and Split Outputs by Exposure Group.

kstachowski · February 1, 2024, 5:38pm

You could also modify the above script (removing the import movies sections, too) to use the total_dose field and make five sets of “good_exposures” and then send those to the external results job.

good_exposures = patch_ctf_exposures.query({'mscope_params/total_dose_e_per_A2': 85.0})

orangeboomerang · February 8, 2024, 2:29pm

hi all,

I did reimport without the “skip header check” and it did indeed find 7 movies without the correct number of frames. Pretty interesting.

However, my pipeline is coming from cryosparc live. I have a particle stack I am happy with. Previously I was linking the “exported exposures” from my live session to run RBMC. But now instead when I try linking the newly imported movies (excluding the 7 bad ones) I get the error below.


    assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ' + missing_inputs + '. Please connect all required inputs.'
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: micrograph.rigid_motion, micrograph.background_blob. Please connect all required inputs.

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

thanks!

Jesse

hsnyder · February 12, 2024, 10:15pm

Hi @orangeboomerang,

You can remove the problematic movies from a live exposure export using a variant of Kye’s script posted above. Something like this:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
)

project_number = "P317"
workspace_number = "W15"
job_number = "J1179" # the live export job
project = cs.find_project(project_number)
movs = job.load_output("accepted_exposures")

num_frames = 40 # set this to the number of frames that the "good" movies have

filtered_movs = movs.mask(movs['movie_blob/shape'][:,0] == num_frames)

cs.save_external_result(
    project_number,
    workspace_number,
    filtered_movs,
    type="exposure",
    name="desired_number_frame_exposures",
)

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

Indeed, we are aware that this currently presents an inconvenience.

– Harris

Anamika1 · February 13, 2024, 5:31pm

Hi I am getting the same error and using the script you mentioned, but my script is not running through and getting following error , could you help pointing my mistake :

(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     license="XXXXXX",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     host="localhost",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     base_port=39000,
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     email="XXXXX ",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     password="XXXXXX"
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project_number = "P29"
bash: project_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ workspace_number = "W1"
bash: workspace_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_1_number = "J170"
bash: job_1_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_2_number = "J130"
bash: job_2_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project = cs.find_project(project_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # Movies with bad number of frames
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ imported_movies_job = project.find_job(job_1_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ failed_movies = imported_movies_job.load_output("failed_movies")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # All exposures from patch CTF estimation
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_job = project.find_job(job_2_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_exposures = patch_ctf_job.load_output("exposures")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ cs.save_external_result(
bash: syntax error near unexpected token `newline'
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     project_number,P29
bash: project_number,P29: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     workspace_number,W1
bash: workspace_number,W1: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     good_exposures,
bash: good_exposures,: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     type="exposure",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     name="desired_number_frame_exposures",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ )
bash: syntax error near unexpected token `)'

wtempel · February 13, 2024, 7:02pm

@Anamika1 The commands you are trying to run are python, not shell commands. To execute the commands on the CryoSPARC master host

Save the relevant commands to a script, say select_exposures.py.

Replace the top portion of the script

hsnyder:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
    )

with

import os
from cryosparc_tools.cryosparc.tools import CryoSPARC
cs = CryoSPARC(host=os.getenv('CRYOSPARC_MASTER_HOSTNAME'),
           base_port=int(os.getenv('CRYOSPARC_BASE_PORT')),
           license=os.getenv('CRYOSPARC_LICENSE_ID'),
           email="your@cryosparc.login.email",
           password="your_cryosparc_password"
           )

Ensure you specify the correct CryoSPARC login email address and password and save the modified script.

Run the modified script

cryosparcm call python /absolute/path/to/select_exposures.py

Anamika1 · February 14, 2024, 2:09pm

Hi @wtempel , I wonder if my issue is something else, but not the in-consistent number of frames as I do not see any failed movies when I import movies again with “skip header check” off. However, this data was collected on falconIV and is in EER format. I saw the post above where they talked about EER data format being an issue and not running in RBMC in Cryosparc.

wtempel · February 14, 2024, 4:54pm

To confirm, please can you post

the full error message you observed
the project and job IDs of the failed job

Anamika1 · February 14, 2024, 5:03pm

This is error message, I am getting and Project ID is P29 and job ID is J164:
[CPU: 620.3 MB Avail: 376.90 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 265, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames

Anamika1 · February 14, 2024, 8:26pm

Getting a new CUDA error now:

wtempel · February 14, 2024, 10:34pm

@Anamika1 Please can you post

the error messages from the screenshot as text, to make it easier for other forum users to find relevant discussions
the version patch level of your CryoSPARC instance
output of the command nvidia-smi on the computer where the job failed.

orangeboomerang · February 15, 2024, 12:25pm

hi all, an update from me. I ended up getting the job to run pretty easily as follows.

I used the script below to identify which movies had an abnormal number of frames. I deleted those movies, I re-imported the movies (good ones only this time), re-ran motion correction, then I used the “reassign particles” job to link my particles to the new movies. Then the RBMC ran without issue, yay!

Jesse

mathewmclaren:

#!/bin/bash

dir="GridSquare*/Data/"
frame_count=54
ext=tiff
for file in ${dir}/*.${ext}; do
    if [ -f "$file" ]; then
        echo $file
        frames=$(header "$file" |grep sections|awk '{print $NF}')
        echo $frames
        if [ $frames -ne $frame_count ]; then
            echo "File: $file - Frames: $frames" >> bad_frames_count.txt
        else
            continue
        fi
    fi
done

orangeboomerang · February 15, 2024, 12:45pm

oops I spoke too soon, I am also getting this error now! Ugh
I am running cryosparc 4.4.0
Note that I received the same error with 1 GPU or 8 GPUs, and there is nothing wrong with the GPU since I have used it perfectly find to process to where I am currently (2.8A).

full error below:

[CPU:   1.37 GB  Avail: 508.03 GB]
Resolution cutoffs: alignment 3.995 A, cross-validation 2.825 A

[CPU:   3.36 GB  Avail: 505.98 GB]
Removed 103 movies with fewer than 2 particles.

[CPU:   3.40 GB  Avail: 505.96 GB]
Removed 43 particles (in micrographs with fewer than 2 total particles).

[CPU:   3.98 GB  Avail: 505.40 GB]
--------------------------------------------------------------
        STARTING: OPTIMIZE HYPERPARAMETERS
--------------------------------------------------------------

[CPU:   3.98 GB  Avail: 505.40 GB]
Working with 35 movies containing 12507 particles

[CPU:   3.98 GB  Avail: 505.40 GB]
Computing intended data cache configuration

[CPU:  232.8 MB  Avail: 509.19 GB]
====== Job process terminated abnormally.

[CPU:  218.9 MB  Avail: 509.40 GB]
DIE: cuCtxCreate: CUDA ERROR: (CUDA_ERROR_DEVICE_UNAVAILABLE) CUDA-capable device(s) is/are busy or unavailable

here is the output from nvidia-smi:
(note that the graphics cards are: NVIDIA GeForce GTX 1080 Ti)

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |     46MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:05:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  On   | 00000000:06:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  On   | 00000000:07:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  On   | 00000000:08:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  On   | 00000000:0B:00.0 Off |                  N/A |
| 29%   17C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   8  NVIDIA GeForce ...  On   | 00000000:0E:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   9  NVIDIA GeForce ...  On   | 00000000:0F:00.0 Off |                  N/A |
| 29%   14C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                 41MiB |
|    1   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    4   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    5   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    6   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    7   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    8   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    9   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

wtempel · February 15, 2024, 5:21pm

There is a patch for v4.4.1 (not v4.4.0) that includes a relevant item: Patch 240110 is available for CryoSPARC v4.4.1.

orangeboomerang · February 16, 2024, 12:37pm

hi,

I just updated to v4.4.1 and the problem persists

any thoughts what else might be causing this?

thanks

Jesse

Anamika1 · February 16, 2024, 2:21pm

Hi , Applying this patch on cryosparc 4.4.1 resolved the issue for me.

Thanks,

Anamika

wtempel · February 16, 2024, 4:41pm

You may have to additionally apply the patch Patch 240110 is available for CryoSPARC v4.4.1.

hsnyder · May 7, 2024, 3:55pm

Hi everyone.

In CryoSPARC v4.5, released today, Ia new parameter was introduced to help alleviate this issue. If the new skip movies with wrong frame count switch is on, then the most common frame count will be assumed to be correct and all movies that don’t have that frame count will be discarded by the job.

– Harris

DanielAsarnow · July 4, 2024, 1:21am

@hsnyder

When I use the import header check, the “failed movies” are the majority population (about 3k, the first 6 of which have a different no. of frames)
in RBMC my job can fail with a wrong frame number error even when the skip wrong frame count is on and all micrographs are accepted as having the same frame number by import w/ header check

(~~re: 1 above Kye’s script works with the “failed” mics swapped for the good ones, just reporting since intended behavior is said to be the most common frame count~~ doesn’t work, failed mics don’t have shapes, etc)

~~I ended up just moving the movies with different frame counts, reimporting then used Kye’s script~~ that doesn’t work either, the “official” frame count from the first mixed import is used and not the physical one from the selected micrographs.

@kstachowski is there any workaround for if the frame count you want to use is not the one chosen?

hsnyder · July 26, 2024, 6:06pm

@DanielAsarnow in your case, is it possible to split the dataset into subsets by frame count, and process them separately?