Reference based motion correction error: All movies must have the same number of frames

mathewmclaren · December 7, 2023, 6:54pm

@matt1 Just a check, are all the movies that provide your particles included in your inputs? I have a job that keeps crashing so decided to split my movies into subsets to see if I can get at least some of my particles to extract. However, my job crashed with the frame error (and obviously the number of frames is correct). It might be worth doing a count of your files and see if it matches with the number of exposures

scottbcohen · December 29, 2023, 10:49pm

Any progress on this issue?

Thanks and Happy New Year!

scottbcohen · January 13, 2024, 4:21am

I have a map from non-uniform refinement to 2.6 Angstrom; the full data set (~380k ptcls) is an accumulation of several separate acquisition sessions over 18 months (mostly at tilt, and few ptcls per micrograph). As alluded to above, Reference-based Motion Correction returned the error “All micrographs must have the same number of frames”.

The job did proceed smoothly when applying a small “test” collection of particles from a SINGLE acquisition. I’m guessing the problem is maybe just a few rogue micrographs somewhere within the full data set?

Is there a way sort of work backwards, taking the full particle stack and separate the particles according to their acquisition sessions (or Import Jobs), such that maybe I can narrow down the culprit and/or motion-correct at least some of the particles?

Alternatively, as suggested above by @martinpacesa , can the Job be modified such that rogue frame counts are simply ignored? Would be a shame to not be able to apply Reference-based MotionCorr after such a long haul.

Thanks for your suggestions!
Scott

wtempel · January 16, 2024, 3:44pm

There may be, as described under Particle sets job - limit number of micrographs used - #5 by mmclean.
If you ran Patch CTF Estimation separately for each acquisition session’s data, you could connect

a single session’s Patch CTF Estimation exposures output
the combined particle set

to a Manually Curate Exposures job and, inside the job and without modifying any selection criteria, push the Done button. The Particles Accepted output should contain particles derived from that single session only.

scottbcohen · January 20, 2024, 1:30am

@wtempel

Thanks heaps - this is very handy and simple too.

Scott (CMRI down under)

kstachowski · January 23, 2024, 3:02pm

Hi @parrot and others,

Here is a small script that @rposert devised as a work-around to this issue that you may find helpful. You will need to run this in a terminal using cryosparc-tools.

Steps:

Reimport movies with Skip header check disabled if that was not initially done. This will output a set of ‘failed_movies’ whose frame count does not match the majority of the movies imported.
Next, run the attached script, replacing the appropriate instance information, project number, workspace number, and job numbers (job_1 = new import movies, job_2 = patch CTF estimation with all movies). This will create a new external job in the appropriate workspace with a set of exposures containing only the number of expected frames.
Create a Manually curate exposures job with the exposures output from the External Results job, and the particles stack you would like to perform RBMC on. Set the parameter Number of picked particles to `1,10000’. This will create an output of exposures where there is at least 1 particle present on the micrographs with the same number of frames.
Connect the Exposures and Particles outputs from the Manually curate exposures job to the RBMC job and the volume associated with those particles.

This should alleviate the error with an inconsistent # of frames.

Script:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
)

project_number = "P317"
workspace_number = "W15"
job_1_number = "J1179"
job_2_number = "J1079"
project = cs.find_project(project_number)

# Movies with bad number of frames
imported_movies_job = project.find_job(job_1_number)
failed_movies = imported_movies_job.load_output("failed_movies")

# All exposures from patch CTF estimation
patch_ctf_job = project.find_job(job_2_number)
patch_ctf_exposures = patch_ctf_job.load_output("exposures")

good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])

cs.save_external_result(
    project_number,
    workspace_number,
    good_exposures,
    type="exposure",
    name="desired_number_frame_exposures",
)

drichman · January 30, 2024, 5:04pm

On a related note (or I can move this to a new topic), I got AssertionError: All movies must have the same total dose per square angstrom.

I had imported 6 datasets that all had the same parameters except slight dose differences, between 45 and 52 e/A2, and combined these datasets during Patch Motion Correction.

Do I need to re-import with the same dose, maybe rounding all to 50 e/A2, and redo Patch Motion, or do you folks think there’s a cryosparc-tools or similar approach to make changes to what already exists?

rbs_sci · January 31, 2024, 1:24am

You should be able to split them out based on movie location/path, RBMC independently and recombine.

Look in Exposure Group Utilities; Exposure, Split, movie_blob/path and Split Outputs by Exposure Group.

kstachowski · February 1, 2024, 5:38pm

Hi @drichman,

You could also modify the above script (removing the import movies sections, too) to use the total_dose field and make five sets of “good_exposures” and then send those to the external results job.

good_exposures = patch_ctf_exposures.query({'mscope_params/total_dose_e_per_A2': 85.0})

orangeboomerang · February 8, 2024, 2:29pm

hi all,

I did reimport without the “skip header check” and it did indeed find 7 movies without the correct number of frames. Pretty interesting.

However, my pipeline is coming from cryosparc live. I have a particle stack I am happy with. Previously I was linking the “exported exposures” from my live session to run RBMC. But now instead when I try linking the newly imported movies (excluding the 7 bad ones) I get the error below.


    assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ' + missing_inputs + '. Please connect all required inputs.'
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: micrograph.rigid_motion, micrograph.background_blob. Please connect all required inputs.

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

thanks!

Jesse

hsnyder · February 12, 2024, 10:15pm

Hi @orangeboomerang,

You can remove the problematic movies from a live exposure export using a variant of Kye’s script posted above. Something like this:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
)

project_number = "P317"
workspace_number = "W15"
job_number = "J1179" # the live export job
project = cs.find_project(project_number)
movs = job.load_output("accepted_exposures")

num_frames = 40 # set this to the number of frames that the "good" movies have

filtered_movs = movs.mask(movs['movie_blob/shape'][:,0] == num_frames)

cs.save_external_result(
    project_number,
    workspace_number,
    filtered_movs,
    type="exposure",
    name="desired_number_frame_exposures",
)

Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?

Indeed, we are aware that this currently presents an inconvenience.

– Harris

Anamika1 · February 13, 2024, 5:31pm

Hi I am getting the same error and using the script you mentioned, but my script is not running through and getting following error , could you help pointing my mistake :

(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     license="XXXXXX",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     host="localhost",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     base_port=39000,
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     email="XXXXX ",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     password="XXXXXX"
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project_number = "P29"
bash: project_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ workspace_number = "W1"
bash: workspace_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_1_number = "J170"
bash: job_1_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_2_number = "J130"
bash: job_2_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project = cs.find_project(project_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # Movies with bad number of frames
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ imported_movies_job = project.find_job(job_1_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ failed_movies = imported_movies_job.load_output("failed_movies")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # All exposures from patch CTF estimation
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_job = project.find_job(job_2_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_exposures = patch_ctf_job.load_output("exposures")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ 
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ cs.save_external_result(
bash: syntax error near unexpected token `newline'
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     project_number,P29
bash: project_number,P29: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     workspace_number,W1
bash: workspace_number,W1: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     good_exposures,
bash: good_exposures,: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     type="exposure",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$     name="desired_number_frame_exposures",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ )
bash: syntax error near unexpected token `)'

wtempel · February 13, 2024, 7:02pm

@Anamika1 The commands you are trying to run are python, not shell commands. To execute the commands on the CryoSPARC master host

Save the relevant commands to a script, say select_exposures.py.

Replace the top portion of the script

hsnyder:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
    )

with

import os
from cryosparc_tools.cryosparc.tools import CryoSPARC
cs = CryoSPARC(host=os.getenv('CRYOSPARC_MASTER_HOSTNAME'),
           base_port=int(os.getenv('CRYOSPARC_BASE_PORT')),
           license=os.getenv('CRYOSPARC_LICENSE_ID'),
           email="your@cryosparc.login.email",
           password="your_cryosparc_password"
           )

Ensure you specify the correct CryoSPARC login email address and password and save the modified script.

Run the modified script

cryosparcm call python /absolute/path/to/select_exposures.py

Anamika1 · February 14, 2024, 2:09pm

Hi @wtempel , I wonder if my issue is something else, but not the in-consistent number of frames as I do not see any failed movies when I import movies again with “skip header check” off. However, this data was collected on falconIV and is in EER format. I saw the post above where they talked about EER data format being an issue and not running in RBMC in Cryosparc.

wtempel · February 14, 2024, 4:54pm

To confirm, please can you post

the full error message you observed
the project and job IDs of the failed job

Anamika1 · February 14, 2024, 5:03pm

This is error message, I am getting and Project ID is P29 and job ID is J164:
[CPU: 620.3 MB Avail: 376.90 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 265, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames

Anamika1 · February 14, 2024, 8:26pm

Getting a new CUDA error now:

wtempel · February 14, 2024, 10:34pm

@Anamika1 Please can you post

the error messages from the screenshot as text, to make it easier for other forum users to find relevant discussions
the version patch level of your CryoSPARC instance
output of the command nvidia-smi on the computer where the job failed.

orangeboomerang · February 15, 2024, 12:25pm

hi all, an update from me. I ended up getting the job to run pretty easily as follows.

I used the script below to identify which movies had an abnormal number of frames. I deleted those movies, I re-imported the movies (good ones only this time), re-ran motion correction, then I used the “reassign particles” job to link my particles to the new movies. Then the RBMC ran without issue, yay!

Jesse

mathewmclaren:

#!/bin/bash

dir="GridSquare*/Data/"
frame_count=54
ext=tiff
for file in ${dir}/*.${ext}; do
    if [ -f "$file" ]; then
        echo $file
        frames=$(header "$file" |grep sections|awk '{print $NF}')
        echo $frames
        if [ $frames -ne $frame_count ]; then
            echo "File: $file - Frames: $frames" >> bad_frames_count.txt
        else
            continue
        fi
    fi
done

orangeboomerang · February 15, 2024, 12:45pm

oops I spoke too soon, I am also getting this error now! Ugh
I am running cryosparc 4.4.0
Note that I received the same error with 1 GPU or 8 GPUs, and there is nothing wrong with the GPU since I have used it perfectly find to process to where I am currently (2.8A).

full error below:

[CPU:   1.37 GB  Avail: 508.03 GB]
Resolution cutoffs: alignment 3.995 A, cross-validation 2.825 A

[CPU:   3.36 GB  Avail: 505.98 GB]
Removed 103 movies with fewer than 2 particles.

[CPU:   3.40 GB  Avail: 505.96 GB]
Removed 43 particles (in micrographs with fewer than 2 total particles).

[CPU:   3.98 GB  Avail: 505.40 GB]
--------------------------------------------------------------
        STARTING: OPTIMIZE HYPERPARAMETERS
--------------------------------------------------------------

[CPU:   3.98 GB  Avail: 505.40 GB]
Working with 35 movies containing 12507 particles

[CPU:   3.98 GB  Avail: 505.40 GB]
Computing intended data cache configuration

[CPU:  232.8 MB  Avail: 509.19 GB]
====== Job process terminated abnormally.

[CPU:  218.9 MB  Avail: 509.40 GB]
DIE: cuCtxCreate: CUDA ERROR: (CUDA_ERROR_DEVICE_UNAVAILABLE) CUDA-capable device(s) is/are busy or unavailable

here is the output from nvidia-smi:
(note that the graphics cards are: NVIDIA GeForce GTX 1080 Ti)

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |     46MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:05:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  On   | 00000000:06:00.0 Off |                  N/A |
| 29%   19C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  On   | 00000000:07:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  On   | 00000000:08:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  On   | 00000000:0B:00.0 Off |                  N/A |
| 29%   17C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  On   | 00000000:0C:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  On   | 00000000:0D:00.0 Off |                  N/A |
| 29%   15C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   8  NVIDIA GeForce ...  On   | 00000000:0E:00.0 Off |                  N/A |
| 29%   13C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   9  NVIDIA GeForce ...  On   | 00000000:0F:00.0 Off |                  N/A |
| 29%   14C    P8     7W / 250W |      8MiB / 11264MiB |      0%   E. Process |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                 41MiB |
|    1   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    4   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    5   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    6   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    7   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    8   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
|    9   N/A  N/A      3401      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+