Here is a small script that @rposert devised as a work-around to this issue that you may find helpful. You will need to run this in a terminal using cryosparc-tools.
Steps:
Reimport movies with Skip header check disabled if that was not initially done. This will output a set of ‘failed_movies’ whose frame count does not match the majority of the movies imported.
Next, run the attached script, replacing the appropriate instance information, project number, workspace number, and job numbers (job_1 = new import movies, job_2 = patch CTF estimation with all movies). This will create a new external job in the appropriate workspace with a set of exposures containing only the number of expected frames.
Create a Manually curate exposures job with the exposures output from the External Results job, and the particles stack you would like to perform RBMC on. Set the parameter Number of picked particles to `1,10000’. This will create an output of exposures where there is at least 1 particle present on the micrographs with the same number of frames.
Connect the Exposures and Particles outputs from the Manually curate exposures job to the RBMC job and the volume associated with those particles.
This should alleviate the error with an inconsistent # of frames.
Script:
from cryosparc.tools import CryoSPARC
cs = CryoSPARC(
license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
host="localhost",
base_port=39000,
email="ali@example.com",
password="password123"
)
project_number = "P317"
workspace_number = "W15"
job_1_number = "J1179"
job_2_number = "J1079"
project = cs.find_project(project_number)
# Movies with bad number of frames
imported_movies_job = project.find_job(job_1_number)
failed_movies = imported_movies_job.load_output("failed_movies")
# All exposures from patch CTF estimation
patch_ctf_job = project.find_job(job_2_number)
patch_ctf_exposures = patch_ctf_job.load_output("exposures")
good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])
cs.save_external_result(
project_number,
workspace_number,
good_exposures,
type="exposure",
name="desired_number_frame_exposures",
)
On a related note (or I can move this to a new topic), I got AssertionError: All movies must have the same total dose per square angstrom.
I had imported 6 datasets that all had the same parameters except slight dose differences, between 45 and 52 e/A2, and combined these datasets during Patch Motion Correction.
Do I need to re-import with the same dose, maybe rounding all to 50 e/A2, and redo Patch Motion, or do you folks think there’s a cryosparc-tools or similar approach to make changes to what already exists?
You could also modify the above script (removing the import movies sections, too) to use the total_dose field and make five sets of “good_exposures” and then send those to the external results job.
I did reimport without the “skip header check” and it did indeed find 7 movies without the correct number of frames. Pretty interesting.
However, my pipeline is coming from cryosparc live. I have a particle stack I am happy with. Previously I was linking the “exported exposures” from my live session to run RBMC. But now instead when I try linking the newly imported movies (excluding the 7 bad ones) I get the error below.
assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ' + missing_inputs + '. Please connect all required inputs.'
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: micrograph.rigid_motion, micrograph.background_blob. Please connect all required inputs.
Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?
You can remove the problematic movies from a live exposure export using a variant of Kye’s script posted above. Something like this:
from cryosparc.tools import CryoSPARC
cs = CryoSPARC(
license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
host="localhost",
base_port=39000,
email="ali@example.com",
password="password123"
)
project_number = "P317"
workspace_number = "W15"
job_number = "J1179" # the live export job
project = cs.find_project(project_number)
movs = job.load_output("accepted_exposures")
num_frames = 40 # set this to the number of frames that the "good" movies have
filtered_movs = movs.mask(movs['movie_blob/shape'][:,0] == num_frames)
cs.save_external_result(
project_number,
workspace_number,
filtered_movs,
type="exposure",
name="desired_number_frame_exposures",
)
Assuming since this seems to be happening to so many people that this behavior of RBMC will be altered in the future to be more flexible?
Indeed, we are aware that this currently presents an inconvenience.
Hi I am getting the same error and using the script you mentioned, but my script is not running through and getting following error , could you help pointing my mistake :
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ license="XXXXXX",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ host="localhost",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ base_port=39000,
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ email="XXXXX ",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ password="XXXXXX"
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project_number = "P29"
bash: project_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ workspace_number = "W1"
bash: workspace_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_1_number = "J170"
bash: job_1_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ job_2_number = "J130"
bash: job_2_number: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project = cs.find_project(project_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # Movies with bad number of frames
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ imported_movies_job = project.find_job(job_1_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ failed_movies = imported_movies_job.load_output("failed_movies")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ # All exposures from patch CTF estimation
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_job = project.find_job(job_2_number)
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ patch_ctf_exposures = patch_ctf_job.load_output("exposures")
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ good_exposures = patch_ctf_exposures.query(lambda row: row['movie_blob/import_sig'] not in failed_movies['movie_blob/import_sig'])
bash: syntax error near unexpected token `('
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ cs.save_external_result(
bash: syntax error near unexpected token `newline'
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ project_number,P29
bash: project_number,P29: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ workspace_number,W1
bash: workspace_number,W1: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ good_exposures,
bash: good_exposures,: command not found...
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ type="exposure",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ name="desired_number_frame_exposures",
(base) [cryosparc_user@delorean CS-p63-25-7-30tilt-krios-9523]$ )
bash: syntax error near unexpected token `)'
Hi @wtempel , I wonder if my issue is something else, but not the in-consistent number of frames as I do not see any failed movies when I import movies again with “skip header check” off. However, this data was collected on falconIV and is in EER format. I saw the post above where they talked about EER data format being an issue and not running in RBMC in Cryosparc.
This is error message, I am getting and Project ID is P29 and job ID is J164:
[CPU: 620.3 MB Avail: 376.90 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 265, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames
hi all, an update from me. I ended up getting the job to run pretty easily as follows.
I used the script below to identify which movies had an abnormal number of frames. I deleted those movies, I re-imported the movies (good ones only this time), re-ran motion correction, then I used the “reassign particles” job to link my particles to the new movies. Then the RBMC ran without issue, yay!
oops I spoke too soon, I am also getting this error now! Ugh
I am running cryosparc 4.4.0
Note that I received the same error with 1 GPU or 8 GPUs, and there is nothing wrong with the GPU since I have used it perfectly find to process to where I am currently (2.8A).
full error below:
[CPU: 1.37 GB Avail: 508.03 GB]
Resolution cutoffs: alignment 3.995 A, cross-validation 2.825 A
[CPU: 3.36 GB Avail: 505.98 GB]
Removed 103 movies with fewer than 2 particles.
[CPU: 3.40 GB Avail: 505.96 GB]
Removed 43 particles (in micrographs with fewer than 2 total particles).
[CPU: 3.98 GB Avail: 505.40 GB]
--------------------------------------------------------------
STARTING: OPTIMIZE HYPERPARAMETERS
--------------------------------------------------------------
[CPU: 3.98 GB Avail: 505.40 GB]
Working with 35 movies containing 12507 particles
[CPU: 3.98 GB Avail: 505.40 GB]
Computing intended data cache configuration
[CPU: 232.8 MB Avail: 509.19 GB]
====== Job process terminated abnormally.
[CPU: 218.9 MB Avail: 509.40 GB]
DIE: cuCtxCreate: CUDA ERROR: (CUDA_ERROR_DEVICE_UNAVAILABLE) CUDA-capable device(s) is/are busy or unavailable
here is the output from nvidia-smi:
(note that the graphics cards are: NVIDIA GeForce GTX 1080 Ti)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:04:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 46MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:05:00.0 Off | N/A |
| 29% 19C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce ... On | 00000000:06:00.0 Off | N/A |
| 29% 19C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce ... On | 00000000:07:00.0 Off | N/A |
| 29% 13C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA GeForce ... On | 00000000:08:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA GeForce ... On | 00000000:0B:00.0 Off | N/A |
| 29% 17C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA GeForce ... On | 00000000:0C:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA GeForce ... On | 00000000:0D:00.0 Off | N/A |
| 29% 15C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 8 NVIDIA GeForce ... On | 00000000:0E:00.0 Off | N/A |
| 29% 13C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 9 NVIDIA GeForce ... On | 00000000:0F:00.0 Off | N/A |
| 29% 14C P8 7W / 250W | 8MiB / 11264MiB | 0% E. Process |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3401 G /usr/lib/xorg/Xorg 41MiB |
| 1 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 4 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 5 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 6 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 7 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 8 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
| 9 N/A N/A 3401 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
In CryoSPARC v4.5, released today, Ia new parameter was introduced to help alleviate this issue. If the new skip movies with wrong frame count switch is on, then the most common frame count will be assumed to be correct and all movies that don’t have that frame count will be discarded by the job.