Hello! This is my first post to the forum, so please let me know if more information is needed.
I am testing reference based motion correction in CryoSPARC 4.4.0, using several particle sets from the same micrographs (raw pixel size of 0.4) representing different structures. One subset of particles (22,663 particles in a box size of 480 pixels) seems to work fine with the RBMC job, but another (26,411 particles in a box size of 400 pixels) fails before completing the first frame of the “Cross-validation scores computed:” part of the job.
I checked the log file within the metadata and the error output is below:
**** handle exception rc
/home/kfreeman/cryoSPARC/cryosparc_worker/cryosparc_compute/jobs/motioncorrection/mic_utils.py:95: NumbaDeprecationWarning: The ‘nopython’ keyword argument was not supplied to the ‘numba.jit’ decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See Deprecation Notices — Numba 0+untagged.4124.gd4460fe.dirty documentation for details. @jit(nogil=True)
/home/kfreeman/cryoSPARC/cryosparc_worker/cryosparc_compute/micrographs.py:563: NumbaDeprecationWarning: The ‘nopython’ keyword argument was not supplied to the ‘numba.jit’ decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See Deprecation Notices — Numba 0+untagged.4124.gd4460fe.dirty documentation for details.
def contrast_normalization(arr_bin, tile_size = 128):
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 322, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 942, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 959, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 622, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.slice_vol
File “cryosparc_master/cryosparc_compute/engine/newengine.py”, line 380, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.setup_comparison
AssertionError
set status to failed
========= main process now complete at 2023-11-21 14:01:26.287948.
========= monitor process now complete at 2023-11-21 14:01:26.291954.
I’m unsure how to troubleshoot this. I did try a few things mentioned in other RBMC threads (like using a single GPU and increasing the oversubscription threshold to 100GB), but to no avail. Any suggestions are welcome!
Hi @Kfreeman, welcome to the forum. Based on the error message, Cryosparc doesn’t like something about your particle or reference volume pixel sizes. Could you describe your workflow? Specifically, any steps like extraction, f-crop, or downsampling, which alter the pixel size or box size?
Also, since you’re processing heterogeneous data, can you confirm that you did the following:
set the number of reference volumes to 2 (which causes an extra volume, particle, and mask input to appear)
connect the references for the two different species to two different volume inputs
connect the particles for the two species to the corresponding particle input groups
Specifically I’m trying to confirm that you didn’t connect multiple particle stacks to the same particles input group.
Thanks for your reply, @hsnyder. One clarification: I’m doing separate homogenous reconstructions of the two different particle types present in the micrographs, not heterogeneous reconstructions of similar structures in overlapping boxes.
Here’s my workflow for the particle set that RBMC fails on:
Initial reconstruction was performed in Relion. To move into CS, I:
Imported raw movies into cryoSPARC; did patch motion correction and patch CTF estimation.
Imported the refined particle set from Relion, connecting the patch CTF job as the source micrographs.
Downsampled the imported particles (crop to 400, f-crop to 400), then did a homogenous reconstruction.
Performed particle subtraction and a local refinement job.
Used the particles, reference volume, and static mask from the local refinement as inputs to the RBMC, along with the movies from the PatchCTF job.
I get the same error if I connect the particles, volume and mask from a NURefine job instead of local refine.
For the second structure from these micrographs (which worked without issues) I worked in CS from start to finish, and used a NURefine job as the input. But box sizes and particle numbers are similar.
Hi @Kfreeman, thanks for the clarification and the more detailed overview of your workflow. It seems like something about the import step is triggering this issue. Could you try the same workflow, but re-extract the particles in cryosparc (including any f-crop that you might want) instead of downsampling, then do a homo refine instead of reconstruct?
If the movies are TIFFs you may need to flip the particles’ y-coordinate in order to get the pick locations right. This can be done with CryoSPARC Tools. Here’s a rough outline for a script that uses cryosparc tools to do so, though you’ll need to adapt it to your particular use case:
from cryosparc.tools import CryoSPARC
cs = CryoSPARC(
license="...", # <-- change (put your license id here)
host="...", # <-- change (put your cryosparc master hostname or IP address here)
base_port=..., # <-- change (put the cryosparc base port here)
email="...", # <-- change (your cryosparc login email)
password="..." # <-- change (your cryosparc login password)
)
assert cs.test_connection()
project = cs.find_project("Pxx") # <-- supply project number, e.g. P16
job = project.find_job("Jyy") # <-- supply job number, e.g. J104
pcls = job.load_output("particles")
pcls['location/center_y_frac'] = 1.0 - pcls['location/center_y_frac']
project.save_external_result(
workspace_uid="Wzz", # <-- supply workspace number, e.g. W1
dataset=pcls,
type="particle",
name="particles",
title="Flip particle Y coordinate",
)
Thanks @hsnyder – this seems to have worked (at least, the job is running smoothly so far). I was working remotely and couldn’t get CryoSPARC Tools running properly, but instead used the “flip mic. in y before extract” option in my Extract job and that worked. Though now I’m wondering if that will cause issues with the motion correction… I’ll find out soon, I suppose.
Hi again, @hsnyder. The rbmc job I described in my last comment (based on particles imported from Relion then re-extracted from CS-motion corrected micrographs that are first flipped along the y axis) finished. Perhaps unsurprisingly, it looks bad and the reconstructed output particles appear to be just noise. I’m assuming this is due to a mismatch of the y-coordinates between the re-extracted particles and the input movies.
So I took another stab at flipping the y coordinate of my imported particles with cryosparc tools and got that seemingly working (at least no error message, and the job appeared in my workspace); thanks for the script. The subsequent refinement jobs look fine and a new reference based motion correction job is underway. I’ll update when that finishes - fingers crossed for a reasonable result this time!
Final reply to close out this discussion: the new reference based motion correction job (with relion-imported particles whose y coordinate was flipped before re-extraction from cryoSPARAC patch motion corrected movies) looked much better and I’ve already got a 0.1 angstrom improvement in resolution; now on to local refinement! Thanks so much, @hsnyder, for helping me get on track.