Patch motion taking hours

Motion Correction
1

Hello, we are processing eer micrographs with pixel size 0.58 binned twice on cryosparc using patch motion and 2600 images took almost 6 hours to complete. Is that normal since relion can do the motion correction of 1000 in less than 1 hour? Are we running it wrong. What is a normal speed?

2

EER upsampling=2 at import perhaps? This does slow things down quite a bit relative to 4K-rendering, which RELION defaults to.

Cheers,
Yang

3

Depends a lot on where the micrographs are, although for high mag EER data file sizes should be small enough not to be a problem.

My guess is, as @leetleyang also hypothesised, that you left EER sampling at 2. This will output 8K movies, which isn’t the issue… the bigger issue will be that CryoSPARC struggles with CTF estimation at very small pixel sizes, so chances are Patch CTF Estimation will fail.

It’s interesting, because I usually find RELION motion correction (using “RELIONCor”) is significantly slower than Patch Motion Correction of EER in CryoSPARC…

4

I am not upsampling. We did patch motion on 9000 images in 20 hours, pixel size 0.58, 45 e/A dose.
Is that normal when I get 1000 movies/hour in Relion?

5

I see.

Can you confirm the number of GPUs you’re parallelising Patch Motion Correction across, and in comparison, the MPI/thread values, nodes, and cores-per-node in your RELION scenario?

Cheers,
Yang

6

I am using 1 GPU for the patch motion and for relion I am using 32 MPI/20 threads, no GPUs.
would using more CPUs make the jobs in cryosparc faster?

7

Thank you. In that case, the throughput you’re seeing is quite normal, on both fronts.

Your RELION parameters instruct RelionCor to process 32 movies in parallel (each in 20-frame batches). What CPU-processing loses out in compute, it makes up for in multithreadedness and parallelisation. I assume the MPIs are spread across several nodes in your cluster, which alleviates network I/O bottlenecks.

On the cryoSPARC front, 9000 EER moviestacks over 20 hours works out to 7.5 stacks/min, which feels normal to me given the 1 GPU-assignment. The good news is that this will scale somewhat linearly with the number of GPUs you assign to the task–within reason. Try increasing Number of GPUs to parallelize to 2 or 4 (depending on your workstation/node configuration) and see if that helps.

Cheers,
Yang

1 Like
8

We have limited number of tokens and cannot really afford to use 4 GPUs. Is there any way to accelerate the process by increasing the threads in the cryosparc jobs and not making is so realiant on GPUs?

9

Unfortunately not. All compute-heavy tasks in cryoSPARC are, by design, GPU-accelerated.

You can limit downtime by preprocessing movies in cryoSPARC Live as they’re being generated/transferred. A modern GPU would be able to keep pace with typical collection throughputs.

Another option is to perform motion correction in RELION and import the averaged micrographs into cryoSPARC for downstream processing. I believe the community has generally found Patch Motion Correction to do a better job, for whatever reason, but RelionCor is perfectly workable. However, this workflow will preclude Reference-Based Motion Correction. As Sanofi seems to have access to a lot CPU resources, Bayesian polishing would be a viable alternative.

Cheers,
Yang

10

Confirming the post from @leetleyang, CryoSPARC’s Patch motion correction is GPU accelerated and does not have a CPU-only option. Parallelism over multiple GPUs generally provides a near-linear performance increase unless there are other bottlenecks such as the filesystem. Performance will depend on a number of factors:

  • For EER data, EER Upsampling Factor parameter (default 2x) causes the data to be processed in 2x super resolution. We set this default in order to be conservative and ensure that the available signal in the EER files can be made use of downstream. However, in some cases a 2x upsampling factor may not be required and can be set to 1x instead. This is especially the case if micrographs will be downsampled via Output F-crop factor anyway. Changing to 1x reduces the number of pixels processed during motion correct by a factor of 4.
  • Filesystem performance can have a large impact. You can see the relative times of loading vs. processing data in the output of Patch motion jobs, e.g.:
-- 0.0: processing 2 of 20: J1/imported/14sep05c_00024sq_00004hl_00002es.frames.tif
        loading /bulk6/data/cryosparcdev_projects/P103/J1/imported/14sep05c_00024sq_00004hl_00002es.frames.tif
        Loading raw movie data from J1/imported/14sep05c_00024sq_00004hl_00002es.frames.tif ...
        Done in 2.26s
        Loading gain data from J1/imported/norm-amibox05-0.mrc ...
        Done in 0.00s
        Processing ...
        Done in 6.99s
        Completed rigid and patch motion with (Z:5,Y:8,X:8) knots
        Writing non-dose-weighted result to J2/motioncorrected/14sep05c_00024sq_00004hl_00002es.frames_patch_aligned.mrc ...
        Done in 0.39s
        Writing 120x120 micrograph thumbnail to J2/thumbnails/14sep05c_00024sq_00004hl_00002es.frames_thumb_@1x.png ...
        Done in 0.31s
        Writing 240x240 micrograph thumbnail to J2/thumbnails/14sep05c_00024sq_00004hl_00002es.frames_thumb_@2x.png ...
        Done in 0.00s
        Writing dose-weighted result to J2/motioncorrected/14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted.mrc ...
        Done in 0.38s
        Writing background estimate to J2/motioncorrected/14sep05c_00024sq_00004hl_00002es.frames_background.mrc ...
        Done in 0.01s
        Writing motion estimates...
        Done in 0.00s

Hope this helps!