Hi there, did anyone use the Patch motion correction (multi) in CryoSparc 2.12.2? Right now, I am using this function with K2 data in the superresolution model. It takes about 300s for one GPU (1080) to process one tif image. After one night with a 4-GPU cluster node, it only processed 700 images. Does anyone know how to speed up?
Thank you.
Tinghai
setting:
Hey @PeterXTH,
Thanks for providing screenshots - it looks like you’re actually on cryoSPARC v2.13.2.
The bulk of the time seems to be coming from the loading of the raw data (255.3s in the screenshot). The motion estimation itself only took about 21.6s. This may be an issue with the speed of the file transfer between the filesystem where the raw movies are stored and the GPU Workstation that is trying to process the data.
You can check out this post I found online to test and make any optimizations:
I just tested it the speed is about 300 MB/s. I do not know which step or parameter I entered is wrong.
(when I use Motioncorr2 in RELION, it takes only about 7 hours.) Any ideas?
Thanks
Hi @PeterXTH,
This indicates your raw movies are stored on the same machine as where this job ran, is that correct?
The raw image is store at HPC storage and runs at HPC storage not local storage disk. The I/O speed of the HPC storage is roughly about 300 MB/s. The cryosparc is installed at a HPC cluster node.
Hi @PeterXTH,
Could it be possible that another process was hogging system resources on the HPC cluster? Can you try running the job again and see if you still get the same timings?
I will delete all jobs and try to run one job at a time.
I have tried with only single job for patch motion correction. It was still very slow with same log.
Hey @PeterXTH,
It turns out that this may have something to do with some performance regressions introduced in v2.13.2- we made some changes to the core data structure class in cryoSPARC which may have caused the slowdowns. We’re releasing an update soon, v2.14.0, which will have lots of optimizations to this code. Can you try it out and see if your problem goes away? I’ll update this thread to let you know when it’s released.
Hey @PeterXTH,
The newest version of cryoSPARC, v2.14.0, has been released. It includes the optimizations I spoke of in my earlier post. Can you try your jobs again and see if you experience a speedup? Thanks!
Hi Sarulthasan,
I just updated to v2.14.2. It is still very slow loading the raw image, almost identical as previous. One thing I should point out is that I used the old version of CUDA8. Is this caused the slow loading (230s/image)?
Thank you.
Hey @PeterXTH,
The version of CUDA shouldn’t affect the movie read times- we use a custom wrapper on top of the python libtiff wrapper (https://github.com/pearu/pylibtiff) which uses threads to read in multiple frames of the TIFF image at the same time. How many frames are your movies?
I am having similar issues with .mrc and .tif movie files. Using v2.14, my file load times vary substantially (from 10 seconds-2000 seconds). If I repeat the job, the load time for each movie is about the same as the run the run before. The fast files still all go in about 15 seconds, but the files that go long have more variability. For example, a file that takes 300 seconds the first run can take 1000 seconds the next run.
HI @WGL,
Is it possible that this variance in times is related to network activity? Are your raw movies stored locally on a hard drive, or are they stored on a file system that is mounted to the machine?
Hi, Iam also facing same issue.
I have 8000 movies in .eer format. Using the latest version of CryoSPARC.
150 movies got patch CTF corrected in first two hours and it taking hours, still running.
Why it is slow. Did you find any solution?
You can compare the performance of your compute infrastructure, including GPU and storage subsystems using tools released in CryoSPARC v4.3.
Thank you, will check for sure.
