Full-frame motioncorrection fails on second movie, processes one fine (cuMemAlloc failure)

Troubleshooting
1

Hi,

I’m trying to perform full-frame motion correction using RTX2080s. Interestingly, if I give the job only one movie to process, it works fine. However, if it is processing more than one movie, it typically seems to process either zero or one movies and then fail a memory allocation. It appears, to me, that it’s trying to load another movie onto the GPU before the first is done? But that’s a guess.

I’m on cryoSPARC v2.14.2, cuda 10.2, nvidia driver version 440.64.00. This is a standalone install with 4x RTX 2080s, CentOS 7.7.1908, and I just rebooted the system just to be sure :wink:

Error:

[CPU: 6.27 GB]   Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 149, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 150, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py", line 215, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py", line 240, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/cryosparc/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

image
image1319×410 80.8 KB

J6 is a clone of J3, except J6 is trying to process all 4k movies instead of 1 (i.e., changed “Only process this many movies” box and nothing else)

Any help would be appreciated.

2

To give some more information, I have tried motion correction on CUDA 10.1 as well, but this did not resolve the issue. Since they’re RTX cards, non-10 versions of CUDA are not an option.

3

Hi @posertinlab

Thanks for reaching out, we will look into this. Could you please let us know the following:

  • Number of frames per movie
  • Dimensions of the frames, in pixels
  • Raw pixel size (A)
  • Total electron dose
4
  • 50 frames / movie
  • 11520 px wide x 8184 px tall
  • 0.5395 A/px
  • Total dose of 50 e-/A^2

See below for imod header output:

$ header imported/stack_00000.tif

 RO image file on unit   1 : imported/stack_00000.tif     Size=     628218 K

                    This is a TIFF file.

 Number of columns, rows, sections .....   11520    8184      50
 Map mode ..............................    0   (byte)
 Start cols, rows, sects, grid x,y,z ...    0     0     0   11520   8184     50
 Pixel spacing (Angstroms)..............  0.5395     0.5395     0.5395
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................   0.0000
 Maximum density .......................   70.000
 Mean density ..........................   35.000
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

     2 Titles :
SerialEMCCD: Dose frac. image, scaled by 1.00  r/f 0
  SuperRef_stack_00000.dm4
5

Thanks for providing those details. We were able to reproduce the issue, and are working on a fix. We will let you know as soon as the fix is available.

1 Like
6

@hsnyder updating to cryoSPARC 2.15 has resolved my issue. Thank you for the quick patch!

2 Likes
7

@hsnyder it seems the same issue occurs with patch CTF as well. One CTF estimation succeeds, but trying to estimate all results in a cufftAllocFailed error.

8

We will definitely look into it. Did the issue arise using the same data discussed above?
i.e.

50 frames / movie
11520 px wide x 8184 px tall
0.5395 A/px
Total dose of 50 e-/A^2
9

Yep, same data! Thanks!

10

Hi @posertinlab, I was able to confirm the issue again. This will be fixed in an upcoming release. In the meantime, if you’re able to use patch motion correction instead of full-frame, you can use the output f-crop feature to reduce the size of the images being fed to patch CTF estimation. I was able to get the memory requirement well under 8 GB that way.

If you do decide to go that route, you’ll need to latest cryosparc patch, which adds a switch for patch motion correction that reduces the GPU memory usage. See the linked section of the new CryoSPARC guide for instructions on how to apply patches: https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches

11

Great, thanks! I look forward to the future release