Patch Motion MemoryError: cuArrayCreate failed: out of memory

Dear all,

With our current installation, we can do patch motion correction from movies in .mrc format collected with a K3 camera without using super resolution mode. However, with a dataset containing movies collected in super resolution mode on the K3, in .tiff format, patch motion correction fails.

We are using G Force RTX 2080 and 2080 Titanium. 
CUDA version 10.0
The SLURM script is the following: 

#SBATCH --job-name cryosparc_P12_J21
#SBATCH -n 12
#SBATCH --gres=gpu:2
#SBATCH -p gpu
#SBATCH --mem=64000MB             
#SBATCH -o /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/output.log
#SBATCH -e /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/error.log

export MODULEPATH=/opt/ohpc/pub/modulefiles:/g/easybuild/x86_64/CentOS/7/haswell/modules/all
source /opt/ohpc/admin/lmod/lmod/init/bash
echo CUDA_VISIBLE_DEVICE=$CUDA_VISIBLE_DEVICES
echo hostname=`hostname`
module load cryosparc2_worker/2.14.2-foss-2018b 
module list

cryosparcw run --project P12 --job J21 --master_hostname cryosparc.embl-hamburg.de --master_command_core_port 39002 > /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/job.log 2>&1 

The error message is the following:

[CPU: 498.3 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/g/easybuild/x86_64/CentOS/7/haswell/software/cryosparc2_worker/2.14.2-foss-2018b/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 446, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 197, in cryosparc2_compute.engine.cuda_core.transfer_ndarray_to_cudaarray
MemoryError: cuArrayCreate failed: out of memory

[CPU: 163.6 MB]  Outputting partial results now...
[CPU: 163.7 MB]  Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 82, in cryosparc2_compute.run.main
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 349, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 64102 has terminated unexpectedly!

Has anyone experienced the same issue and have a solution ?
Thank you in advance,
Maxime.

Hi @mkiller,

Thanks for reporting - K3 superres movies (regardless of compression format) do require more GPU RAM to process through patch motion. Can you let us know the following:

  • raw pixel size of the super-res movies
  • number of frames
  • total electron dose

And for now can you try to run the job with the “Output F-crop factor” set to e.g. 1/2 ? this will downsample the output micrograph and cause the job to require slightly less GPU RAM.

Hi @apunjani ,

Thanks for your reply.

-pixel size in super resolution: 0.55 (1.1 real)
-40 frames
-Total electron dose: 60 e/A^2

I have run the job with “Output F-crop factor” set to 1/2 and got the same error.

Best,
Maxime.

Hi @mkiller,

Can you try the job with “Maximum alignment resolution” set to 10A, and Output F-Crop factor set to 1/2 ?
This may just fit in 8GB RAM but it’s hard to say.

Hello, I am having the same problem, K3 super-res, pix size 0.546, 42 frames, 44 e/pix/s. I specified 4 GPUs.

I tried F-crop 1/4, F-crop 1/2 and Max align res 10A with F-crop 1/4, but always get the same error described by mkiller.

It looks like cryosparc is calling 64GB memory. Is there a way to increase the RAM available to the GPUs? Thank you.

Hi @DaveT and @mkiller,

After running some jobs with your settings, it appears that both of your jobs currently require about 12 GB of GPU memory. We are working on an optimisation that will reduce the memory usage by several GB, and we will let you know when it’s available. Based on our preliminary testing with the optimisation, these jobs will fit in 8 GB of memory if an output f-crop factor of 3/4 is used.

@DaveT, Unfortunately, there’s no way to increase the memory available to the GPU. The GPU has it’s own RAM, separate from system RAM, that’s physically located on the card. Could you let us know what GPU model you’re using so we have some idea of the memory constraints?

Thanks for looking into this. GPUs are RTX 2080 Ti, 11 GB memory. I’ve tried F-crops of 1/2 & 1/4, which gave the same errror. I’ll test it, but wouldn’t 1/2 & 1/4 use less memory the 3/4?

Hi @DaveT,

Thanks for getting back to me with the GPU information.

To clarify, we’ve been getting the f-crop 3/4 setting to work (with under 8GB) only with the memory usage optimisation we’re working on internally. You’re correct that if 1/4 and 1/2 don’t work for you currently, then 3/4 won’t either. We are working to make this improvement available as soon as possible, and will keep you advised.

Hi @hsnyder,
Great thank you, looking forward.
Best,
Maxime.

Hi @mkiller and @DaveT,

A patch is now available that will address this problem. After you have updated to v2.15, please see the following section of the new CryoSPARC guide regarding how to apply the patch (depending on whether your setup is single or multiple nodes)
https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches

After the patch is applied, these jobs should complete successfully. You’ll notice a new switch in the job builder for patch motion correction, called “Reduce GPU memory usage”. It is on by default, but we consider this a beta feature and if you have any trouble with it, you can turn it off and revert to the previous behaviour.