pycuda._driver.MemoryError: cuMemAlloc failed: out of memory

Troubleshooting
1

Hello,

I am using Cuda 11.2 on Ubuntu 16.04 with GeForce GTX 980 Ti/PCIe/SSE2 and cryosparc v3.1.0. I am trying to perform Patch motion correction (multi) with ~3000 tiff movies, and I am recieving this error:

Error occurred while processing J6/imported/VLP_30k_1003_Oct14.tif
Traceback (most recent call last):
File “/home/salmen/programs/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec
return self.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 602, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 339, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
File “/home/salmen/programs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory

Marking J6/imported/VLP_30k_1003_Oct14.tif as incomplete and continuing…

Please let me know what I should do to overcome this error.

Best,
Wil

1 Like
2

Hi @wsalmen,

Unfortunately the 980 Ti, with only 6 GB of VRAM, is very limited, and technically below the minimum specification required to run cryoSPARC. I suspect that the only way to overcome that error is to upgrade your GPU. You can see more information about cryoSPARC’s minimum system requirements here: https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements

Harris

3

Hi @hsnyder,
I have the same problem, but I am using Cuda 11.3.0-1 on Ubuntu 18.04.5 with GeForce GTX 1660 Ti/PCIe/SSE2 (16GB Ram) and cryosparc v3.2.0. I’m running single Workstation and my notenook meets the minimum requirements but this error appeared.

Please let me know what I should do to overcome this error.

Best,
Roman

4

@KulR, I believe the GTX 1660 Ti only has 6GB VRAM as well. Can you post the output of nvidia-smi?

5

@spunjani, yes it has 6GB VRAM, but requirements says that I need 16 GB RAM

requirements
requirements790×593 36.8 KB

I take this from https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements, so do I need to change my videocard?

And will I need to change my videocard if I start worker node in cluster?

6

Also this is the result of nvidia-smi:

nvidia-smi
nvidia-smi736×443 50.7 KB

7

Your CPU and RAM requirements are fine, but you’re getting the pycuda._driver.MemoryError: cuMemAlloc failed: out of memory error because your GPU doesn’t have enough VRAM (we require 11GB+).

Do you mean that you will be scheduling jobs to a different workstation that is managed by a cluster? If so, you don’t need a GPU. In that case, your notebook will be acting as the “master” node.

8

Yes, I mean that my notebook will be master node and cluster as worker node. Thank you for your answer!

9

I’ll jump into this discussion, a bit late. We have several GTX 1080 cards (8GB of VRAM) that we would like to continue using with cryoSPARC (v 3.2 on Ubuntu 20.04). Is there a way, other than running CPU-only, to continue using these cards with cryoSPARC?

Michael