CUDA_ERROR_OUT_OF_MEMORY error for 2D classification after upgrading to v5

2D Classification
1

Hi All,

I got CUDA_ERROR_OUT_OF_MEMORY error for 2D classification after upgrading to v5 (currently on v5.0.2). This error always happens when using a class number of 400, either with 1 or 2 GPUs (GeForce RTX 4090 with 24 GB of VRAM). The 2D jobs work fine with a class number of 300 or 250. I used to be able to run two instances of 2D classification with 400 classes on one such GPU on v4.7.1, but now I have to run the same job on GPUs with 48 GB of VRAM (RTX A6000). Below is the error message I received:

Traceback (most recent call last):
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 851, in _attempt_allocation
    return allocator()
           ^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 1054, in allocator
    return driver.cuMemAlloc(size)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 348, in safe_cuda_api_call
    return self._check_cuda_python_error(fname, libfn(*args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 408, in _check_cuda_python_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cli/run.py", line 105, in cli.run.run_job
  File "cli/run.py", line 210, in cli.run.run_job_function
  File "compute/jobs/class2D/run.py", line 295, in compute.jobs.class2D.run.run_class_2D
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/compute/alignment.py", line 652, in greedy_align_2D_noqueue
    align_res = align_pairs(
                ^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/compute/alignment.py", line 440, in align_pairs
    NET.ensure_allocated("denom", (N_H, N_KK, N_S, N_R), n.float32)
  File "compute/gpu/gpucore.py", line 399, in compute.gpu.gpucore.EngineBaseThread.ensure_allocated
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/compute/gpu/gpuarray.py", line 377, in empty
    return device_array(shape, dtype, stream=stream)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/compute/gpu/gpuarray.py", line 333, in device_array
    arr = GPUArray(shape=shape, strides=strides, dtype=dtype, stream=stream)  # type: ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/compute/gpu/gpuarray.py", line 122, in __init__
    super().__init__(shape, strides, dtype, stream, gpu_data)
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/devicearray.py", line 103, in __init__
    gpu_data = devices.get_context().memalloc(self.alloc_size)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 1372, in memalloc
    return self.memory_manager.memalloc(bytesize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 1056, in memalloc
    ptr = self._attempt_allocation(allocator)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 863, in _attempt_allocation
    return allocator()
           ^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 1054, in allocator
    return driver.cuMemAlloc(size)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 348, in safe_cuda_api_call
    return self._check_cuda_python_error(fname, libfn(*args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cryosparcuser/Applications/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/driver.py", line 408, in _check_cuda_python_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [2] Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY

I have been monitoring the VRAM usage and noticed that the VRAM usage is around 7-8GB during initial iterations and increases to ~15GB as soon as the 2D classification job starts the first full iteration, which is immediately followed by the CUDA_ERROR_OUT_OF_MEMORY error. I did not see the full 24GB VRAM being filled before the job crash.

Now I have downgraded to v4.7.1 due to this issue, but would really hope this issue can be solved soon for v5.

Thanks.

2

Thanks @YYang for your post. In v5.0.2, one may be able to avoid Call to cuMemAlloc results in CUDA_ERROR_OUT_OF_MEMORY in 2D classification with hundreds of classes by setting the Plotting sort method parameter to size.

CSv5.0.5 GPU memory allocation error
3

Thanks @wtempel for the suggestion. I will try it and post later whether it solves the issue.

Best,

Yang

4

Hi,

we have the same problem with 5.0.4 and can also resolve it with sorting method set to size. Is this a bug (which might be resolved) or is it a real Memory problem? We have this even with small (256px) boxes on 3090 (24GB VRAM) cards…

Can you clarify: The sorting should only affect the visualisation, but not the results of the the 2D classes?

Thanks for the great software by the way,
Jan

5

Thanks @gebauer for your questions and the kind words.
In v5.0.4 the sorting method only affects visualization, not the results of the classes.
We hope to reduce the memory usage of the similarity sorting implementation in a future release.

1 Like
6

Dear developers,

I’ve done an analysis of the GPU memory usage in 2D classification using the parameter “Plotting sort method” kept at the default value “similarity”, run on two A4000 GPUs, each with 16 GB GPU memory. I’ve recorded the GPU memory usage with the “nvidia-smi” tool every second. The maximum GPU memory usage per iteration was stable at ~5 GB per GPU, but in the last iteration suddenly three short spikes at ~11 GB appeared only on GPU 0, which explains the observed CUDA_OUT_OF_MEMORY issues. A control run with “Plotting sort method” set to “size” did not show any spikes above ~5GB. Hopefully, this GPU memory usage issue can be solved in a future version. I’ve appended a plot of the GPU memory usage for GPU 0.

Best regards,

Dirk

2D-Classification-GPU0
2D-Classification-GPU0640Ă—481 35 KB

1 Like
7

Thanks so much @dirk for this visualization and testing, it’s helpful to see!

A fix for this excessive memory usage in the final three iterations is currently in progress! In the meantime as you said, changing the Plotting sort method to size or class_index should work-around the issue!

Michael

8

Hi,

I experienced the same issue on both RTX 4080 and Tesla V100 (16 GB) during 2D classification with similarity sorting (1,000,000 particles, 128px box). Interestingly, switching to an older Tesla P100 (physically, everything else remained the same, including drivers) solved the problem! This clearly indicates it is not just memory, but also GPU architecture dependent issue.