Max box size in refinement

Hi,
I’m try to to refinement of virus with big box size (1200 pix). When I try scale the box above 600 pix I getting mem error.
I have 4 1080Ti and 512G RAM

this is the error:
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/nonuniform_refine/run.py”, line 696, in cryosparc2_compute.jobs.nonuniform_refine.run.run_non_uni_refine
File “cryosparc2_worker/cryosparc2_compute/jobs/nonuniform_refine/run.py”, line 698, in cryosparc2_compute.jobs.nonuniform_refine.run.run_non_uni_refine
File “cryosparc2_worker/cryosparc2_compute/jobs/local_filter/run.py”, line 292, in cryosparc2_compute.jobs.local_filter.run.standalone_locfilter
File “cryosparc2_worker/cryosparc2_compute/jobs/local_filter/run.py”, line 361, in cryosparc2_compute.jobs.local_filter.run.standalone_locfilter
File “/home/binshtem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 1018, in conj
result = self._new_like_me(order=order)
File “/home/binshtem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 420, in _new_like_me
allocator=self.allocator, strides=strides, order=order)
File “/home/binshtem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

The maximum size is ~512 or 768 px for your GPUs. Unfortunately, there is no way to reduce the Fourier padding in cryoSPARC, or I would recommend trying 1.5x or no padding instead of 2. You can try skipping padding, or using 1.5x padding in Relion GPU, or use cisTEM or Relion CPU instead.

I am just curious about the GPU memory calculation. With 512px in 3D, double precision, and 2X padding, the memory is about 4GB (512^3 * 2^3 * 4). So it should be OK, right? Or there is something that I forgot to include? Thanks.

You need to fit at least two (Fourier/complex) reconstructions, some particles, a various other things. A good formula is ~box**3 * 20 * K. So up to ~800 px box on a 11GB card for legacy homogeneous refinement.

Hi @DanielAsarnow @Liguo @Elad,

in v2.13 we’ve optimized refinement box size in the “new refinement” job - it can now fit up to 1024^3 box size on 11GB cards.

Note that this still requires a substantial (160+GB) of CPU ram. We are working on optimizing the CPU memory usage as well.

1 Like

I tried Non-Uniform Refinement (NEW) for a virus dataset with the box size 720x720 on a GPU node of Nvidia Titan V (12G memory), and failed during iteration 1 with this error message (Job process terminated abnormally). I am using newest cryosparc (version: v3.2.0+211012).