Hello, I am getting a GPU memory when I try to run 3D flex refinement. The GPU is a 16 GB 3080 Ti. If I downsample the particles it does not reduce the amount of memory the job is asking for and the job fails with the same error.
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main
File “/home/lnd/Cryosparc/cryosparc_worker/cryosparc_compute/jobs/flex_refine/run_highres.py”, line 150, in run
flexmod.do_hr_refinement_flex(numiter=params[‘flex_bfgs_num_iters’])
File “cryosparc_master/cryosparc_compute/jobs/flex_refine/flexmod.py”, line 1640, in cryosparc_compute.jobs.flex_refine.flexmod.do_hr_refinement_flex.lambda7
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py”, line 198, in fmin_l_bfgs_b
**opts)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/lbfgsb.py”, line 308, in _minimize_lbfgsb
finite_diff_rel_step=finite_diff_rel_step)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/optimize.py”, line 262, in _prepare_scalar_function
finite_diff_rel_step, bounds, epsilon=epsilon)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py”, line 140, in init
self._update_fun()
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py”, line 233, in _update_fun
self._update_fun_impl()
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py”, line 137, in update_fun
self.f = fun_wrapped(self.x)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py”, line 134, in fun_wrapped
return fun(np.copy(x), *args)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/optimize.py”, line 74, in call
self._compute_if_needed(x, *args)
File “/home/lnd/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/scipy/optimize/optimize.py”, line 68, in _compute_if_needed
fg = self.fun(x, *args)
File “cryosparc_master/cryosparc_compute/jobs/flex_refine/flexmod.py”, line 1640, in cryosparc_compute.jobs.flex_refine.flexmod.do_hr_refinement_flex.lambda7
File “cryosparc_master/cryosparc_compute/jobs/flex_refine/flexmod.py”, line 1611, in cryosparc_compute.jobs.flex_refine.flexmod.errfunc_flex
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.57 GiB (GPU 1; 11.77 GiB total capacity; 9.33 GiB already allocated; 980.38 MiB free; 10.13 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF