Is this a node with a single physical GPU?
Are other GPU tasks (Linux GUI, other CryoSPARC or non-CryoSPARC GPU calculations) running on the node at the same time?
Does defining
export NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT=0
inside cryosparc_worker/config.sh
(see Patch Motion Correction - RuntimeError: Could not allocate GPU array: CUDA_ERROR_OUT_OF_MEMORY - #32 by nfrasser) mitigate the problem?