Hi @olibclarke, I have a potential workaround that may address this. For background, v4.4 includes a new GPU memory management system (using the numba Python library) that does not immediately free memory when it’s no longer required. Instead, it frees in batches or when memory is low.
Your Patch Motion job appears to fail during a special allocation step that is unaware of this memory management system. So there may be some GPU memory that could be freed to make this work.
We’ll should have a fix for this in a future version of CryoSPARC, but in the mean time you could try disabling batched-memory deallocation by adding the following line to cryosparc_worker/config.sh:
export NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT=0
Let me know if you get a chance to try this and it works for you.
@AlexHouser I am not sure whether the same fix will apply to you; based on the text, the error appears to be coming from a different place that is correctly managed. We are still investigating other memory usage changes in v4.4.