I recently encountered a problem during particle extraction from micrographs. Here is the screenshot showing the error message.
I feel this error is related to GPU out of memory, because I noticed that the used GPU memory was about 5 GB when the job was initially launched and gradually increased as the job ran. Eventually, as one or more of the GPUs ran out of memory, I started to see the error messages. Using the CPU version of the job, particle extraction successfully completed for all the micrographs without any issue. And the system memory usage stayed about the same through the whole process of the job. It seems like the GPU version of the particle extraction job was unable to release the GPU memory after it finished extraction from previous batches of micrographs. Below is some information about my system:
CPU: AMD Threadripper Pro 5975WX, 32 Core
Memory: 256 GB
GPUs: 4x NVIDIA RTX A5500, each with 24 GB of VRAM
I don’t know if this issue is dependent on the version of cryoSPARC. Although I never encountered this problem before while using cryoSPARC v2 and v3, the datasets I worked with before were also much smaller than this one (~2000 movies before vs 6453 movies here). So it was possible that in previous cases, particle extraction completed before the GPU ran out of memory.
Please let me know if you have any suggestions or ways to fix this problem.