Hi,
Every time I try to run the “new refinement” module, it runs out of GPU memory (box size 600 px, RTX-2080Ti). Tried resampling down to 512px on the fly, but apparently that isn’t yet supported for the “new refinement” module.
When I run a refinement with otherwise identical parameters, but via the “legacy” homogeneous refinement module, it runs fine.
What is the max box size one can accommodate on an 11GB GPU for the new refinement module?
Cheers
Oli
It seems this is due to the CTF refinement. Running the global or local CTF refinement on 600px particles separately also fails, running out of GPU mem. Is there any way to make this consume less GPU memory?
(or a way to run this via the CPU would also be useful)
This is another good time to plug 1.5x Fourier padding (vs. 2x). 
Have you tried reducing the GPU batch size in the defocus refinement section?
I did try reducing the batch size to 100 - no difference - and unfortunately there is no way to alter the Fourier padding in CTF refinement as far as I can see
Cheers
Oli
I think it’s only a user parameter in 2D classification.
Tried with a box size of 512 (downsampling particles), and it gets through three iterations but dies on the fourth. What is the max box size for the “new” homogeneous refinement on an 11GB card?
Oli
Hi @olibclarke @DanielAsarnow,
Sorry about this - the new refinement is intended to actually be a lot more memory efficient than before! We’ll look into what the max box size is/should be on an 11GB card.
For now, @olibclarke can you run the separate CTF refinement jobs by changing the “GPU batch size of images” parameter to ~100?
This parameter is present in the “new refinement” as well but does not work correctly unfortunately…
Hi Ali - reducing the batch size worked for the local CTF refinement, but not for global, unfortunately
Cheers
Oli
Also @apunjani - I also run out of GPU mem with these large particles in 3D-VA - for 3D-VA maybe it might be possible to downsample on the fly to avoid this?
Cheers
Oli
Dear Oli, dear Ali, dear Daniel,
I think I’ve been running into similar issues as described above (I’m running CS version 2.15.0), trying to refine a structure with a box size of 800pix on a cluster equipped with NVIDIA GV100GL Tesla V100 with 16GB of GPU RAM per GPU node. I was using the new homogeneous refinement, all parameters default, getting this error after the start of iteration 0:
I tried to set the GPU batch size to 100 which made the jobs run further but they also failed in the end. Incidentally, non-uniform refinement jobs also fail with this kind of box size.
My structure is very large but I can downsample a bit (box size 600pix) without reaching the Nyquist limit which now made the homogeneous refinement run all the way to the end, but the NU refinement failed again. I now have a whole collection of error messages 
So my question is should these jobs actually run, do I do something wrong? Have there been any developments since last year December? Has cryoSPARC done some sort of benchmarking for memory for running refinement jobs? Happy to provide further details if needed…
Thank you,
Claudia
