Does "new refinement" require more GPU memory?

open

#1

Hi,

Every time I try to run the “new refinement” module, it runs out of GPU memory (box size 600 px, RTX-2080Ti). Tried resampling down to 512px on the fly, but apparently that isn’t yet supported for the “new refinement” module.

When I run a refinement with otherwise identical parameters, but via the “legacy” homogeneous refinement module, it runs fine.

What is the max box size one can accommodate on an 11GB GPU for the new refinement module?

Cheers
Oli


#2

It seems this is due to the CTF refinement. Running the global or local CTF refinement on 600px particles separately also fails, running out of GPU mem. Is there any way to make this consume less GPU memory?


#3

(or a way to run this via the CPU would also be useful)


#4

This is another good time to plug 1.5x Fourier padding (vs. 2x). :grinning:

Have you tried reducing the GPU batch size in the defocus refinement section?


#5

I did try reducing the batch size to 100 - no difference - and unfortunately there is no way to alter the Fourier padding in CTF refinement as far as I can see

Cheers
Oli


#6

I think it’s only a user parameter in 2D classification.


#7

Tried with a box size of 512 (downsampling particles), and it gets through three iterations but dies on the fourth. What is the max box size for the “new” homogeneous refinement on an 11GB card?

Oli


#8

Hi @olibclarke @DanielAsarnow,

Sorry about this - the new refinement is intended to actually be a lot more memory efficient than before! We’ll look into what the max box size is/should be on an 11GB card.
For now, @olibclarke can you run the separate CTF refinement jobs by changing the “GPU batch size of images” parameter to ~100?
This parameter is present in the “new refinement” as well but does not work correctly unfortunately…


#9

Hi Ali - reducing the batch size worked for the local CTF refinement, but not for global, unfortunately

Cheers
Oli


#10

Also @apunjani - I also run out of GPU mem with these large particles in 3D-VA - for 3D-VA maybe it might be possible to downsample on the fly to avoid this?

Cheers
Oli