3D Flex reconstruction uses a very large amount of memory. I’ve gotten around this issue by creating huge swap files on our reconstruction nodes, but this is obviously undesirable, both for scratch space and because in effect cryosparc is now shuffling the same files around on the disk before starting a reconstruction.
At least from watching netdata, it seems that much of this memory is consumed loading in the particle images/CTF data into system memory before starting. Is there any chance we could have a low-memory mode, similar to the low (GPU) memory mode for motion correction, loads the images in smaller batches?
2 Likes
Seconded.
I’m used to systems with 1TB+ of RAM running out of memory with extreme box sizes but it’s a bit crazy when it happens with box sizes <512 pixels.
1 Like
Hi @posertinlab @rbs_sci you are correct that 3DFlex training and reconstruction job currently both expect to fit the entire particle stack (and pre-computed CTF stacks) in RAM before they’re able to run. This is a remnant of implementation constraints from early on in the 3DFlex development.
We are working on switching 3DFlex over to the standard CryoSPARC IO/caching systems so that we can read particles on the fly and deal with arbitrary sized datasets without exhausting system RAM.
Thanks!
4 Likes
@apunjani - Look forward to it! Thanks for the reply.