3DFlex train job - RAM unavailable

Hi @AnokhiShah,

Thanks for reporting. 3DFlex requires a lot of RAM because 1) it stores all particles in RAM during training and reconstruction and 2) the model itself requires a lot of RAM to store and train. We don’t currently have precise bounds/estimates on how much RAM the method will use, so the 64GB requirement is more of a placeholder. For some datasets (depending on many factors like box size, number of voxels in the mask, number of particles, pixel size, mesh params, number of mesh cells, etc) the actual usage may be a lot less or a lot more than this number.
Likewise we do not have tight bounds on GPU memory usage, though we have run experiments on cards with as little as 16GB of GPU RAM.
Eventually we hope to better characterize memory usage, but for now you may have to find another node to run on or a different solution.

Thanks!

1 Like