Homogeneous Refinement Memory Error for 600px box using 64GB RAM

Troubleshooting
1

Hi, I am getting “memory error” again, posted similar error back on Dec. 18th. This time run job with 550 box size images, 600 failed after 0 iteration. 550 box job ran 9 iterations last night and failed on 10th. Restarted with last map calculated, had to import it as 3D volume. this time it ran successfully 5 iterations and failed again with error:
Done iteration 5 in 5034.457s. Total time so far 9776.073s

– Iteration 6

Using Full Dataset (split 14232 in A, 14231 in B)

Using Max Alignment Radius 205.958 (3.088A)

Using dynamic mask.

– DEV 0 THR 0 NUM 7231 TOTAL 2213.7991 ELAPSED 4429.5787 –

Processed 28463.000 images in 4433.721s.

Computing FSCs…

Done in 282.455s

Optimizing FSC Mask…
Traceback (most recent call last): File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main File “cryosparc2_worker/cryosparc2_compute/jobs/refine/run.py”, line 390, in cryosparc2_compute.jobs.refine.run.run_homo_refine File “cryosparc2_compute/sigproc.py”, line 909, in find_best_fsc tight_near_ang=near, tight_far_ang=far, initmask=initmask) File “cryosparc2_compute/sigproc.py”, line 885, in compute_all_fscs radwns, fsc_true, fsc_noisesub = noise_sub_fsc (rMA, rMB, mask, radwn_noisesub_start, radwn_max) File “cryosparc2_compute/sigproc.py”, line 762, in noise_sub_fsc fMBrand = fourier.fft(fourier.ifft(randomphases(fourier.fft(MB), radwn_ns)) * mask) File “cryosparc2_compute/sigproc.py”, line 757, in randomphases return fM * mask + randphases * amp * (~mask) MemoryError.
I am using 2.4.2 version, CentOS 7 (64 bit, 64 Gb RAM, M6000 GPU with 24 Gb memory, CUDA 8, no SSD).
could somebody tell me what went wrong? Is it possible to continue if previous job failed at some iteration step?

2

Hi @mbs,

Sorry for the trouble with this - unfortunately the memory error is happening on the CPU (I.e. system ram is running out). This should not really be happening (your box size is not unreasonably large) but there are several inefficiencies in our FSC computing code that waste memory. We are hoping to fix this (or accelerate on the GPU) but for now unfortunately the only solution is for you to run the job ensuring that nothing else on the machine is using system RAM (no other cryosparc jobs or jobs of any kind), or to acquire more RAM for the machine.

Sorry!

3

Hi @apunjani,
Thank you for reply. I understand a little more now what is happening with job fails. I use a different machine with more memory and could get further with refinement, although not much. Could you give me an idea how much memory you think I would need for 600^3 refinement? Both, uniform- and non-uniform ones? It s looks so far that all jobs fail at FSC step. And, no other jobs were running at that time on the machines I used so far, I was the only user at the time.
Thank you,
Michael

4

Hi @mbs,

Using the current version of cryoSPARC (v2.13.2), you can expect a 640px homogeneous refinement to fit on <11GB of GPU memory and <55GB of CPU memory. Please update to the latest version of cryoSPARC and let us know if you get around this error.

Further optimizations of CPU memory requirements are in the pipeline- we are expecting these optimizations to result in <15GB of CPU memory used for this box size.

5

Hi @stephan,
Yes, I can run homogeneous refinement now with 600 px box size. But nu refinement crashes with 512 px box size, when it increases alignment radius beyond 3A resolution (64 Gb RAM). Is that normal?
Thanks, Michael

6

Hi @mbs

Yes this seems correct- I have also seen similar memory requirements with larger box sizes while using Non-Uniform Refinement. Is it possible if you can turn on the “Enable Higher Order CTF” parameter? This uses a different code path that may work out for your memory limitations.

7

Hi @stephan
I usually have that parameter enabled. But jobs still fail.
Michael

8

@mbs we are hoping to soon get to memory optimization of the various advanced refinements in cryoSPARC (beyond homogeneous refinement) but this work is not yet done so there may not be much we can do at this point (other that downsampling/using a smaller box size).

9

I hope to see less memory usage with NU refinements. 640px requires more than 160GB RAM for 2.5A reconstruction. In order to use 4 GPUs, we would need almost a TB of RAM.