Hi all,
The 3D Flex Reconstruction job keeps getting the following error:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 96, in cryosparc_compute.run.main
File “/opt/apps/alces/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/flex_refine/run_highres.py”, line 113, in run
ctfs_cpu = n.empty((N_D, N_highres, N_highres//2+1), n.float32)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 105. GiB for an array with shape (276000, 450, 226) and data type float32
The box size is 450 (pxl=0.75A), the training box is 128, and I have about 275k particles.
I also tried to change the computation parameter Override scheduler, but it didn’t help - I got the same error.
How much memory does the system have? 3D Flex still (last I saw) requires the entire particle stack and CTF parameters loaded into memory - this requires double the storage space of the stack, in 32-bit. So if you have 16-bit output and your stack in 32GB, before any analysis is carried out, 3D Flex will be trying to load 128GB of data into volatile memory.
A quick check would be downsample the stack to, say, 150 pixels and see if the error still occurs with the downsampled particles.
Thanks both
The odd thing is, that I ran the same job in another workspace with similar data set (same parameters, different sample) and it runs smoothly.
Serena, how do you turn on the parameter Load all particles in RAM from the job? Or is there somewhere else?
Thanks, Serena and others,
We updated to version 4.5, but the problem persists -
“Job is unresponsive - no heartbeat received in 60 seconds.”
It happens regardless if the followings are on or off- “load all particles to RAM”, cache particle images on SSD, overide scheduler.
Any combination of parameters to try that might be helpful, or another parameter I haven’t specified?
Cheers…
