3DFlex Recontruction error

3D Flexible Refinement
1

Hi, when I launch 3dFlex Reconstruction I have this error in the reading step of the *_ctf.mrc files. The error is as follows:
“Job is unresponsive - no heartbeat received in 60 seconds.” Reading some reports about it, I have modified the CRYOSPARC_HEARTBEAT_SECONDS variable in the master config and put a big number to give it more time. However, every time I run the job, it keeps failing for the same reason. Is there any way to solve this?

2

Please can you provide the following information

  • the type of the CryoSPARC worker node:
    • CryoSPARC-managed, separate worker or
    • combined master/worker
    • cluster
  • is the CryoSPARC instance up-to-date: v4.2.1 + 230621 patch for v4.2.1
  • is the job alive on the worker and sending_ heartbeats. Heartbeat timestamps can be found in the job log (Metadata|Log).
  • is RAM exhausted (check output of free -g) on either master or worker host when you observe “Job is unresponsive”?
1 Like
3

Free memory is always the first thing to check for 3D flex failing at reading the CTFs.

If your particle stack is 200GB, on a system with 512GB you’re OK (just) because it also loads the CTFs in a separate 200GB stack.

If your particles are 260GB+ on a 512GB system, you’ll have problems.

4

@cfonseca This Happened to me as well. Just try experimenting with a smaller set of particles. It worked for me.

5

wondering how to go about using all the particles after splitting into sets for the 3d flex reconstruction?

6

Thanks @Shashank. Yes, with a smaller dataset, the 3DFlex Reconstruction worked!!! Maybe this is a bug that should be fixed by the CS guys.

7

I have a standalong configuration. But I have been able to prove that with a small particle set it works. With half a million particles it fails, but with 100k it works fine. Maybe this is something to look at.

8

@cfonseca No problem! As much as I remember the 3D flex does not copy/read particles on the SSD yet (scratch) and it is using RAM for it now, that’s why we need a lot of RAM to run 3D flex with 1M particle (based on pixel size and box size).

9

In our testing, the system RAM requirements for 3dflex adhere loosely the following templates. Note the dependence on particle number and image size. These figures were discerned from our scheduler’s MaxRSS figures. YMMV.

flex_train: (<size_of_training_data> * 1.6) +20GB
flex_highres: (<size_of_full_res_data> * 1.6)

Where dataset size = <num_particles> * <box_size>^2 * 4 / 1024^3 (GB)

Cheers,
Yang

1 Like
10

As others have suggested, it may indeed be the case that your CryoSPARC worker has sufficient system RAM for the smaller particle set, but not enough for including all particles in the reconstruction. This is a known limitation of 3DFlex reconstruction.