Hey, here’s an update on the problem.
I have now tested 3D Flex on a different dataset.
Training was perfomed with 277,000 particles and box size was as follow
[2022-12-26 23:53:12.79]
[CPU: 886.1 MB]
Input particles have box size 512
[2022-12-26 23:53:12.80]
[CPU: 886.1 MB]
Input particles have pixel size 1.0580 A
[2022-12-26 23:53:12.82]
[CPU: 886.1 MB]
Cropping input particles to box size 448 for reconstruction
[2022-12-26 23:53:12.82]
[CPU: 886.1 MB]
Downsampling cropped particles to box size 112 (pixel size 4.2320 A) for training
[2022-12-26 23:53:12.82]
[CPU: 886.1 MB]
Nyquist resolution at training time will be 8.4640 A
Flex Mesh and Flex Train worked nicely, as well as Flex Generate.
I then ran Flex Reconstruct, and it again failed at It0, this time without the red error message, only this:
[2022-12-27 19:18:56.18]
[CPU: 323.13 GB]
====== High resolution flexible refinement =======
[2022-12-27 19:18:56.18]
[CPU: 323.13 GB]
Max num L-BFGS iterations was set to 20
[2022-12-27 19:18:56.18]
[CPU: 323.13 GB]
Starting L-BFGS.
[2022-12-27 19:18:56.18]
[CPU: 323.13 GB]
Reconstructing half-map A
[2022-12-27 19:18:56.19]
[CPU: 323.13 GB]
Iteration 0 : 138000 / 138500 particles
[2022-12-27 19:34:40.17]
[CPU: 172.0 MB]
====== Job process terminated abnormally.
Again, to make sure, I ran again Flex Data Prep, same parameters, but with 80,000 particles instead of the full 277,000. I ran Flex Reconstruct with the model trained on the 277,000 particles and again got the same problem:
[2022-12-28 0:51:39.23]
[CPU: 97.55 GB]
====== High resolution flexible refinement =======
[2022-12-28 0:51:39.24]
[CPU: 97.55 GB]
Max num L-BFGS iterations was set to 20
[2022-12-28 0:51:39.24]
[CPU: 97.55 GB]
Starting L-BFGS.
[2022-12-28 0:51:39.24]
[CPU: 97.55 GB]
Reconstructing half-map A
[2022-12-28 0:51:39.25]
[CPU: 97.55 GB]
Iteration 0 : 39000 / 40000 particles
[2022-12-28 0:57:47.78]
[CPU: 174.1 MB]
====== Job process terminated abnormally.
What I noticed:
When loading the particles by batch of 1000 at this step:
Reconstructing half-map A
Iteration 0:
It first loads everything once properly (the 138500 particles or 40000 particles), then it restarts again the same process (is it normal? shouldn’t it do half-map B next instead of restarting?) and when it reaches the last 1000 batch of the 2nd loading, it fails.
Let me know if you need anything else from our side to understand what’s going on.
Thanks in advance