Hi,
I often find that streaming refinement stalls at “computing FSCs”, requiring a kill and restart to get past it. There doesn’t seem to be much pattern to it, so I’m not sure how to isolate the bug. But not infrequently, it will be sitting on one iteration at computing FSCs in the evening, and then it will still be stuck at the same place in the morning. Is there any place I can get more detailed logs to help isolate the issue?
Cheers
Oli
Hi @olibclarke this is unforutnately due to the large memory usage of FSC computation in the current implementation… for noise substitution etc many volumes are created and stored simultaneously. For larger box sizes this means a short spike in RAM usage that often makes a system start to swap if other things are going on, and then FFTs and etc all stall and the job can hang for hours.
We have a new GPU implementation of all this but it’s not yet deployed- will be coming soon and hopefully will make FSCs much faster as well as stop this from happening.
Hmmm - maybe - but this was with a 320px box size, and I have 384GB RAM…
hmm, that does seem like plenty. I’m not sure what it could be then.
We will deploy the optimized/GPU version of FSC and see if this kind of issue still happens.
