Local refinement memory error with NU-refinement ON and Fulcrum coordinates

Hi @lizellelubbe,

Are you running into errors in the local refinement job, or the non-uniform refinement job? As well, are you running the legacy or new version? With regards to the slowdown, I would ask where in the job are you seeing the most slowdown – for a typical iteration, most of the time would be spent either in the alignment/backprojection process, cross validation, or in FSC computation. Could you check the stream log for the last iteration and show the times for the following steps?

  • “Processed __ images in x seconds.”
  • “Computing FSCs… Done in z seconds”
  • “Local cross validation A done in y seconds” (if using the legacy NU-refine, I don’t think this part is timed)

This might help us see if the slowdown is related to the GPU or CPU processing. There’s also been reports of a similar slowdown in the new local refine/NU-refine job (Local refinement (new) error), and seemingly a system restart helped alleviate the issue, which could help in this case (but it would be helpful to first check the GPU’s status using nvidia-smi to make sure there aren’t other processes using memory).

Best,
Michael