Local refinement hanging

Local Refinement
1

A local refinement job of a relatively large number of particles (~3 million) seems to be hanging in iteration 4 for more than two days. Is it possible to use the intermediate results (volume and particles of previous iterations in subsequent jobs?

If I try to drag the output results (iteration 3) by dragging the volume and particles into another job the new job says that it is waiting for the local refinement job to finish Is there a way to start another job with the intermediate results even if the refinement job has not finished or has been killed?

Thks

H,

2

I would set up another test run to verify behavior. run the local refine on a tiny subset of the particles, then kill it after iteration 1, then “mark job as complete”, then try to use that output to see if a new job will accept the input.

IF YES, then yes you should be able to mark as complete and it will have saved the progress so far.

3

I’ve recently seen a local refinement taking a very long time - I’d never let it going for 2 days… - I noticed that restarting the same job with limited search angle to 20° (which is still generous I think) made it finish in reasonable time.
I wonder why would one launch a local refinement with so many particles, though… If you can share, that’d satisfy my curiosity.

4

Can’t speak for @hsosa, but when we have conducted local refinements with millions of particles, it is usually after symmetry expansion

In answer to the original question - you should be able to kill it and mark it as complete, and use the latest particles & volume

1 Like
6

@hsosa Please can you post additional details

  • the CryoSPARC version
  • is particle caching enabled for this job?
  • outputs of these commands on the CryoSPARC worker:
uname -a
free -h
cat /sys/kernel/mm/transparent_hugepage/enabled
nvidia-smi -L
df -Th /path/to/cryosparc/cache # if applicable, substitute actual cache path
7

The option to mark the job as complete seems like a good alternative to use next time I run into this problem. However, I just restarted the job after killing it and this time it ran until completion in about 10h which I think is normal. Regrading the other questions:

  • the CryoSPARC version
  • Cryosparc version 4.6.0
  • is particle caching enabled for this job?
  • Yes particle caching was enabled for this job

Thanks

Hernando

1 Like