Downsample particles hangs after 40th batch

Troubleshooting
1

Hi,
I have a large number of particles I am attempting to split in order to use the local SSD cache for processing. I’ve split my particles into stacks of 1M and would like to use the downsample particles job to generate new particle stacks with smaller total file sizes.

When I try running downsample particles, it consistently hangs indefinitely after processing the 40th stack, regardless of which particle subset it is given or the number of threads indicated in the settings. Has anyone encountered this problem?

Thanks,
Alan

2

May we ask for some additional information:

  1. From the cryoSPARC perspective, are these just “regular” Downsample Particles jobs with 1 million particles each?
  2. How were the particles divided into subsets and how was the actual number of particles in each subset confirmed?
  3. What is the box size before the downsample operation?
  4. How much DRAM does the compute node have?
  5. How many threads were requested per job?
  6. Please paste the text from the bottom of the Overview tab, starting high enough to capture some lines that display the memory usage (in brackets at the beginning of some lines).
3

Hi there,
I was facing the same issue today. My job repeatedly got stuck after a specific number of batches (in my case 64/88). But then, when I for once waited long enough (maybe another 30-60 min for some 180.000 particles in a 560 px box originally), it did finish (without writing out any logs for the remaining batches). So you just need to be patient, I guess.
Cheers,
mce

4

Thanks @mce For posting this observation.

If you were to run a clone for this job, it would be interesting to watch the job directory and see if files are being added to the directory or downsample subdirectory during the 30-60 min period of apparent inactivity.
Another interesting test would another clone of the job with blank Crop to box size and Fourier crop to box size parameters.
In case you run these tests please can you update us on your findings?

5

Regarding your first suggestion: yes, files are constantly being saved to the downsample directory during that period of apparent inactivity. I haven’t done the other test yet.

6

Did you also check the job log (Metadata|Log) for error messages that could indicate a problem connecting to CryoSPARC master services, or other problems?