My case is a bit extreme, but I thought it is relevant with a big dataset.
I have 2.6 million particles of 720 x 720 pixels after signal subtraction.
I want to crop the particles in real space using Downsample Particles to 400 x 400 pixels. This takes 80s per 1000 particles (on a cluster worker with 16 CPUs requested). That means it will take 57 hours to complete.
Is there any way to make this process faster? I thought it was long in terms of time, as signal subtraction of 2.6 million symmetry expanded particles only takes 36 hours, and I believe signal subtraction requires a lot more computation.
Just a quick update here: The speed is heavy dependent on if other jobs are running, ranging from 80 s to 300 s per 1000 particles. I assume that jobs that use more read/writing affect this process a lot more. Also, using 16 cores (double the default 8 cores), the reported time for a batch is for 8 cores so the time is halved. In the end, it took 31 hrs for me. It took 20hr to reach 50% but the last 50% was done in 11 hrs as I made sure no other job is running at that time.
Hi! We reproduced the conditions you described on our systems and found that the job processes approximately 420 particles per second (about 2.4 seconds per 1,000 particles). Given this, it may be that the processing speed on your system is limited by either i) the available read or write performance of your filesystem where the particle stacks are stored or ii) network bandwidth between the worker where the job runs and the filesystem. Either of these causes would also manifest as performance becoming poorer when other jobs are also running.
