Run time doubled

Troubleshooting
1

Hi CryoSPARC team,

I have a worker workstation that repeatedly develops a performance issue after running jobs continuously for some time, although I have not been able to determine exactly how many jobs it takes to trigger it. Once this happens, the runtime of subsequent jobs (most noticeably heterogeneous refinement and ab initio reconstruction) roughly doubles. A reboot immediately restores normal performance.

The system is running Ubuntu 24.04, and I observed the same behavior previously under Ubuntu 22.04 as well.

Hardware configuration:

2xIntel Xeon 6526Y, 1TB RAM, 4×RTX A6000ada, 2×4TB Samsung 990 Pro SSDs used for cache

Do you have suggestions for how to troubleshoot this issue, or what logs / performance metrics I should monitor when the slowdown occurs?

Thank you

2

is it possible you’ve filled cache and need to manually purge? cache-purge can be taking substantial time in the jobs, and running without cache because it’s full the same

3

I’m not sure I fully understand your question. All of the jobs I described use the same particle stack in the cache.

4

can disregard, I misread the issue (notably that a reboot restores performance)