Hi everyone,
We are seeing a severe performance issue on our VBC CBE cluster and are trying to determine whether this is a CryoSPARC v4.7.1+251124 issue or an infrastructure problem.
Our local IT team is already investigating, but I wanted to ask whether anyone has experienced similar behavior.
Observation:
We compared two 3D Classification jobs with essentially identical parameters.
- Cluster: ~28k particles → runtime ~6 hours
- Workstation: ~55k particles → runtime ~7 minutes
The workstation processes roughly twice as many particles but finishes almost 50× faster.
This behavior is not limited to 3D Classification. We see the same issue consistently for multiple CryoSPARC job types, including Homogeneous Refinement, Local Refinement, and 2D Classification.
Initially we suspected SSD cache population, but even after the cache is populated, the actual computation remains extremely slow.
For example:
Consensus reconstruction takes 1438 s on the cluster and 16 s on the workstation.
Optimal per-particle scales takes 3019 s on the cluster and 30 s on the workstation.
O-EM iterations take 150-230 s on the cluster and 2 s on the workstation.
Additional observation:
The slowdown is not limited to the initial cache copy.
Interestingly, jobs actually run faster when the CryoSPARC SSD cache is disabled.
This makes us wonder whether there could be a bottleneck when CryoSPARC accesses particle images from the local SSD cache during processing, or whether there are known issues with the SSD cache implementation on certain cluster configurations.
Question:
Has anyone observed similar behavior?
Are there known issues where enabling the SSD cache causes severe slowdowns on a cluster, even though the cache itself populates successfully?
Any suggestions for additional diagnostics or things to check would be appreciated.
Thanks!