Hi cryoSPARC team,
To make SSD caching in CS possible, we have just installed NVMe drives in all our GPU nodes.
Holy moly! that gives some significant speedup in processing, especially if one has large particle stacks.
Thanks for that option!
But this brings back an old tropic that I have tried raise some times before.
For most cryoSPARCs jobs, their ability to pull data off our Cephfs disk cluster, is rather poor performing.
It would be nice if all jobs in CS, were given the opportunity to use “parallel disc I/O” like the option Relion has. I believe this would give another significant speedup of cryoSPARC, as bandwitdh of Relion easily is above 15 Gbps from our Cephfs, while cryosparc (except for import) only is around 1.5 Gbps.
I hope you will consider this in future version of CS?
P.S. Although cryoSPARC has move to Python3, it still looks like shutil.copyfile only has a buffersize (length) of 16 KB, is this true? If so, I think an increase to e.g. 16MB would make a huge impact on bandwidth on network drives?