Hi Wolfram,
thank you for confirming that this is not a bug in CryoSPARC, but rather a local configuration issue! This brought me on the right track.
We never had an issue with sshfs before, until the update to CryoSPARC 4.6.0.
Apparently, the new version opens all particle images at once while running the helix_refine job. Since there were 1452 particle images, it required 1452 file handles. However, the maximum allowed number of file handles per user on Ubuntu is just 1024 per default. Since the CryoSPARC project folder was hosted on an Ubuntu box, it ran out of file handles. This caused an I/O error on sshfs on the worker node.
However, if CryoSPARC transferred the particle images on the local SSD cache instead, it only opened a limited number of particle images on the Ubuntu box at one time to do the transfer, so sufficient file handles were available. The maximum allowed number of file handles per user on the worker node was set to 12288. Once all particle images had been transferred to the SSD, the worker node could open all of them simultaneously as there were sufficient file handles, and the job could complete.
I now changed the maximum allowed number of file handles for the CryoSPARC user to 65536 on the master node, all worker nodes and all data servers. This resolved the issue.
1 Like