We has encountered this http timeout error message for the past three weeks. Now it almost fully blocked our cryosparc usage. Here are the error message:
We tried to add the “export CRYOSPARC_CLIENT_TIMEOUT=1800” to the config.sh under the “cryosparc_master” folder. However, it looks like this environment setting was not used after restarting cryosparc with “cryosparc restart”. We still have the error message saying “timeout=300”.
The ports 39000-39010 have been opened for both master and worker node. Trying “curl master node:39002 from the worker nodes” can return “hello world” without issue.
Now if we run 2D or 3D jobs without an SSD, the chance of successful running will increase. But with SSD, jobs usually fail before transferring data and throw out the timeout error. It’s worth noting that the particle dataset was not large at all (only 20G).
The cryosparc version we use is 3.3.2. We have 1 master node and two worker nodes.
Any suggestions would be much appreciated!