Job is unresponsive - no heartbeat received in 30 second

Features and Functionality
1

I often get these error messages when my jobs are running. The jobs still complete and are not dead. This causes problems with queueing, as the next job in the queue will launch. This can then crash both jobs as they are launched on the same GPU.

I think this is often caused by the slightly laggy networked file system I am storing my project directories in.

Would it be possible for you to extend the ‘Unresponsive time’ maybe to 120 secs to reduce the regularity of these warnings and make it a bit more amenable to slightly sub-optimal file storage setups?

Thanks a lot.

2

Hi @donaldb,

This is a good idea - we can make it a configurable number in the next version!

Ali

3

Hi - I get often this error too, is there a way to change that number ? Do I remember correctly that in cryosparc v1 one could change it ?
Many thanks !

4

Hi @marino-j,

Yes, there is a way to change this number. You have to set the environment variable CRYOSPARC_HEARTBEAT_SECONDS in the cryosparc2_master/config.sh file.
For example, append:
export CRYOSPARC_HEARTBEAT_SECONDS=180 to cryosparc2_master/config.sh, and restart cryoSPARC: cryosparcm restart

5

done ! Thanks a lot for your help.

6

@apunjani, I am getting this error during Heterogeneous Refinement using v2.15.0 and JOB DETAILS already showed FAILED. However, my job somehow continued automatically after some time (not sure after how many seconds). Is it possible to default this parameter to 180 seconds (export CRYOSPARC_HEARTBEAT_SECONDS=180) in the next version? I don’t want to bother our cluster admin to make the change at this moment. Thanks so much!