Command_rtp and command_vis are always restarting and jobs can't submit to cluster

Troubleshooting
1

After a shutdown of our cluster, the CryoSPARC instance can’t submit jobs to cluster…

cryosparcm status shows that command_rtp and command_vis are always restarting.

Any cryosparcm commands failed with the following error:
cryosparc_tools.cryosparc.errors.CommandError: *** (http://login:39002, code 403) Encounted error from JSONRPC function "system.describe" with params ()

2

BTW, force update did not work, and manually submitting the job with command sbatch /XXX/XXX/XXX/queue_sub_script.sh is available temporarily…

3

… suggests a problem that extends beyond job submission. Is the UI accessible and functional (except job submission)?
Please can you post the outputs of these commands on the CryoSPARC master host:

uname -a 
host login
curl login:39002
curl 127.0.0.1:39002
grep HOST /path/to/cryosparc_master/config.sh # substitute actual path
ps -eo user:16,pid,ppid,start,command | grep -e cryosparc_ -e mongo
ls -l /tmp/mongo*.sock /tmp/cryosparc*.sock
cryosparcm call env | grep -i proxy
cryosparcm log supervisord | tail -n 40
cryosparcm log command_core | tail -n 40
4

Sorry, this was a proxy-related issue from our cluster…The Internet access was achieved by a proxy from another node, and the environment variable $no_proxy only contains 127.0.0.1 and localhost. After adding the hostname login into $no_proxy and run cryosparcm restart, the problem solved…

Thank you again for your reply :slight_smile:

1 Like