Unknown 504 error when dealing with jobs (after reboot)

Install, Configure and Update
1

Hi everyone, I rebooted the master node, and after that started getting “Unable to clear P43 J68: Unknown 504 error” (or unable to kill/schedule for other jobs). I tried restarting the cryosparc with cryosparcm restart – didn’t help.

2
  1. Do these errors also show up in any of the logs inside cryosparc_master/run/?
    If they do, please post them verbatim along with relevant preceding and following lines for context.
  2. Are there any recent errors in your command_core log?
  3. Does the output of
    hostname -f match the value assigned to CRYOSPARC_MASTER_HOSTNAME (inside cryosparc_master/config.sh?
  4. Is there a chance that the ip address of your master host has changed?
  5. Is $CRYOSPARC_MASTER_HOSTNAME resolving accurately on your network?
3

Do these errors also show up in any of the logs inside cryosparc_master/run/?
If they do, please post them verbatim along with relevant preceding and following lines for context.

The GUI doesn’t show any logs, if that’s what you mean. Neither did the job.log files exist.

Are there any recent errors in your command_core log?

Nothing by cryosparcm log command_core | grep -i error

Does the output of hostname -f match the value assigned to CRYOSPARC_MASTER_HOSTNAME (inside cryosparc_master/config.sh?

There is no CRYOSPARC_MASTER_HOSTNAME assigned in the config.sh file.

Is there a chance that the ip address of your master host has changed?

No, it’s the same.

Is $CRYOSPARC_MASTER_HOSTNAME resolving accurately on your network?

There’s no such environmental variable.

4

I was referring to the log files in the
/path/to/cryosparc_master/run directory. You might begin your search for relevant events with

grep -i err /path/to/cryosparc_master/run/*.log

(additional greping likely required to omit false-positives)
and then post relevant hits along with their context.

Interesting. Please can you post the output of

grep -v LICENSE /path/to/cryosparc_master/config.sh

(/path/to/ inside commands above needs to be replaced by actual paths according to your CryoSPARC installation)

5

sorry, I initially checked cryosparc_worker/config.sh, which doesn’t have it set.
What you asked for is:

(base) cryosparcuser@cmm-1:~$ grep -v LICENSE /opt/cryosparc/cryosparc_master/config.sh

export CRYOSPARC_MASTER_HOSTNAME="cmm-1"
export CRYOSPARC_DB_PATH="/data/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_SLACK_WEBHOOK_URL="http://localhost:8000"
export CRYOSPARC_DISABLE_IMPORT_ON_MASTER=true

So, regarding your previous question:

Is $CRYOSPARC_MASTER_HOSTNAME resolving accurately on your network?

It is:

marin@cmm2:~$ nc -zv cmm-1 39000
Connection to cmm-1 39000 port [tcp/*] succeeded!

About run logs with grep -i err /path/to/cryosparc_master/run/*.log – I skimmed through them and didn’t show anything meaningful on either 500 code or time-code when I had these troubles.

6

What is the output when you try this command on cmm-1 itself?

7

Success:

marin@cmm-1:~$ nc -zv cmm-1 39000
Connection to cmm-1 39000 port [tcp/*] succeeded!
8

Would you like to email us the log archive created by
cryosparcm snaplogs
and also let us know the approximate time when you observed