No respond from cryosparc

Troubleshooting
#1

Hi all,

my cryosparc instance just hung completely.
Stoping/Restarting doesn’t help, I do not even see the login page in my browser.
I was running a live process, here the last entry from command_rtp.log

AssertionError: {‘code’: 500, ‘data’: None, ‘message’: ‘OtherError: Circular reference in job tree!! Cannot layout’, ‘name’: ‘OtherError’}
2020-12-26 21:24:29,033 RTP.LIVE_SESSION ERROR Failed computing statistics for P123, S1: cannot encode object: 0, of type: <class ‘numpy.int64’>

Any ideas how to fix?

Best,
Tarek

#2

Anyone?
Cryosparc is dead right now.

#3

Upon some investigation I suspect the circular reference during a live process to caus the problem.
It is S1 in P123.
Any advise how to remove all traces of that corrupted project from my database?

#4

Hi @tarek, to help me further troubleshoot this, please send me the output of these two terminal commands:

cryosparcm log command_core | tail -n 1000
cryosparcm log command_rtp | tail -n 1000
#5

Hi @nfrasser,

thanks for reply, attached the two outputs.

command_rtp
command_core

Best,
Tarek

#6

Hi @tarek, looks like those logs were a bit stale, please try the following:

  1. Restart cryoSPARC with command:

    cryosparm restart

  2. Run this command

    cryosparcm status

    Send me the output, excluding the line that starts with export CRYOSPARC_LICENSE_ID=

  3. Try loading the cryoSPARC web interface. Wait until it hangs or stops working

  4. Run the two commands I sent you in my first message again and send me the output

#7

CryoSPARC System master node installed at
/srv/public/cryosparc2/cryosparc2_master
Current cryoSPARC version: v3.0.1

cryosparcm process status:

app STOPPED Not started
app_dev STOPPED Not started
command_core RUNNING pid 19597, uptime 0:02:00
command_rtp RUNNING pid 19657, uptime 0:01:57
command_vis RUNNING pid 19636, uptime 0:01:58
database RUNNING pid 19521, uptime 0:02:03
liveapp STOPPED Not started
liveapp_dev STOPPED Not started
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started

global config variables:

export CRYOSPARC_MASTER_HOSTNAME=“master”
export CRYOSPARC_DB_PATH="/srv/public/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

no difference between both outputs.

#8

It looks like the webapp component (the main cryoSPARC interface) is not running. Can you start it with this command?

cryosparcm start webapp

You can also start the Live interface like this:

cryosparcm start app

Then try loading the web interface again.

#9

If you have trouble with those commands, try this:

#10

Hi @nfrasser.
That indeed brought me cryosparc back. I will re-install the latest version and see if runs smoothly.
Thanks for your help!

3 Likes