Cryosparc does not function after an installation

yurotakagi · May 24, 2022, 1:49am

Hi
I did a clean installation of the latest version of cryosparc on Centos 7.9 with CUDA 11.2. The installation went good. When started, everything looked very good. However, once I tried to start a actual job. I got the following error message regardless of which type of job I tried to initiate. I got this error message for a simple job of importing 3D volume. I tried to follow some of trouble shooting methods but none worked. Does anyone have any idea what is wrong? It looks like it is a sever error

Thanks for your help. Best. Yuro

ServerError: Traceback (most recent call last): File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 146, in wrapper res = func(*args, **kwargs) File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1857, in scheduler_run scheduler_run_core(do_run) File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1949, in scheduler_run_core ‘lane’ : lanes[0][‘name’], IndexError: list index out of range The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 146, in wrapper res = func(*args, **kwargs) File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 5112, in enqueue_job scheduler_run() File “/home/takagilab/diskarray/software/cryoem/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 153, in wrapper raise ServerError(s.getvalue(), code=400) from e flask_jsonrpc.exceptions.ServerError

dgirodat · May 24, 2022, 6:20pm

Hi,

I am getting the same error, I just did a fresh install of cryosparc today and I am getting this error when importing movies.

Thank you,
Dylan

ServerError: Traceback (most recent call last): File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 150, in wrapper res = func(*args, **kwargs) File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 1861, in scheduler_run scheduler_run_core(do_run) File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 1953, in scheduler_run_core ‘lane’ : lanes[0][‘name’], IndexError: list index out of range The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 150, in wrapper res = func(*args, **kwargs) File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 5121, in enqueue_job scheduler_run() File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 157, in wrapper raise ServerError(s.getvalue(), code=400) from e flask_jsonrpc.exceptions.ServerError

dgirodat · May 24, 2022, 7:49pm

I would like to add that the command

cryosparcm log command_core
gives the output of:

2022-05-24 13:42:04,737 COMMAND.CORE 2022-05-24 13:42:04,738 COMMAND.CORE 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:04,791 COMMAND.BG_WORKER 2022-05-24 13:42:05,795 COMMAND.SCHEDULER 2022-05-24 13:42:05,795 COMMAND.SCHEDULER 2022-05-24 13:42:05,796 COMMAND.SCHEDULER 2022-05-24 13:42:05,796 COMMAND.SCHEDULER 2022-05-24 13:42:05,796 COMMAND.SCHEDULER 2022-05-24 13:42:05,851 COMMAND.CORE 2022-05-24 13:42:45,961 COMMAND.CORE 2022-05-24 13:42:45,962 COMMAND.CORE 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:46,015 COMMAND.BG_WORKER 2022-05-24 13:42:47,019 COMMAND.SCHEDULER 2022-05-24 13:42:47,019 COMMAND.SCHEDULER 2022-05-24 13:42:47,020 COMMAND.SCHEDULER 2022-05-24 13:42:47,020 COMMAND.SCHEDULER 2022-05-24 13:42:47,020 COMMAND.SCHEDULER 2022-05-24 13:42:47,074 COMMAND.CORE scheduler_run_core WARNING | Error connecting to cryoSPARC license server. Checking local license file.
scheduler_run_core INFO | Not a commercial instance - heartbeat set to 12 hours.
background_worker ERROR | Job Scheduler failed
background_worker ERROR | Traceback (most recent call last):
background_worker ERROR | File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 233, in background_worker
background_worker ERROR | scheduler_run_core() # sets last run time
background_worker ERROR | File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 1953, in scheduler_run_core
background_worker ERROR | ‘lane’ : lanes[0][‘name’],
background_worker ERROR | IndexError: list index out of range
scheduler_run_core INFO | Running…
scheduler_run_core INFO | Jobs Queued: [(‘P1’, ‘J2’)]
scheduler_run_core INFO | Licenses currently active : 0
scheduler_run_core INFO | Now trying to schedule J2
scheduler_run_core INFO | Scheduling directly onto master node ch-fe2.local
scheduler_run_core WARNING | Insecure mode - no SSL for external requests
scheduler_run_core WARNING | Error connecting to cryoSPARC license server. Checking local license file.
scheduler_run_core INFO | Not a commercial instance - heartbeat set to 12 hours.
background_worker ERROR | Job Scheduler failed
background_worker ERROR | Traceback (most recent call last):
background_worker ERROR | File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 233, in background_worker
background_worker ERROR | scheduler_run_core() # sets last run time
background_worker ERROR | File “/lustre/scratch5/.mdt1/dgirodat/cryosparc_master/cryosparc_command/command_core/init.py”, line 1953, in scheduler_run_core
background_worker ERROR | ‘lane’ : lanes[0][‘name’],
background_worker ERROR | IndexError: list index out of range
scheduler_run_core INFO | Running…
scheduler_run_core INFO | Jobs Queued: [(‘P1’, ‘J2’)]
scheduler_run_core INFO | Licenses currently active : 0
scheduler_run_core INFO | Now trying to schedule J2
scheduler_run_core INFO | Scheduling directly onto master node ch-fe2.local
scheduler_run_core WARNING | Insecure mode - no SSL for external requests

Thanks,
Dylan

ccgauvin94 · May 25, 2022, 2:03am

What is the output of cryosparcm cli "get_scheduler_lanes()" ?

wtempel · May 25, 2022, 1:21pm

@yurotakagi @dgirodat The worker component of your respective cryoSPARC instances may not have been properly configured.
Did you install --standalone or run cryosparcw connect [...] as required for your desired cryoSPARC mode? Did the commands complete without error?

yurotakagi · May 25, 2022, 1:39pm

Hi wtempel
I was installing “standalone” version. I tried to run cryosparcw connect but got the error message. I erased all - so I don’t have the error message to show. I plan to re-install again to see how things go - if it does not work, I will check output of cryosparcm cli "get_scheduler_lanes()"suggested by ccgauvin94.

Best

yurotakagi · May 25, 2022, 2:17pm

Hi

It looks like I was able to fix the problem and now cryosparc is working. What I did is as follows:
When I looked the error message carefully, I noticed the messaging saying something like “for ssdpath, “scratch/cryosparc_cache” does not exist”. I thought that scratch/cryosparc_cache directory would be automatically generated. Previously, I generated the directory after installation but did not work. So, this time, I manually generated the directory and ran the standalone installation script. Then somehow, it worked. I have no idea why. But I just might as well report my experience here.
Best
Yuro

dgirodat · May 25, 2022, 4:25pm

Hi

I could not get the scratch/cryosparc_cache working, I did not find this in my error for ssdpath.
I have been trying to install using the standalone option. My installation looks like:
./install.sh --standalone --license XX --worker_path /lustre/scratch5/dgirodat/cryosparc_worker --cudapath /usr/projects/hpcsoft/cos2/chicoma/cuda/11.6 --ssdpath /lustre/scratch5/dgirodat/cryosparc_ssd --initial_email XX@XX.XX --initial_password XX --initial_username “dgirodat” --initial_firstname “Dylan” --initial_lastname “Girodat” --port 39000

The output of cryosparcm cli “get_sheduler_lanes()” is:

Thanks,
Dylan

wtempel · May 25, 2022, 8:28pm

@dgirodat Am I understanding correctly that you suspect the --ssdpath parameter as the cause for the failed worker configuration?
In this case, please can you rerun ./install.sh, replacing the
--ssdpath /lustre/scratch5/dgirodat/cryosparc_ssd
option with --nossd.
What output does
cryosparcm cli "get_sheduler_lanes()"
show after trying that?

dgirodat · May 25, 2022, 8:34pm

Hi wtempel,

The --nossd option did work for me. I now get:
{‘desc’: ‘’, ‘name’: ‘default’, ‘title’: ‘Lane default (node)’, ‘type’: ‘node’}
for cryosparcm cli “get_scheduler_lanes()”

Thank you,
Dylan

wtempel · August 25, 2022, 7:34pm

A recent patch (Patch 220824 is available for cryoSPARC v3.3.2) provides a clearer notification when workers have not been properly connected to the cryoSPARC instance.