Web app not starting after v3.3 update

Venkata · December 1, 2021, 6:13pm

Hi all,
I am excited with the new features of the cryosparc 3.3. I updated on my standalone machine. It showed update was successful. Database is running. rest all including the web app stopped.
CryoSPARC process status:

app STOPPED Not started
app_dev STOPPED Not started
command_core RUNNING pid 21915, uptime 0:00:10
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 21792, uptime 0:00:13
liveapp STOPPED Not started
liveapp_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started

Can someone help me running it.

Thank you

stephan · December 1, 2021, 6:20pm

Hey @Venkata,

Can you run cryosparcm log command_core and send us the output?

Venkata · December 1, 2021, 6:36pm

2021-12-01 13:06:00,153 COMMAND.DATA dump_project INFO | Exporting project P2
2021-12-01 13:06:00,154 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/02172021/cs/P4 does not exist. Skipping export of P2
2021-12-01 13:06:00,154 COMMAND.DATA dump_project INFO | Exporting project P3
2021-12-01 13:06:00,155 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/02172021/REPROCESS/P3 does not exist. Skipping export of P3
2021-12-01 13:06:00,156 COMMAND.DATA dump_project INFO | Exporting project P4
2021-12-01 13:06:00,156 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/CS/P10 does not exist. Skipping export of P4
2021-12-01 13:06:00,157 COMMAND.DATA dump_project INFO | Exporting project P5
2021-12-01 13:06:00,157 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/CS/P10 does not exist. Skipping export of P5
2021-12-01 13:06:00,158 COMMAND.DATA dump_project INFO | Exporting project P6
2021-12-01 13:06:00,158 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/CS/P10 does not exist. Skipping export of P6
2021-12-01 13:06:00,159 COMMAND.DATA dump_project INFO | Exporting project P7
2021-12-01 13:06:00,160 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/CS/P10 does not exist. Skipping export of P7
2021-12-01 13:06:00,160 COMMAND.DATA dump_project INFO | Exporting project P8
2021-12-01 13:06:00,161 COMMAND.DATA dump_project WARNING | /run/media/Rajendra/DATA/Rajendra_R/P8 does not exist. Skipping export of P8
2021-12-01 13:06:00,218 COMMAND.DATA dump_project INFO | Exporting project P9
2021-12-01 13:06:00,219 COMMAND.DATA dump_project INFO | Exported project P9 to /run/media/Rajendra/DATA/Rajendra_R/11292021_HisMuSKDOK7/CS/P9/project.json in 0.00s
2021-12-01 13:06:00,225 COMMAND.DATA dump_project INFO | Exporting project P9
2021-12-01 13:06:00,226 COMMAND.DATA dump_project INFO | Exported project P9 to /run/media/Rajendra/DATA/Rajendra_R/11292021_HisMuSKDOK7/CS/P9/project.json in 0.00s

stephan · December 1, 2021, 6:49pm

Hi @Venkata,

Everything looks fine in those logs, can you do a restart? cryosparcm restart That should fix things.

Venkata · December 1, 2021, 7:05pm

Hi Stphan I tried to restart

ryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Database configuration is OK.
command_core: started
    command_core connection succeeded
[Rajendra@c102237 CS]$ cryosparcm restart
CryoSPARC is running.
Stopping cryoSPARC 
command_core: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Database configuration is OK.
command_core: started
    command_core connection succeeded

Still the same issue.

cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/run/media/Rajendra/DATA/Rajendra_R/Cryosparcuser/cryosparc_master
Current cryoSPARC version: v3.3.0
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_dev                          STOPPED   Not started
command_core                     RUNNING   pid 8048, uptime 0:00:54
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         RUNNING   pid 7934, uptime 0:00:56
liveapp                          STOPPED   Not started
liveapp_dev                      STOPPED   Not started
webapp                           STOPPED   Not started
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_MASTER_HOSTNAME=""
export CRYOSPARC_DB_PATH="/run/media/Rajendra/DATA/Rajendra_R/Cryosparcuser/cryosparc_database"
export CRYOSPARC_BASE_PORT=61000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

stephan · December 1, 2021, 7:15pm

Hi @Venkata,

Can you run cryosparcm log -l ERROR command_core and send me the output?

Venkata · December 1, 2021, 7:17pm

~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
…skipping…

wilnart · December 1, 2021, 7:21pm

Hi Cryosparc team, I’m facing exactly the same issue

stephan · December 1, 2021, 7:23pm

Hey @wilnart,

Can you send us your logs? cryosparcm log command_core. Try to send as many logs as possible. Thanks!

wilnart · December 1, 2021, 7:26pm

@stephan

2021-12-01 12:16:02,403 COMMAND.CORE         run                  INFO     | === STARTED TASKS WORKER ===
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2021-12-01 12:16:04,249 COMMAND.SCHEDULER    get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2021-12-01 12:16:04,250 COMMAND.JOBS         update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2021-12-01 12:16:04,250 COMMAND.DATA         export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
[2021-12-01 12:16:04,251] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 445, in startup
    com.prune_old_notifications(mongo.db)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_compute/jobs/common.py", line 773, in prune_old_notifications
    created_at = list(db['notifications'].find({}).sort('created_at', -1).skip(num_keep).limit(1))[0]['created_at']
IndexError: list index out of range
2021-12-01 12:16:04,286 COMMAND.SCHEDULER    get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2021-12-01 12:16:04,286 COMMAND.JOBS         update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2021-12-01 12:16:04,286 COMMAND.DATA         export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
[2021-12-01 12:16:04,287] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 445, in startup
    com.prune_old_notifications(mongo.db)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_compute/jobs/common.py", line 773, in prune_old_notifications
    created_at = list(db['notifications'].find({}).sort('created_at', -1).skip(num_keep).limit(1))[0]['created_at']
IndexError: list index out of range
2021-12-01 12:16:04,311 COMMAND.SCHEDULER    get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2021-12-01 12:16:04,311 COMMAND.JOBS         update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2021-12-01 12:16:04,311 COMMAND.DATA         export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
[2021-12-01 12:16:04,313] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/wilson/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 445, in startup
    com.prune_old_notifications(mongo.db)
  File "/home/wilson/cryosparc/cryosparc_master/cryosparc_compute/jobs/common.py", line 773, in prune_old_notifications
    created_at = list(db['notifications'].find({}).sort('created_at', -1).skip(num_keep).limit(1))[0]['created_at']
IndexError: list index out of range

stephan · December 1, 2021, 7:43pm

Hi @wilnart,

Thank you very much for sending that, a fix will be out shortly.

stephan · December 1, 2021, 8:05pm

Hi @wilnart, @Venkata,

Thanks again for reporting, and apologies for the inconvenience. A fix for this has been released, you can update to it by running cryosparcm update, which should bring you to v3.3.1.

Juha · December 1, 2021, 8:26pm

v3.3.1 worked for me

Venkata · December 1, 2021, 8:41pm

Thank you so much Stephan for the quick fix. v3.3.1 worked for me.

singularity91 · December 1, 2021, 8:46pm

The v3.3.1 release resolved the issue. Thanks for the quick fix, Stephan.

wilnart · December 1, 2021, 9:50pm

@stephan Thanks for a speedy fix

Hua · January 26, 2022, 8:24pm

@stephan I encountered the same issue when update to v3.3.1. And tried to stop, restart…

CryoSPARC process status:

app RUNNING pid 562, uptime 0:00:13
app_dev STOPPED Not started
command_core RUNNING pid 32091, uptime 0:00:25
command_rtp RUNNING pid 32531, uptime 0:00:21
command_vis RUNNING pid 32130, uptime 0:00:22
database RUNNING pid 31644, uptime 0:00:28
liveapp STOPPED Not started
liveapp_dev STOPPED Not started
webapp STARTING
webapp_dev STOPPED Not started

Hua · January 26, 2022, 8:53pm

Sorry for disturbing. cryosparc v3.3.1 works well. Seems it does not matter even liveapp is stopped. @stephan