Web app doesn't load, status is normal

shockacone · October 24, 2019, 3:17pm

Hi,

This behavior seems random. I’ve tried checking some logs and haven’t found anything out of place. Basically the issue is cryoSPARC v2 is up and running as you can see below:

But Webapp doesn’t load and shows the error above.

When I restart cryoSPARC v2 the behavior is corrected. This has been an ongoing issue for sometime now, we’re using version 2.11.0.

Any advice would be appreciated, thanks.

sdawood · October 24, 2019, 3:31pm

Hi @shockacone,

Could you paste the contents of cryosparcm log webapp please?

Thanks,
Suhail

shockacone · October 24, 2019, 3:53pm

Blockquote
cryosparc2@semcworkshop:~/cryosparc2_master/bin$ cryosparcm log webapp
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “index_abs_path”,
“params”: [
“/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004”,
“5afb254c4dba25102c4b1a85”
],
“id”: “rc3qSoz6nDiGN5ZYd”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “job_set_param”,
“params”: [
“P144”,
“J71”,
“particle_meta_path”,
“/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004”
],
“id”: “B2HBEuNHAeFFfr9Da”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_state_var”,
“params”: {
“user_id”: “5afb254c4dba25102c4b1a85”,
“key”: “browserPath”,
“value”: “/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004”
},
“id”: “wsWKsF9yn2RZYx9S8”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “index_abs_path”,
“params”: [
“/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004”,
“5afb254c4dba25102c4b1a85”
],
“id”: “qhWWdJMSc4K9zAJuS”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “index_abs_path”,
“params”: [
“/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004/particles.star”,
“5afb254c4dba25102c4b1a85”
],
“id”: “JduNwdxEqukPhpWic”
},
“json”: true
}
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
[PUB] job.events.checkpoints: { project_uid: ‘P137’, job_uid: ‘J578’, type: ‘checkpoint’ }
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_job”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P137”,
“W13”,
“J578”
],
“id”: “6zSbJAzaPSjf7zP5d”
},
“json”: true
}
[PUB] job.events: { project_uid: ‘P137’, job_uid: ‘J578’ } 100 0
[PUB] events.countAfterCheckpoint
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “job_set_param”,
“params”: [
“P144”,
“J71”,
“particle_meta_path”,
“/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004/particles.star”
],
“id”: “MWousDwN9fzMFDesp”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_state_var”,
“params”: {
“user_id”: “5afb254c4dba25102c4b1a85”,
“key”: “browserPath”,
“value”: “/gpfs/appion/chernandez/THG/18nov14c/relion/Extract/job004/particles.star”
},
“id”: “HdDGPxYEpvB8YpKAu”
},
“json”: true
}
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “enqueue_job”,
“params”: {
“project_uid”: “P144”,
“job_uid”: “J71”,
“lane”: “default”
},
“id”: “woyRzDPeipqqkNh56”
},
“json”: true
}
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
[PUB] job.events.checkpoints: { project_uid: ‘P144’, job_uid: ‘J71’, type: ‘checkpoint’ }
{
“method”: “POST”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_job”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P144”,
“W3”,
“J71”
],
“id”: “2dQJTSqBsNanskANA”
},
“json”: true
}
[PUB] job.events: { project_uid: ‘P144’, job_uid: ‘J71’ } 100 0
[PUB] events.countAfterCheckpoint
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false

shockacone · October 29, 2019, 4:10pm

Hi, any update on this? It’s happened again and I’m not sure how to interpret the log

Blockquote
cryosparc2@semcworkshop:~$ cryosparcm log webapp
}
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_workspace”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P137”,
“W13”
],
“id”: “LfZc7xPGtyCqmzee3”
},
“json”: true
}
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
[PUB] job.events.checkpoints: { project_uid: ‘P137’, job_uid: ‘J600’, type: ‘checkpoint’ }
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_job”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P137”,
“W13”,
“J600”
],
“id”: “vsiPTNQeH4gbGmxpF”
},
“json”: true
}
[PUB] job.events: { project_uid: ‘P137’, job_uid: ‘J600’ } 100 0
[PUB] events.countAfterCheckpoint
[PUB] job.events: { project_uid: ‘P137’,
job_uid: ‘J600’,
created_at: { ‘$gt’: 2019-10-26T03:45:21.349Z } } 100 0
[PUB] events.countAfterCheckpoint
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: null
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “get_result_download_abs_path”,
“params”: {
“project_uid”: “P137”,
“result_spec”: “J600.volume.map_sharp”
},
“id”: “yyTrr9kkkx5my8TGh”
},
“json”: true
}
download result /gpfs/appion/jgorman/19mar08b/relion/P137/J600/cryosparc_P137_J600_009_volume_map_sharp.mrc
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
[PUB] job.events.checkpoints: { project_uid: ‘P137’, job_uid: ‘J598’, type: ‘checkpoint’ }
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_job”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P137”,
“W13”,
“J598”
],
“id”: “XGf8WCMitTbu5mpgv”
},
“json”: true
}
[PUB] job.events: { project_uid: ‘P137’, job_uid: ‘J598’ } 100 0
[PUB] events.countAfterCheckpoint
[PUB] job.events: { project_uid: ‘P137’,
job_uid: ‘J598’,
created_at: { ‘$gt’: 2019-10-26T02:08:33.669Z } } 100 0
[PUB] events.countAfterCheckpoint
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: null
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “get_result_download_abs_path”,
“params”: {
“project_uid”: “P137”,
“result_spec”: “J598.volume.map_sharp”
},
“id”: “j4QSq8Bh2JhkApRNo”
},
“json”: true
}
download result /gpfs/appion/jgorman/19mar08b/relion/P137/J598/cryosparc_P137_J598_009_volume_map_sharp_local.mrc
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
[PUB] job.events.checkpoints: { project_uid: ‘P137’, job_uid: ‘J601’, type: ‘checkpoint’ }
{
“method”: “POST”,
“uri”: “://semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: “5afb254c4dba25102c4b1a85”
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “set_user_viewed_job”,
“params”: [
“5afb254c4dba25102c4b1a85”,
“P137”,
“W13”,
“J601”
],
“id”: “3EfB8Ek5b7rywtqwY”
},
“json”: true
}
[PUB] job.events: { project_uid: ‘P137’, job_uid: ‘J601’ } 100 0
[PUB] events.countAfterCheckpoint
[PUB] job.events: { project_uid: ‘P137’,
job_uid: ‘J601’,
created_at: { ‘$gt’: 2019-10-26T08:03:33.538Z } } 100 0
[PUB] events.countAfterCheckpoint
{
“method”: “POST”,
“uri”: “semcworkshop:39002/api”,
“headers”: {
“CRYOSPARC-USER”: null
},
“body”: {
“jsonrpc”: “2.0”,
“method”: “get_result_download_abs_path”,
“params”: {
“project_uid”: “P137”,
“result_spec”: “J601.volume.map_sharp”
},
“id”: “P8TWgWECzNfv7soHp”
},
“json”: true
}
download result /gpfs/appion/jgorman/19mar08b/relion/P137/J601/cryosparc_P137_J601_007_volume_map_sharp.mrc
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false
==== project query user 5afb254c4dba25102c4b1a85 test false

sdawood · November 1, 2019, 4:48pm

Hi @shockacone,

Sorry for the delay. Are you sure the machine you’re trying to connect via the browser has port 39000 forwarded?

- Suhail

shockacone · November 1, 2019, 5:51pm

Hi Suhail,

Thanks for the response. Are you asking whether we are including the port in the URL when trying to connect? We connect by entering “cryoSPARC_host:39000” in our browsers. Other than doing that we haven’t had to do any port forwarding on the client machines (I’m assuming your question is about the clients which are trying to connect to the CryoSPARC v2 host/server).

The staff connects from multiple machines at once MAC/Windows/Linux. Each staff member simply enters the URL along with the port number and is able to access CryoSPARC v2 interface. This interruption in service seems to happen randomly and once we restart CryoSPARC v2 the behavior is normal again.

Thank you,
Shaker

sdawood · November 15, 2019, 8:49pm

Hi Shaker,

Sorry for the delay - are you still experiencing issues connecting to the web app or does restarting cryoSPARC solve it?

- Suhail

shockacone · December 2, 2019, 4:06pm

Hi Suhail,

Unfortunately we’re still experiencing this issue. Restarting does fix it most of the time but when we have ~10 or so users using the same installation it’s a big delay when jobs are stopped randomly and we’re not always able to respond immediately (after-hours, overnight). Please let us know if there is a better fix to this issue.

This has happened again today and upon restarting CryoSPARC V2 I got the following message:

cryosparc2@semcworkshop:~$ cryosparcm stop
CryoSPARC is running.
Stopping cryosparc.
command_proxy: stopped
webapp: stopped
command_core: stopped
database: stopped
Shut down
cryosparc2@semcworkshop:~$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
  cryosparc command core startup complete.
command_vis: ERROR (spawn error)
command_proxy: started
webapp: started
-----------------------------------------------------

CryoSPARC master started.
 From this machine, access cryoSPARC at
    http://localhost:39000

 From other machines on the network, access cryoSPARC at
    http://semcworkshop:39000

Startup can take several minutes. Point your browser to the address
and refresh until you see the cryoSPARC web interface.

cryosparc2@semcworkshop:~$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.12.2
----------------------------------------------------------------------------

cryosparcm process status:

app                              STOPPED   Not started
app_dev                          STOPPED   Not started
command_core                     RUNNING   pid 5222, uptime 0:04:13
command_proxy                    RUNNING   pid 5312, uptime 0:04:10
command_rtp                      STOPPED   Not started
command_vis                      FATAL     Exited too quickly (process log may have details)
database                         RUNNING   pid 5145, uptime 0:04:15
watchdog_dev                     STOPPED   Not started
webapp                           RUNNING   pid 5320, uptime 0:04:09
webapp_dev                       STOPPED   Not started

global config variables:

export CRYOSPARC_LICENSE_ID="xxxxxxx"
export CRYOSPARC_MASTER_HOSTNAME="semcworkshop"
export CRYOSPARC_DB_PATH="/home/cryosparc2/database/"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

sdawood · December 2, 2019, 6:45pm

Hi @shockacone,

Can you report the output of:

cryosparcm log command_vis

Thanks,
Suhail

shockacone · December 2, 2019, 6:57pm

cryosparc2@semcworkshop:~$ cryosparcm log command_vis
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/skimage/__init__.py", line 158, in <module>
    from .util.dtype import *
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/skimage/util/__init__.py", line 5, in <module>
    from .apply_parallel import apply_parallel
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/skimage/util/apply_parallel.py", line 8, in <module>
    import dask.array as da
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/dask/array/__init__.py", line 8, in <module>
    from .routines import (take, choose, argwhere, where, coarsen, insert,
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/dask/array/routines.py", line 252, in <module>
    @wraps(np.matmul)
  File "/home/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'numpy.ufunc' object has no attribute '__module__'

Thanks Suhail,
Shaker

stephan · December 12, 2019, 7:18pm

@shockacone,

Can you try restarting the visualization server? cryosparcm stop command_vis, cryosparcm start command_vis

shockacone · February 6, 2020, 6:28pm

This issue stopped happening when we updated to a newer version.