Database error after v4.1.0 update

wtempel · December 17, 2022, 11:37pm

Do you really mean ~/bash_profile or ~/.bash_profile?

wtempel · December 17, 2022, 11:44pm

@Vruiz @rfronzes Please can you post the output of the command
env | grep -i http
run under the Linux account that also runs the CryoSPARC master processes.

Vruiz · December 19, 2022, 5:52pm

Sorry, I meant:

~/.bash_profile

The output of env | grep -i http is:

[victor@c105627 ~]$ env | grep -i http
HTTPS_PROXY=http://proxy.swmed.edu:3128
HTTP_PROXY=http://proxy.swmed.edu:3128

victor is the account that runs cryosparc

wtempel · December 19, 2022, 7:38pm

@Vruiz Please can you confirm that

these settings were already in place when CryoSPARC v3.4.0 was running
CryoSPARC v3.4.0 functioned properly with these proxy variables defined
CryoSPARC v < 4 did not function without these proxy variables defined

Vruiz · December 19, 2022, 8:06pm

these settings were already in place when CryoSPARC v3.4.0 was running
Yes
CryoSPARC v3.4.0 functioned properly with these proxy variables defined
CryoSPARC v < 4 did not function without these proxy variables defined
CS 3.4.0 works with and without these proxy settings

wtempel · December 21, 2022, 9:26pm

If CryoSPARC failed to start with a pymongo error after an update to CryoSPARC v4.1.0 or v4.1.1, you may try the following:

ensure that cryosparc_master software is at either v4.1.0 or v4.1.1
download cryosparc_master_patch.tar.gz
confirm file integrity. Output of
md5sum cryosparc_master_patch.tar.gz
should match: 8df9953103829632756b809f3c637d2f
cryosparcm stop
confirm no more CryoSPARC processes:
ps xww | grep -e cryosparc -e mongod
kill, but not kill -9, any CryoSPARC-related processes that may still exist

# substitute correct paths
mv cryosparc_master_patch.tar.gz /path/to/cryosparc_master
cd /path/to/cryosparc_master
tar -xzf cryosparc_master_patch.tar.gz --overwrite --strip-components=1 --directory ./

if cryosparcm start
still fails, please post error messages on the forum. Otherwise:

update worker(s)

# substitute correct paths
cp /path/to/cryosparc_master/cryosparc_worker.tar.gz /path/to/cryosparc_worker/
cd /path/to/cryosparc_worker/
./bin/cryosparcw update

Vruiz · April 29, 2023, 10:35pm

Hi,

Finally, I could install CS v.4.3.1 by stopping CS, killing all processes, and removing mongod.lock:

rm /home/victor/cryosparc/cryosparc_database/mongod.lock

Then cryosparcm update worked well, however, now I can´t access the web interface, I check the CS status and everything looks normal to me:

[victor@c105627 ~]$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
    configuration complete
database: started
checkdb success
command_core: started
    command_core connection succeeded
    command_core startup successful
command_vis: started
command_rtp: started
    command_rtp connection succeeded
    command_rtp startup successful
app: started
app_api: started
-----------------------------------------------------

CryoSPARC master started. 
 From this machine, access cryoSPARC and cryoSPARC Live at
    http://localhost:39000

 From other machines on the network, access cryoSPARC and cryoSPARC Live at
    http://c105627.dhcp.swmed.org:39000


Startup can take several minutes. Point your browser to the address
and refresh until you see the cryoSPARC web interface.

However I´m unable to connect to the web interface through the LAN network, and it worked well before running CS v3.4. This made me suspect that the port 39000 was closed but it´s not since I didn´t change any settings but somehow I can´t connect to this port:
telnet c105627.dhcp.swmed.org 39000
Trying 129.112.162.193…
telnet: connect to address 129.112.162.193: Connection refused
telnet: Unable to connect to remote host

But I can access to 39001:
telnet c105627.dhcp.swmed.org 39001
Trying 129.112.162.193…
Connected to c105627.dhcp.swmed.org.
Escape character is ‘^]’.

I´m really confused and don´t know how to tackle the problem, I would appreciate your help.

wtempel · April 30, 2023, 10:49pm

Please ensure not to use kill -9 or kill -KILL when shutting down CryoSPARC processes (see this warning).

On the CryoSPARC master computer, please can you run:

ip addr
cryosparcm status | grep ^app
curl 127.0.0.1:39000

and post the commands’ outputs.

Vruiz · May 1, 2023, 2:35pm

Hi wtempel,

I used

kill #process

The output are:

[victor@c105627 ~]$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:42:3f:3a:4e:90 brd ff:ff:ff:ff:ff:ff
    inet 129.112.162.193/24 brd 129.112.162.255 scope global noprefixroute dynamic enp1s0f0
       valid_lft 218378sec preferred_lft 218378sec
    inet6 fe80::fc51:4f1e:63be:c539/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether a0:42:3f:3a:4e:91 brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:6c:7d:75 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:6c:7d:75 brd ff:ff:ff:ff:ff:ff

[victor@c105627 ~]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/victor/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              EXITED    Apr 29 05:22 PM
app_api                          RUNNING   pid 47405, uptime 0:00:16
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 122490, uptime 1 day, 17:08:30
command_rtp                      RUNNING   pid 122562, uptime 1 day, 17:08:23
command_vis                      RUNNING   pid 122520, uptime 1 day, 17:08:24
database                         RUNNING   pid 122383, uptime 1 day, 17:08:34

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="redacted"
export CRYOSPARC_MASTER_HOSTNAME="c105627.dhcp.swmed.org"
export CRYOSPARC_DB_PATH="/home/victor/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,swmed.org,swmed.edu,*.swmed.org,*.swmed.edu,swlxnexpose.swmed.org,129.112.7.*
export NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,swmed.org,swmed.edu,*.swmed.org,*.swmed.edu,swlxnexpose.swmed.org,129.112.7.*

[victor@c105627 ~]$ curl 127.0.0.1:39000
curl: (7) Failed connect to 127.0.0.1:39000; Connection refused

wtempel · May 1, 2023, 7:28pm

It seems the app has exited, likely abnormally, and does not accept connections. Please can you inspect the
app.log and supervisord.log files for possible causes of the exit?

Vruiz · May 2, 2023, 3:05pm

Hi wtempel,

This is part of the app.log:

Computing instance stats
Done instance stats 364.455 ms
(node:6253) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:6253) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:6253) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:6253) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 108.481 ms
(node:14956) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:14956) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:14956) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:14956) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 111.535 ms
(node:15325) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:15325) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:15325) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:15325) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 109.563 ms
(node:33650) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:33650) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:33650) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:33650) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 111.214 ms
(node:34028) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:34028) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:34028) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:34028) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 107.623 ms
(node:110388) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:110388) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:110388) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:110388) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
(node:110753) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:110753) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:110753) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:110753) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 383.193 ms
(node:111880) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:111880) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:111880) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:111880) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 109.675 ms
(node:114865) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:114865) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:114865) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:114865) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 108.161 ms
(node:115234) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:115234) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:115234) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:115234) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 110.009 ms
(node:9889) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:9889) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:9889) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:9889) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 31.085 ms
(node:73916) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:73916) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:73916) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:73916) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
Computing instance stats
Done instance stats 27.765 ms
2023-04-23 13:08:31,355 INFO | DDP port: 39006
2023-04-23 13:08:31,357 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-23 13:09:02,232 INFO | DDP port: 39006
2023-04-23 13:09:02,235 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-23 13:09:32,274 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-23 13:09:32,274 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-23 13:09:32,274 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-23 13:09:32,274 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-23 13:09:32,274 ERROR |   reason: Eg {
2023-04-23 13:09:32,274 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-23 13:09:32,274 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-23 13:09:32,274 ERROR |     stale: false,
2023-04-23 13:09:32,274 ERROR |     compatible: true,
2023-04-23 13:09:32,274 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-23 13:09:32,274 ERROR |     localThresholdMS: 15,
2023-04-23 13:09:32,274 ERROR |     setName: 'meteor',
2023-04-23 13:09:32,274 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000063"),
2023-04-23 13:09:32,274 ERROR |     maxSetVersion: 1,
2023-04-23 13:09:32,274 ERROR |     commonWireVersion: 0,
2023-04-23 13:09:32,274 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-23 13:09:32,274 ERROR |   },
2023-04-23 13:09:32,274 ERROR |   code: undefined,
2023-04-23 13:09:32,274 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-23 13:09:32,274 ERROR | }
2023-04-23 13:09:32,282 ERROR | Database initialization failed
2023-04-23 13:18:15,436 INFO | DDP port: 39006
2023-04-23 13:18:15,444 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-23 13:18:45,493 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-23 13:18:45,493 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-23 13:18:45,493 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-23 13:18:45,493 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-23 13:18:45,493 ERROR |   reason: Eg {
2023-04-23 13:18:45,493 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-23 13:18:45,493 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-23 13:18:45,493 ERROR |     stale: false,
2023-04-23 13:18:45,493 ERROR |     compatible: true,
2023-04-23 13:18:45,493 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-23 13:18:45,493 ERROR |     localThresholdMS: 15,
2023-04-23 13:18:45,493 ERROR |     setName: 'meteor',
2023-04-23 13:18:45,493 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000065"),
2023-04-23 13:18:45,493 ERROR |     maxSetVersion: 1,
2023-04-23 13:18:45,493 ERROR |     commonWireVersion: 0,
2023-04-23 13:18:45,493 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-23 13:18:45,493 ERROR |   },
2023-04-23 13:18:45,493 ERROR |   code: undefined,
2023-04-23 13:18:45,493 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-23 13:18:45,493 ERROR | }
2023-04-23 13:18:45,500 ERROR | Database initialization failed
2023-04-24 14:57:55,428 INFO | DDP port: 39006
2023-04-24 14:57:55,433 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-24 14:58:25,476 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-24 14:58:25,476 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-24 14:58:25,476 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-24 14:58:25,476 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-24 14:58:25,476 ERROR |   reason: Eg {
2023-04-24 14:58:25,476 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-24 14:58:25,476 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-24 14:58:25,476 ERROR |     stale: false,
2023-04-24 14:58:25,476 ERROR |     compatible: true,
2023-04-24 14:58:25,476 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-24 14:58:25,476 ERROR |     localThresholdMS: 15,
2023-04-24 14:58:25,476 ERROR |     setName: 'meteor',
2023-04-24 14:58:25,476 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000067"),
2023-04-24 14:58:25,476 ERROR |     maxSetVersion: 1,
2023-04-24 14:58:25,476 ERROR |     commonWireVersion: 0,
2023-04-24 14:58:25,476 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-24 14:58:25,476 ERROR |   },
2023-04-24 14:58:25,476 ERROR |   code: undefined,
2023-04-24 14:58:25,476 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-24 14:58:25,476 ERROR | }
2023-04-24 14:58:25,483 ERROR | Database initialization failed
2023-04-29 16:07:45,294 INFO | DDP port: 39006
2023-04-29 16:07:45,296 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-29 16:08:15,336 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-29 16:08:15,336 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-29 16:08:15,336 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-29 16:08:15,336 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-29 16:08:15,336 ERROR |   reason: Eg {
2023-04-29 16:08:15,336 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-29 16:08:15,336 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-29 16:08:15,336 ERROR |     stale: false,
2023-04-29 16:08:15,336 ERROR |     compatible: true,
2023-04-29 16:08:15,336 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-29 16:08:15,336 ERROR |     localThresholdMS: 15,
2023-04-29 16:08:15,336 ERROR |     setName: 'meteor',
2023-04-29 16:08:15,336 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000069"),
2023-04-29 16:08:15,336 ERROR |     maxSetVersion: 1,
2023-04-29 16:08:15,336 ERROR |     commonWireVersion: 0,
2023-04-29 16:08:15,336 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-29 16:08:15,336 ERROR |   },
2023-04-29 16:08:15,336 ERROR |   code: undefined,
2023-04-29 16:08:15,336 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-29 16:08:15,336 ERROR | }
2023-04-29 16:08:15,344 ERROR | Database initialization failed
2023-04-29 16:23:51,024 INFO | DDP port: 39006
2023-04-29 16:23:51,026 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-29 16:24:21,066 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-29 16:24:21,066 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-29 16:24:21,066 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-29 16:24:21,066 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-29 16:24:21,066 ERROR |   reason: Eg {
2023-04-29 16:24:21,066 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-29 16:24:21,066 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-29 16:24:21,066 ERROR |     stale: false,
2023-04-29 16:24:21,066 ERROR |     compatible: true,
2023-04-29 16:24:21,066 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-29 16:24:21,066 ERROR |     localThresholdMS: 15,
2023-04-29 16:24:21,066 ERROR |     setName: 'meteor',
2023-04-29 16:24:21,066 ERROR |     maxElectionId: new ObjectId("7fffffff000000000000006b"),
2023-04-29 16:24:21,066 ERROR |     maxSetVersion: 1,
2023-04-29 16:24:21,066 ERROR |     commonWireVersion: 0,
2023-04-29 16:24:21,066 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-29 16:24:21,066 ERROR |   },
2023-04-29 16:24:21,066 ERROR |   code: undefined,
2023-04-29 16:24:21,066 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-29 16:24:21,066 ERROR | }
2023-04-29 16:24:21,073 ERROR | Database initialization failed
2023-04-29 16:25:51,750 INFO | DDP port: 39006
2023-04-29 16:25:51,752 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-29 16:26:21,791 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-29 16:26:21,791 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-29 16:26:21,791 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-29 16:26:21,791 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-29 16:26:21,791 ERROR |   reason: Eg {
2023-04-29 16:26:21,791 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-29 16:26:21,791 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-29 16:26:21,791 ERROR |     stale: false,
2023-04-29 16:26:21,791 ERROR |     compatible: true,
2023-04-29 16:26:21,791 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-29 16:26:21,791 ERROR |     localThresholdMS: 15,
2023-04-29 16:26:21,791 ERROR |     setName: 'meteor',
2023-04-29 16:26:21,791 ERROR |     maxElectionId: new ObjectId("7fffffff000000000000006d"),
2023-04-29 16:26:21,791 ERROR |     maxSetVersion: 1,
2023-04-29 16:26:21,791 ERROR |     commonWireVersion: 0,
2023-04-29 16:26:21,791 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-29 16:26:21,791 ERROR |   },
2023-04-29 16:26:21,791 ERROR |   code: undefined,
2023-04-29 16:26:21,791 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-29 16:26:21,791 ERROR | }
2023-04-29 16:26:21,800 ERROR | Database initialization failed
2023-04-29 16:49:31,344 INFO | DDP port: 39006
2023-04-29 16:49:31,346 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-29 16:50:01,385 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-29 16:50:01,385 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-29 16:50:01,385 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-29 16:50:01,385 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-29 16:50:01,385 ERROR |   reason: Eg {
2023-04-29 16:50:01,385 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-29 16:50:01,385 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-29 16:50:01,385 ERROR |     stale: false,
2023-04-29 16:50:01,385 ERROR |     compatible: true,
2023-04-29 16:50:01,385 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-29 16:50:01,385 ERROR |     localThresholdMS: 15,
2023-04-29 16:50:01,385 ERROR |     setName: 'meteor',
2023-04-29 16:50:01,385 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000070"),
2023-04-29 16:50:01,385 ERROR |     maxSetVersion: 1,
2023-04-29 16:50:01,385 ERROR |     commonWireVersion: 0,
2023-04-29 16:50:01,385 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-29 16:50:01,385 ERROR |   },
2023-04-29 16:50:01,385 ERROR |   code: undefined,
2023-04-29 16:50:01,385 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-29 16:50:01,385 ERROR | }
2023-04-29 16:50:01,392 ERROR | Database initialization failed
2023-04-29 17:21:59,291 INFO | DDP port: 39006
2023-04-29 17:21:59,293 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-04-29 17:22:29,333 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-04-29 17:22:29,333 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-04-29 17:22:29,333 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-04-29 17:22:29,333 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-04-29 17:22:29,333 ERROR |   reason: Eg {
2023-04-29 17:22:29,333 ERROR |     type: 'ReplicaSetNoPrimary',
2023-04-29 17:22:29,333 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-04-29 17:22:29,333 ERROR |     stale: false,
2023-04-29 17:22:29,333 ERROR |     compatible: true,
2023-04-29 17:22:29,333 ERROR |     heartbeatFrequencyMS: 10000,
2023-04-29 17:22:29,333 ERROR |     localThresholdMS: 15,
2023-04-29 17:22:29,333 ERROR |     setName: 'meteor',
2023-04-29 17:22:29,333 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000072"),
2023-04-29 17:22:29,333 ERROR |     maxSetVersion: 1,
2023-04-29 17:22:29,333 ERROR |     commonWireVersion: 0,
2023-04-29 17:22:29,333 ERROR |     logicalSessionTimeoutMinutes: null
2023-04-29 17:22:29,333 ERROR |   },
2023-04-29 17:22:29,333 ERROR |   code: undefined,
2023-04-29 17:22:29,333 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-04-29 17:22:29,333 ERROR | }
2023-04-29 17:22:29,340 ERROR | Database initialization failed
2023-05-01 10:33:39,854 INFO | DDP port: 39006
2023-05-01 10:33:39,856 INFO | /websocket proxy target: http://c105627.dhcp.swmed.org:39006
[HPM] Proxy created: /websocket  -> http://c105627.dhcp.swmed.org:39006
2023-05-01 10:34:09,895 ERROR | [MongoDB] Unable to connect Server selection timed out after 30000 ms OV [MongoServerSelectionError]: Server selection timed out after 30000 ms
2023-05-01 10:34:09,895 ERROR |     at Timeout._onTimeout (/home/victor/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2023-05-01 10:34:09,895 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2023-05-01 10:34:09,895 ERROR |     at processTimers (node:internal/timers:502:7) {
2023-05-01 10:34:09,895 ERROR |   reason: Eg {
2023-05-01 10:34:09,895 ERROR |     type: 'ReplicaSetNoPrimary',
2023-05-01 10:34:09,895 ERROR |     servers: Map(1) { 'localhost:39001' => [XW] },
2023-05-01 10:34:09,895 ERROR |     stale: false,
2023-05-01 10:34:09,895 ERROR |     compatible: true,
2023-05-01 10:34:09,895 ERROR |     heartbeatFrequencyMS: 10000,
2023-05-01 10:34:09,895 ERROR |     localThresholdMS: 15,
2023-05-01 10:34:09,895 ERROR |     setName: 'meteor',
2023-05-01 10:34:09,895 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000074"),
2023-05-01 10:34:09,895 ERROR |     maxSetVersion: 1,
2023-05-01 10:34:09,895 ERROR |     commonWireVersion: 0,
2023-05-01 10:34:09,895 ERROR |     logicalSessionTimeoutMinutes: null
2023-05-01 10:34:09,895 ERROR |   },
2023-05-01 10:34:09,895 ERROR |   code: undefined,
2023-05-01 10:34:09,895 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2023-05-01 10:34:09,895 ERROR | }
2023-05-01 10:34:09,902 ERROR | Database initialization failed

supervisord.log

2023-05-02 10:57:03,957 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:57:04,962 INFO spawned: 'app_api' with pid 104824
2023-05-02 10:57:06,728 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 10:57:35,740 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:57:36,746 INFO spawned: 'app_api' with pid 104846
2023-05-02 10:57:38,513 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 10:58:07,540 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:58:08,545 INFO spawned: 'app_api' with pid 104860
2023-05-02 10:58:10,341 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 10:58:39,334 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:58:40,340 INFO spawned: 'app_api' with pid 104882
2023-05-02 10:58:42,103 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 10:59:11,120 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:59:12,125 INFO spawned: 'app_api' with pid 104896
2023-05-02 10:59:13,868 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 10:59:42,887 INFO exited: app_api (exit status 1; not expected)
2023-05-02 10:59:43,893 INFO spawned: 'app_api' with pid 104918
2023-05-02 10:59:45,660 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 11:00:14,677 INFO exited: app_api (exit status 1; not expected)
2023-05-02 11:00:15,683 INFO spawned: 'app_api' with pid 104965
2023-05-02 11:00:17,460 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 11:00:46,473 INFO exited: app_api (exit status 1; not expected)
2023-05-02 11:00:47,478 INFO spawned: 'app_api' with pid 104987
2023-05-02 11:00:49,258 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-05-02 11:01:18,282 INFO exited: app_api (exit status 1; not expected)
2023-05-02 11:01:19,288 INFO spawned: 'app_api' with pid 105040
2023-05-02 11:01:21,042 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

wtempel · May 2, 2023, 7:10pm

It is possible that the coordination between some CryoSPARC-related service processes has failed.
Under the Linux account that usually runs CryoSPARC (victor?), please can you run

If that second ps command doesn’t show any more CryoSPARC-related processes, please try
cryosparcm start
Can you now connect to the CryoSPARC web interface?

Vruiz · May 2, 2023, 7:56pm

I found a solution in this threat:

My /etc/hosts file looked like this:

cat /etc/hosts_BACKUP 
7.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomai

I modified to:

127.0.0.1 c105627 localhost localhost.localdomain localhost4 localhost4.localdomain4
7.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain4
129.112.162.193 c105627.dhcp.swmed.org c105627

I also modified the /cryosparc/cryosparc_master file to:

export CRYOSPARC_LICENSE_ID="redacted"
export CRYOSPARC_MASTER_HOSTNAME="c105627"
export CRYOSPARC_DB_PATH="/home/victor/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

export no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com,swmed.org,swmed.edu,*.swmed.org,*.swmed.edu,swlxnexpose.swmed.org,129.122.162.193
export NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,swmed.org,swmed.edu,*.swmed.org,*.swmed.edu,swlxnexpose.swmed.org,129.112.162.193

I use Centos 7

Please let me know if I should change any other settings.

wtempel · May 2, 2023, 8:01pm

Could this have been a typo or copy-paste error that led to two digits missing at beginning of the line? In this case, you may want to comment out or delete this line.
Also, I am not sure of the effect of/precedence rules for multiple ipv4 entries with the same hostname c105627[.dhcp.swmed.org].

Vruiz · May 2, 2023, 8:12pm

It is present on the original file so I guess the first “12” digits were deleted at some point.

wtempel · May 2, 2023, 9:00pm

@Vruiz [Edited:] Is the CryoSPARC interface accessible after

full shutdown of CryoSPARC (Database error after v4.1.0 update - #23 by wtempel)
changing /etc/hosts to

127.0.0.1 c105627 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain4

cryosparcm restart

?

Vruiz · May 2, 2023, 9:23pm

@wtempel after applying those modifications I can access the web interface:

127.0.0.1 c105627 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain4

But I can’t import any movies, cryosparcm status is:

[victor@c105627 ~]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/victor/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 147427, uptime 0:10:06
app_api                          RUNNING   pid 147456, uptime 0:10:05
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 147393, uptime 0:10:15
command_rtp                      RUNNING   pid 149282, uptime 0:00:03
command_vis                      RUNNING   pid 149266, uptime 0:00:06
database                         RUNNING   pid 147286, uptime 0:10:19

----------------------------------------------------------------------------
*** CommandClient: (http://c105627:39002/api) HTTP Error 403 notresolvable; please check cryosparcm log command for additional information.
Response from server: b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n<html>\n<!-- FileName: index.html\n     Language: [en]\n-->\n<!--Head-->\n<head>\n  <meta http-equiv="X-UA-Compatible" content="IE=7" />\n  <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">\n  <title>McAfee Web Gateway - Notification</title>\n  <script src="/mwg-internal/de5fs23hu73ds/files/javascript/sw.js" type="text/javascript" ></script>\n  <link rel="stylesheet" href="/mwg-internal/de5fs23hu73ds/files/default/stylesheet.css" />\n</head>\n<!--/Head-->\n\n<!--Body-->\n<body onload="swOnLoad();">\n  <table class=\'bodyTable\'>\n    <tr>\n      <td class=\'bodyData\' background=\'/mwg-internal/de5fs23hu73ds/files/utsw/img/bg_body.gif\'>\n<!--Logo-->\n<table class=\'logoTable\'>\n  <tr>\n    <td class=\'logoData\'>\n      <img src=\'/mwg-internal/de5fs23hu73ds/files/utsw/img/utswLogo.jpg\'>\n    </td>\n  </tr>\n</table>\n<!--/Logo-->\n\n<!--Contents-->\n<!-- FileName: notresolvable.html\n     Language: [en]\n-->\n<!--Title-->\n<table class=\'titleTable\'>\n  <tr>\n    <td class=\'titleData\'>\n      Host Not Resolvable\n    </td>\n  </tr>\n</table>\n<!--/Title-->\n\n<!--Content-->\n<table class="contentTable">\n  <tr>\n    <td class="contentData">\n      The Host is not resolvable.<br>\n      This website is down. Please contact the site owner if you believe this is an error.\n    </td>\n  </tr>\n</table>\n<!--/Content-->\n\n<!--Info-->\n<table class="infoTable">\n  <tr>\n    <td class="infoData">\n      <b>URL: </b><script type="text/javascript">break_line("http://c105627:39002/api");</script><br />\n    </td>\n  </tr>\n</table>\n<!--/Info-->\n\n<!--/Contents-->\n\n<!--Foot-->\n<table class=\'footTable\'>\n  <tr>\n   <td class=\'helpDeskData\'>\n      For assistance, please contact the IR Service Desk at 214-648-7600.\n    </td>\n  </tr>\n  <tr>\n    <td class=\'footData\'>\n      generated <span id="time">2023-05-02 16:21:02</span> by McAfee Web Gateway [129.112.115.43/m-proxy4] \n      <br />\n      Python-urllib/3.8\n      <br />\n     Give Not Resolvable for NXDOMAIN Web Sites\n      <br />\n      Client IP [129.112.162.193]\n    </td>\n  </tr>\n</table>\n<!--/Foot-->\n\n      </td>\n    </tr>\n  </table>\n</body>\n<!--/Body-->\n</html>'
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

Vruiz · May 2, 2023, 9:50pm

Adding the hostname to cryosparc_master/config.sh solved the proxy issue:

export no_proxy=c105627
export NO_PROXY=c105627

wtempel · May 2, 2023, 9:56pm

Nice. Did this also resolve

?

Vruiz · May 2, 2023, 10:10pm

@wtempel, CS was unable to check the license and jobs were indefinitely in the queue.