Cryosparc crashes after update to 4.0.0

Install, Configure and Update
1

Hi,

I have updated cryosparc as described on the official documentation website. Everything was finished successfully, however, when I tried to open the website, the app crashes:

Current cryoSPARC version: v4.0.0
----------------------------------------------------------------------------

CryoSPARC process status:

app                              EXITED    Oct 03 02:55 PM
app_api                          RUNNING   pid 81018, uptime 0:00:29
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 80893, uptime 0:00:43
command_rtp                      RUNNING   pid 80960, uptime 0:00:35
command_vis                      RUNNING   pid 80925, uptime 0:00:37
database                         RUNNING   pid 80773, uptime 0:00:46

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="XXX"
export CRYOSPARC_MASTER_HOSTNAME="10.0.90.57"
export CRYOSPARC_DB_PATH="/home_old/cryosparc_user/software/cryosparc/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_FORCE_HOSTNAME=true

After the restart of cryosparc, everything is running, including the app. However, as soon as I try to connect using the browser app crashes and says status exited.

in the app log:

2022-10-03 14:55:54,082 ERROR | [MongoDB] Unable to connect connect ECONNREFUSED ::1:39001 eV [MongoServerSelectionError]: connect ECONNREFUSED ::1:39001
2022-10-03 14:55:54,082 ERROR |     at Timeout._onTimeout (/home_old/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2022-10-03 14:55:54,082 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2022-10-03 14:55:54,082 ERROR |     at processTimers (node:internal/timers:502:7) {
2022-10-03 14:55:54,082 ERROR |   reason: hg {
2022-10-03 14:55:54,082 ERROR |     type: 'ReplicaSetNoPrimary',
2022-10-03 14:55:54,082 ERROR |     servers: Map(1) { 'localhost:39001' => [AW] },
2022-10-03 14:55:54,082 ERROR |     stale: false,
2022-10-03 14:55:54,082 ERROR |     compatible: true,
2022-10-03 14:55:54,082 ERROR |     heartbeatFrequencyMS: 10000,
2022-10-03 14:55:54,082 ERROR |     localThresholdMS: 15,
2022-10-03 14:55:54,082 ERROR |     setName: 'meteor',
2022-10-03 14:55:54,082 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000050"),
2022-10-03 14:55:54,082 ERROR |     maxSetVersion: 1,
2022-10-03 14:55:54,082 ERROR |     commonWireVersion: 0,
2022-10-03 14:55:54,082 ERROR |     logicalSessionTimeoutMinutes: null
2022-10-03 14:55:54,082 ERROR |   },
2022-10-03 14:55:54,082 ERROR |   code: undefined,
2022-10-03 14:55:54,082 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2022-10-03 14:55:54,082 ERROR | }
2022-10-03 14:55:54,086 ERROR | Database initialization failed

I would be grateful for some help with troubleshooting.
Best,
Dawid

2

Hi Dawid,

Please can you email us screen output during update, if still available, as well as the outputs of

  • cryosparcm errorreport
  • cryosparcm checkdb
  • cryosparcm test i

[edited to correct command spelling]

3

Hi @dzyla,

We have investigated this issue internally but haven’t yet come to a definitive reason for the cause.

Could you please try the following and let us know if it helps:

Check that no MongoDB zombie processes are running on the master node. First stop the running CryoSPARC instance:

cryosparcm stop

On the CryoSPARC master node, list all processes associated with MongoDB with the following command:

ps -ax | grep mongo

If you see any processes that appear associated with MongoDB launched by a previous instance of CryoSPARC, kill them with the command kill <PID> , where <PID> is the process ID number at the beginning of each line.

Then start CryoSPARC on the master note or workstation:

cryosparcm start

4

Hi,
So I still couldn’t solve the issue. I killed all mongo* instances and/or restarted/ re-installed cryosparc from scratch (new database and installation location), and the problem is still there.

Interestingly, Ubuntu 22.04, with the old database, could boot and run the cryosparc webserver. Is Centos7 still supported?

Thanks!

5

I am glad you got the CryoSPARC master and database to run on Ubuntu 22.04.
Unfortunately, I am not sure about the cause of the database initialization failure and if/how the failure may be related to CentOS-7.

6

Hi,

I’m still having the issue even after upgrading 4.0.2.

Basically, the cryoSPARC process status is:

app                              EXITED    Oct 26 11:47 PM
app_api                          RUNNING   pid 254605, uptime 0:00:03
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 252549, uptime 0:00:50
command_rtp                      RUNNING   pid 252903, uptime 0:00:42
command_vis                      RUNNING   pid 252824, uptime 0:00:43
database                         RUNNING   pid 252365, uptime 0:00:52

here’s cryosparcm log database

----------------------------------------------------------------------------
License is valid
2022-10-26T23:46:22.612-0400 I REPL     [replexec-0] transition to RECOVERING from STARTUP2
2022-10-26T23:46:22.612-0400 I REPL     [replexec-0] Starting replication fetcher thread
2022-10-26T23:46:22.612-0400 I REPL     [replexec-0] Starting replication applier thread
2022-10-26T23:46:22.612-0400 I REPL     [replexec-0] Starting replication reporter thread
2022-10-26T23:46:22.613-0400 I REPL     [rsSync] transition to SECONDARY from RECOVERING
2022-10-26T23:46:22.613-0400 I REPL     [rsSync] conducting a dry run election to see if we could be elected. current term: 84
2022-10-26T23:46:22.614-0400 I REPL     [replexec-0] dry election run succeeded, running for election in term 85
2022-10-26T23:46:22.629-0400 I REPL     [replexec-0] election succeeded, assuming primary role in term 85
2022-10-26T23:46:22.629-0400 I REPL     [replexec-0] transition to PRIMARY from SECONDARY
2022-10-26T23:46:22.629-0400 I REPL     [replexec-0] Resetting sync source to empty, which was :27017
2022-10-26T23:46:22.629-0400 I REPL     [replexec-0] Entering primary catch-up mode.
2022-10-26T23:46:22.629-0400 I REPL     [replexec-0] Exited primary catch-up mode.
2022-10-26T23:46:24.441-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42194 #1 (1 connection now open)
2022-10-26T23:46:24.448-0400 I NETWORK  [conn1] received client metadata from 127.0.0.1:42194 conn1: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.23" }, os: { type: "Linux", name: "CentOS Linux release 7.6.1810 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-957.21.3.el7.x86_64" } }
2022-10-26T23:46:24.463-0400 I ACCESS   [conn1] Successfully authenticated as principal cryosparc_admin on admin from client 127.0.0.1:42194
2022-10-26T23:46:24.467-0400 I ACCESS   [conn1] Successfully authenticated as principal cryosparc_admin on admin from client 127.0.0.1:42194
2022-10-26T23:46:24.470-0400 I NETWORK  [conn1] end connection 127.0.0.1:42194 (0 connections now open)
2022-10-26T23:46:24.615-0400 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2022-10-26T23:46:30.226-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42196 #2 (1 connection now open)
2022-10-26T23:46:30.226-0400 I NETWORK  [conn2] received client metadata from 127.0.0.1:42196 conn2: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:30.229-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42198 #3 (2 connections now open)
2022-10-26T23:46:30.229-0400 I NETWORK  [conn3] received client metadata from 127.0.0.1:42198 conn3: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:30.246-0400 I ACCESS   [conn3] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:42198
2022-10-26T23:46:30.957-0400 I WRITE    [conn3] update meteor.jobs command: { q: { cluster_job_id: { $exists: false } }, u: { $set: { cluster_job_id: null } }, multi: true, upsert: false } planSummary: COLLSCAN keysExamined:0 docsExamined:2517 nMatched:0 nModified:0 numYields:22 locks:{ Global: { acquireCount: { r: 23, w: 23 } }, Database: { acquireCount: { w: 23 } }, Collection: { acquireCount: { w: 23 } } } 355ms
2022-10-26T23:46:30.957-0400 I COMMAND  [conn3] command meteor.$cmd command: update { update: "jobs", ordered: true, lsid: { id: UUID("22af8554-8e2e-435b-8d9b-51f164233cf9") }, $clusterTime: { clusterTime: Timestamp(1666842390, 2), signature: { hash: BinData(0, 6D808EFFCD3D792A2E53F54D95C25BD8CDE275ED), keyId: 7150458212343349249 } }, $db: "meteor", $readPreference: { mode: "primary" } } numYields:0 reslen:229 locks:{ Global: { acquireCount: { r: 23, w: 23 } }, Database: { acquireCount: { w: 23 } }, Collection: { acquireCount: { w: 23 } } } protocol:op_msg 355ms
2022-10-26T23:46:31.013-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42204 #4 (3 connections now open)
2022-10-26T23:46:31.015-0400 I NETWORK  [conn4] received client metadata from 127.0.0.1:42204 conn4: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:31.017-0400 I ACCESS   [conn4] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:42204
2022-10-26T23:46:33.030-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42216 #5 (4 connections now open)
2022-10-26T23:46:33.030-0400 I NETWORK  [conn5] received client metadata from 127.0.0.1:42216 conn5: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:33.031-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42218 #6 (5 connections now open)
2022-10-26T23:46:33.032-0400 I NETWORK  [conn6] received client metadata from 127.0.0.1:42218 conn6: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:33.043-0400 I ACCESS   [conn6] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:42218
2022-10-26T23:46:38.182-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42254 #7 (6 connections now open)
2022-10-26T23:46:38.185-0400 I NETWORK  [conn7] received client metadata from 127.0.0.1:42254 conn7: { driver: { name: "nodejs", version: "4.9.0" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v16.14.2, LE (unified)|Node.js v16.14.2, LE (unified)" }
2022-10-26T23:46:38.189-0400 I NETWORK  [conn7] end connection 127.0.0.1:42254 (5 connections now open)
2022-10-26T23:46:40.083-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42264 #8 (6 connections now open)
2022-10-26T23:46:40.093-0400 I NETWORK  [conn8] received client metadata from 127.0.0.1:42264 conn8: { driver: { name: "nodejs", version: "4.3.1" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)" }
2022-10-26T23:46:40.098-0400 I NETWORK  [conn8] end connection 127.0.0.1:42264 (5 connections now open)
2022-10-26T23:46:41.600-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42278 #9 (6 connections now open)
2022-10-26T23:46:41.600-0400 I NETWORK  [conn9] received client metadata from 127.0.0.1:42278 conn9: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:41.603-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42280 #10 (7 connections now open)
2022-10-26T23:46:41.604-0400 I NETWORK  [conn10] received client metadata from 127.0.0.1:42280 conn10: { driver: { name: "PyMongo", version: "3.12.2" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "CPython 3.7.12.final.0" }
2022-10-26T23:46:41.630-0400 I ACCESS   [conn10] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:42280
2022-10-26T23:47:11.775-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42528 #11 (8 connections now open)
2022-10-26T23:47:11.779-0400 I NETWORK  [conn11] received client metadata from 127.0.0.1:42528 conn11: { driver: { name: "nodejs", version: "4.3.1" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)" }
2022-10-26T23:47:11.784-0400 I NETWORK  [conn11] end connection 127.0.0.1:42528 (7 connections now open)
2022-10-26T23:47:43.498-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42660 #12 (8 connections now open)
2022-10-26T23:47:43.502-0400 I NETWORK  [conn12] received client metadata from 127.0.0.1:42660 conn12: { driver: { name: "nodejs", version: "4.3.1" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)" }
2022-10-26T23:47:43.507-0400 I NETWORK  [conn12] end connection 127.0.0.1:42660 (7 connections now open)
2022-10-26T23:48:15.198-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42782 #13 (8 connections now open)
2022-10-26T23:48:15.202-0400 I NETWORK  [conn13] received client metadata from 127.0.0.1:42782 conn13: { driver: { name: "nodejs", version: "4.3.1" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)" }
2022-10-26T23:48:15.207-0400 I NETWORK  [conn13] end connection 127.0.0.1:42782 (7 connections now open)
2022-10-26T23:48:46.919-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:42906 #14 (8 connections now open)
2022-10-26T23:48:46.924-0400 I NETWORK  [conn14] received client metadata from 127.0.0.1:42906 conn14: { driver: { name: "nodejs", version: "4.3.1" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-957.21.3.el7.x86_64" }, platform: "Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)" }
2022-10-26T23:48:46.928-0400 I NETWORK  [conn14] end connection 127.0.0.1:42906 (7 connections now open)

here’s cryosparcm log app

2022-10-26 23:46:38,167 INFO | DDP port: 39006
2022-10-26 23:46:38,169 INFO | /websocket proxy target: http://localhost:39006
[HPM] Proxy created: /websocket  -> http://localhost:39006
2022-10-26 23:47:08,187 ERROR | [MongoDB] Unable to connect connect ECONNREFUSED ::1:39001 eV [MongoServerSelectionError]: connect ECONNREFUSED ::1:39001
2022-10-26 23:47:08,187 ERROR |     at Timeout._onTimeout (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/custom-server/dist/server/index.js:24:651839)
2022-10-26 23:47:08,187 ERROR |     at listOnTimeout (node:internal/timers:559:17)
2022-10-26 23:47:08,187 ERROR |     at processTimers (node:internal/timers:502:7) {
2022-10-26 23:47:08,187 ERROR |   reason: hg {
2022-10-26 23:47:08,187 ERROR |     type: 'ReplicaSetNoPrimary',
2022-10-26 23:47:08,187 ERROR |     servers: Map(1) { 'localhost:39001' => [AW] },
2022-10-26 23:47:08,187 ERROR |     stale: false,
2022-10-26 23:47:08,187 ERROR |     compatible: true,
2022-10-26 23:47:08,187 ERROR |     heartbeatFrequencyMS: 10000,
2022-10-26 23:47:08,187 ERROR |     localThresholdMS: 15,
2022-10-26 23:47:08,187 ERROR |     setName: 'meteor',
2022-10-26 23:47:08,187 ERROR |     maxElectionId: new ObjectId("7fffffff0000000000000055"),
2022-10-26 23:47:08,187 ERROR |     maxSetVersion: 1,
2022-10-26 23:47:08,187 ERROR |     commonWireVersion: 0,
2022-10-26 23:47:08,187 ERROR |     logicalSessionTimeoutMinutes: null
2022-10-26 23:47:08,187 ERROR |   },
2022-10-26 23:47:08,187 ERROR |   code: undefined,
2022-10-26 23:47:08,187 ERROR |   [Symbol(errorLabels)]: Set(0) {}
2022-10-26 23:47:08,187 ERROR | }
2022-10-26 23:47:08,194 ERROR | Database initialization failed

given logs, it seems like app is unable to connect to the database.
Please let me know if you need any additional info.
Thank you.

p.s.
cryosparcm checkdb & test i both all passed without any errors.

7

If you upgraded from a functioning CryoSPARC instance, what was the version when the instance was still functioning?
What is the output of
curl localhost:39001
on the master node?

8

I believe it was the version right before 4.0.0

curl localhost:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
9

and this is cryosparcm log app_api

/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/api/bundle/programs/server/node_modules/fibers/future.js:313
                                                throw(ex);
                                                ^

MongoServerSelectionError: connect ECONNREFUSED ::1:39001
    at Timeout._onTimeout (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/api/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/sdam/topology.js:312:38)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  reason: TopologyDescription {
    type: 'ReplicaSetNoPrimary',
    servers: Map(1) {
      'localhost:39001' => ServerDescription {
        _hostAddress: HostAddress { isIPv6: false, host: 'localhost', port: 39001 },
        address: 'localhost:39001',
        type: 'Unknown',
        hosts: [],
        passives: [],
        arbiters: [],
        tags: {},
        minWireVersion: 0,
        maxWireVersion: 0,
        roundTripTime: -1,
        lastUpdateTime: 17703036763,
        lastWriteDate: 0,
        error: MongoNetworkError: connect ECONNREFUSED ::1:39001
            at connectionFailureError (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/api/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connect.js:381:20)
            at Socket.<anonymous> (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/api/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/cmap/connect.js:301:22)
            at Object.onceWrapper (events.js:520:26)
            at Socket.emit (events.js:400:28)
            at Socket.emit (domain.js:475:12)
            at emitErrorNT (internal/streams/destroy.js:106:8)
            at emitErrorCloseNT (internal/streams/destroy.js:74:3)
            at processTicksAndRejections (internal/process/task_queues.js:82:21)
      }
    },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: 'meteor',
    maxSetVersion: 1,
    maxElectionId: ObjectId {
      [Symbol(id)]: Buffer(12) [Uint8Array] [
        127, 255, 255, 255, 0,
          0,   0,   0,   0, 0,
          0,  93
      ]
    },
    commonWireVersion: 6,
    logicalSessionTimeoutMinutes: undefined
  }
}

given that hostname is not localhost but calling localhost and wonder if that’s the cause? (even though technically localhost should work though)

$ curl localhost:39001
curl: (7) Failed connect to localhost:39001; Connection refused
$ curl __hostname__:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
10

Fixed the issue by fixing /etc/hosts to correct the routing for ‘localhost’.

3 Likes
11

Could you share the corrected /etc/hosts file content? That would be useful. Thank you!

12

I can confirm that chaning the /etc/hosts to:

127.0.0.1   localhost
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

It works well and solves the issue. If the computer has a hostname after reinstalling cryosparc, one also should change the config.sh in a master folder to the IP address of the machine. I hope it helps!

1 Like
13

Ooops. Sorry.
I didn’t get any notifications and I found the reply purely by luck.

here’s my version:

127.0.0.1			Your_Hostname localhost localhost.localdomain localhost4 localhost4.localdomain4
::1				localhost localhost.localdomain localhost6 localhost6.localdomain6
XXX.XXX.XXX.XXX     Your_Hostname.domain.tld Your_Hostname

You need to change two things, Your_Hostname for $hostname value and XXX.XXX.XXX.XXX for your ip address accordingly.
I believe dzyla’s should be sufficient but I just have additional decorations for my personal convenience/preference.
Hope this helps.

1 Like