Cryosparc v4 start error

Troubleshooting
1

I searched a lot and tried killing cyrosparc and mongod processes, but nothing works.

$cryosparcm start

Starting cryoSPARC System master process…
CryoSPARC is not already running.
configuring database
configuration complete
/home/patrick/Downloads/cryosparc/cryosparc_master/bin/cryosparcm: line 363: 15619 Segmentation fault (core dumped) command python -c “from cryosparc_compute import database_management; database_management.configure_mongo(logfile=‘${CRYOSPARC_ROOT_DIR}/run/database.log’)”
[2023-12-12T14:52:50+08:00] Error configuring database. Most recent database log lines:
2023-12-12T14:52:49.913+0800 I NETWORK [conn4] received client metadata from 127.0.0.1:48808 conn4: { driver: { name: “PyMongo”, version: “3.13.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.14.0-390.el9.x86_64” }, platform: “CPython 3.8.17.final.0” }
2023-12-12T14:52:49.933+0800 I NETWORK [conn4] end connection 127.0.0.1:48808 (1 connection now open)
2023-12-12T14:52:49.933+0800 I NETWORK [conn3] end connection 127.0.0.1:48804 (0 connections now open)
2023-12-12T14:52:49.933+0800 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2023-12-12T14:52:49.933+0800 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2023-12-12T14:52:49.933+0800 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] shutting down replication subsystems
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication reporter thread
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication fetcher thread
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication applier thread

$ cryosparcm status

CryoSPARC System master node installed at
/home/patrick/Downloads/cryosparc/cryosparc_master
Current cryoSPARC version: v4.4.1

CryoSPARC process status:

app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
command_core STOPPED Not started
command_rtp STOPPED Not started
command_vis STOPPED Not started
database STOPPED Not started

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

$ curl 127.0.0.1:39000
curl: (7) Failed to connect to 127.0.0.1 port 39000: Connection refused

supervisord.log

2023-12-12 14:52:46,299 INFO RPC interface ‘supervisor’ initialized
2023-12-12 14:52:46,299 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2023-12-12 14:52:46,300 INFO daemonizing the supervisord process
2023-12-12 14:52:46,301 INFO supervisord started with pid 15592

can anyone please help?

2

forgot to mention, this happened after restarting the server. before that everything’s fine.
/etc/hosts like this:
…
127.0.0.1 gpunode3

3

Welcome to the forum @biolish .

Please can you post additional information:

  1. Outputs of the commands
    ps -weo pid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep
ls -l /tmp/*.sock
  2. Additional lines from the database log with a time stamp around 2023-12-12T14:52:49:
    cryosparcm log database | grep "^2023-12-12T14:52:4"
5

Hi wtempel,

thanks in advance. bye the way, it’s centos9 stream OS.
here’s the outputs:

nothing after

$ ps -weo pid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep
$

nothing after

$ ls -l /tmp/.sock
ls: cannot access '/tmp/.sock’: No such file or directory

$ cryosparcm log database | grep “^2023-12-12T14:52:4”
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] MongoDB starting : pid=15621 port=39001 dbpath=/home/patrick/Downloads/cryosparc/cryosparc_database 64-bit host=gpunode3
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] db version v3.6.23
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] allocator: tcmalloc
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] modules: none
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] build environment:
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] distarch: x86_64
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] target_arch: x86_64
2023-12-12T14:52:46.916+0800 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/home/patrick/Downloads/cryosparc/cryosparc_database” } }
2023-12-12T14:52:46.916+0800 I - [initandlisten] Detected data files in /home/patrick/Downloads/cryosparc/cryosparc_database created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2023-12-12T14:52:46.916+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=128011M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release=“3.0”,require_max=“3.0”),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-12-12T14:52:47.401+0800 I STORAGE [initandlisten] WiredTiger message [1702363967:401016][15621:0x7f3a5befba40], txn-recover: Main recovery loop: starting at 26/6784
2023-12-12T14:52:47.476+0800 I STORAGE [initandlisten] WiredTiger message [1702363967:476289][15621:0x7f3a5befba40], txn-recover: Recovering log 26 through 27
2023-12-12T14:52:47.518+0800 I STORAGE [initandlisten] WiredTiger message [1702363967:518859][15621:0x7f3a5befba40], txn-recover: Recovering log 27 through 27
2023-12-12T14:52:47.555+0800 I STORAGE [initandlisten] WiredTiger message [1702363967:555691][15621:0x7f3a5befba40], txn-recover: Set global recovery timestamp: 0
2023-12-12T14:52:47.561+0800 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2023-12-12T14:52:47.561+0800 I STORAGE [initandlisten] The size storer reports that the oplog contains 112953 records totaling to 66583448 bytes
2023-12-12T14:52:47.561+0800 I STORAGE [initandlisten] Sampling the oplog to determine where to place markers for truncation
2023-12-12T14:52:47.563+0800 I STORAGE [initandlisten] Sampling from the oplog between Nov 30 11:38:05:2 and Dec 12 14:36:58:2 to determine where to place markers for truncation
2023-12-12T14:52:47.563+0800 I STORAGE [initandlisten] Taking 99 samples and assuming that each section of oplog contains approximately 11385 records totaling to 6711221 bytes
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Nov 30 11:40:02:19
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Nov 30 11:44:49:5
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Nov 30 11:49:36:16
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Nov 30 12:05:49:4
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Nov 30 12:11:19:1
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Dec 2 01:47:49:1
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Dec 4 11:26:10:1
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Dec 6 18:49:45:1
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] Placing a marker at optime Dec 9 07:36:41:1
2023-12-12T14:52:47.586+0800 I STORAGE [initandlisten] WiredTiger record store oplog processing took 24ms
2023-12-12T14:52:47.586+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.586+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** Start the server with --bind_ip to specify which IP
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2023-12-12T14:52:47.587+0800 I CONTROL [initandlisten]
2023-12-12T14:52:47.593+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/home/patrick/Downloads/cryosparc/cryosparc_database/diagnostic.data’
2023-12-12T14:52:47.595+0800 I REPL [initandlisten] Rollback ID is 1
2023-12-12T14:52:47.596+0800 I REPL [initandlisten] No oplog entries to apply for recovery. appliedThrough and checkpointTimestamp are both null.
2023-12-12T14:52:47.596+0800 I NETWORK [initandlisten] listening via socket bound to 127.0.0.1
2023-12-12T14:52:47.596+0800 I NETWORK [initandlisten] listening via socket bound to /tmp/mongodb-39001.sock
2023-12-12T14:52:47.596+0800 I NETWORK [initandlisten] waiting for connections on port 39001
2023-12-12T14:52:47.596+0800 I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2023-12-12T14:52:47.596+0800 I REPL [replexec-0] New replica set config in use: { _id: “meteor”, version: 1, protocolVersion: 1, members: [ { _id: 0, host: “localhost:39001”, arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId(‘655c14be8835105e6df90515’) } }
2023-12-12T14:52:47.596+0800 I REPL [replexec-0] This node is localhost:39001 in the config
2023-12-12T14:52:47.596+0800 I REPL [replexec-0] transition to STARTUP2 from STARTUP
2023-12-12T14:52:47.596+0800 I REPL [replexec-0] Starting replication storage threads
2023-12-12T14:52:47.597+0800 I REPL [replexec-0] transition to RECOVERING from STARTUP2
2023-12-12T14:52:47.597+0800 I REPL [replexec-0] Starting replication fetcher thread
2023-12-12T14:52:47.597+0800 I REPL [replexec-0] Starting replication applier thread
2023-12-12T14:52:47.597+0800 I REPL [replexec-0] Starting replication reporter thread
2023-12-12T14:52:47.598+0800 I REPL [rsSync] transition to SECONDARY from RECOVERING
2023-12-12T14:52:47.598+0800 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 18
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] dry election run succeeded, running for election in term 19
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] election succeeded, assuming primary role in term 19
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] transition to PRIMARY from SECONDARY
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] Resetting sync source to empty, which was :27017
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] Entering primary catch-up mode.
2023-12-12T14:52:47.598+0800 I REPL [replexec-0] Exited primary catch-up mode.
2023-12-12T14:52:49.599+0800 I REPL [rsSync] transition to primary complete; database writes are now permitted
2023-12-12T14:52:49.891+0800 I NETWORK [listener] connection accepted from 127.0.0.1:48772 #1 (1 connection now open)
2023-12-12T14:52:49.891+0800 I NETWORK [conn1] received client metadata from 127.0.0.1:48772 conn1: { driver: { name: “PyMongo”, version: “3.13.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.14.0-390.el9.x86_64” }, platform: “CPython 3.8.17.final.0” }
2023-12-12T14:52:49.892+0800 I NETWORK [listener] connection accepted from 127.0.0.1:48788 #2 (2 connections now open)
2023-12-12T14:52:49.892+0800 I NETWORK [conn2] received client metadata from 127.0.0.1:48788 conn2: { driver: { name: “PyMongo”, version: “3.13.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.14.0-390.el9.x86_64” }, platform: “CPython 3.8.17.final.0” }
2023-12-12T14:52:49.911+0800 I NETWORK [conn2] end connection 127.0.0.1:48788 (1 connection now open)
2023-12-12T14:52:49.911+0800 I NETWORK [conn1] end connection 127.0.0.1:48772 (0 connections now open)
2023-12-12T14:52:49.913+0800 I NETWORK [listener] connection accepted from 127.0.0.1:48804 #3 (1 connection now open)
2023-12-12T14:52:49.913+0800 I NETWORK [conn3] received client metadata from 127.0.0.1:48804 conn3: { driver: { name: “PyMongo”, version: “3.13.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.14.0-390.el9.x86_64” }, platform: “CPython 3.8.17.final.0” }
2023-12-12T14:52:49.913+0800 I NETWORK [listener] connection accepted from 127.0.0.1:48808 #4 (2 connections now open)
2023-12-12T14:52:49.913+0800 I NETWORK [conn4] received client metadata from 127.0.0.1:48808 conn4: { driver: { name: “PyMongo”, version: “3.13.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.14.0-390.el9.x86_64” }, platform: “CPython 3.8.17.final.0” }
2023-12-12T14:52:49.933+0800 I NETWORK [conn4] end connection 127.0.0.1:48808 (1 connection now open)
2023-12-12T14:52:49.933+0800 I NETWORK [conn3] end connection 127.0.0.1:48804 (0 connections now open)
2023-12-12T14:52:49.933+0800 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2023-12-12T14:52:49.933+0800 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2023-12-12T14:52:49.933+0800 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] shutting down replication subsystems
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication reporter thread
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication fetcher thread
2023-12-12T14:52:49.934+0800 I REPL [signalProcessingThread] Stopping replication applier thread
6

$ ls -l /tmp/.sock
ls: cannot access '/tmp/.sock’: No such file or directory

cannot display the asterisk

7

firewalld was stopped and disabled.

8

I cannot (yet) resolve the original problem (the startup error), for which I will ask our team.

The “missing” asterisks are probably due to the asterisks’ interpretation as markdown control characters (details).
You could try showing the command and its output in a code block:


```
$ ls -l /tmp/*.sock
ls: cannot access '/tmp/*.sock': No such file or directory

```
9

Thanks~

*Thank you again*
10

@biolish Our team is investigating a bug that may be related to the problem you reported.

A possible workaround, assuming the database is already configured, may be to

  1. ensure that CryoSPARC is stopped completely
  2. modify the file
    /home/patrick/Downloads/cryosparc/cryosparc_master/bin/cryosparcm
    by commenting out the command (on line 978 on CryoSPARC v4.4.1)
    cryosparcm configuredb
  3. try cryosparcm start
    This can be done on your installation with the command
    sed -i.bak20231214 '978s/\(^\s*cryosparcm\s\+configuredb\)/#\1/' /home/patrick/Downloads/cryosparc/cryosparc_master/bin/cryosparcm

Does this work?

11

Hi wtempel,

the cryosparc is completely stopped.
the I modified the cryosparcm file as you said, it looks like this:

978 #     cryosparcm configuredb

then run cryosparc start:

$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
/home/patrick/Downloads/cryosparc/cryosparc_master/bin/cryosparcm: line 363: 178451 Segmentation fault      (core dumped) command python -c "from cryosparc_compute import database_management; database_management.check_mongo()"
[2023-12-15T11:03:08+08:00] Error checking database. Most recent database log lines:
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] dry election run succeeded, running for election in term 21
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] election succeeded, assuming primary role in term 21
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] transition to PRIMARY from SECONDARY
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] Resetting sync source to empty, which was :27017
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] Entering primary catch-up mode.
2023-12-15T11:03:07.398+0800 I REPL     [replexec-0] Exited primary catch-up mode.
2023-12-15T11:03:08.399+0800 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2023-12-15T11:03:08.545+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:58972 #1 (1 connection now open)
2023-12-15T11:03:08.555+0800 I NETWORK  [conn1] received client metadata from 127.0.0.1:58972 conn1: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.14.0-390.el9.x86_64" }, platform: "CPython 3.8.17.final.0" }
2023-12-15T11:03:08.563+0800 I NETWORK  [conn1] end connection 127.0.0.1:58972 (0 connections now open)

then I realized there is a #, I tried removing it,

978     cryosparcm configuredb

the check all the processes, make sure cryosparc completely stopped, then

$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
/home/patrick/Downloads/cryosparc/cryosparc_master/bin/cryosparcm: line 363: 178649 Segmentation fault      (core dumped) command python -c "from cryosparc_compute import database_management; database_management.configure_mongo(logfile='${CRYOSPARC_ROOT_DIR}/run/database.log')"
[2023-12-15T11:06:42+08:00] Error configuring database. Most recent database log lines:
2023-12-15T11:06:42.076+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:47228 #1 (1 connection now open)
2023-12-15T11:06:42.081+0800 I NETWORK  [conn1] received client metadata from 127.0.0.1:47228 conn1: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.14.0-390.el9.x86_64" }, platform: "CPython 3.8.17.final.0" }
2023-12-15T11:06:42.082+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:47242 #2 (2 connections now open)
2023-12-15T11:06:42.086+0800 I NETWORK  [conn2] received client metadata from 127.0.0.1:47242 conn2: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.14.0-390.el9.x86_64" }, platform: "CPython 3.8.17.final.0" }
2023-12-15T11:06:42.103+0800 I NETWORK  [conn1] end connection 127.0.0.1:47228 (1 connection now open)
2023-12-15T11:06:42.103+0800 I NETWORK  [conn2] end connection 127.0.0.1:47242 (0 connections now open)
2023-12-15T11:06:42.108+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:47256 #3 (1 connection now open)
2023-12-15T11:06:42.113+0800 I NETWORK  [conn3] received client metadata from 127.0.0.1:47256 conn3: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "5.14.0-390.el9.x86_64" }, platform: "CPython 3.8.17.final.0" }
2023-12-15T11:06:42.279+0800 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2023-12-15T11:06:42.654+0800 I NETWORK  [conn3] end connection 127.0.0.1:47256 (0 connections now open)

there’s some info might be useful:

we have 2 centos stream 9 server, with the same error of cryosparc,
but on another centos stream 8 server, everything works well.

12

Please can you confirm that the number “978” is not inside the file, but only shown here to indicate the line number?

This probably is useful info. I have not tested CryoSPARC on CentOS stream. Did you consider a more “feature-stable” distribution like Ubuntu-LTS (22.04) or Rocky (9)?

13

Hi wtempel,

yes the number 978 was added by me. it’s not in the file.

i’ll try ubuntu or something else later.

many thanks.

14

Hi, wtempel,

I also suffered from same problem with cryosparc v4.4.1 in CentOS stream 9 server, but not in CentOS stream 7 server.

I roll back to v4.4.0 in CentOS 9, but the error is persisted.

I will try to remove the cryosparc and install again.

15

A non-“stream” Linux distribution with a longer support cycle, like Ubuntu-22.04 or Rocky 9, may work better.