Cryosparc cannot connect to database "Could not get database status" after restart

bioxyuan · September 10, 2025, 5:17pm

Cryosparc cannot connect to database “Could not get database status” after restart. Noticed that the IP address changed, Could it be the cause? see below the output:

(base) (base) [ruan@unicorn ~]$ Starting CryoSPARC System master process…
CryoSPARC is not already running.
configuring database…
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
File “”, line 1, in
File “/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py”, line 47, in configure_mongo
initialize_replica_set()
File “/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py”, line 84, in initialize_replica_set
admin_db = try_get_pymongo_db(mongo_client)
File “/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py”, line 251, in try_get_pymongo_db
admin_db.command(({‘serverStatus’: 1}))
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py”, line 108, in csot_wrapper
return func(self, *args, **kwargs)
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py”, line 893, in command
with self.__client._conn_for_reads(read_preference, session, operation=command_name) as (
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py”, line 1375, in _conn_for_reads
server = self._select_server(read_preference, session, operation)
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py”, line 1322, in _select_server
server = topology.select_server(
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 368, in select_server
server = self._select_server(
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 346, in _select_server
servers = self.select_servers(
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 253, in select_servers
server_descriptions = self._select_servers_loop(
File “/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 303, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 20.0s, Topology Description: <TopologyDescription id: 68c1b1e14f7e0723e78db871, topology_type: Single, servers: [<ServerDescription (‘localhost’, 39001) server_type: Unknown, rtt: None, error=AutoReconnect(‘localhost:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)’)>]>
[2025-09-10T13:15:20-0400] Error configuring database. Most recent database log lines:
2025-09-10T13:15:20.435-0400 I REPL [signalProcessingThread] Stopping replication reporter thread
2025-09-10T13:15:20.435-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread
2025-09-10T13:15:20.435-0400 I REPL [signalProcessingThread] Stopping replication applier thread
2025-09-10T13:15:20.520-0400 I REPL [signalProcessingThread] Stopping replication storage threads
2025-09-10T13:15:20.521-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2025-09-10T13:15:20.527-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2025-09-10T13:15:20.528-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2025-09-10T13:15:20.605-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2025-09-10T13:15:20.605-0400 I CONTROL [signalProcessingThread] now exiting
2025-09-10T13:15:20.605-0400 I CONTROL [signalProcessingThread] shutting down with code:0

wtempel · September 10, 2025, 5:31pm

Welcome to the forum @bioxyuan . Please can you post the relevant lines from the database log that precede

2025-09-10T13:15:20.435-0400 I REPL [signalProcessingThread] Stopping replication reporter thread

The database log can be accessed with the command

cryosparcm log database

bioxyuan · September 10, 2025, 5:45pm

Here is the output from database log:

2025-09-10T13:15:20.520-0400 I REPL [signalProcessingThread] Stopping replication storage threads
2025-09-10T13:15:20.521-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2025-09-10T13:15:20.527-0400 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2025-09-10T13:15:20.528-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2025-09-10T13:15:20.605-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2025-09-10T13:15:20.605-0400 I CONTROL [signalProcessingThread] now exiting
2025-09-10T13:15:20.605-0400 I CONTROL [signalProcessingThread] shutting down with code:0
2025-09-10T13:28:21.251-0400 I CONTROL [initandlisten] MongoDB starting : pid=16847 port=39001 dbpath=/home/ruan/programs/cryosparc/cryosparc2_database 64-bit host=unicorn
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] db version v3.6.23
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] allocator: tcmalloc
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] modules: none
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] build environment:
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] distarch: x86_64
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] target_arch: x86_64
2025-09-10T13:28:21.252-0400 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/home/ruan/programs/cryosparc/cryosparc2_database” } }
2025-09-10T13:28:21.252-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=128251M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release=“3.0”,require_max=“3.0”),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2025-09-10T13:28:21.739-0400 I STORAGE [initandlisten] WiredTiger message [1757525301:739919][16847:0x7fe97dd8ba40], txn-recover: Set global recovery timestamp: 0
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** Start the server with --bind_ip to specify which IP
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2025-09-10T13:28:21.788-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine.
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options]
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is ‘always’.
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is ‘always’.
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten] ** We suggest setting it to ‘never’
2025-09-10T13:28:21.789-0400 I CONTROL [initandlisten]
2025-09-10T13:28:21.791-0400 I STORAGE [initandlisten] createCollection: local.startup_log with no UUID.
2025-09-10T13:28:21.812-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/home/ruan/programs/cryosparc/cryosparc2_database/diagnostic.data’
2025-09-10T13:28:21.813-0400 I STORAGE [initandlisten] createCollection: local.me with no UUID.
2025-09-10T13:28:21.835-0400 I STORAGE [initandlisten] createCollection: local.replset.minvalid with no UUID.
2025-09-10T13:28:21.856-0400 I STORAGE [initandlisten] createCollection: local.replset.election with no UUID.
2025-09-10T13:28:21.878-0400 I REPL [initandlisten] Did not find local initialized voted for document at startup.
2025-09-10T13:28:21.878-0400 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one.
2025-09-10T13:28:21.878-0400 I STORAGE [initandlisten] createCollection: local.system.rollback.id with no UUID.
2025-09-10T13:28:21.900-0400 I REPL [initandlisten] Initialized the rollback ID to 1
2025-09-10T13:28:21.900-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2025-09-10T13:28:21.901-0400 I NETWORK [initandlisten] listening via socket bound to 127.0.0.1
2025-09-10T13:28:21.901-0400 I NETWORK [initandlisten] listening via socket bound to /tmp/mongodb-39001.sock
2025-09-10T13:28:21.901-0400 I NETWORK [initandlisten] waiting for connections on port 39001
2025-09-10T13:28:21.901-0400 I CONTROL [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
2025-09-10T13:29:35.156-0400 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2025-09-10T13:29:35.161-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2025-09-10T13:29:35.161-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
2025-09-10T13:29:35.161-0400 I REPL [signalProcessingThread] shutdown: removing all drop-pending collections…
2025-09-10T13:29:35.161-0400 I REPL [signalProcessingThread] shutdown: removing checkpointTimestamp collection…
2025-09-10T13:29:35.161-0400 I REPL [signalProcessingThread] shutting down replication subsystems
2025-09-10T13:29:35.162-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2025-09-10T13:29:35.167-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2025-09-10T13:29:35.406-0400 I STORAGE [signalProcessingThread] WiredTiger message [1757525375:406690][16847:0x7fe97c4cb700], txn-recover: Main recovery loop: starting at 1/24960
2025-09-10T13:29:35.487-0400 I STORAGE [signalProcessingThread] WiredTiger message [1757525375:487720][16847:0x7fe97c4cb700], txn-recover: Recovering log 1 through 2
2025-09-10T13:29:35.544-0400 I STORAGE [signalProcessingThread] WiredTiger message [1757525375:543998][16847:0x7fe97c4cb700], txn-recover: Recovering log 2 through 2
2025-09-10T13:29:35.597-0400 I STORAGE [signalProcessingThread] WiredTiger message [1757525375:597016][16847:0x7fe97c4cb700], txn-recover: Set global recovery timestamp: 0
2025-09-10T13:29:35.705-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2025-09-10T13:29:35.705-0400 I CONTROL [signalProcessingThread] now exiting
2025-09-10T13:29:35.705-0400 I CONTROL [signalProcessingThread] shutting down with code:0
Waiting for data… (interrupt to abort)

wtempel · September 10, 2025, 6:16pm

Thanks for posting the lines from the log

Depending on your CryoSPARC and network configurations, disruptions of CryoSPARC may occur.

Please can you post the outputs of these commands

hostname -f
host $(hostname -f)
grep -v LICENSE_ID /home/ruan/programs/cryosparc/cryosparc2_master/config.sh
cat /home/ruan/programs/cryosparc/cryosparc2_master/version
ip -br -4 a

and describe how the ip address is assigned to your computer, and under what circumstances the ip address may change.

bioxyuan · September 10, 2025, 6:25pm

Thank you for the quick response. see the output below:

(base) [ruan@unicorn ~]$ hostname -f
unicorn.uchc.edu
(base) [ruan@unicorn ~]$ host $(hostname -f)
unicorn.uchc.edu has address 10.133.255.214
(base) [ruan@unicorn ~]$ grep -v LICENSE_ID /home/ruan/programs/cryosparc/cryosparc2_master/config.sh

export CRYOSPARC_MASTER_HOSTNAME="unicorn"
export CRYOSPARC_DB_PATH="/home/ruan/programs/cryosparc/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_FORCE_HOSTNAME=true
(base) [ruan@unicorn ~]$ cat /home/ruan/programs/cryosparc/cryosparc2_master/version
v4.6.2
(base) [ruan@unicorn ~]$ ip -br -4 a
lo               UNKNOWN        127.0.0.1/8 
enp5s0           UP             10.133.255.214/16 
virbr0           DOWN           192.168.122.1/24 
(base) [ruan@unicorn ~]$

wtempel · September 10, 2025, 6:49pm

Thanks @bioxyuan .
Please can you

follow the multiple steps for a thorough, confirmed shutdown of CryoSPARC.
try running cryosparcm start. If the startup does not succeed, please post
the error message

from a separate terminal window, outputs of the commands

ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
tail -n 50 /home/ruan/programs/cryosparc/cryosparc2_master/run/database.log
tail -n 50 /home/ruan/programs/cryosparc/cryosparc2_master/run/supervisord.log

details of the address change you mentioned

wtempel:

how the ip address is assigned to your computer, and under what circumstances the ip address may change.

bioxyuan · September 10, 2025, 7:02pm

cryosparcm start failed after I followed the multip steps of cryosparc shut down, below is the outputs:

(base) [ruan@unicorn ~]$ cryosparcm start
Starting CryoSPARC System master process...
CryoSPARC is not already running.
configuring database...
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py", line 47, in configure_mongo
    initialize_replica_set()
  File "/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py", line 84, in initialize_replica_set
    admin_db = try_get_pymongo_db(mongo_client)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py", line 251, in try_get_pymongo_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py", line 893, in command
    with self.__client._conn_for_reads(read_preference, session, operation=command_name) as (
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1375, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1322, in _select_server
    server = topology.select_server(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 368, in select_server
    server = self._select_server(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 346, in _select_server
    servers = self.select_servers(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 253, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 303, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 20.0s, Topology Description: <TopologyDescription id: 68c1c9eb0319a8db0671bd4c, topology_type: Single, servers: [<ServerDescription ('localhost', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>
[2025-09-10T14:57:54-0400] Error configuring database. Most recent database log lines:
2025-09-10T14:57:53.858-0400 I REPL     [signalProcessingThread] shutting down replication subsystems
2025-09-10T14:57:53.858-0400 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2025-09-10T14:57:53.863-0400 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2025-09-10T14:57:54.160-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:160014][20883:0x7fc75a195700], txn-recover: Main recovery loop: starting at 4/5760
2025-09-10T14:57:54.253-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:253209][20883:0x7fc75a195700], txn-recover: Recovering log 4 through 5
2025-09-10T14:57:54.315-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:315882][20883:0x7fc75a195700], txn-recover: Recovering log 5 through 5
2025-09-10T14:57:54.369-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:369950][20883:0x7fc75a195700], txn-recover: Set global recovery timestamp: 0
2025-09-10T14:57:54.465-0400 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2025-09-10T14:57:54.465-0400 I CONTROL  [signalProcessingThread] now exiting
2025-09-10T14:57:54.465-0400 I CONTROL  [signalProcessingThread] shutting down with code:0

Then I run the commands provided, see the outputs below:

(base) [ruan@unicorn ~]$ ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo
ruan      20856      1 14:56:38 python /home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/ruan/programs/cryosparc/cryosparc2_master/supervisord.conf
ruan      21635  20950 14:59:32 grep --color=auto -e cryosparc_ -e mongo
(base) [ruan@unicorn ~]$ ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
ls: cannot access /tmp/mongo*.sock: No such file or directory
srwx------ 1 ruan ruan 0 Sep 10 14:56 /tmp/cryosparc-supervisor-5af56541fa4e1b5aac46299dfe93cb5c.sock
(base) [ruan@unicorn ~]$ tail -n 50 /home/ruan/programs/cryosparc/cryosparc2_master/run/database.log
2025-09-10T14:56:40.223-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=128251M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2025-09-10T14:56:40.807-0400 I STORAGE  [initandlisten] WiredTiger message [1757530600:807045][20883:0x7fc75ba55a40], txn-recover: Main recovery loop: starting at 3/768
2025-09-10T14:56:40.889-0400 I STORAGE  [initandlisten] WiredTiger message [1757530600:889457][20883:0x7fc75ba55a40], txn-recover: Recovering log 3 through 4
2025-09-10T14:56:40.945-0400 I STORAGE  [initandlisten] WiredTiger message [1757530600:945149][20883:0x7fc75ba55a40], txn-recover: Recovering log 4 through 4
2025-09-10T14:56:40.997-0400 I STORAGE  [initandlisten] WiredTiger message [1757530600:997410][20883:0x7fc75ba55a40], txn-recover: Set global recovery timestamp: 0
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server. 
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP 
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2025-09-10T14:56:41.035-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2025-09-10T14:56:41.036-0400 I CONTROL  [initandlisten] 
2025-09-10T14:56:41.041-0400 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/home/ruan/programs/cryosparc/cryosparc2_database/diagnostic.data'
2025-09-10T14:56:41.045-0400 I REPL     [initandlisten] Did not find local initialized voted for document at startup.
2025-09-10T14:56:41.047-0400 I REPL     [initandlisten] Rollback ID is 1
2025-09-10T14:56:41.048-0400 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2025-09-10T14:56:41.048-0400 I NETWORK  [initandlisten] listening via socket bound to 127.0.0.1
2025-09-10T14:56:41.048-0400 I NETWORK  [initandlisten] listening via socket bound to /tmp/mongodb-39001.sock
2025-09-10T14:56:41.048-0400 I NETWORK  [initandlisten] waiting for connections on port 39001
2025-09-10T14:56:41.048-0400 I CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
2025-09-10T14:57:53.857-0400 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2025-09-10T14:57:53.857-0400 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2025-09-10T14:57:53.858-0400 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
2025-09-10T14:57:53.858-0400 I REPL     [signalProcessingThread] shutdown: removing all drop-pending collections...
2025-09-10T14:57:53.858-0400 I REPL     [signalProcessingThread] shutdown: removing checkpointTimestamp collection...
2025-09-10T14:57:53.858-0400 I REPL     [signalProcessingThread] shutting down replication subsystems
2025-09-10T14:57:53.858-0400 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2025-09-10T14:57:53.863-0400 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2025-09-10T14:57:54.160-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:160014][20883:0x7fc75a195700], txn-recover: Main recovery loop: starting at 4/5760
2025-09-10T14:57:54.253-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:253209][20883:0x7fc75a195700], txn-recover: Recovering log 4 through 5
2025-09-10T14:57:54.315-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:315882][20883:0x7fc75a195700], txn-recover: Recovering log 5 through 5
2025-09-10T14:57:54.369-0400 I STORAGE  [signalProcessingThread] WiredTiger message [1757530674:369950][20883:0x7fc75a195700], txn-recover: Set global recovery timestamp: 0
2025-09-10T14:57:54.465-0400 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2025-09-10T14:57:54.465-0400 I CONTROL  [signalProcessingThread] now exiting
2025-09-10T14:57:54.465-0400 I CONTROL  [signalProcessingThread] shutting down with code:0
(base) [ruan@unicorn ~]$ tail -n 50 /home/ruan/programs/cryosparc/cryosparc2_master/run/supervisord.log
2025-09-09 18:09:54,011 INFO daemonizing the supervisord process
2025-09-09 18:09:54,015 INFO supervisord started with pid 10963
2025-09-09 18:18:22,828 INFO RPC interface 'supervisor' initialized
2025-09-09 18:18:22,828 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-09 18:18:22,830 INFO daemonizing the supervisord process
2025-09-09 18:18:22,833 INFO supervisord started with pid 12859
2025-09-09 20:24:18,744 INFO RPC interface 'supervisor' initialized
2025-09-09 20:24:18,744 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-09 20:24:18,746 INFO daemonizing the supervisord process
2025-09-09 20:24:18,750 INFO supervisord started with pid 18523
2025-09-09 21:48:50,279 INFO RPC interface 'supervisor' initialized
2025-09-09 21:48:50,279 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-09 21:48:50,282 INFO daemonizing the supervisord process
2025-09-09 21:48:50,285 INFO supervisord started with pid 21876
2025-09-09 21:51:51,666 INFO RPC interface 'supervisor' initialized
2025-09-09 21:51:51,666 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-09 21:51:51,668 INFO daemonizing the supervisord process
2025-09-09 21:51:51,672 INFO supervisord started with pid 22334
2025-09-10 09:12:28,429 INFO RPC interface 'supervisor' initialized
2025-09-10 09:12:28,429 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 09:12:28,431 INFO daemonizing the supervisord process
2025-09-10 09:12:28,435 INFO supervisord started with pid 48248
2025-09-10 11:25:57,330 INFO RPC interface 'supervisor' initialized
2025-09-10 11:25:57,330 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 11:25:57,333 INFO daemonizing the supervisord process
2025-09-10 11:25:57,337 INFO supervisord started with pid 5945
2025-09-10 11:39:40,011 INFO RPC interface 'supervisor' initialized
2025-09-10 11:39:40,011 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 11:39:40,013 INFO daemonizing the supervisord process
2025-09-10 11:39:40,016 INFO supervisord started with pid 8116
2025-09-10 12:44:47,426 INFO RPC interface 'supervisor' initialized
2025-09-10 12:44:47,426 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 12:44:47,428 INFO daemonizing the supervisord process
2025-09-10 12:44:47,432 INFO supervisord started with pid 12698
2025-09-10 12:56:51,056 INFO RPC interface 'supervisor' initialized
2025-09-10 12:56:51,056 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 12:56:51,058 INFO daemonizing the supervisord process
2025-09-10 12:56:51,061 INFO supervisord started with pid 13770
2025-09-10 13:14:06,001 INFO RPC interface 'supervisor' initialized
2025-09-10 13:14:06,001 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 13:14:06,003 INFO daemonizing the supervisord process
2025-09-10 13:14:06,007 INFO supervisord started with pid 16021
2025-09-10 13:28:20,558 INFO RPC interface 'supervisor' initialized
2025-09-10 13:28:20,558 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 13:28:20,560 INFO daemonizing the supervisord process
2025-09-10 13:28:20,564 INFO supervisord started with pid 16820
2025-09-10 14:56:39,545 INFO RPC interface 'supervisor' initialized
2025-09-10 14:56:39,545 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-09-10 14:56:39,547 INFO daemonizing the supervisord process
2025-09-10 14:56:39,551 INFO supervisord started with pid 20856

For the IP address changing, I run ifconfig:

enp4s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether a8:5e:45:3e:6b:6b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x9c300000-9c37ffff  

enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet **10.133.255.214**  netmask 255.255.0.0  broadcast 10.133.255.255
        inet6 fe80::aa5e:45ff:fe3e:6b6c  prefixlen 64  scopeid 0x20<link>
        ether a8:5e:45:3e:6b:6c  txqueuelen 1000  (Ethernet)
        RX packets 64984  bytes 17788076 (16.9 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 20614  bytes 3647120 (3.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x9c200000-9c27ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 255  bytes 22346 (21.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 255  bytes 22346 (21.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:38:0a:99  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The bold IP change from 10.23.10.22 to current 10.133.255.214 because of the insititute network switch

wtempel · September 10, 2025, 7:47pm

Thanks. I am not sure what’s causing

localhost:39001: [Errno 111] Connection refused

Please can you post the outputs of these commands, just to confirm a few assumptions:

host localhost
curl localhost:22
curl 127.0.0.1:22
/home/ruan/programs/cryosparc/cryosparc2_master/bin/cryosparcm call env | grep -i proxy

bioxyuan · September 10, 2025, 7:49pm

here we go:

(base) [ruan@unicorn ~]$ host localhost

localhost.uchc.edu has address 10.140.50.243

(base) [ruan@unicorn ~]$ curl localhost:22

curl: (7) Failed connect to localhost:22; Connection refused

(base) [ruan@unicorn ~]$ curl 127.0.0.1:22

SSH-2.0-OpenSSH_7.4

Protocol mismatch.

curl: (56) Recv failure: Connection reset by peer

(base) [ruan@unicorn ~]$ /home/ruan/programs/cryosparc/cryosparc2_master/bin/cryosparcm call env | grep -i proxy

no_proxy=127.0.0.1, localhost

(base) [ruan@unicorn ~]$

wtempel · September 10, 2025, 8:52pm

This resolution deviates from a popular notion of localhost and may interfere with CryoSPARC function. Would it be possible to reconfigure the resolution such that localhost resolves to 127.0.0.1? The details/specific steps depend on your system configuration and network policies.
The appropriate reconfiguration may additionally depend on whether this CryoSPARC installation includes worker nodes external to the CryoSPARC master host.

bioxyuan · September 10, 2025, 9:27pm

Thank you. I tried to reconfigure the resolution by adding the following lines to the hosts file:

127.0.0.1       localhost localhost.localdomain localhost.uchc.edu
::1             localhost localhost.localdomain localhost.uchc.edu

10.140.50.243   unicorn.uchc.edu unicorn

then restart the cryotsparc, now I got:

(base) [ruan@unicorn ~]$ cryosparcm start&
[1] 28434
(base) [ruan@unicorn ~]$ Starting CryoSPARC System master process...
CryoSPARC is not already running.
configuring database...
    creating cryosparc_admin
    cryosparc_admin created
    creating cryosparc_user
    cryosparc_user created
    configuration complete
database: started
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Could not get replica set status; please reconfigure the database with `cryosparcm configuredb`
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py", line 270, in check_mongo
    admin_db = try_get_pymongo_db(mongo_client)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/cryosparc_compute/database_management.py", line 251, in try_get_pymongo_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py", line 893, in command
    with self.__client._conn_for_reads(read_preference, session, operation=command_name) as (
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1375, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1322, in _select_server
    server = topology.select_server(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 368, in select_server
    server = self._select_server(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 346, in _select_server
    servers = self.select_servers(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 253, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/home/ruan/programs/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py", line 303, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: unicorn:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 20.0s, Topology Description: <TopologyDescription id: 68c1ec6f4f94b3678cd917d5, topology_type: Unknown, servers: [<ServerDescription ('unicorn', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('unicorn:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>
[2025-09-10T17:25:09-0400] Error checking database. Most recent database log lines:
2025-09-10T17:23:57.577-0400 I REPL     [replexec-0] Starting replication reporter thread
2025-09-10T17:23:57.578-0400 I REPL     [rsSync] transition to SECONDARY from RECOVERING
2025-09-10T17:23:57.578-0400 I REPL     [rsSync] conducting a dry run election to see if we could be elected. current term: 1
2025-09-10T17:23:57.578-0400 I REPL     [replexec-0] dry election run succeeded, running for election in term 2
2025-09-10T17:23:57.580-0400 I REPL     [replexec-0] election succeeded, assuming primary role in term 2
2025-09-10T17:23:57.580-0400 I REPL     [replexec-0] transition to PRIMARY from SECONDARY
2025-09-10T17:23:57.580-0400 I REPL     [replexec-0] Resetting sync source to empty, which was :27017
2025-09-10T17:23:57.580-0400 I REPL     [replexec-0] Entering primary catch-up mode.
2025-09-10T17:23:57.580-0400 I REPL     [replexec-0] Exited primary catch-up mode.
2025-09-10T17:23:59.580-0400 I REPL     [rsSync] transition to primary complete; database writes are now permitted

looks like, we can configure the database, but start failed.

wtempel · September 10, 2025, 9:45pm

Is not this IP address outdated?

bioxyuan · September 10, 2025, 10:05pm

sorry I made a mistake here. current IP address is 10.133.255.214. I correct this, and now it is able to start. However, I was told that the user is not exist

wtempel · September 10, 2025, 10:09pm

Excellent.

Please can you open a new topic for this question, and provide details, including the command or UI interaction that triggered the error.

bioxyuan · September 10, 2025, 10:44pm

I have created a new topic for the user not existing issue.