Cryosparc cannot connect to database following restart: "Could not get database status"

zochert · July 29, 2024, 6:26pm

Good afternoon,

Last week I downloaded and set up Cryosparc for a single workstation, and was able to successfully use the software (and shut down/restart the computer) for several days. However, as of today whenever I attempt to start the software using ‘cryosparcm start’, I receive an error message stating that it ‘could not get database status’ after three attempts. Rebooting has not resolved the error, nor has using ‘cryosparcm configuredb’ as suggested in the traceback.
Any advice or direction would be very welcome!

Starting CryoSPARC System master process…
CryoSPARC is not already running.
configuring database…
configuration complete
database: started
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Could not get replica set status; please reconfigure the database with cryosparcm configuredb
Traceback (most recent call last):
File “”, line 1, in
File “/home/zocher/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 270, in check_mongo
admin_db = try_get_pymongo_db(mongo_client)
File “/home/zocher/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 251, in try_get_pymongo_db
admin_db.command(({‘serverStatus’: 1}))
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py”, line 107, in csot_wrapper
return func(self, *args, **kwargs)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py”, line 890, in command
with self.__client._conn_for_reads(read_preference, session) as (
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py”, line 1346, in _conn_for_reads
server = self._select_server(read_preference, session)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py”, line 1303, in _select_server
server = topology.select_server(server_selector)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 302, in select_server
server = self._select_server(selector, server_selection_timeout, address)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 286, in _select_server
servers = self.select_servers(selector, server_selection_timeout, address)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 237, in select_servers
server_descriptions = self._select_servers_loop(selector, server_timeout, address)
File “/home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py”, line 259, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: pop-os.sickkids.ca:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 20.0s, Topology Description: <TopologyDescription id: 66a7da28b6904f5a914c754e, topology_type: Unknown, servers: [<ServerDescription (‘pop-os.sickkids.ca’, 39001) server_type: Unknown, rtt: None, error=AutoReconnect(‘pop-os.sickkids.ca:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)’)>]>
[2024-07-29T14:07:43-04:00] Error checking database. Most recent database log lines:
2024-07-29T14:06:31.143-0400 I REPL [replexec-0] Starting replication reporter thread
2024-07-29T14:06:31.143-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING
2024-07-29T14:06:31.144-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 6
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] dry election run succeeded, running for election in term 7
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] election succeeded, assuming primary role in term 7
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] transition to PRIMARY from SECONDARY
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] Resetting sync source to empty, which was :27017
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] Entering primary catch-up mode.
2024-07-29T14:06:31.144-0400 I REPL [replexec-0] Exited primary catch-up mode.
2024-07-29T14:06:33.145-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted

wtempel · July 30, 2024, 7:00pm

Welcome to the forum @zochert.
Please can you try the following commands (and post their outputs) when CryoSPARC is in the state that corresponds to the output you posted (incomplete startup: database started):

cryosparcm status
grep HOSTNAME /home/zocher/cryosparc/cryosparc_master/config.sh
hostname -f
host $(hostname -f)
grep ^127 /etc/hosts
curl pop-os.sickkids.ca:39001
curl 127.0.0.1:39001
ps -eo user,pid,ppid,start,command | grep -e cryosparc_ -e mongo
ls -l /tmp/cryosparc*sock /tmp/mongo*sock

zochert · August 1, 2024, 6:48pm

Hello,

Here are the outputs of all the requested commands. ‘Cryosparcm status’ gives the readout listed and then idles until manually cancelled.

zocher@pop-os:~/cryosparc/cryosparc_master$ /home/zocher/cryosparc/cryosparc_master/bin/cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/zocher/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_api                          STOPPED   Not started
app_api_dev                      STOPPED   Not started
command_core                     STOPPED   Not started
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         RUNNING   pid 7449, uptime 0:04:24

----------------------------------------------------------------------------
^C
zocher@pop-os:~/cryosparc/cryosparc_master$ grep HOSTNAME /home/zocher/cryosparc/cryosparc_master/config.sh
export CRYOSPARC_MASTER_HOSTNAME="pop-os.sickkids.ca"
zocher@pop-os:~/cryosparc/cryosparc_master$ hostname -f
pop-os.sickkids.ca
zocher@pop-os:~/cryosparc/cryosparc_master$ host $(hostname -f)
pop-os.sickkids.ca has address 172.26.116.236
zocher@pop-os:~/cryosparc/cryosparc_master$ grep ^127 /etc/hosts
127.0.0.1	localhost
zocher@pop-os:~/cryosparc/cryosparc_master$ curl pop-os.sickkids.ca:39001
curl: (7) Failed to connect to pop-os.sickkids.ca port 39001 after 116 ms: Connection refused
zocher@pop-os:~/cryosparc/cryosparc_master$ curl 127.0.0.1:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
zocher@pop-os:~/cryosparc/cryosparc_master$ ps -eo user,pid,ppid,start,command | grep -e cryosparc_ -e mongo
zocher      7345    2031 14:37:53 python /home/zocher/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/zocher/cryosparc/cryosparc_master/supervisord.conf
zocher      7449    7345 14:38:00 mongod --auth --dbpath /home/zocher/cryosparc/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
zocher      7860    2726 14:45:02 grep --color=auto -e cryosparc_ -e mongo
zocher@pop-os:~/cryosparc/cryosparc_master$ ls -l /tmp/cryosparc*sock /tmp/mongo*sock
srwx------ 1 zocher zocher 0 Aug  1 14:37 /tmp/cryosparc-supervisor-410c32c67bb9a0cedcc5648319c073f9.sock
srwx------ 1 zocher zocher 0 Aug  1 14:38 /tmp/mongodb-39001.sock

I will note that since I last posted I attempted a full reinstall, which worked for about 24 hours before failing once more.

wtempel · August 1, 2024, 7:28pm

A firewall or other component of you network may block non-loopback access to the CryoSPARC database port.
If

this is a --standalone CryoSPARC installation without additional worker cluster or node(s)
and no other software on your computer would be “confused” by the proposed change

you can try adding the line
127.0.0.99 pop-os.sickkids.ca
to the /etc/hosts file and restarting CryoSPARC.
This (possible) fix would not work if additional worker nodes are connected to this CryoSPARC instance.
Does this help?

zochert · August 1, 2024, 8:05pm

Thank you for your help-- this immediately solved my problem! I am using a single-workstation setup for Cryosparc and it’s unlikely we’ll ever need to add a second worker node, so this fix works wonderfully. I really appreciate your advice!