Error starting cryosparc: "Could not get database status"

Install, Configure and Update
1

Hello,
I have been using cryosparc on a standalone workstation for some time. For an unrelated issue, I had to restart the computer. Upon running ‘cryosparc start’ I get an error that the system could not get database status (after 3 attempts) and that I should run ‘cryosparcm configuredb’. I have tried, after stopping cryosparc, and it seems to run to completion saying ‘configuration complete’ but I still have the same error when I run ‘cryosparcm start’. The full traceback is below. I tried rebooting and a fresh install but that did not solve the problem. Any advice would be welcome.
Thanks,
Eric Gibbs

From cryosparcm start:
Starting cryoSPARC System master process…
CryoSPARC is not already running.
configuring database
** creating cryosparc_admin**
** cryosparc_admin created**
** creating cryosparc_user**
** cryosparc_user created**
** configuration complete**
database: started
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
checkdb error - could not get replica set status; please reconfigure the database with cryosparcm configuredb
Traceback (most recent call last):
** File “”, line 1, in **
** File “/home/beta/Applications/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 268, in check_mongo**
** admin_db = try_get_pymongo_admin_db(mongo_client)**
** File “/home/beta/Applications/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 249, in try_get_pymongo_admin_db**
** admin_db.command(({‘serverStatus’: 1}))**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py”, line 827, in command**
** with self.__client._socket_for_reads(read_preference, session) as (sock_info, secondary_ok):**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/contextlib.py”, line 113, in enter**
** return next(self.gen)**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 1478, in _socket_for_reads**
** server = self._select_server(read_preference, session)**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 1436, in _select_server**
** server = topology.select_server(server_selector)**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 250, in select_server**
** return random.choice(self.select_servers(selector, server_selection_timeout, address))**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 211, in select_servers**
** server_descriptions = self._select_servers_loop(selector, server_timeout, address)**
** File “/home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 226, in _select_servers_loop**
** raise ServerSelectionTimeoutError(**
pymongo.errors.ServerSelectionTimeoutError: beta.ugen.cwru.edu:47001: timed out, Timeout: 20.0s, Topology Description: <TopologyDescription id: 6425dd1f72e51a9c19cbdc55, topology_type: Unknown, servers: [<ServerDescription (‘beta.ugen.cwru.edu’, 47001) server_type: Unknown, rtt: None, error=NetworkTimeout(‘beta.ugen.cwru.edu:47001: timed out’)>]>
[2023-03-30T15:05:11-0400] Error checking database. Most recent database log lines:
2023-03-30T15:03:58.033-0400 I REPL [replexec-0] Starting replication reporter thread
2023-03-30T15:03:58.034-0400 I REPL [rsSync] transition to SECONDARY from RECOVERING
2023-03-30T15:03:58.034-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 1
2023-03-30T15:03:58.034-0400 I REPL [replexec-0] dry election run succeeded, running for election in term 2
2023-03-30T15:03:58.045-0400 I REPL [replexec-0] election succeeded, assuming primary role in term 2
2023-03-30T15:03:58.045-0400 I REPL [replexec-0] transition to PRIMARY from SECONDARY
2023-03-30T15:03:58.045-0400 I REPL [replexec-0] Resetting sync source to empty, which was :27017
2023-03-30T15:03:58.045-0400 I REPL [replexec-0] Entering primary catch-up mode.
2023-03-30T15:03:58.045-0400 I REPL [replexec-0] Exited primary catch-up mode.
2023-03-30T15:04:00.037-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted

2

Please can you run these commands

whoami
stat /home/beta/Applications/cryosparc/cryosparc_master/bin/cryosparcm

to ensure that the active user and file ownershp matches and
post the output of:

hostname
grep HOSTNAME /home/beta/Applications/cryosparc/cryosparc_master/config.sh
cryosparcm stop
ps xww | grep -e cryosparc -e mongo
3

Here are the outputs of the first suggestion. It seems that it matches to me…:
(base) [beta@beta GlyRabDE_200uMStryFirst_20uMIvm]$ whoami
beta
(base) [beta@beta GlyRabDE_200uMStryFirst_20uMIvm]$ stat /home/beta/Applications/cryosparc/cryosparc_master/bin/cryosparcm
File: ‘/home/beta/Applications/cryosparc/cryosparc_master/bin/cryosparcm’
Size: 76436 Blocks: 152 IO Block: 4096 regular file
Device: 802h/2050d Inode: 405426303 Links: 1
Access: (0775/-rwxrwxr-x) Uid: ( 1000/ beta) Gid: ( 1000/ beta)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2023-03-30 13:23:10.349271334 -0400
Modify: 2023-03-15 10:08:38.000000000 -0400
Change: 2023-03-30 13:16:13.791152166 -0400
Birth: -

Here is the output of the latter portion:
(base) [beta@beta GlyRabDE_200uMStryFirst_20uMIvm]$ grep HOSTNAME /home/beta/Applications/cryosparc/cryosparc_master/config.sh
export CRYOSPARC_MASTER_HOSTNAME=“beta.UGEN.cwru.edu”
(base) [beta@beta GlyRabDE_200uMStryFirst_20uMIvm]$ cryosparcm stop
CryoSPARC is running.
Stopping cryoSPARC
database: stopped
Shut down
(base) [beta@beta GlyRabDE_200uMStryFirst_20uMIvm]$ ps xww | grep -e cryosparc -e mongo
78713 pts/1 S+ 0:00 grep --color=auto -e cryosparc -e mongo

4

Thanks @edg42 These outputs rule out some possible causes. I’ll ask within our team what other causes there might be.

5

Next tests could be (as beta@beta):

  1. cryosparcm start
  2. record outputs of (in a separate shell window, in case the previous command stalled)
    • ps xww | grep -e cryosparc -e mongo
    • curl localhost:47001
    • curl beta.ugen.cwru.edu:47001
    • host beta.ugen.cwru.edu
6

Here are the results from those. I don’t know if this is indicative of a problem, but the host IP address does not match the address I use to connect to the workstation via ssh.

[beta@beta ~]\$ ps xww | grep -e cryosparc -e mongo
122950 ?        Ss     0:00 python /home/beta/Applications/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/beta/Applications/cryosparc/cryosparc_master/supervisord.conf
123061 ?        Sl     0:01 mongod --auth --dbpath /home/beta/Applications/cryosparc/cryosparc_database --port 47001 --oplogSize 64 --replSet meteor --nojournal --wiredTigerCacheSizeGB 4 --bind_ip_all
123173 pts/0    S+     0:00 grep --color=auto -e cryosparc -e mongo
(topaz) [beta@beta ~]\$ curl localhost:47001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
(topaz) [beta@beta ~]\$ curl beta.ugen.cwru.edu:47001
curl: (28) Failed to connect to beta.ugen.cwru.edu port 47001: Connection timed out
(topaz) [beta@beta ~]\$ host beta.ugen.cwru.edu
beta.UGEN.cwru.edu has address 129.22.182.189
(topaz) [beta@beta ~]\$ curl 129.22.194.57:47000
7

Any further suggestions?

8

I was able to complete installation by adding the flag --hostname and the IP address that I use for ssh to the installation command. I then had to add “export CRYOSPARC_FORCE_HOSTNAME=true” to the master config.sh file. This allowed it to start but didn’t assign a user so I had to do cryosparcm createuser etc.

It seems there was some disconnect between the hostname and the actual ip address. I don’t know what caused it, my guess is it is a server thing on my end. Above my expertise. Hopefully this is helpful to anyone else facing a similar issue.

I also had to reattach my projects.

9

The drawback is that your installation may break again if the server is assigned a different IP address. Additional complications will arise when/if you want to add separate GPU workers to your CryoSPARC instance.
I recommend ensuring that the server is guaranteed to be assigned a specific IP address between reboots and that the server hostname resolves to this IP address. Your IT support may be able to help with that.

11