Database not detected

I was in the process of trying to upgrade from Centos 7 to 8. I wasn’t able to complete the upgrade and now Cryosparc is not connecting to the database. Here is what happens when I try to start CryoSPARC:

[cryosparc_user@X ~]$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
    configuration complete
database: started
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
checkdb error - could not get replica set status; please reconfigure the database with `cryosparcm configuredb`
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 268, in check_mongo
    admin_db = try_get_pymongo_admin_db(mongo_client)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 249, in try_get_pymongo_admin_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 827, in command
    with self.__client._socket_for_reads(read_preference, session) as (sock_info, secondary_ok):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1478, in _socket_for_reads
    server = self._select_server(read_preference, session)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1436, in _select_server
    server = topology.select_server(server_selector)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 250, in select_server
    return random.choice(self.select_servers(selector, server_selection_timeout, address))
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 211, in select_servers
    server_descriptions = self._select_servers_loop(selector, server_timeout, address)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: defcon2.bio.uci.edu:39001: [Errno 113] No route to host, Timeout: 20.0s, Topology Description: <TopologyDescription id: 646278909ed7920fe7478880, topology_type: Unknown, servers: [<ServerDescription ('defcon2.bio.uci.edu', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('defcon2.bio.uci.edu:39001: [Errno 113] No route to host')>]>
[2023-05-15T11:24:23-0700] Error checking database. Most recent database log lines:
2023-05-15T11:23:10.427-0700 I REPL     [replexec-0] Starting replication reporter thread
2023-05-15T11:23:10.428-0700 I REPL     [rsSync] transition to SECONDARY from RECOVERING
2023-05-15T11:23:10.428-0700 I REPL     [rsSync] conducting a dry run election to see if we could be elected. current term: 108
2023-05-15T11:23:10.428-0700 I REPL     [replexec-0] dry election run succeeded, running for election in term 109
2023-05-15T11:23:10.433-0700 I REPL     [replexec-0] election succeeded, assuming primary role in term 109
2023-05-15T11:23:10.433-0700 I REPL     [replexec-0] transition to PRIMARY from SECONDARY
2023-05-15T11:23:10.433-0700 I REPL     [replexec-0] Resetting sync source to empty, which was :27017
2023-05-15T11:23:10.433-0700 I REPL     [replexec-0] Entering primary catch-up mode.
2023-05-15T11:23:10.433-0700 I REPL     [replexec-0] Exited primary catch-up mode.
2023-05-15T11:23:12.429-0700 I REPL     [rsSync] transition to primary complete; database writes are now permitted
[cryosparc_user@defcon2 ~]$ cd /home/cryosparc_user/software/cryosparc/cryosparc_worker/
[cryosparc_user@X cryosparc_worker]$ ls
bin  check_install_deps.sh  config.sh  cryosparc_compute  cryosparc_tools  deps  deps_bundle  deps_bundle_hashes  deps_hashes  environment.yml  install.sh  version
[cryosparc_user@X cryosparc_worker]$ ls ../cryosparc_master/cryosparc_worker.tar.gz
../cryosparc_master/cryosparc_worker.tar.gz
[cryosparc_user@X cryosparc_worker]$ ln -s ../cryosparc_master/cryosparc_worker.tar.gz
[cryosparc_user@X cryosparc_worker]$ ./bin/cryosparcw update
Updating... checking versions
Current version v4.2.1 - New version v4.2.1 
Already up to date
[cryosparc_user@X cryosparc_worker]$

Your network-related system configuration may have changed during the upgrade attempt (side note: CentOS EOL).
Please can you post the outputs of commands

host defcon2.bio.uci.edu
curl 127.0.0.1:39001
cryosparcm status | grep HOSTNAME
[cryosparc_user@defcon2 cryosparc_worker]$ host defcon2.bio.uci.edu
defcon2.bio.uci.edu has address 128.195.151.92
[cryosparc_user@defcon2 cryosparc_worker]$ curl 127.0.0.1:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
[cryosparc_user@defcon2 cryosparc_worker]$ cryosparcm status | grep HOSTNAME
[cryosparc_user@defcon2 cryosparc_worker]$

A firewall may be blocking access to port 39001 (and other CryoSPARC ports). What are the outputs of

ip -4 addr
curl 128.195.151.92:39001

?

[cryosparc_user@defcon2 ~]$ ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: enp179s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 128.195.151.98/25 brd 128.195.151.127 scope global noprefixroute dynamic enp179s0f1
       valid_lft 28275sec preferred_lft 28275sec
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
curl 128.195.151.92:39001
curl: (7) Failed connect to 128.195.151.92:39001; No route to host
[cryosparc_user@defcon2 ~]$

So maybe not a firewall issue. Has this computer been assigned a new ip address?
The resolution of your issue may depend on

  • whether your CryoSPARC instance is running in standalone/single workstation mode
  • whether you are planning to add additional workers to your instance
  • whether the maintainers of your network can guarantee the assignment of a constant ip address to your computer for the foreseeable future