Database configuration Error

Install, Configure and Update
1

When I try to run cryosparcm start, it failed at configuring database. Here is the following error code when I ran crysoarcm restart:

cryosparcm restart
CryoSPARC is running.
Stopping cryoSPARC 
command_core: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
    configuration complete
database: started
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
checkdb error - could not get replica set status; please reconfigure the database with `cryosparcm configuredb`
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 268, in check_mongo
    admin_db = try_get_pymongo_admin_db(mongo_client)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 249, in try_get_pymongo_admin_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 827, in command
    with self.__client._socket_for_reads(read_preference, session) as (sock_info, secondary_ok):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1478, in _socket_for_reads
    server = self._select_server(read_preference, session)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1436, in _select_server
    server = topology.select_server(server_selector)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 250, in select_server
    return random.choice(self.select_servers(selector, server_selection_timeout, address))
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 211, in select_servers
    server_descriptions = self._select_servers_loop(selector, server_timeout, address)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: linda:39001: [Errno -2] Name or service not known, Timeout: 20.0s, Topology Description: <TopologyDescription id: 6491d8bee03670373a0fc27c, topology_type: Unknown, servers: [<ServerDescription ('linda', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('linda:<port_number>: [Errno -2] Name or service not known')>]>
[2023-06-20T09:51:16-07:00] Error checking database. Most recent database log lines:
2023-06-20T09:50:04.542-0700 I REPL     [replexec-0] Starting replication reporter thread
2023-06-20T09:50:04.543-0700 I REPL     [rsSync] transition to SECONDARY from RECOVERING
2023-06-20T09:50:04.543-0700 I REPL     [rsSync] conducting a dry run election to see if we could be elected. current term: 34
2023-06-20T09:50:04.543-0700 I REPL     [replexec-0] dry election run succeeded, running for election in term 35
2023-06-20T09:50:04.547-0700 I REPL     [replexec-0] election succeeded, assuming primary role in term 35
2023-06-20T09:50:04.547-0700 I REPL     [replexec-0] transition to PRIMARY from SECONDARY
2023-06-20T09:50:04.547-0700 I REPL     [replexec-0] Resetting sync source to empty, which was :27017
2023-06-20T09:50:04.547-0700 I REPL     [replexec-0] Entering primary catch-up mode.
2023-06-20T09:50:04.547-0700 I REPL     [replexec-0] Exited primary catch-up mode.
2023-06-20T09:50:06.544-0700 I REPL     [rsSync] transition to primary complete; database writes are now permitted


I also checked with crysparcm status and gave the following:

cryosparcm status
----------------------------------------------------------------------------'
CryoSPARC System master node installed at
/home/cryosparc_user/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1
----------------------------------------------------------------------------'

CryoSPARC process status:

app                              STOPPED   Not started
app_api                          STOPPED   Not started
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     STOPPED   Not started
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         RUNNING   pid 1090680, uptime 0:10:23

----------------------------------------------------------------------------
*** CommandClient: (http://LINDA:<PORT_NUMBER>/api) URL Error [Errno 111] Connection refused
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?
2

Welcome to the forum @CGAO123.
You must ensure that whatever is configured for CRYOSPARC_MASTER_HOSTNAME inside /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh is correctly resolved by DNS or /etc/hosts to the computer that runs cryosparc_master services.
Moreover, capital letters in the value defined for the CRYOSPARC_MASTER_HOSTNAME will likely cause problems when running CryoSPARC jobs, for example if the definition were in fact

export CRYOSPARC_MASTER_HOSTNAME="LINDA"

instead of, say,

export CRYOSPARC_MASTER_HOSTNAME="linda"

.
What is the output of the command

grep HOSTNAME /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh

?
Is this a “Single Workstation” installation where there is no plan to connect additional GPU worker hosts?

3

The output is

export CRYOSPARC_MASTER_HOSTNAME=“LINDA”

export CRYOSPARC_FORCE_HOSTNAME=true

I believe this is a single workstation installation but I am unsure of future plans regarding addtional GPUs. Will this change the way database should be configured?

4

In the single workstation case, a greatly simplified configuration is possible. See Failed to launch! 255 Host key verification failed? for an example.
Briefly summarized (and omitting important details/steps), one could in the single workstation case:

  • define the master hostname as localhost
  • define the worker hostname as localhost

and consequently bypass a number of issues related to domain name resolution, remote port access and ssh logins.

To make, on the other hand, the CryoSPARC instance expandable for the future addition of separate GPU workers, one has to ensure:

  • password-less ssh logins from the master host to GPU nodes
  • GPU nodes must be able to correctly resolve the master’s hostname and contact the master host via various network ports
1 Like
5

Thank you so much for the explanation, I believe there is a plan in the future to connect to additional GPU workers. In that case, how should I set up cryosparc correctly?

6

Please see our guide for general requirements. A few additional suggestions: master and worker hosts should be connected to the same private network such that

  • the network’s dhcp service or some other mechanism ensures that each node’s IP address remains constant from reboot to reboot
  • hostnames, host/domain name resolution and firewall configurations ensure
    • host names do not include capital letters, which can cause unexpected outcomes of hostname comparisons
    • worker nodes and the master node (and those nodes only) can use the hostname returned by running
      hostname -f on the master host to connect to the CryoSPARC ports on the master node
    • the master host can connect to each specific worker node via ssh
1 Like
7

I tried following the guide and several other protocols and wasn’t able to fix my situation. For example, I tried running

cryosparcm status | grep MASTER_HOSTNAME

and it wouldn’t return anything beuase running

cryosparcm status

alone ends in this error message:

CryoSPARC System master node installed at
/home/cryosparc_user/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1

CryoSPARC process status:

app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
app_legacy STOPPED Not started
app_legacy_dev STOPPED Not started
command_core STOPPED Not started
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 1090680, uptime 2 days, 5:37:23

*** CommandClient: (http://LINDA:39002/api) URL Error [Errno 111] Connection refused
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

8

This requirement increases the complexity of the fix.

  1. Request from your IT/network support that your computer will be assigned IP address and hostname that

    • remain constant from reboot to reboot and
    • that are correctly resolved by other computers on your internal network.

    Also ask them for help in configuring your computer

    • with that newly assigned IP address and
    • so that the command hostname -f prints out that newly assigned “permanent” hostname

    For the purpose of this discussion, suppose they assign the hostname linda1.somedept.abcu.edu.

  2. Ensure CryoSPARC is stopped:

    • cryosparcm stop
    • ps -eo pid,ppid,cmd | grep -e cryosparc -e mongo
      to confirm no CryoSPARC related processes remain. If you are unsure about processes listed, post ps output here.

  3. Edit the CRYOSPARC_MASTER_HOSTNAME definition inside /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh:

    export CRYOSPARC_MASTER_HOSTNAME="linda1.somedept.abcu.edu" # use actually assigned hostname instead

  4. Comment out the CRYOSPARC_FORCE_HOSTNAME definition:

    # export CRYOSPARC_FORCE_HOSTNAME=true

  5. Try cryosparcm start.
    In case the command fails, please post error messages in this forum topic. If the command succeeds, please post the output of the command
    cryosparcm cli "get_scheduler_targets()"
    so we can see what changes to your worker configuration may be needed.

1 Like