Warning: Could not get database status (attempt 1/3) - issue after storage outage

Troubleshooting
4

The database seems to be corrupt. As a first step, I recommend eliminating, to the extent possible, the risk of a future storage outage.
For pre-requisites of running
cryosparcm restore, please see the guide. For an important limitation of
cryosparcm restore, please see below.
Ensure all CryoSPARC processes are terminated by running (under the CryoSPARC-related Linux account)

  1. cryosparcm stop
    then
  2. ps -u $USER -opid,pid,ppid | grep -e cryosparc -e mongo
    to identify potentially left-over CryoSPARC-related processes, which should be terminated with
    kill -TERM

If additional processing was performed on the CryoSPARC instance after the most recent database backup, a hypothetical database restoration may result in the overwriting of more current information in project directories by out-of-date information from the restored database. If this is a concern, I recommend

  1. updating the CRYOSPARC_DB_PATH= definition inside /XXX/cryosparc_master/config.sh with a new, suitable path.
  2. then running
    cryosparcm start
    which should start CryoSPARC with a blank database.
  3. re-creating CryoSPARC users in the database with
    cryosparcm createuser
    commands (guide)
  4. registering CryoSPARC workers in the database with
    cryosparcw connect commands (on the worker(s), guide) or
    cryosparcm cluster connect command(s) (guide), as applicable.
  5. attaching project directories. You may have to delete the cs.lock file from any given project directory before attachment because the instance id stored in the new, blank database differs from the old instance id stored in the old database. This is an exception from the rule that cs.lock files should generally not be deleted.

[edited for spelling 2024-04-12]