Could not get replica set status; please reconfigure the database with cryosparcm configuredb Traceback (most recent call last): File , line 1, in File /apps/cryosparc/v4.7.1_251124/cryosparc_master/cryosparc_compute/database_management.py , line 270, in check_mongo admin_db = try_get_pymongo_db(mongo_client) File /apps/cryosparc/v4.7.1_251124/cryosparc_master/cryosparc_compute/database_management.py , line 251, in try_get_pymongo_db admin_db.command(({ serverStatus : 1})) File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py , line 108, in csot_wrapper return func(self, *args, **kwargs) File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py , line 893, in command with self.__client._conn_for_reads(read_preference, session, operation=command_name) as ( File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py , line 1375, in _conn_for_reads server = self._select_server(read_preference, session, operation) File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/mongo_client.py , line 1322, in _select_server server = topology.select_server( File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py , line 368, in select_server server = self._select_server( File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py , line 346, in _select_server servers = self.select_servers( File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py , line 253, in select_servers server_descriptions = self._select_servers_loop( File /apps/cryosparc/v4.7.1_251124/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/topology.py , line 303, in _select_servers_loop raise ServerSelectionTimeoutError( pymongo.errors.ServerSelectionTimeoutError: ip-10-220-65-10:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 20.0s, Topology Description: <TopologyDescription id: 6a5630b494a8387a6686bd37, topology_type: Unknown, servers: [<ServerDescription ( ip-10-220-65-10 , 39001) server_type: Unknown, rtt: None, error=AutoReconnect( ip-10-220-65-10:39001: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms) )>]> [2026-07-14T08:52:10-04:00] Error checking database. Most recent database log lines: 2026-07-14T08:41:05.231-0400 I CONTROL [initandlisten] shutting down with code:100 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] MongoDB starting : pid=6527 port=39001 dbpath=/projects-prd/cryosparcdatabase/v4.7.1_251124 64-bit host=ip-10-220-65-10 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] db version v3.6.23 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] allocator: tcmalloc 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] modules: none 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] build environment: 2026-07-14T08:50:58.341-0400 I CONTROL [initandlisten] distarch: x86_64
Hi @wtempel , Could you please help me
@satyashetty Please can you provide additional information:
- Did you just update to v4.7.1? If so, from which version?
- Or did you just update from v4.7.1? If so, to which version?
- What are the outputs of these commands:
ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
Hi @wtempel ,
Thank you for your response.
Sorry for the confusion. This was not an upgrade to or from v4.7.1. The CryoSPARC version was already v4.7.1.
We recently deployed a new Ubuntu cluster to replace our previous Amazon Linux cluster and migrated the existing CryoSPARC 4.7.1 installation to the new environment. After the migration, we encountered what appeared to be a MongoDB/database issue.
Regarding the requested commands:
-
ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongoAt the time of the issue, the MongoDB/CryoSPARC processes continued running even after attempting
killandkill -9. The processes were only terminated after we unmounted the Lustre filesystem. -
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sockThese socket files were not present.
We also discovered an issue with the FSx for Lustre filesystem on the Ubuntu cluster. After deploying a new cluster and resolving the Lustre issue, CryoSPARC started working as expected, which makes us believe the MongoDB issue may have been a side effect of the filesystem problem.
We do have a few questions regarding the recommended recovery procedure in this type of situation:
-
If MongoDB does not stop cleanly, what is the recommended sequence of steps? For example:
-
Run
cryosparcm stop -
Verify that all
mongodprocesses have exited -
If necessary, terminate remaining processes
-
Remove
mongod.lock(if appropriate) -
Start or restart CryoSPARC
Is this the correct sequence, or is there a different recommended recovery procedure?
-
-
Could you explain the purpose of the
WiredTiger.lockfile? Under what circumstances, if any, should it be removed manually? -
When should
mongod --repairbe used? What types of database issues does it resolve, and when is it not recommended?
Additionally, during this issue we were unable to perform a CryoSPARC backup. Since the root cause appears to have been related to the Lustre filesystem rather than database corruption, we would appreciate any guidance on the best recovery approach if we encounter a similar situation in the future.
Thank you for your assistance.
Thanks @satyashetty for the additional details and questions.
Use of the kill -9 option is discouraged.
We have not tested storing the database on a Lustre filesystem. Please see mongodb documentation for filesystem recommendations.
I do not recall any situations that involved manual removal of mongod.lock or WiredTiger.lock.
If cryosparcm stop did not stop the database, consider the steps for a through shutdown procedure. As of July 2026, the steps described on the linked section apply to CryoSPARC v5.
mongod --repair should probably not be used to resolve problems with the CryoSPARC database (why?).
CryoSPARC v5 includes Instance Recovery functionality for recovery from damage to the CryoSPARC database.
Thank you for the details.