MongoDB Replica Set Error / DBPathInUse / Ghost Process Deadlock after v4.7.1 Upgrade

Hi @wtempel ,

Thank you for your response.

Sorry for the confusion. This was not an upgrade to or from v4.7.1. The CryoSPARC version was already v4.7.1.

We recently deployed a new Ubuntu cluster to replace our previous Amazon Linux cluster and migrated the existing CryoSPARC 4.7.1 installation to the new environment. After the migration, we encountered what appeared to be a MongoDB/database issue.

Regarding the requested commands:

  • ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo

    At the time of the issue, the MongoDB/CryoSPARC processes continued running even after attempting kill and kill -9. The processes were only terminated after we unmounted the Lustre filesystem.

  • ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock

    These socket files were not present.

We also discovered an issue with the FSx for Lustre filesystem on the Ubuntu cluster. After deploying a new cluster and resolving the Lustre issue, CryoSPARC started working as expected, which makes us believe the MongoDB issue may have been a side effect of the filesystem problem.

We do have a few questions regarding the recommended recovery procedure in this type of situation:

  1. If MongoDB does not stop cleanly, what is the recommended sequence of steps? For example:

    • Run cryosparcm stop

    • Verify that all mongod processes have exited

    • If necessary, terminate remaining processes

    • Remove mongod.lock (if appropriate)

    • Start or restart CryoSPARC

    Is this the correct sequence, or is there a different recommended recovery procedure?

  2. Could you explain the purpose of the WiredTiger.lock file? Under what circumstances, if any, should it be removed manually?

  3. When should mongod --repair be used? What types of database issues does it resolve, and when is it not recommended?

Additionally, during this issue we were unable to perform a CryoSPARC backup. Since the root cause appears to have been related to the Lustre filesystem rather than database corruption, we would appreciate any guidance on the best recovery approach if we encounter a similar situation in the future.

Thank you for your assistance.