Hi @wtempel ,
Thank you for your response.
Sorry for the confusion. This was not an upgrade to or from v4.7.1. The CryoSPARC version was already v4.7.1.
We recently deployed a new Ubuntu cluster to replace our previous Amazon Linux cluster and migrated the existing CryoSPARC 4.7.1 installation to the new environment. After the migration, we encountered what appeared to be a MongoDB/database issue.
Regarding the requested commands:
-
ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongoAt the time of the issue, the MongoDB/CryoSPARC processes continued running even after attempting
killandkill -9. The processes were only terminated after we unmounted the Lustre filesystem. -
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sockThese socket files were not present.
We also discovered an issue with the FSx for Lustre filesystem on the Ubuntu cluster. After deploying a new cluster and resolving the Lustre issue, CryoSPARC started working as expected, which makes us believe the MongoDB issue may have been a side effect of the filesystem problem.
We do have a few questions regarding the recommended recovery procedure in this type of situation:
-
If MongoDB does not stop cleanly, what is the recommended sequence of steps? For example:
-
Run
cryosparcm stop -
Verify that all
mongodprocesses have exited -
If necessary, terminate remaining processes
-
Remove
mongod.lock(if appropriate) -
Start or restart CryoSPARC
Is this the correct sequence, or is there a different recommended recovery procedure?
-
-
Could you explain the purpose of the
WiredTiger.lockfile? Under what circumstances, if any, should it be removed manually? -
When should
mongod --repairbe used? What types of database issues does it resolve, and when is it not recommended?
Additionally, during this issue we were unable to perform a CryoSPARC backup. Since the root cause appears to have been related to the Lustre filesystem rather than database corruption, we would appreciate any guidance on the best recovery approach if we encounter a similar situation in the future.
Thank you for your assistance.