The RAID controller on our cryosparc master server crashed, leaving the database in a corrupted state. From a previous bout with this, I’m aware that the steps to recovery are
- make a backup copy of the database directory
- cd database directory
- mongod --dbpath . --repair
In attempting to back up the database, I noticed it’s taking up almost 1TB of space, which seems not right. On further investigation it seem the folder contains a couple of very large backup files that I didn’t create:
cryosparc_user@cerebro:~/cryosparc_database$ pwd /local/home/cryosparc_user/cryosparc_database cryosparc_user@cerebro:~/cryosparc_database$ ls -lh backup total 413G -rw-rw-r-- 1 cryosparc_user cryosparc_user 206G Jun 12 19:15 cryosparc_backup_2021_06_12_14h03.archive -rw-rw-r-- 1 cryosparc_user cryosparc_user 207G Jun 15 19:02 cryosparc_backup_2021_06_15_13h51.archive
Is there some process which would have created these files automatically?
Second, one of the .wt files looks to be out of control:
rw-r--r-- 1 cryosparc_user cryosparc_user 339G Nov 6 18:19 collection-36--7747720921166270324.wt
I’m guessing the controller crashed while cryosparc was attempting to write something, started spinning its wheels, and filled the database with cruft before we could catch it.
So, my question is just proceed with the prescribed recovery method? Backup up the current database folder is going to take a a lot of time under the circumstances.
Any other suggestions? Understood that this is kind of a worst case scenario for software.