Corrupted mongodb

Troubleshooting
1

Hi,
After some maintenance issues, I can no longer restart the cryosparc db, I receive the following error:

2021-11-29T19:56:34.980-0600 I CONTROL [initandlisten] target_arch: x86_64
2021-11-29T19:56:34.981-0600 I CONTROL [initandlisten] options: { net: { port: 40001 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/atavium/home/jfrench/cryosparc_jf/cryosparc_lithium_standalone”, journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2021-11-29T19:56:35.037-0600 I - [initandlisten] Detected data files in /atavium/home/jfrench/cryosparc_jf/cryosparc_lithium_standalone created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2021-11-29T19:56:35.038-0600 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),log=(enabled=false),
2021-11-29T19:56:35.262-0600 E STORAGE [initandlisten] WiredTiger error (0) [1638237395:262804][8093:0x7f8fd97c3d40], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2021-11-29T19:56:35.262-0600 E STORAGE [initandlisten] WiredTiger error (-31804) [1638237395:262886][8093:0x7f8fd97c3d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2021-11-29T19:56:35.262-0600 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2021-11-29T19:56:35.262-0600 I - [initandlisten]

***aborting after fassert() failure

2021-11-29T19:56:35.297-0600 F - [initandlisten] Got signal: 6 (Aborted).

I tried using the mongo tools to repair it but those crash as well. We do not have daily backups, is there any other way to recover the db?

Thanks,
Jeff

2

Can you post the failure message from the attempted repair?

3

…/software/cryosparc/cryosparc_master/deps/external/mongodb/bin/mongod --dbpath ./ --repair
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] MongoDB starting : pid=17433 port=27017 dbpath=./ 64-bit host=lithium.hi.umn.edu
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] db version v3.4.10
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] allocator: tcmalloc
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] modules: none
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] build environment:
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] distarch: x86_64
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] target_arch: x86_64
2021-11-29T21:24:53.157-0600 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: “./” } }
2021-11-29T21:24:53.197-0600 I - [initandlisten] Detected data files in ./ created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2021-11-29T21:24:53.198-0600 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=127686M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),log=(enabled=false),
2021-11-29T21:24:53.239-0600 E STORAGE [initandlisten] WiredTiger error (0) [1638242693:239489][17433:0x7f1948503d40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 40960: block header checksum of 2155646527 doesn’t match expected checksum of 1511640665
2021-11-29T21:24:53.239-0600 E STORAGE [initandlisten] WiredTiger error (0) [1638242693:239548][17433:0x7f1948503d40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2021-11-29T21:24:53.239-0600 E STORAGE [initandlisten] WiredTiger error (-31804) [1638242693:239571][17433:0x7f1948503d40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2021-11-29T21:24:53.239-0600 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2021-11-29T21:24:53.239-0600 I - [initandlisten]

***aborting after fassert() failure

2021-11-29T21:24:53.271-0600 F - [initandlisten] Got signal: 6 (Aborted).

0x55c9e2696ac1 0x55c9e2695cd9 0x55c9e26961bd 0x7f19477de630 0x7f1947437387 0x7f1947438a78 0x55c9e196ae97 0x55c9e23cdb66 0x55c9e1974b46 0x55c9e1974d62 0x55c9e1974fc4 0x55c9e2f9ac9f 0x55c9e2f9b1eb 0x55c9e2f977dd 0x55c9e2f9c3b7 0x55c9e2fb5ff3 0x55c9e2fef6cb 0x55c9e307a50b 0x55c9e307aa4d 0x55c9e307ad0c 0x55c9e2ffddf1 0x55c9e3070838 0x55c9e303a7fe 0x55c9e303a8db 0x55c9e2feb8b9 0x55c9e23b288a 0x55c9e23ab0c5 0x55c9e229d5d7 0x55c9e195780c 0x55c9e197607b 0x7f1947423555 0x55c9e19d0c41
----- BEGIN BACKTRACE -----
{“backtrace”:[{“b”:“55C9E1163000”,“o”:“1533AC1”,“s”:"_ZN5mongo15printStackTraceERSo"},{“b”:“55C9E1163000”,“o”:“1532CD9”},{“b”:“55C9E1163000”,“o”:“15331BD”},{“b”:“7F19477CF000”,“o”:“F630”},{“b”:“7F1947401000”,“o”:“36387”,“s”:“gsignal”},{“b”:“7F1947401000”,“o”:“37A78”,“s”:“abort”},{“b”:“55C9E1163000”,“o”:“807E97”,“s”:"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{“b”:“55C9E1163000”,“o”:“126AB66”},{“b”:“55C9E1163000”,“o”:“811B46”,“s”:"__wt_eventv"},{“b”:“55C9E1163000”,“o”:“811D62”,“s”:"__wt_err"},{“b”:“55C9E1163000”,“o”:“811FC4”,“s”:"__wt_panic"},{“b”:“55C9E1163000”,“o”:“1E37C9F”,“s”:"__wt_block_extlist_read"},{“b”:“55C9E1163000”,“o”:“1E381EB”,“s”:"__wt_block_extlist_read_avail"},{“b”:“55C9E1163000”,“o”:“1E347DD”,“s”:"__wt_block_checkpoint_load"},{“b”:“55C9E1163000”,“o”:“1E393B7”},{“b”:“55C9E1163000”,“o”:“1E52FF3”,“s”:"__wt_btree_open"},{“b”:“55C9E1163000”,“o”:“1E8C6CB”,“s”:"__wt_conn_btree_open"},{“b”:“55C9E1163000”,“o”:“1F1750B”,“s”:"__wt_session_get_btree"},{“b”:“55C9E1163000”,“o”:“1F17A4D”,“s”:"__wt_session_get_btree"},{“b”:“55C9E1163000”,“o”:“1F17D0C”,“s”:"__wt_session_get_btree_ckpt"},{“b”:“55C9E1163000”,“o”:“1E9ADF1”,“s”:"__wt_curfile_open"},{“b”:“55C9E1163000”,“o”:“1F0D838”},{“b”:“55C9E1163000”,“o”:“1ED77FE”,“s”:"__wt_metadata_cursor_open"},{“b”:“55C9E1163000”,“o”:“1ED78DB”,“s”:"__wt_metadata_cursor"},{“b”:“55C9E1163000”,“o”:“1E888B9”,“s”:“wiredtiger_open”},{“b”:“55C9E1163000”,“o”:“124F88A”,“s”:"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{“b”:“55C9E1163000”,“o”:“12480C5”},{“b”:“55C9E1163000”,“o”:“113A5D7”,“s”:"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{“b”:“55C9E1163000”,“o”:“7F480C”},{“b”:“55C9E1163000”,“o”:“81307B”,“s”:“main”},{“b”:“7F1947401000”,“o”:“22555”,“s”:"__libc_start_main"},{“b”:“55C9E1163000”,“o”:“86DC41”}],“processInfo”:{ “mongodbVersion” : “3.4.10”, “gitVersion” : “078f28920cb24de0dd479b5ea6c66c644f6326e9”, “compiledModules” : [], “uname” : { “sysname” : “Linux”, “release” : “3.10.0-1160.15.2.el7.x86_64”, “version” : “#1 SMP Wed Feb 3 15:06:38 UTC 2021”, “machine” : “x86_64” }, “somap” : [ { “b” : “55C9E1163000”, “elfType” : 3, “buildId” : “D9AB5C91FBC6F740604F4BC28348FE33EC87DEC2” }, { “b” : “7FFCE7AFC000”, “elfType” : 3, “buildId” : “9F1A0F30C88CB271F585FB4AD18A128DCC15BBCF” }, { “b” : “7F1948107000”, “path” : “/lib64/librt.so.1”, “elfType” : 3, “buildId” : “3E44DF7055942478D052E40FDD1F5B7862B152B0” }, { “b” : “7F1947F03000”, “path” : “/lib64/libdl.so.2”, “elfType” : 3, “buildId” : “7F2E9CB0769D7E57BD669B485A74B537B63A57C4” }, { “b” : “7F1947C01000”, “path” : “/lib64/libm.so.6”, “elfType” : 3, “buildId” : “7615604EAF4A068DFAE5085444D15C0DEE93DFBD” }, { “b” : “7F19479EB000”, “path” : “/lib64/libgcc_s.so.1”, “elfType” : 3, “buildId” : “EDF51350C7F71496149D064AA8B1441F786DF88A” }, { “b” : “7F19477CF000”, “path” : “/lib64/libpthread.so.0”, “elfType” : 3, “buildId” : “E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B” }, { “b” : “7F1947401000”, “path” : “/lib64/libc.so.6”, “elfType” : 3, “buildId” : “A317B42B15368ADCAE21C11107691A03EC91059D” }, { “b” : “7F194830F000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “62C449974331341BB08DCCE3859560A22AF1E172” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55c9e2696ac1]
mongod(+0x1532CD9) [0x55c9e2695cd9]
mongod(+0x15331BD) [0x55c9e26961bd]
libpthread.so.0(+0xF630) [0x7f19477de630]
libc.so.6(gsignal+0x37) [0x7f1947437387]
libc.so.6(abort+0x148) [0x7f1947438a78]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c9e196ae97]
mongod(+0x126AB66) [0x55c9e23cdb66]
mongod(__wt_eventv+0x3D7) [0x55c9e1974b46]
mongod(__wt_err+0x9D) [0x55c9e1974d62]
mongod(__wt_panic+0x2E) [0x55c9e1974fc4]
mongod(__wt_block_extlist_read+0x8F) [0x55c9e2f9ac9f]
mongod(__wt_block_extlist_read_avail+0x2B) [0x55c9e2f9b1eb]
mongod(__wt_block_checkpoint_load+0x26D) [0x55c9e2f977dd]
mongod(+0x1E393B7) [0x55c9e2f9c3b7]
mongod(__wt_btree_open+0xB43) [0x55c9e2fb5ff3]
mongod(__wt_conn_btree_open+0x19B) [0x55c9e2fef6cb]
mongod(__wt_session_get_btree+0xFB) [0x55c9e307a50b]
mongod(__wt_session_get_btree+0x63D) [0x55c9e307aa4d]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55c9e307ad0c]
mongod(__wt_curfile_open+0x161) [0x55c9e2ffddf1]
mongod(+0x1F0D838) [0x55c9e3070838]
mongod(__wt_metadata_cursor_open+0x6E) [0x55c9e303a7fe]
mongod(__wt_metadata_cursor+0x4B) [0x55c9e303a8db]
mongod(wiredtiger_open+0x1659) [0x55c9e2feb8b9]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x73A) [0x55c9e23b288a]
mongod(+0x12480C5) [0x55c9e23ab0c5]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x55c9e229d5d7]
mongod(+0x7F480C) [0x55c9e195780c]
mongod(main+0x96B) [0x55c9e197607b]
libc.so.6(__libc_start_main+0xF5) [0x7f1947423555]
mongod(+0x86DC41) [0x55c9e19d0c41]
----- END BACKTRACE -----
Aborted (core dumped)

4

I’d love to be wrong, but that looks unrecoverable to me.

5

Just went through this myself. You have to do daily backups of the mongo database – it’s way too brittle to be left to its own devices. Move your database folder to database-OLD and start re-entering your projects by hand. That is your option. Here are the instructions for setting up a daily backup of MongoDB. Be aware that over time this will start to take up considerable space even though the backups are rotated.

https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring/cryosparcm#cryosparcm-backup