Our Cryosparc v4.3.1 system was upgraded from CentOS 7 to Rocky 8. It appears that Cryosparc was not shut down cleanly beforehand. Now cryosparcm start fails with:
configuring database
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
<snip>
Below is the relevant section from the database log. It appears that Mongo encountered corrupt data.
How should I proceed?
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] MongoDB starting : pid=285980 port=39001 dbpath=/home/cryosparc_user/cryosparc2/cryosparc2_database 64-bit host=<snip>
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] db version v3.6.23
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] allocator: tcmalloc
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] modules: none
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] build environment:
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] distarch: x86_64
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] target_arch: x86_64
2023-12-18T14:54:19.499-0800 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/home/cryosparc_user/cryosparc2/cryosp
arc2_database", journal: { enabled: false } } }
2023-12-18T14:54:19.499-0800 W - [initandlisten] Detected unclean shutdown - /home/cryosparc_user/cryosparc2/cryosparc2_database/mongod.lock is not empty.
2023-12-18T14:54:19.500-0800 I - [initandlisten] Detected data files in /home/cryosparc_user/cryosparc2/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active sto
rage engine to 'wiredTiger'.
2023-12-18T14:54:19.500-0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2023-12-18T14:54:19.500-0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=128398M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_b
ase=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0)
,verbose=(recovery_progress),,log=(enabled=false),
2023-12-18T14:54:19.996-0800 I STORAGE [initandlisten] WiredTiger message [1702940059:996896][285980:0x7f7932a9e480], txn-recover: Set global recovery timestamp: 0
2023-12-18T14:54:20.000-0800 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2023-12-18T14:54:20.001-0800 I STORAGE [initandlisten] The size storer reports that the oplog contains 5498 records totaling to 63521131 bytes
2023-12-18T14:54:20.001-0800 I STORAGE [initandlisten] Sampling the oplog to determine where to place markers for truncation
2023-12-18T14:54:20.001-0800 E STORAGE [initandlisten] WiredTiger error (0) [1702940060:1443][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_block_read_off, 29
1: collection-4-6563032916698143400.wt: read checksum error for 8192B block at offset 12791808: block header checksum of 443346175 doesn't match expected checksum of 3118558262 Raw: [1702940060:1443][
285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_block_read_off, 291: collection-4-6563032916698143400.wt: read checksum error for 8192B block at offset 12791808:
block header checksum of 443346175 doesn't match expected checksum of 3118558262
2023-12-18T14:54:20.001-0800 E STORAGE [initandlisten] WiredTiger error (0) [1702940060:1540][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_bm_corrupt_dump, 1
44: {12791808, 8192, 3118558262}: (chunk 1 of 8): 00 00 00
<snip>
2023-12-18T14:54:20.002-0800 E STORAGE [initandlisten] WiredTiger error (-31802) [1702940060:2258][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_block_read_of
f, 302: collection-4-6563032916698143400.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1702940060:2258][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSO
R.prev: __wt_block_read_off, 302: collection-4-6563032916698143400.wt: fatal read error: WT_ERROR: non-specific WiredTiger error
2023-12-18T14:54:20.002-0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1702940060:2270][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_panic, 523: t
he process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1702940060:2270][285980:0x7f7932a9e480], file:collection-4-6563032916698143400.wt, WT_CURSOR.prev: __wt_panic, 523: the proce
ss must exit and restart: WT_PANIC: WiredTiger library panic
2023-12-18T14:54:20.002-0800 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420
2023-12-18T14:54:20.002-0800 F - [initandlisten] \n\n***aborting after fassert() failure\n\n
2023-12-18T14:54:20.017-0800 F - [initandlisten] Got signal: 6 (Aborted).
0x55a75dbe0f21 0x55a75dbe0139 0x55a75dbe061d 0x7f7931761cf0 0x7f79313d8acf 0x7f79313abea5 0x55a75c2c4dec 0x55a75c39fd76 0x55a75c411ad1 0x55a75c261a94 0x55a75c261eb4 0x55a75c4c83b5 0x55a75c4c84f5 0x55
a75c43bec1 0x55a75c4427bd 0x55a75c45d07d 0x55a75c4d20c1 0x55a75c47e98c 0x55a75c385934 0x55a75c38f057 0x55a75c38ff4a 0x55a75c390231 0x55a75c390436 0x55a75c3790df 0x55a75c4fe66e 0x55a75c4f78f6 0x55a75c3
60bf5 0x55a75c570fb6 0x55a75c33d5b8 0x55a75c34063c 0x55a75c2c6bc9 0x7f79313c4d85 0x55a75c328741
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55A75B93B000","o":"22A5F21","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55A75B93B000","o":"22A5139"},{"b":"55A75B93B000","o":"22A561D"},{"b":"7F793174F000","o":"12CF0"},{"b":"7F793
138A000","o":"4EACF","s":"gsignal"},{"b":"7F793138A000","o":"21EA5","s":"abort"},{"b":"55A75B93B000","o":"989DEC","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55A75B93B000","o":"A64D
76"},{"b":"55A75B93B000","o":"AD6AD1"},{"b":"55A75B93B000","o":"926A94","s":"__wt_err_func"},{"b":"55A75B93B000","o":"926EB4","s":"__wt_panic"},{"b":"55A75B93B000","o":"B8D3B5","s":"__wt_block_read_of
f"},{"b":"55A75B93B000","o":"B8D4F5","s":"__wt_bm_read"},{"b":"55A75B93B000","o":"B00EC1","s":"__wt_bt_read"},{"b":"55A75B93B000","o":"B077BD","s":"__wt_page_in_func"},{"b":"55A75B93B000","o":"B2207D"
},{"b":"55A75B93B000","o":"B970C1","s":"__wt_btcur_prev"},{"b":"55A75B93B000","o":"B4398C"},{"b":"55A75B93B000","o":"A4A934","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv"},{"b":"55A75B93B000
","o":"A54057","s":"_ZN5mongo21WiredTigerRecordStore11OplogStones26_calculateStonesBySamplingEPNS_16OperationContextEll"},{"b":"55A75B93B000","o":"A54F4A","s":"_ZN5mongo21WiredTigerRecordStore11OplogS
tones16_calculateStonesEPNS_16OperationContextEm"},{"b":"55A75B93B000","o":"A55231","s":"_ZN5mongo21WiredTigerRecordStore11OplogStonesC1EPNS_16OperationContextEPS0_"},{"b":"55A75B93B000","o":"A55436",
"s":"_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE"},{"b":"55A75B93B000","o":"A3E0DF","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextEN
S_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE"},{"b":"55A75B93B000","o":"BC366E","s":"_ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_strin
gIcSt11char_traitsIcESaIcEEEb"},{"b":"55A75B93B000","o":"BBC8F6","s":"_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBa
seESt14default_deleteIS8_EENS_10StringDataEPS0_EE"},{"b":"55A75B93B000","o":"A25BF5"},{"b":"55A75B93B000","o":"C35FB6","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55A
75B93B000","o":"A025B8"},{"b":"55A75B93B000","o":"A0563C","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55A75B93B000","o":"98BBC9","s":"main"},{"b":"7F793138A000","o":"3AD85","s":"__libc_start_main"},{"
b":"55A75B93B000","o":"9ED741"}],"processInfo":{ "mongodbVersion" : "3.6.23", "gitVersion" : "d352e6a4764659e0d0350ce77279de3c1f243e5c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "relea
se" : "4.18.0-477.27.1.el8_8.x86_64", "version" : "#1 SMP Wed Sep 20 15:55:39 UTC 2023", "machine" : "x86_64" }, "somap" : [ { "b" : "55A75B93B000", "elfType" : 3, "buildId" : "B0818C001F2B63D4533D208
F68F08AE2A599CA9E" }, { "b" : "7FFCD4793000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "567BF6B6F8DD052AF6A5C62B4FAE34D1B55FF41D" }, { "b" : "7F793252D000", "path" : "/home/cryosparc_use
r/cryosparc2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.8.so", "elfType" : 3 }, { "b" : "7F7932315000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "7A6C
3C40109A388930064033096DF1003BA05570" }, { "b" : "7F793210D000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "8B4FA9BBD0DA01252F3B06439520B3A305DACC0A" }, { "b" : "7F7931F09000", "path" :
"/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2419CB1F884781832B478F8AD70F96BAF48FC497" }, { "b" : "7F7931B87000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "15F38A7F64D5A4E1BC0B7F44
91CD17E630A8809B" }, { "b" : "7F793196F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DE767DAE55C9183AE8DDB79CF70D17602287CAE1" }, { "b" : "7F793174F000", "path" : "/lib64/libpthre
ad.so.0", "elfType" : 3, "buildId" : "8FF3B7D14ABCC882A7201334E28BE9BC785CD741" }, { "b" : "7F793138A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D4708E8BDFB99D8EBDD2DF46A639E70BF100
1AF8" }, { "b" : "7F7932887000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "006CAC5ED3D76E365A92DEFD2635A744A019E54D" }, { "b" : "7F7931186000", "path" : "/lib64/libutil.so.1"
, "elfType" : 3, "buildId" : "D3DFAD0DC1C464EBF973745AE7C1AA9B6D043594" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a75dbe0f21]
mongod(+0x22A5139) [0x55a75dbe0139]
mongod(+0x22A561D) [0x55a75dbe061d]
libpthread.so.0(+0x12CF0) [0x7f7931761cf0]
libc.so.6(gsignal+0x10F) [0x7f79313d8acf]
libc.so.6(abort+0x127) [0x7f79313abea5]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55a75c2c4dec]
mongod(+0xA64D76) [0x55a75c39fd76]
mongod(+0xAD6AD1) [0x55a75c411ad1]
mongod(__wt_err_func+0x90) [0x55a75c261a94]
mongod(__wt_panic+0x3F) [0x55a75c261eb4]
mongod(__wt_block_read_off+0x585) [0x55a75c4c83b5]
mongod(__wt_bm_read+0x135) [0x55a75c4c84f5]
mongod(__wt_bt_read+0xB1) [0x55a75c43bec1]
mongod(__wt_page_in_func+0x178D) [0x55a75c4427bd]
mongod(+0xB2207D) [0x55a75c45d07d]
mongod(__wt_btcur_prev+0x731) [0x55a75c4d20c1]
mongod(+0xB4398C) [0x55a75c47e98c]
mongod(_ZN5mongo31WiredTigerRecordStoreCursorBase4nextEv+0x264) [0x55a75c385934]
mongod(_ZN5mongo21WiredTigerRecordStore11OplogStones26_calculateStonesBySamplingEPNS_16OperationContextEll+0x157) [0x55a75c38f057]
mongod(_ZN5mongo21WiredTigerRecordStore11OplogStones16_calculateStonesEPNS_16OperationContextEm+0x2EA) [0x55a75c38ff4a]
mongod(_ZN5mongo21WiredTigerRecordStore11OplogStonesC1EPNS_16OperationContextEPS0_+0x241) [0x55a75c390231]
mongod(_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE+0xB6) [0x55a75c390436]
mongod(_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE+0x27F) [0x55a75c3790df]
mongod(_ZN5mongo26KVDatabaseCatalogEntryBase14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x54E) [0x55a75c4fe66e]
mongod(_ZN5mongo15KVStorageEngineC2EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE+0xA46) [0x55
a75c4f78f6]
mongod(+0xA25BF5) [0x55a75c360bf5]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x266) [0x55a75c570fb6]
mongod(+0xA025B8) [0x55a75c33d5b8]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x26C) [0x55a75c34063c]
mongod(main+0x9) [0x55a75c2c6bc9]
libc.so.6(__libc_start_main+0xE5) [0x7f79313c4d85]
mongod(+0x9ED741) [0x55a75c328741]
----- END BACKTRACE -----