Database crashes (ab-initio/heterogeneous ref)

Troubleshooting
1

Hi, I get database crashing and I need to restart cryosparc whenever I try to browse the results from a ab-initio job or heterogeneous refinement with multiple (5+) classes.

The cryosparcm log database output is the following.

88A9C42D9E3CB338836544A" }, { “b” : “7FCBDE152000”, “path” : “/lib/x86_64-linux-gnu/libpthread.so.0”, “elfType” : 3, “buildId” : “7B4536F41CDAA5888408E82D0836E33DCF436466” }, { “b” : “7FCBDDF60000”, “path” : “/lib/x86_64-linux-gnu/libc.so.6”, “elfType” : 3, “buildId” : “1878E6B475720C7C51969E69AB2D276FAE6D1DEE” }, { “b” : “7FCBDE667000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “4587364908DE169DEC62FFA538170118C1C3A078” }, { “b” : “7FCBDDF5B000”, “path” : “/lib/x86_64-linux-gnu/libutil.so.1”, “elfType” : 3, “buildId” : “4F3EE75C38F09D6346DE1E8ECA0F8D8A41071D9F” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x557e91ee6f21]
mongod(+0x22A5139) [0x557e91ee6139]
mongod(+0x22A561D) [0x557e91ee661d]
libpthread.so.0(+0x14420) [0x7fcbde166420]
libc.so.6(gsignal+0xCB) [0x7fcbddfa300b]
libc.so.6(abort+0x12B) [0x7fcbddf82859]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x557e905cadec]
mongod(+0xA64D76) [0x557e906a5d76]
mongod(+0xAD6AD1) [0x557e90717ad1]
mongod(__wt_err_func+0x90) [0x557e90567a94]
mongod(__wt_panic+0x3F) [0x557e90567eb4]
mongod(__wt_block_read_off+0x585) [0x557e907ce3b5]
mongod(__wt_bm_read+0x135) [0x557e907ce4f5]
mongod(__wt_bt_read+0xB1) [0x557e90741ec1]
mongod(__wt_page_in_func+0x178D) [0x557e907487bd]
mongod(__wt_row_search+0x76D) [0x557e9076cbcd]
mongod(__wt_btcur_search+0x722) [0x557e907e00e2]
mongod(+0xB4723A) [0x557e9078823a]
mongod(_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE+0x55) [0x557e9068b295]
mongod(_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEE+0xAF) [0x557e90edecbf]
mongod(_ZN5mongo10FetchStage6doWorkEPm+0x106) [0x557e90e8eb06]
mongod(_ZN5mongo9PlanStage4workEPm+0x6B) [0x557e90eb3ebb]
mongod(_ZN5mongo14MultiPlanStage6doWorkEPm+0xB9) [0x557e90ea9809]
mongod(_ZN5mongo9PlanStage4workEPm+0x6B) [0x557e90eb3ebb]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x43E) [0x557e90c7444e]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x557e90c74f7b]
mongod(_ZN5mongo15DistinctCommand3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x143B) [0x557e908c87eb]
mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x557e9181aa36]
mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x557e91815d6f]
mongod(+0xC3AF2B) [0x557e9087bf2b]
mongod(+0xC3C93C) [0x557e9087d93c]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x314) [0x557e9087e974]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x557e9088eaaa]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x557e90889467]
mongod(+0xC4C891) [0x557e9088d891]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x557e917ecec2]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x557e908882a0]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xEE4) [0x557e9088abe4]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x241) [0x557e9088b961]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x557e908894ed]
mongod(+0xC4C891) [0x557e9088d891]
mongod(+0x1BAC425) [0x557e917ed425]
mongod(+0x215CC94) [0x557e91d9dc94]
libpthread.so.0(+0x8609) [0x7fcbde15a609]
libc.so.6(clone+0x43) [0x7fcbde07f133]
----- END BACKTRACE -----

2

Please can you post a few lines of the log just above the beginning of the BACKTRACE.

1 Like
3

Blockquote
2022-10-27T11:06:43.284+0200 E STORAGE [conn34] WiredTiger error (0) [1666861603:284475][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_block_read_off, 291: collection-4–4098553455242
555341.wt: read checksum error for 8192B block at offset 91856896: block header checksum of 0 doesn’t match expected checksum of 2587758584 Raw: [1666861603:284475][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.
wt, WT_CURSOR.search: __wt_block_read_off, 291: collection-4–4098553455242555341.wt: read checksum error for 8192B block at offset 91856896: block header checksum of 0 doesn’t match expected checksum of 2587758584

2022-10-27T11:06:43.285+0200 E STORAGE [conn34] WiredTiger error (0) [1666861603:285815][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_bm_corrupt_dump, 144: {91856896, 8192, 258775858
4}: (chunk 8 of 8): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2022-10-27T11:06:43.285+0200 E STORAGE [conn34] WiredTiger error (-31802) [1666861603:285842][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_block_read_off, 302: collection-4–40985534
55242555341.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1666861603:285842][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_block_read_off, 302: collection-4–4098
553455242555341.wt: fatal read error: WT_ERROR: non-specific WiredTiger error
2022-10-27T11:06:43.285+0200 E STORAGE [conn34] WiredTiger error (-31804) [1666861603:285852][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_panic, 523: the process must exit and resta
rt: WT_PANIC: WiredTiger library panic Raw: [1666861603:285852][3814398:0x7f3e9e241700], file:collection-4–4098553455242555341.wt, WT_CURSOR.search: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger librar
y panic
2022-10-27T11:06:43.285+0200 F - [conn34] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420
2022-10-27T11:06:43.285+0200 F - [conn34] \n\naborting after fassert() failure\n\n
2022-10-27T11:06:43.316+0200 F - [conn23] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 74
2022-10-27T11:06:43.316+0200 F - [conn23] \n\naborting after fassert() failure\n\n
2022-10-27T11:06:43.317+0200 F - [conn25] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 74
2022-10-27T11:06:43.317+0200 F - [conn25] \n\naborting after fassert() failure\n\n
2022-10-27T11:06:43.412+0200 F - [conn4] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 74
2022-10-27T11:06:43.412+0200 F - [conn4] \n\naborting after fassert() failure\n\n
2022-10-27T11:06:43.699+0200 F - [conn34] Got signal: 6 (Aborted).

4

@stavros The database engine encountered an inconsistency in a database file. Please see this related topic for some recommendations.