MongoDB has crashed twice this week - not quite sure how to debug - here are some of the relevant entries from the logfile:
2026-03-11T11:27:15.577-0400 E STORAGE [WTOplogJournalThread] WiredTiger error (5) [1773242835:577013][615551:0x14d1a16b0700], WT_SESSION.l
og_flush: __posix_sync, 99: /home/npa2pc/cryosparc/cryodatabase/journal/WiredTigerLog.0000002511: handle-sync: fdatasync: Input/output error
Raw: [1773242835:577013][615551:0x14d1a16b0700], WT_SESSION.log_flush: __posix_sync, 99: /home/npa2pc/cryosparc/cryodatabase/journal/WiredT
igerLog.0000002511: handle-sync: fdatasync: Input/output error
2026-03-11T11:27:15.577-0400 E STORAGE [WTOplogJournalThread] WiredTiger error (-31804) [1773242835:577083][615551:0x14d1a16b0700], WT_SESS
ION.log_flush: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1773242835:577083][615551:0x14d1
a16b0700], WT_SESSION.log_flush: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
2026-03-11T11:27:15.577-0400 F - [WTOplogJournalThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp
423
2026-03-11T11:27:15.577-0400 F - [WTOplogJournalThread] \n\n***aborting after fassert() failure\n\n
2026-03-11T11:27:15.577-0400 E STORAGE [conn1352] WiredTiger error (116) [1773242835:577343][615551:0x14d19bb5e700], WT_SESSION.commit_tran
saction: __posix_file_write, 546: /home/npa2pc/cryosparc/cryodatabase/journal/WiredTigerLog.0000002511: handle-write: pwrite: failed to writ
e 141184 bytes at offset 75975296: Stale file handle Raw: [1773242835:577343][615551:0x14d19bb5e700], WT_SESSION.commit_transaction: __posix
_file_write, 546: /home/npa2pc/cryosparc/cryodatabase/journal/WiredTigerLog.0000002511: handle-write: pwrite: failed to write 141184 bytes a
t offset 75975296: Stale file handle
2026-03-11T11:27:15.577-0400 E STORAGE [conn1352] WiredTiger error (116) [1773242835:577393][615551:0x14d19bb5e700], WT_SESSION.commit_tran
saction: __log_fs_write, 220: journal/WiredTigerLog.0000002511: fatal log failure: Stale file handle Raw: [1773242835:577393][615551:0x14d19
bb5e700], WT_SESSION.commit_transaction: __log_fs_write, 220: journal/WiredTigerLog.0000002511: fatal log failure: Stale file handle
2026-03-11T11:27:15.577-0400 E STORAGE [conn1352] WiredTiger error (-31804) [1773242835:577415][615551:0x14d19bb5e700], WT_SESSION.commit_t
ransaction: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1773242835:577415][615551:0x14d19bb
5e700], WT_SESSION.commit_transaction: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
2026-03-11T11:27:15.577-0400 F - [conn1352] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 423
2026-03-11T11:27:15.577-0400 F - [conn1352] \n\n***aborting after fassert() failure\n\n
2026-03-11T11:27:15.577-0400 I WRITE [LogicalSessionCacheRefresh] update config.system.sessions command: { q: { _id: { id: UUID("95acd6d5
-af9b-4d0c-802e-3fe6fcf8bf23"), uid: BinData(0, 47E73090B3C4005D91E94C0BDB3DE0C17283FD6CFD9CFC0DCCF34DD3D82EE78F) } }, u: { $currentDate: {
lastUse: true }, $setOnInsert: { user: { name: “cryosparc_user@admin” } } }, multi: false, upsert: true } planSummary: IDHACK keysExamined:1
docsExamined:1 nMatched:1 nModified:1 keysInserted:1 keysDeleted:1 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database:
{ acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } storage:{} 184277ms
2026-03-11T11:27:15.577-0400 I COMMAND [conn1396] warning: log line attempted (13kB) over max size (10kB)
2026-03-11T11:27:15.588-0400 F - [WTOplogJournalThread] Got signal: 6 (Aborted).
0x559c82b48b41 0x559c82b47d59 0x559c82b4823d 0x14d1a38ab990 0x14d1a351152f 0x14d1a34e4e65 0x559c810b00ee 0x559c811bb88f 0x559c81229a01 0x55
9c81037065 0x559c8103747f 0x559c811fada3 0x559c812cafce 0x559c812d18cb 0x559c8121b4df 0x559c811b37ec 0x559c81198eb0 0x559c8119914a 0x559c82c
59680 0x14d1a38a11ca 0x14d1a34fc8d3
----- BEGIN BACKTRACE -----
{“backtrace”:[{“b”:“559C8068B000”,“o”:“24BDB41”,“s”:“_ZN5mongo15printStackTraceERSo”},{“b”:“559C8068B000”,“o”:“24BCD59”},{“b”:“559C8068B000”
,“o”:“24BD23D”},{“b”:“14D1A3899000”,“o”:“12990”},{“b”:“14D1A34C3000”,“o”:“4E52F”,“s”:“gsignal”},{“b”:“14D1A34C3000”,“o”:“21E65”,“s”:“abort”}
,{“b”:“559C8068B000”,“o”:“A250EE”,“s”:“_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj”},{“b”:“559C8068B000”,“o”:“B3088F”},{“b”:"559C8068B
000",“o”:“B9EA01”},{“b”:“559C8068B000”,“o”:“9AC065”,“s”:“__wt_err_func”},{“b”:“559C8068B000”,“o”:“9AC47F”,“s”:“__wt_panic”},{“b”:"559C8068B0
00",“o”:“B6FDA3”},{“b”:“559C8068B000”,“o”:“C3FFCE”,“s”:“__wt_log_force_sync”},{“b”:“559C8068B000”,“o”:“C468CB”,“s”:“__wt_log_flush”},{“b”:"5
59C8068B000",“o”:“B904DF”},{“b”:“559C8068B000”,“o”:“B287EC”,“s”:“_ZN5mongo22WiredTigerSessionCache16waitUntilDurableEbb”},{“b”:"559C8068B000
“,“o”:“B0DEB0”,“s”:”_ZN5mongo22WiredTigerOplogManager23_oplogJournalThreadLoopEPNS_22WiredTigerSessionCacheEPNS_21WiredTigerRecordStoreEb"},
{“b”:“559C8068B000”,“o”:“B0E14A”,“s”:"_ZNSt6thread5_ImplISt12_Bind_simpleIFZN5mongo4stdx6threadC4IMNS2_22WiredTigerOplogManagerEFvPNS2_22Wir
edTigerSessionCacheEPNS2_21WiredTigerRecordStoreEbEJPS6_S8_RSA_RbELi0EEET_DpOT0_EUlvE_vEEE6_M_runEv"},{“b”:“559C8068B000”,“o”:“25CE680”},{"b
":“14D1A3899000”,“o”:“81CA”},{“b”:“14D1A34C3000”,“o”:“398D3”,“s”:“clone”}],“processInfo”:{ “mongodbVersion” : “4.0.28”, “gitVersion” : "af1a
9dc12adcfa83cc19571cb3faba26eeddac92", “compiledModules” : , “uname” : { “sysname” : “Linux”, “release” : “4.18.0-553.47.1.el8_10.x86_64”,
“version” : “#1 SMP Thu Apr 3 14:27:08 UTC 2025”, “machine” : “x86_64” }, “somap” : [ { “b” : “559C8068B000”, “elfType” : 3, “buildId” : "0
4032FD6593DE6CA52ADF5D29FD4A4DD1DB5C3EF" }, { “b” : “7FFEF9FB0000”, “path” : “linux-vdso.so.1”, “elfType” : 3, “buildId” : "5F7FE0AE4A9FDC0A
F2FD642D6DDCB8C93842B869" }, { “b” : “14D1A4677000”, “path” : "/sfs/ceph/standard/takcryoem/cryosparc/cryosparc_master/.pixi/envs/master/lib
/libpython3.12.so", “elfType” : 3 }, { “b” : “14D1A445F000”, “path” : “/lib64/libresolv.so.2”, “elfType” : 3, “buildId” : "531C1B01384C2A5CA
D2F8B3551F9816ABE1D1EDA" }, { “b” : “14D1A4257000”, “path” : “/lib64/librt.so.1”, “elfType” : 3, “buildId” : "7635C7433232A17C1C1892E7469B53
9EAD753BCE" }, { “b” : “14D1A4053000”, “path” : “/lib64/libdl.so.2”, “elfType” : 3, “buildId” : “C3055F965736231C924FAD099A662D4F6CA8F672” }
mongod(+0xB3088F) [0x559c811bb88f]
, { “b” : “14D1A3CD1000”, “path” : “/lib64/libm.so.6”, “elfType” : 3, “buildId” : “DC6FE3A171227FC2EACDC1AC5F6BE9747DA5DC69” }, { “b” : “14D1A3AB9000”, “path” : “/lib64/libgcc_s.so.1”, “elfType” : 3, “buildId” : “971D2AC8B1C07585F3BCD66AD0988225A4EA2F8F” }, { “b” : “14D1A3899000”, “path” : “/lib64/libpthread.so.0”, “elfType” : 3, “buildId” : “17340BC9F8F0AAFCA761B8330627A59F866128E8” }, { “b” : “14D1A34C3000”, “path” : “/lib64/libc.so.6”, “elfType” : 3, “buildId” : “21AC34704743C50FC3D67D887383443FBEDC7195” }, { “b” : “14D1A4D0D000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “E38E7DDAB9B4628212F243508003412A30BBEDEE” }, { “b” : “14D1A32BF000”, “path” : “/lib64/libutil.so.1”, “elfType” : 3, “buildId” : “33B61615FDC876BBF6284ABAE49FE521B8B7696A” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x559c82b48b41]
mongod(+0x24BCD59) [0x559c82b47d59]
mongod(+0x24BD23D) [0x559c82b4823d]
libpthread.so.0(+0x12990) [0x14d1a38ab990]
libc.so.6(gsignal+0x10F) [0x14d1a351152f]
libc.so.6(abort+0x127) [0x14d1a34e4e65]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x559c810b00ee]
mongod(+0xB3088F) [0x559c811bb88f]
mongod(+0xB9EA01) [0x559c81229a01]
mongod(__wt_err_func+0x90) [0x559c81037065]
mongod(__wt_panic+0x39) [0x559c8103747f]
mongod(+0xB6FDA3) [0x559c811fada3]
mongod(__wt_log_force_sync+0x2FE) [0x559c812cafce]
mongod(__wt_log_flush+0xDB) [0x559c812d18cb]
mongod(+0xB904DF) [0x559c8121b4df]
mongod(_ZN5mongo22WiredTigerSessionCache16waitUntilDurableEbb+0x36C) [0x559c811b37ec]
mongod(_ZN5mongo22WiredTigerOplogManager23_oplogJournalThreadLoopEPNS_22WiredTigerSessionCacheEPNS_21WiredTigerRecordStoreEb+0x5A0) [0x559c81198eb0]
mongod(_ZNSt6thread5_ImplISt12_Bind_simpleIFZN5mongo4stdx6threadC4IMNS2_22WiredTigerOplogManagerEFvPNS2_22WiredTigerSessionCacheEPNS2_21WiredTigerRecordStoreEbEJPS6_S8_RSA_RbELi0EEET_DpOT0_EUlvE_vEEE6_M_runEv+0x6A) [0x559c8119914a]
mongod(+0x25CE680) [0x559c82c59680]
libpthread.so.0(+0x81CA) [0x14d1a38a11ca]
libc.so.6(clone+0x43) [0x14d1a34fc8d3]