Database - fatal - exited too quickly

irv555 · November 15, 2019, 7:01pm

I have a stand alone system with the following:

Centos 7   3.10.0-957.5.1.el7.x86_64
Current cryoSPARC version: v2.11.2-live_privatebeta
export CRYOSPARC_CUDA_PATH="/usr/local/cuda"

[cryosparc03sa@cryosparc03:~]$ cryosparcm log database

***aborting after fassert() failure

2019-11-15T10:49:16.765-0700 F - [initandlisten] Got signal: 6 (Aborted).

 0x55ceb2d03d6b 0x55ceb2d02fe6 0x55ceb2d034c3 0x2accf132d5d0 0x2accf1570207 0x2accf15718f8 0x55ceb1f22887 0x55ceb29fa597 0x55ceb1f2ce75 0x55ceb1f2d08c 0x55ceb1f2d2f5 0x55ceb36fa9d7 0x55ceb36f91d9 0x55ceb36ae893 0x55ceb3739974 0x55ceb3739f1f 0x55ceb373a1cc 0x55ceb36bc8a9 0x55ceb372f618 0x55ceb36f88fe 0x55ceb36f89eb 0x55ceb36aae6a 0x55ceb29df685 0x55ceb29d79ca 0x55ceb28bd50a 0x55ceb1f2ebab 0x2accf155c3d5 0x55ceb1f92cff

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"55CEB18E3000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55CEB18E3000","o":"141FFE6"},{"b":"55CEB18E3000","o":"14204C3"},{"b":"2ACCF131E000","o":"F5D0"},{"b":"2ACCF153A000","o":"36207","s":"gsignal"},{"b":"2ACCF153A000","o":"378F8","s":"abort"},{"b":"55CEB18E3000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55CEB18E3000","o":"1117597"},{"b":"55CEB18E3000","o":"649E75","s":"__wt_eventv"},{"b":"55CEB18E3000","o":"64A08C","s":"__wt_err"},{"b":"55CEB18E3000","o":"64A2F5","s":"__wt_panic"},{"b":"55CEB18E3000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"55CEB18E3000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"55CEB18E3000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"55CEB18E3000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"55CEB18E3000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"55CEB18E3000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"55CEB18E3000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"55CEB18E3000","o":"1E4C618"},{"b":"55CEB18E3000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"55CEB18E3000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"55CEB18E3000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"55CEB18E3000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55CEB18E3000","o":"10F49CA"},{"b":"55CEB18E3000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55CEB18E3000","o":"64BBAB","s":"main"},{"b":"2ACCF153A000","o":"223D5","s":"__libc_start_main"},{"b":"55CEB18E3000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.5.1.el7.x86_64", "version" : "#1 SMP Fri Feb 1 14:54:57 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55CEB18E3000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFD7E26D000", "elfType" : 3, "buildId" : "733D1C681CFBD8BBEB11E8B7F80876A9AED6BB74" }, { "b" : "2ACCF01A2000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2ACCF03A3000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2ACCF0619000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2ACCF0A9C000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "2ACCF0CA4000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "2ACCF0EA8000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2ACCF101C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "2ACCF0385000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2ACCF131E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "2ACCF153A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A644E865A1F672B46A1232F7C275F53B06042696" }, { "b" : "2ACCF1907000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "2ACCF017F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x55ceb2d03d6b]

 mongod(+0x141FFE6) [0x55ceb2d02fe6]

 mongod(+0x14204C3) [0x55ceb2d034c3]

 libpthread.so.0(+0xF5D0) [0x2accf132d5d0]

 libc.so.6(gsignal+0x37) [0x2accf1570207]

 libc.so.6(abort+0x148) [0x2accf15718f8]

 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55ceb1f22887]

 mongod(+0x1117597) [0x55ceb29fa597]

 mongod(__wt_eventv+0x3E4) [0x55ceb1f2ce75]

 mongod(__wt_err+0xA0) [0x55ceb1f2d08c]

 mongod(__wt_panic+0x2F) [0x55ceb1f2d2f5]

 mongod(__wt_turtle_read+0x227) [0x55ceb36fa9d7]

 mongod(__wt_metadata_search+0x99) [0x55ceb36f91d9]

 mongod(__wt_conn_btree_open+0x73) [0x55ceb36ae893]

 mongod(__wt_session_get_btree+0xE4) [0x55ceb3739974]

 mongod(__wt_session_get_btree+0x68F) [0x55ceb3739f1f]

 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55ceb373a1cc]

 mongod(__wt_curfile_open+0x169) [0x55ceb36bc8a9]

 mongod(+0x1E4C618) [0x55ceb372f618]

 mongod(__wt_metadata_cursor_open+0x6E) [0x55ceb36f88fe]

 mongod(__wt_metadata_cursor+0x4B) [0x55ceb36f89eb]

 mongod(wiredtiger_open+0x183A) [0x55ceb36aae6a]

 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x55ceb29df685]

 mongod(+0x10F49CA) [0x55ceb29d79ca]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x55ceb28bd50a]

 mongod(main+0xE9B) [0x55ceb1f2ebab]

 libc.so.6(__libc_start_main+0xF5) [0x2accf155c3d5]

 mongod(+0x6AFCFF) [0x55ceb1f92cff]

----- END BACKTRACE -----

ISSUE:
We had a user try to stop and start the cryosparc service as himself when the box was being unresponsive. He didn’t become the user that cryosparc runs as ‘cryosparc03sa’. There was other nfs mounted filesystems that had made the box appear unresponsive. I don’t know how to get it running again.

Thanks

sdawood · November 15, 2019, 8:53pm

Hi @irv555,

Unfortunately we haven’t come across this error before. It seems like the database logs you posted don’t include the actual MongoDB error, could you try pasting more of the output from cryosparcm log database?

Regards,
Suhail

irv555 · November 15, 2019, 9:13pm

This is all I get. The command doesn’t seem to finish (it never returns to a command prompt.

[cryosparc03sa@cryosparc03:~]$ cryosparcm log database

***aborting after fassert() failure

2019-11-15T10:49:16.765-0700 F - [initandlisten] Got signal: 6 (Aborted).

 0x55ceb2d03d6b 0x55ceb2d02fe6 0x55ceb2d034c3 0x2accf132d5d0 0x2accf1570207 0x2accf15718f8 0x55ceb1f22887 0x55ceb29fa597 0x55ceb1f2ce75 0x55ceb1f2d08c 0x55ceb1f2d2f5 0x55ceb36fa9d7 0x55ceb36f91d9 0x55ceb36ae893 0x55ceb3739974 0x55ceb3739f1f 0x55ceb373a1cc 0x55ceb36bc8a9 0x55ceb372f618 0x55ceb36f88fe 0x55ceb36f89eb 0x55ceb36aae6a 0x55ceb29df685 0x55ceb29d79ca 0x55ceb28bd50a 0x55ceb1f2ebab 0x2accf155c3d5 0x55ceb1f92cff

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"55CEB18E3000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55CEB18E3000","o":"141FFE6"},{"b":"55CEB18E3000","o":"14204C3"},{"b":"2ACCF131E000","o":"F5D0"},{"b":"2ACCF153A000","o":"36207","s":"gsignal"},{"b":"2ACCF153A000","o":"378F8","s":"abort"},{"b":"55CEB18E3000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55CEB18E3000","o":"1117597"},{"b":"55CEB18E3000","o":"649E75","s":"__wt_eventv"},{"b":"55CEB18E3000","o":"64A08C","s":"__wt_err"},{"b":"55CEB18E3000","o":"64A2F5","s":"__wt_panic"},{"b":"55CEB18E3000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"55CEB18E3000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"55CEB18E3000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"55CEB18E3000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"55CEB18E3000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"55CEB18E3000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"55CEB18E3000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"55CEB18E3000","o":"1E4C618"},{"b":"55CEB18E3000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"55CEB18E3000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"55CEB18E3000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"55CEB18E3000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55CEB18E3000","o":"10F49CA"},{"b":"55CEB18E3000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55CEB18E3000","o":"64BBAB","s":"main"},{"b":"2ACCF153A000","o":"223D5","s":"__libc_start_main"},{"b":"55CEB18E3000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.5.1.el7.x86_64", "version" : "#1 SMP Fri Feb 1 14:54:57 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55CEB18E3000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFD7E26D000", "elfType" : 3, "buildId" : "733D1C681CFBD8BBEB11E8B7F80876A9AED6BB74" }, { "b" : "2ACCF01A2000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2ACCF03A3000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2ACCF0619000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2ACCF0A9C000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "2ACCF0CA4000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "2ACCF0EA8000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2ACCF101C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "2ACCF0385000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2ACCF131E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "2ACCF153A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A644E865A1F672B46A1232F7C275F53B06042696" }, { "b" : "2ACCF1907000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "2ACCF017F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x55ceb2d03d6b]

 mongod(+0x141FFE6) [0x55ceb2d02fe6]

 mongod(+0x14204C3) [0x55ceb2d034c3]

 libpthread.so.0(+0xF5D0) [0x2accf132d5d0]

 libc.so.6(gsignal+0x37) [0x2accf1570207]

 libc.so.6(abort+0x148) [0x2accf15718f8]

 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55ceb1f22887]

 mongod(+0x1117597) [0x55ceb29fa597]

 mongod(__wt_eventv+0x3E4) [0x55ceb1f2ce75]

 mongod(__wt_err+0xA0) [0x55ceb1f2d08c]

 mongod(__wt_panic+0x2F) [0x55ceb1f2d2f5]

 mongod(__wt_turtle_read+0x227) [0x55ceb36fa9d7]

 mongod(__wt_metadata_search+0x99) [0x55ceb36f91d9]

 mongod(__wt_conn_btree_open+0x73) [0x55ceb36ae893]

 mongod(__wt_session_get_btree+0xE4) [0x55ceb3739974]

 mongod(__wt_session_get_btree+0x68F) [0x55ceb3739f1f]

 mongod(__wt_session_get_btree_ckpt+0x14C) [0x55ceb373a1cc]

 mongod(__wt_curfile_open+0x169) [0x55ceb36bc8a9]

 mongod(+0x1E4C618) [0x55ceb372f618]

 mongod(__wt_metadata_cursor_open+0x6E) [0x55ceb36f88fe]

 mongod(__wt_metadata_cursor+0x4B) [0x55ceb36f89eb]

 mongod(wiredtiger_open+0x183A) [0x55ceb36aae6a]

 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x55ceb29df685]

 mongod(+0x10F49CA) [0x55ceb29d79ca]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x55ceb28bd50a]

 mongod(main+0xE9B) [0x55ceb1f2ebab]

 libc.so.6(__libc_start_main+0xF5) [0x2accf155c3d5]

 mongod(+0x6AFCFF) [0x55ceb1f92cff]

----- END BACKTRACE -----

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] MongoDB starting : pid=18985 port=39001 dbpath=/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database 64-bit host=cryosparc03.chpc.utah.edu

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] allocator: tcmalloc

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] modules: none

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] build environment:

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] distarch: x86_64

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] target_arch: x86_64

2019-11-15T10:49:18.019-0700 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }

2019-11-15T10:49:18.046-0700 I - [initandlisten] Detected data files in /uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2019-11-15T10:49:18.047-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),

2019-11-15T10:49:18.059-0700 E STORAGE [initandlisten] WiredTiger error (0) [1573840158:59758][18985:0x2b31377a3e00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

2019-11-15T10:49:18.059-0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1573840158:59812][18985:0x2b31377a3e00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic

2019-11-15T10:49:18.059-0700 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361

2019-11-15T10:49:18.059-0700 I - [initandlisten] 

***aborting after fassert() failure

2019-11-15T10:49:18.066-0700 F - [initandlisten] Got signal: 6 (Aborted).

 0x562742a57d6b 0x562742a56fe6 0x562742a574c3 0x2b31387355d0 0x2b3138978207 0x2b31389798f8 0x562741c76887 0x56274274e597 0x562741c80e75 0x562741c8108c 0x562741c812f5 0x56274344e9d7 0x56274344d1d9 0x562743402893 0x56274348d974 0x56274348df1f 0x56274348e1cc 0x5627434108a9 0x562743483618 0x56274344c8fe 0x56274344c9eb 0x5627433fee6a 0x562742733685 0x56274272b9ca 0x56274261150a 0x562741c82bab 0x2b31389643d5 0x562741ce6cff

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"562741637000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562741637000","o":"141FFE6"},{"b":"562741637000","o":"14204C3"},{"b":"2B3138726000","o":"F5D0"},{"b":"2B3138942000","o":"36207","s":"gsignal"},{"b":"2B3138942000","o":"378F8","s":"abort"},{"b":"562741637000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"562741637000","o":"1117597"},{"b":"562741637000","o":"649E75","s":"__wt_eventv"},{"b":"562741637000","o":"64A08C","s":"__wt_err"},{"b":"562741637000","o":"64A2F5","s":"__wt_panic"},{"b":"562741637000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"562741637000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"562741637000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"562741637000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"562741637000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"562741637000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"562741637000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"562741637000","o":"1E4C618"},{"b":"562741637000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"562741637000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"562741637000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"562741637000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"562741637000","o":"10F49CA"},{"b":"562741637000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"562741637000","o":"64BBAB","s":"main"},{"b":"2B3138942000","o":"223D5","s":"__libc_start_main"},{"b":"562741637000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.5.1.el7.x86_64", "version" : "#1 SMP Fri Feb 1 14:54:57 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "562741637000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFEAB3E6000", "elfType" : 3, "buildId" : "733D1C681CFBD8BBEB11E8B7F80876A9AED6BB74" }, { "b" : "2B31375AA000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2B31377AB000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2B3137A21000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2B3137EA4000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "2B31380AC000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "2B31382B0000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2B3138424000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "2B313778D000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2B3138726000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "2B3138942000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A644E865A1F672B46A1232F7C275F53B06042696" }, { "b" : "2B3138D0F000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "2B3137587000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x562742a57d6b]

 mongod(+0x141FFE6) [0x562742a56fe6]

 mongod(+0x14204C3) [0x562742a574c3]

 libpthread.so.0(+0xF5D0) [0x2b31387355d0]

 libc.so.6(gsignal+0x37) [0x2b3138978207]

 libc.so.6(abort+0x148) [0x2b31389798f8]

 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x562741c76887]

 mongod(+0x1117597) [0x56274274e597]

 mongod(__wt_eventv+0x3E4) [0x562741c80e75]

 mongod(__wt_err+0xA0) [0x562741c8108c]

 mongod(__wt_panic+0x2F) [0x562741c812f5]

 mongod(__wt_turtle_read+0x227) [0x56274344e9d7]

 mongod(__wt_metadata_search+0x99) [0x56274344d1d9]

 mongod(__wt_conn_btree_open+0x73) [0x562743402893]

 mongod(__wt_session_get_btree+0xE4) [0x56274348d974]

 mongod(__wt_session_get_btree+0x68F) [0x56274348df1f]

 mongod(__wt_session_get_btree_ckpt+0x14C) [0x56274348e1cc]

 mongod(__wt_curfile_open+0x169) [0x5627434108a9]

 mongod(+0x1E4C618) [0x562743483618]

 mongod(__wt_metadata_cursor_open+0x6E) [0x56274344c8fe]

 mongod(__wt_metadata_cursor+0x4B) [0x56274344c9eb]

 mongod(wiredtiger_open+0x183A) [0x5627433fee6a]

 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x562742733685]

 mongod(+0x10F49CA) [0x56274272b9ca]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x56274261150a]

 mongod(main+0xE9B) [0x562741c82bab]

 libc.so.6(__libc_start_main+0xF5) [0x2b31389643d5]

 mongod(+0x6AFCFF) [0x562741ce6cff]

----- END BACKTRACE -----

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] MongoDB starting : pid=19007 port=39001 dbpath=/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database 64-bit host=cryosparc03.chpc.utah.edu

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] allocator: tcmalloc

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] modules: none

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] build environment:

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] distarch: x86_64

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] target_arch: x86_64

2019-11-15T10:49:20.147-0700 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }

2019-11-15T10:49:20.173-0700 I - [initandlisten] Detected data files in /uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2019-11-15T10:49:20.173-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),

2019-11-15T10:49:20.192-0700 E STORAGE [initandlisten] WiredTiger error (0) [1573840160:192088][19007:0x2b93148a0e00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

2019-11-15T10:49:20.192-0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1573840160:192153][19007:0x2b93148a0e00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic

2019-11-15T10:49:20.192-0700 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361

2019-11-15T10:49:20.192-0700 I - [initandlisten] 

***aborting after fassert() failure

2019-11-15T10:49:20.201-0700 F - [initandlisten] Got signal: 6 (Aborted).

 0x563911ebad6b 0x563911eb9fe6 0x563911eba4c3 0x2b93158325d0 0x2b9315a75207 0x2b9315a768f8 0x5639110d9887 0x563911bb1597 0x5639110e3e75 0x5639110e408c 0x5639110e42f5 0x5639128b19d7 0x5639128b01d9 0x563912865893 0x5639128f0974 0x5639128f0f1f 0x5639128f11cc 0x5639128738a9 0x5639128e6618 0x5639128af8fe 0x5639128af9eb 0x563912861e6a 0x563911b96685 0x563911b8e9ca 0x563911a7450a 0x5639110e5bab 0x2b9315a613d5 0x563911149cff

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"563910A9A000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"563910A9A000","o":"141FFE6"},{"b":"563910A9A000","o":"14204C3"},{"b":"2B9315823000","o":"F5D0"},{"b":"2B9315A3F000","o":"36207","s":"gsignal"},{"b":"2B9315A3F000","o":"378F8","s":"abort"},{"b":"563910A9A000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"563910A9A000","o":"1117597"},{"b":"563910A9A000","o":"649E75","s":"__wt_eventv"},{"b":"563910A9A000","o":"64A08C","s":"__wt_err"},{"b":"563910A9A000","o":"64A2F5","s":"__wt_panic"},{"b":"563910A9A000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"563910A9A000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"563910A9A000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"563910A9A000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"563910A9A000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"563910A9A000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"563910A9A000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"563910A9A000","o":"1E4C618"},{"b":"563910A9A000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"563910A9A000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"563910A9A000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"563910A9A000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"563910A9A000","o":"10F49CA"},{"b":"563910A9A000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"563910A9A000","o":"64BBAB","s":"main"},{"b":"2B9315A3F000","o":"223D5","s":"__libc_start_main"},{"b":"563910A9A000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.5.1.el7.x86_64", "version" : "#1 SMP Fri Feb 1 14:54:57 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "563910A9A000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFF99AC2000", "elfType" : 3, "buildId" : "733D1C681CFBD8BBEB11E8B7F80876A9AED6BB74" }, { "b" : "2B93146A7000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2B93148A8000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2B9314B1E000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2B9314FA1000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "2B93151A9000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "2B93153AD000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2B9315521000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "2B931488A000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2B9315823000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "2B9315A3F000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A644E865A1F672B46A1232F7C275F53B06042696" }, { "b" : "2B9315E0C000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "2B9314684000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x563911ebad6b]

 mongod(+0x141FFE6) [0x563911eb9fe6]

 mongod(+0x14204C3) [0x563911eba4c3]

 libpthread.so.0(+0xF5D0) [0x2b93158325d0]

 libc.so.6(gsignal+0x37) [0x2b9315a75207]

 libc.so.6(abort+0x148) [0x2b9315a768f8]

 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5639110d9887]

 mongod(+0x1117597) [0x563911bb1597]

 mongod(__wt_eventv+0x3E4) [0x5639110e3e75]

 mongod(__wt_err+0xA0) [0x5639110e408c]

 mongod(__wt_panic+0x2F) [0x5639110e42f5]

 mongod(__wt_turtle_read+0x227) [0x5639128b19d7]

 mongod(__wt_metadata_search+0x99) [0x5639128b01d9]

 mongod(__wt_conn_btree_open+0x73) [0x563912865893]

 mongod(__wt_session_get_btree+0xE4) [0x5639128f0974]

 mongod(__wt_session_get_btree+0x68F) [0x5639128f0f1f]

 mongod(__wt_session_get_btree_ckpt+0x14C) [0x5639128f11cc]

 mongod(__wt_curfile_open+0x169) [0x5639128738a9]

 mongod(+0x1E4C618) [0x5639128e6618]

 mongod(__wt_metadata_cursor_open+0x6E) [0x5639128af8fe]

 mongod(__wt_metadata_cursor+0x4B) [0x5639128af9eb]

 mongod(wiredtiger_open+0x183A) [0x563912861e6a]

 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x563911b96685]

 mongod(+0x10F49CA) [0x563911b8e9ca]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x563911a7450a]

 mongod(main+0xE9B) [0x5639110e5bab]

 libc.so.6(__libc_start_main+0xF5) [0x2b9315a613d5]

 mongod(+0x6AFCFF) [0x563911149cff]

----- END BACKTRACE -----

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] MongoDB starting : pid=19020 port=39001 dbpath=/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database 64-bit host=cryosparc03.chpc.utah.edu

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] allocator: tcmalloc

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] modules: none

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] build environment:

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] distarch: x86_64

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] target_arch: x86_64

2019-11-15T10:49:23.290-0700 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }

2019-11-15T10:49:23.318-0700 I - [initandlisten] Detected data files in /uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2019-11-15T10:49:23.318-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),

2019-11-15T10:49:23.331-0700 E STORAGE [initandlisten] WiredTiger error (0) [1573840163:331761][19020:0x2aad0be2de00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

2019-11-15T10:49:23.331-0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1573840163:331820][19020:0x2aad0be2de00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic

2019-11-15T10:49:23.331-0700 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361

2019-11-15T10:49:23.331-0700 I - [initandlisten] 

***aborting after fassert() failure

2019-11-15T10:49:23.337-0700 F - [initandlisten] Got signal: 6 (Aborted).

 0x558e47a7cd6b 0x558e47a7bfe6 0x558e47a7c4c3 0x2aad0cdbf5d0 0x2aad0d002207 0x2aad0d0038f8 0x558e46c9b887 0x558e47773597 0x558e46ca5e75 0x558e46ca608c 0x558e46ca62f5 0x558e484739d7 0x558e484721d9 0x558e48427893 0x558e484b2974 0x558e484b2f1f 0x558e484b31cc 0x558e484358a9 0x558e484a8618 0x558e484718fe 0x558e484719eb 0x558e48423e6a 0x558e47758685 0x558e477509ca 0x558e4763650a 0x558e46ca7bab 0x2aad0cfee3d5 0x558e46d0bcff

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"558E4665C000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"558E4665C000","o":"141FFE6"},{"b":"558E4665C000","o":"14204C3"},{"b":"2AAD0CDB0000","o":"F5D0"},{"b":"2AAD0CFCC000","o":"36207","s":"gsignal"},{"b":"2AAD0CFCC000","o":"378F8","s":"abort"},{"b":"558E4665C000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"558E4665C000","o":"1117597"},{"b":"558E4665C000","o":"649E75","s":"__wt_eventv"},{"b":"558E4665C000","o":"64A08C","s":"__wt_err"},{"b":"558E4665C000","o":"64A2F5","s":"__wt_panic"},{"b":"558E4665C000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"558E4665C000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"558E4665C000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"558E4665C000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"558E4665C000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"558E4665C000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"558E4665C000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"558E4665C000","o":"1E4C618"},{"b":"558E4665C000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"558E4665C000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"558E4665C000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"558E4665C000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"558E4665C000","o":"10F49CA"},{"b":"558E4665C000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"558E4665C000","o":"64BBAB","s":"main"},{"b":"2AAD0CFCC000","o":"223D5","s":"__libc_start_main"},{"b":"558E4665C000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.5.1.el7.x86_64", "version" : "#1 SMP Fri Feb 1 14:54:57 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "558E4665C000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFFB02C1000", "elfType" : 3, "buildId" : "733D1C681CFBD8BBEB11E8B7F80876A9AED6BB74" }, { "b" : "2AAD0BC34000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2AAD0BE35000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2AAD0C0AB000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2AAD0C52E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "2AAD0C736000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "2AAD0C93A000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2AAD0CAAE000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "2AAD0BE17000", "path" : "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2AAD0CDB0000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "2AAD0CFCC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A644E865A1F672B46A1232F7C275F53B06042696" }, { "b" : "2AAD0D399000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "BFB22A176189006BBF996A7664CCC6BDB2EB9BD9" }, { "b" : "2AAD0BC11000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" } ] }}

 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x558e47a7cd6b]

 mongod(+0x141FFE6) [0x558e47a7bfe6]

 mongod(+0x14204C3) [0x558e47a7c4c3]

 libpthread.so.0(+0xF5D0) [0x2aad0cdbf5d0]

 libc.so.6(gsignal+0x37) [0x2aad0d002207]

 libc.so.6(abort+0x148) [0x2aad0d0038f8]

 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x558e46c9b887]

 mongod(+0x1117597) [0x558e47773597]

 mongod(__wt_eventv+0x3E4) [0x558e46ca5e75]

 mongod(__wt_err+0xA0) [0x558e46ca608c]

 mongod(__wt_panic+0x2F) [0x558e46ca62f5]

 mongod(__wt_turtle_read+0x227) [0x558e484739d7]

 mongod(__wt_metadata_search+0x99) [0x558e484721d9]

 mongod(__wt_conn_btree_open+0x73) [0x558e48427893]

 mongod(__wt_session_get_btree+0xE4) [0x558e484b2974]

 mongod(__wt_session_get_btree+0x68F) [0x558e484b2f1f]

 mongod(__wt_session_get_btree_ckpt+0x14C) [0x558e484b31cc]

 mongod(__wt_curfile_open+0x169) [0x558e484358a9]

 mongod(+0x1E4C618) [0x558e484a8618]

 mongod(__wt_metadata_cursor_open+0x6E) [0x558e484718fe]

 mongod(__wt_metadata_cursor+0x4B) [0x558e484719eb]

 mongod(wiredtiger_open+0x183A) [0x558e48423e6a]

 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x558e47758685]

 mongod(+0x10F49CA) [0x558e477509ca]

 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x558e4763650a]

 mongod(main+0xE9B) [0x558e46ca7bab]

 libc.so.6(__libc_start_main+0xF5) [0x2aad0cfee3d5]

 mongod(+0x6AFCFF) [0x558e46d0bcff]

----- END BACKTRACE -----

irv555 · November 15, 2019, 9:13pm

Just a note here. I have a cron job that backs up the database everyday at 3:30am. The last backup ran 11.11 at 3:31 and then later that same day is when the problem was reported. I tried to restore from backup, but got this:

[cryosparc03sa@cryosparc03:~]$ cryosparcm restore --file=/scratch/general/lustre/cryosparc03sa-db-bak/cryosparc_backup_2019_11_11_03h31.archive

Restoring from /scratch/general/lustre/cryosparc03sa-db-bak/cryosparc_backup_2019_11_11_03h31.archive

CryoSPARC is running.

Starting mongod incase it's not already running.

database: ERROR (spawn error)

Starting mongorestore.

2019-11-15T14:10:35.259-0700 Failed: error connecting to db server: no reachable servers

stephan · November 15, 2019, 9:27pm

Hi @irv555 & @hansenbry,

This might be caused by the following series of events:

The initial sock file goes missing (this can happen if the /tmp directory is cleared
The original mongo process is still running, but supervisor doesn’t know that its running because there is no sock file
The backup/restore commands use supervisor to start the database if its not already running
Supervisor reports to the backup/restore commands that the mongo process is not running
The backup/restore commands turn on the database using supervisor, creating a new sock file
The mongo database throws exceptions because it has two processes trying to acquire a lock on the db

Please kill the mongo processes, delete the lock files, and restart cryoSPARC.

ps -ax | grep “mongod”
kill <process_pid>
//delete the .lock file
cryosparcm start

irv555 · November 15, 2019, 9:36pm

I didn’t have any mongod processes. I deleted the mongod.lock file in ~/cryosparc2_database, but still get the same message when I try to start:

[cryosparc03sa@cryosparc03:tmp]$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
    "listDatabases")["databases"]]
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
    with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
    with self._get_socket(read_preference) as sock_info:
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/uufs/chpc.utah.edu/common/home/cryoem-group1/cryosparc03sa/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused

irv555 · November 20, 2019, 7:37pm

Since it appears that there was no easy resolution to this issue, I reinstalled v2.11.0 and restored our last good database backup. We had upgraded to v2.11.2-private_beta, but it would just download an empty tar file. Has it been pulled or moved?

irv555 · May 19, 2021, 9:20pm

Hello again. I have another new cryosSPARC system (A40 GPU’s) that also has the database spawn error today. It’s running cryoSPARC v3.0.1. I have tried the database repair steps you gave me earlier with no luck. I have tried killing all processes and the lock file. Nothing is letting the database start again. Thus, I can’t do a restore of the database from backup. What is the best way to proceed so that I can restore from backups.

nfrasser · May 21, 2021, 8:31pm

@irv555 I’m still investigating the issue; in the mean time could you try the following?

Make sure the base port for the new system is the same as the previous system
Send me the output you see when you run the cryosparcm restore command
Send me the latest output/error messages from the following commands:
```
cryosparcm status
cryosparcm restart
cryosparcm status
```

nfrasser · May 26, 2021, 7:37pm

@irv555 could you additionally try and send me the following?

Restart the machine and try restoring again
Send me the database log at cryosparc_master/log/database.log
Send me the database’s replica set configuration with the following commands:

cryosparcm restart
cryosparcm mongo
rs.conf()

(Press Ctrl+C to exit the mongo prompt)