HI,
I have this error when I try to restart cryosparc.
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
"listDatabases")["databases"]]
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
server = self._get_topology().select_server(selector)
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
address))
File "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused
I have try to stop it and check with ps -ax | grep “supervisord” and there is no process.
This is the cryosparc status:
bash-4.2$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/binshtem/software/cryosparc/cryosparc2_master
Current cryoSPARC version: v2.12.4
----------------------------------------------------------------------------
cryosparcm is not running.
----------------------------------------------------------------------------
global config variables:
export CRYOSPARC_LICENSE_ID="xxxxxx"
export CRYOSPARC_MASTER_HOSTNAME="xxxxxxx"
export CRYOSPARC_DB_PATH="/home/xxxxxx/software/cryosparc/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
cryosparcm process status after restart with error:
app STOPPED Not started
app_dev STOPPED Not started
command_core STOPPED Not started
command_proxy STOPPED Not started
command_rtp STOPPED Not started
command_vis STOPPED Not started
database FATAL Exited too quickly (process log may have details)
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started
this is the database log:
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] MongoDB starting : pid=36141 port=39001 dbpath=/home/binshtem/software/cryosparc/cryosparc2_database 64-bit host=cryogpu2.csb.vanderbilt.edu
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] allocator: tcmalloc
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] modules: none
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] build environment:
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] distarch: x86_64
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] target_arch: x86_64
2020-01-02T10:21:25.033-0600 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/home/binshtem/software/cryosparc/cryosparc2_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2020-01-02T10:21:25.036-0600 W - [initandlisten] Detected unclean shutdown - /home/binshtem/software/cryosparc/cryosparc2_database/mongod.lock is not empty.
2020-01-02T10:21:25.072-0600 I - [initandlisten] Detected data files in /home/binshtem/software/cryosparc/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-01-02T10:21:25.072-0600 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-01-02T10:21:25.073-0600 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2020-01-02T10:21:25.087-0600 E STORAGE [initandlisten] WiredTiger error (0) [1577982085:87787][36141:0x2ac87000be00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2020-01-02T10:21:25.087-0600 E STORAGE [initandlisten] WiredTiger error (-31804) [1577982085:87825][36141:0x2ac87000be00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-01-02T10:21:25.087-0600 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2020-01-02T10:21:25.087-0600 I - [initandlisten]
***aborting after fassert() failure
2020-01-02T10:21:25.095-0600 F - [initandlisten] Got signal: 6 (Aborted).
0x55c81c8b5d6b 0x55c81c8b4fe6 0x55c81c8b54c3 0x2ac870f805e0 0x2ac8711c21f7 0x2ac8711c38e8 0x55c81bad4887 0x55c81c5ac597 0x55c81badee75 0x55c81badf08c 0x55c81badf2f5 0x55c81d2ac9d7 0x55c81d2ab1d9 0x55c81d260893 0x55c81d2eb974 0x55c81d2ebf1f 0x55c81d2ec1cc 0x55c81d26e8a9 0x55c81d2e1618 0x55c81d2aa8fe 0x55c81d2aa9eb 0x55c81d25ce6a 0x55c81c591685 0x55c81c5899ca 0x55c81c46f50a 0x55c81bae0bab 0x2ac8711aec05 0x55c81bb44cff
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55C81B495000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55C81B495000","o":"141FFE6"},{"b":"55C81B495000","o":"14204C3"},{"b":"2AC870F71000","o":"F5E0"},{"b":"2AC87118D000","o":"351F7","s":"gsignal"},{"b":"2AC87118D000","o":"368E8","s":"abort"},{"b":"55C81B495000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55C81B495000","o":"1117597"},{"b":"55C81B495000","o":"649E75","s":"__wt_eventv"},{"b":"55C81B495000","o":"64A08C","s":"__wt_err"},{"b":"55C81B495000","o":"64A2F5","s":"__wt_panic"},{"b":"55C81B495000","o":"1E179D7","s":"__wt_turtle_read"},{"b":"55C81B495000","o":"1E161D9","s":"__wt_metadata_search"},{"b":"55C81B495000","o":"1DCB893","s":"__wt_conn_btree_open"},{"b":"55C81B495000","o":"1E56974","s":"__wt_session_get_btree"},{"b":"55C81B495000","o":"1E56F1F","s":"__wt_session_get_btree"},{"b":"55C81B495000","o":"1E571CC","s":"__wt_session_get_btree_ckpt"},{"b":"55C81B495000","o":"1DD98A9","s":"__wt_curfile_open"},{"b":"55C81B495000","o":"1E4C618"},{"b":"55C81B495000","o":"1E158FE","s":"__wt_metadata_cursor_open"},{"b":"55C81B495000","o":"1E159EB","s":"__wt_metadata_cursor"},{"b":"55C81B495000","o":"1DC7E6A","s":"wiredtiger_open"},{"b":"55C81B495000","o":"10FC685","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55C81B495000","o":"10F49CA"},{"b":"55C81B495000","o":"FDA50A","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55C81B495000","o":"64BBAB","s":"main"},{"b":"2AC87118D000","o":"21C05","s":"__libc_start_main"},{"b":"55C81B495000","o":"6AFCFF"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.27.2.el7.x86_64", "version" : "#1 SMP Mon Jul 29 17:46:05 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55C81B495000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFF486D6000", "elfType" : 3, "buildId" : "88212B24946DE81A7B05A4F4422F5A8B37CBD908" }, { "b" : "2AC86FE12000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2AC870014000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2AC87028A000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2AC8706EF000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "6D322588B36D2617C03C0F3B93677E62FCFFDA81" }, { "b" : "2AC8708F7000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1E42EBFB272D37B726F457D6FE3C33D2B094BB69" }, { "b" : "2AC870AFB000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2AC870C6F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "808BD35686C193F218A5AAAC6194C49214CFF379" }, { "b" : "2AC86FFF5000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2AC870F71000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "A48D21B2578A8381FBD8857802EAA660504248DC" }, { "b" : "2AC87118D000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "95FF02A4BEBABC573C7827A66D447F7BABDDAA44" }, { "b" : "2AC871550000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "C1CAEB92FAB92F5D9AFBA13867535EFD4BB55055" }, { "b" : "2AC86FDF0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "22FA66DA7D14C88BF36C69454A357E5F1DEFAE4E" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x55c81c8b5d6b]
mongod(+0x141FFE6) [0x55c81c8b4fe6]
mongod(+0x14204C3) [0x55c81c8b54c3]
libpthread.so.0(+0xF5E0) [0x2ac870f805e0]
libc.so.6(gsignal+0x37) [0x2ac8711c21f7]
libc.so.6(abort+0x148) [0x2ac8711c38e8]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c81bad4887]
mongod(+0x1117597) [0x55c81c5ac597]
mongod(__wt_eventv+0x3E4) [0x55c81badee75]
mongod(__wt_err+0xA0) [0x55c81badf08c]
mongod(__wt_panic+0x2F) [0x55c81badf2f5]
mongod(__wt_turtle_read+0x227) [0x55c81d2ac9d7]
mongod(__wt_metadata_search+0x99) [0x55c81d2ab1d9]
mongod(__wt_conn_btree_open+0x73) [0x55c81d260893]
mongod(__wt_session_get_btree+0xE4) [0x55c81d2eb974]
mongod(__wt_session_get_btree+0x68F) [0x55c81d2ebf1f]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55c81d2ec1cc]
mongod(__wt_curfile_open+0x169) [0x55c81d26e8a9]
mongod(+0x1E4C618) [0x55c81d2e1618]
mongod(__wt_metadata_cursor_open+0x6E) [0x55c81d2aa8fe]
mongod(__wt_metadata_cursor+0x4B) [0x55c81d2aa9eb]
mongod(wiredtiger_open+0x183A) [0x55c81d25ce6a]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x55c81c591685]
mongod(+0x10F49CA) [0x55c81c5899ca]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x55c81c46f50a]
mongod(main+0xE9B) [0x55c81bae0bab]
libc.so.6(__libc_start_main+0xF5) [0x2ac8711aec05]
mongod(+0x6AFCFF) [0x55c81bb44cff]
----- END BACKTRACE -----
Thanks
