Database: ERROR (spawn error) after updated V2.8

closed

(lifei fu) #1

Hi
I updated cryosparc from V2.5 to V2.8 in ubuntu. It installed successfully without error messages.
but when I started cryosparc. there is error about the database can not be started.

CryoSPARC is running.
Stopping cryosparc.
command_proxy: stopped
command_vis: stopped
webapp: stopped
command_core: stopped
database: stopped
Shut down
Starting cryoSPARC System master process…
CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
File “”, line 1, in
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 1149, in database_names
“listDatabases”)[“databases”]]
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py”, line 491, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py”, line 17, in enter
return self.gen.next()
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 859, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py”, line 17, in enter
return self.gen.next()
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 823, in _get_socket
server = self._get_topology().select_server(selector)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 214, in select_server
address))
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused

run with cryosparcm log command_core command

*** BG WORKER START
======== Creating User: marcel@gmail.com marcel
Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 180, in background_worker
check_heartbeats()
File “cryosparc2_command/command_core/init.py”, line 1373, in check_heartbeats
‘heartbeat_at’ : {’$lt’ : deadline} }, {‘project_uid’ : 1, ‘uid’ : 1}))
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1114, in next
if len(self.__data) or self._refresh():
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1036, in _refresh
self.__collation))
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 873, in __send_message
**kwargs)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 905, in _send_message_with_response
exhaust)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 916, in _reset_on_error
return func(*args, **kwargs)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/server.py”, line 136, in send_message_with_response
response_data = sock_info.receive_message(1, request_id)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/pool.py”, line 452, in receive_message
self._raise_connection_failure(error)

File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/pool.py”, line 550, in _raise_connection_failure
_raise_connection_failure(self.address, error)
File “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/pool.py”, line 211, in _raise_connection_failure
raise AutoReconnect(msg)
AutoReconnect: ubuntu-hellion:39001: [Errno 104] Connection reset by peer
****** Heartbeat check failed ****

Has anyone any suggestion, please?

Best,
Lifei


(lifei fu) #2

here is the database log file:
2019-06-13T12:30:56.097+0200 F - [initandlisten] Got signal: 6 (Aborted).

0x55ecd4b95d6b 0x55ecd4b94fe6 0x55ecd4b954c3 0x7f792637a390 0x7f7925fd4428 0x7f7925fd602a 0x55ecd3db4887 0x55ecd488c597 0x55ecd3dbee75 0x55ecd3dbf08c 0x55ecd3dbf2f5 0x55ecd54e875f 0x55ecd54e8c9b 0x55ecd54e5087 0x55ecd54e9e27 0x55ecd5503823 0x55ecd55409b9 0x55ecd55cb974 0x55ecd55cbf1f 0x55ecd55cc1cc 0x55ecd554e8a9 0x55ecd55c1618 0x55ecd558a8fe 0x55ecd558a9eb 0x55ecd553ce6a 0x55ecd4871685 0x55ecd48699ca 0x55ecd474f50a 0x55ecd3dc0bab 0x7f7925fbf830 0x55ecd3e24cff
----- BEGIN BACKTRACE -----
{“backtrace”:[{“b”:“55ECD3775000”,“o”:“1420D6B”,“s”:"_ZN5mongo15printStackTraceERSo"},{“b”:“55ECD3775000”,“o”:“141FFE6”},{“b”:“55ECD3775000”,“o”:“14204C3”},{“b”:“7F7926369000”,“o”:“11390”},{“b”:“7F7925F9F000”,“o”:“35428”,“s”:“gsignal”},{“b”:“7F7925F9F000”,“o”:“3702A”,“s”:“abort”},{“b”:“55ECD3775000”,“o”:“63F887”,“s”:"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{“b”:“55ECD3775000”,“o”:“1117597”},{“b”:“55ECD3775000”,“o”:“649E75”,“s”:"__wt_eventv"},{“b”:“55ECD3775000”,“o”:“64A08C”,“s”:"__wt_err"},{“b”:“55ECD3775000”,“o”:“64A2F5”,“s”:"__wt_panic"},{“b”:“55ECD3775000”,“o”:“1D7375F”,“s”:"__wt_block_extlist_read"},{“b”:“55ECD3775000”,“o”:“1D73C9B”,“s”:"__wt_block_extlist_read_avail"},{“b”:“55ECD3775000”,“o”:“1D70087”,“s”:"__wt_block_checkpoint_load"},{“b”:“55ECD3775000”,“o”:“1D74E27”},{“b”:“55ECD3775000”,“o”:“1D8E823”,“s”:"__wt_btree_open"},{“b”:“55ECD3775000”,“o”:“1DCB9B9”,“s”:"__wt_conn_btree_open"},{“b”:“55ECD3775000”,“o”:“1E56974”,“s”:"__wt_session_get_btree"},{“b”:“55ECD3775000”,“o”:“1E56F1F”,“s”:"__wt_session_get_btree"},{“b”:“55ECD3775000”,“o”:“1E571CC”,“s”:"__wt_session_get_btree_ckpt"},{“b”:“55ECD3775000”,“o”:“1DD98A9”,“s”:"__wt_curfile_open"},{“b”:“55ECD3775000”,“o”:“1E4C618”},{“b”:“55ECD3775000”,“o”:“1E158FE”,“s”:"__wt_metadata_cursor_open"},{“b”:“55ECD3775000”,“o”:“1E159EB”,“s”:"__wt_metadata_cursor"},{“b”:“55ECD3775000”,“o”:“1DC7E6A”,“s”:“wiredtiger_open”},{“b”:“55ECD3775000”,“o”:“10FC685”,“s”:"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{“b”:“55ECD3775000”,“o”:“10F49CA”},{“b”:“55ECD3775000”,“o”:“FDA50A”,“s”:"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{“b”:“55ECD3775000”,“o”:“64BBAB”,“s”:“main”},{“b”:“7F7925F9F000”,“o”:“20830”,“s”:"__libc_start_main"},{“b”:“55ECD3775000”,“o”:“6AFCFF”}],“processInfo”:{ “mongodbVersion” : “3.4.10-4-g67ee356c6b”, “gitVersion” : “67ee356c6be377cda547d16423daef5beb4e8377”, “compiledModules” : [], “uname” : { “sysname” : “Linux”, “release” : “4.4.0-150-generic”, “version” : “#176-Ubuntu SMP Wed May 29 18:56:26 UTC 2019”, “machine” : “x86_64” }, “somap” : [ { “b” : “55ECD3775000”, “elfType” : 3, “buildId” : “A14732EAAA95508462633A5030B501E08B1B106F” }, { “b” : “7FFCE0D67000”, “elfType” : 3, “buildId” : “06792BE30644E35D9A29CC94FE43976F73FFB31A” }, { “b” : “7F79274F9000”, “path” : “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0”, “elfType” : 3 }, { “b” : “7F792723F000”, “path” : “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/bin/…/lib/libssl.so.1.0.0”, “elfType” : 3 }, { “b” : “7F7926DFE000”, “path” : “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/bin/…/lib/libcrypto.so.1.0.0”, “elfType” : 3 }, { “b” : “7F7926BD4000”, “path” : “/lib/x86_64-linux-gnu/librt.so.1”, “elfType” : 3, “buildId” : “69143E8B39040C964D3958490535322675F15DD3” }, { “b” : “7F79269D0000”, “path” : “/lib/x86_64-linux-gnu/libdl.so.2”, “elfType” : 3, “buildId” : “37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD” }, { “b” : “7F792688F000”, “path” : “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/bin/…/lib/libstdc++.so.6”, “elfType” : 3 }, { “b” : “7F7926586000”, “path” : “/lib/x86_64-linux-gnu/libm.so.6”, “elfType” : 3, “buildId” : “BAD67A84E56E73D031AE507261DA066B35949D34” }, { “b” : “7F79274E2000”, “path” : “/home/hellion/cryosparcv2/cryosparc2_master/deps/anaconda/bin/…/lib/libgcc_s.so.1”, “elfType” : 3 }, { “b” : “7F7926369000”, “path” : “/lib/x86_64-linux-gnu/libpthread.so.0”, “elfType” : 3, “buildId” : “B17C21299099640A6D863E423D99265824E7BB16” }, { “b” : “7F7925F9F000”, “path” : “/lib/x86_64-linux-gnu/libc.so.6”, “elfType” : 3, “buildId” : “1CA54A6E0D76188105B12E49FE6B8019BF08803A” }, { “b” : “7F7925D9C000”, “path” : “/lib/x86_64-linux-gnu/libutil.so.1”, “elfType” : 3, “buildId” : “3A65026AA054865D6CBB84A15ADBF178DC14CAE5” }, { “b” : “7F79274B5000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x55ecd4b95d6b]
mongod(+0x141FFE6) [0x55ecd4b94fe6]
mongod(+0x14204C3) [0x55ecd4b954c3]
libpthread.so.0(+0x11390) [0x7f792637a390]
libc.so.6(gsignal+0x38) [0x7f7925fd4428]
libc.so.6(abort+0x16A) [0x7f7925fd602a]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55ecd3db4887]
mongod(+0x1117597) [0x55ecd488c597]
mongod(__wt_eventv+0x3E4) [0x55ecd3dbee75]
mongod(__wt_err+0xA0) [0x55ecd3dbf08c]
mongod(__wt_panic+0x2F) [0x55ecd3dbf2f5]
mongod(__wt_block_extlist_read+0x8F) [0x55ecd54e875f]
mongod(__wt_block_extlist_read_avail+0x2B) [0x55ecd54e8c9b]
mongod(__wt_block_checkpoint_load+0x277) [0x55ecd54e5087]
mongod(+0x1D74E27) [0x55ecd54e9e27]
mongod(__wt_btree_open+0xAC3) [0x55ecd5503823]
mongod(__wt_conn_btree_open+0x199) [0x55ecd55409b9]
mongod(__wt_session_get_btree+0xE4) [0x55ecd55cb974]
mongod(__wt_session_get_btree+0x68F) [0x55ecd55cbf1f]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x55ecd55cc1cc]
mongod(__wt_curfile_open+0x169) [0x55ecd554e8a9]
mongod(+0x1E4C618) [0x55ecd55c1618]
mongod(__wt_metadata_cursor_open+0x6E) [0x55ecd558a8fe]
mongod(__wt_metadata_cursor+0x4B) [0x55ecd558a9eb]
mongod(wiredtiger_open+0x183A) [0x55ecd553ce6a]
mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x815) [0x55ecd4871685]
mongod(+0x10F49CA) [0x55ecd48699ca]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x69A) [0x55ecd474f50a]
mongod(main+0xE9B) [0x55ecd3dc0bab]
libc.so.6(__libc_start_main+0xF0) [0x7f7925fbf830]
mongod(+0x6AFCFF) [0x55ecd3e24cff]
----- END BACKTRACE -----


(Stephan Arulthasan) #3

Hi @biolifei,

Is it possible that there was a zombie process running while you were updating cryoSPARC? Please stop cryoSPARC cryosparcm stop then check if there are any running cryoSPARC processes ps -ax | grep "supervisord" and kill any processes related to cryoSPARC. You can then start cryosparcm start and see if you get the same error.


(lifei fu) #4

Hi,
thanks for your reply. I think it is not about supervisord this zombie process running. I tried to kill the supervisord" process and it didn’t work.
I backed up the database before cryosparc is updated to 2.8. So, when I deleted all the files in the database fold and restart cryosparc, in this way, all the projects will be removed, it works without any error messages.
But when I copy the old database files back to the database folder, it showed the same error again.
Now, I updated it to the cryosparc V2.8.3, it has same problem, the old projects can be loaded anymore.


(Christine Gee) #5

Hi, I just had an issue that was due to ownership problems. Some files in the database folder were owned by root somehow and a log file in the cryosparc_master/run folder. This is because somehow a job was launced as root. I don’t understand how…