Hi, after upgrading from v2.0.27 to v2.3.0 things don’t work anymore. Any ideas what pymongo is doing (why is it not connecting)?
here’s the full output:
> cryosparcm update
CryoSPARC current version v2.0.27
update starting on Mon Oct 1 17:00:56 CEST 2018
No version specified - updating to latest version.
=============================
Updating to version v2.3.0.
=============================
CryoSPARC is running.
Stopping cryosparc.
unix:///tmp/cryosparc-supervisor-6539593591723470764.sock refused connection
ERROR: unix:///tmp/cryosparc-supervisor-6539593591723470764.sock refused connection (already shut down?)
Downloading master update...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 772M 100 772M 0 0 1337k 0 0:09:51 0:09:51 --:--:-- 4779k
Downloading worker update...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 470M 100 470M 0 0 1389k 0 0:05:46 0:05:46 --:--:-- 2046k
Done.
Update will now be applied to the master installation,
followed by worker installations on other node.
Deleting old files...
Extracting...
Done.
Updating dependencies...
Checking dependencies...
Legacy hash (v2.2 & older) detected. No new python dependencies need to be installed.
Currently checking hash for mongodb
Dependencies for mongodb have changed - reinstalling...
Completed.
Completed dependency check.
Successfully updated master from version v2.0.27 to version v2.3.0.
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
"listDatabases")["databases"]]
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
server = self._get_topology().select_server(selector)
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
address))
File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused
mongo troubleshooting:
> mongo localhost:39001
MongoDB shell version v3.4.10-4-g67ee356c6b
connecting to: localhost:39001
2018-10-04T10:46:30.126+0200 W NETWORK [thread1] Failed to connect to 127.0.0.1:39001, in(checking socket for error after poll), reason: Connection refused
2018-10-04T10:46:30.126+0200 E QUERY [thread1] Error: couldn't connect to server localhost:39001, connection attempt failed :
connect@src/mongo/shell/mongo.js:237:13
@(connect):1:6
exception: connect failed
incidentally, by ctrl-c during start process I think I killed, well corrupted now my database. this is bad news for me right?
before it was:
> cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.3.0
----------------------------------------------------------------------------
cryosparcm process status:
command_core STOPPED Not started
command_proxy STOPPED Not started
command_vis STOPPED Not started
database EXITED Oct 01 05:18 PM
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started
----------------------------------------------------------------------------
global config variables:
export CRYOSPARC_LICENSE_ID="05539ab2-a2f5-11e8-844b-d7d811127706"
export CRYOSPARC_MASTER_HOSTNAME="hpcl5006"
export CRYOSPARC_DB_PATH="/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
now status is saying:
> cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.3.0
----------------------------------------------------------------------------
cryosparcm process status:
command_core STOPPED Not started
command_proxy STOPPED Not started
command_vis STOPPED Not started
database FATAL Exited too quickly (process log may have details)
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started
----------------------------------------------------------------------------
global config variables:
export CRYOSPARC_LICENSE_ID="05539ab2-a2f5-11e8-844b-d7d811127706"
export CRYOSPARC_MASTER_HOSTNAME="hpcl5006"
export CRYOSPARC_DB_PATH="/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
I didn’t get those errors when I made sure to do “cryosparcm stop” and then manually killed the cryosparc python process before running the update command.
It is possible that a cryosparc process was orphaned unintentionally before updating, causing the update process to fail. Expanding on what @donaldb said, make sure cryosparc is stopped (cryosparcm stop) and then run ps -ax | grep "supervisord" to find the PID’s of cryoSPARC 2 processes which you can then kill. You might also need to delete this file: unix:///tmp/cryosparc-supervisor-6539593591723470764.sock
In terms of mongo, check the process log/mongo log mentioned for possibly more details in how to revive it. If you’re able to salvage the database, you can reinstall the cryosparc master and your instance should be fine.