Mongo connection refused

Hi, after upgrading from v2.0.27 to v2.3.0 things don’t work anymore. Any ideas what pymongo is doing (why is it not connecting)?

here’s the full output:

> cryosparcm update 
CryoSPARC current version v2.0.27
          update starting on Mon Oct  1 17:00:56 CEST 2018

No version specified - updating to latest version.

=============================
Updating to version v2.3.0.
=============================
CryoSPARC is running.
Stopping cryosparc.
unix:///tmp/cryosparc-supervisor-6539593591723470764.sock refused connection
ERROR: unix:///tmp/cryosparc-supervisor-6539593591723470764.sock refused connection (already shut down?)
  Downloading master update...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  772M  100  772M    0     0  1337k      0  0:09:51  0:09:51 --:--:-- 4779k
  Downloading worker update...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  470M  100  470M    0     0  1389k      0  0:05:46  0:05:46 --:--:-- 2046k
  Done.

 Update will now be applied to the master installation, 
 followed by worker installations on other node.

  Deleting old files...
  Extracting...
  Done.
  Updating dependencies...
Checking dependencies... 
Legacy hash (v2.2 & older) detected. No new python dependencies need to be installed.
Currently checking hash for mongodb
Dependencies for mongodb have changed - reinstalling...
Completed.
Completed dependency check. 
 
Successfully updated master from version v2.0.27 to version v2.3.0.
 
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
    "listDatabases")["databases"]]
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
    with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
    with self._get_socket(read_preference) as sock_info:
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused

mongo troubleshooting:

> mongo localhost:39001
MongoDB shell version v3.4.10-4-g67ee356c6b
connecting to: localhost:39001
2018-10-04T10:46:30.126+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:39001, in(checking socket for error after poll), reason: Connection refused
2018-10-04T10:46:30.126+0200 E QUERY    [thread1] Error: couldn't connect to server localhost:39001, connection attempt failed :
connect@src/mongo/shell/mongo.js:237:13
@(connect):1:6
exception: connect failed

incidentally, by ctrl-c during start process I think I killed, well corrupted now my database. this is bad news for me right?

before it was:

> cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.3.0
----------------------------------------------------------------------------

cryosparcm process status:

command_core                     STOPPED   Not started
command_proxy                    STOPPED   Not started
command_vis                      STOPPED   Not started
database                         EXITED    Oct 01 05:18 PM
watchdog_dev                     STOPPED   Not started
webapp                           STOPPED   Not started
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID="05539ab2-a2f5-11e8-844b-d7d811127706"
export CRYOSPARC_MASTER_HOSTNAME="hpcl5006"
export CRYOSPARC_DB_PATH="/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

now status is saying:

> cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.3.0
----------------------------------------------------------------------------

cryosparcm process status:

command_core                     STOPPED   Not started
command_proxy                    STOPPED   Not started
command_vis                      STOPPED   Not started
database                         FATAL     Exited too quickly (process log may have details)
watchdog_dev                     STOPPED   Not started
webapp                           STOPPED   Not started
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID="05539ab2-a2f5-11e8-844b-d7d811127706"
export CRYOSPARC_MASTER_HOSTNAME="hpcl5006"
export CRYOSPARC_DB_PATH="/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

I had the same problem.

I didn’t get those errors when I made sure to do “cryosparcm stop” and then manually killed the cryosparc python process before running the update command.

Hi @mturk,

It is possible that a cryosparc process was orphaned unintentionally before updating, causing the update process to fail. Expanding on what @donaldb said, make sure cryosparc is stopped (cryosparcm stop) and then run ps -ax | grep "supervisord" to find the PID’s of cryoSPARC 2 processes which you can then kill. You might also need to delete this file: unix:///tmp/cryosparc-supervisor-6539593591723470764.sock

In terms of mongo, check the process log/mongo log mentioned for possibly more details in how to revive it. If you’re able to salvage the database, you can reinstall the cryosparc master and your instance should be fine.

Please let me know what happens.

stopped cryosparc, and checked no other sparc supervisord processes are running

tried reinstalling cryosparc:

  • conda install fails in the deps_bundle packages
link_names = {index[d]['name'] for d in link_dists}
    KeyError: Dist(_Dist__initd=True, channel=u'<unknown>', dist_name=u'libstdcxx-ng-7.2.0-h7a57d05_2', name=u'libstdcxx-ng', version=u'7.2.0', build_string=u'h7a57d05_2', build_number=2, with_features_depends=None, base_url=u'file:///fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps_bundle/conda_pkgs', platform=None)

sometimes the package in the KeyError is get_terminal_size-1.0.0-haa9412d_0, or libgfortran-3.0.0-1

cryosparc ld_lib is empty string?

CRYOSPARC_LD_LIBRARY_PATH=

python

> which python
/fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/bin/python

conda paths

> conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.31
       conda is private : False
      conda-env version : 4.3.31
    conda-build version : not installed
         python version : 2.7.14.final.0
       requests version : 2.18.4
       root environment : /fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda  (writable)
    default environment : /fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda
       envs directories : /fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/envs
                          /fs/home/mturk/.conda/envs
          package cache : /fs/fs04/lv01/home/b_baumei/mturk/cryosparc2/cryosparc2_master/deps/anaconda/pkgs
                          /fs/home/mturk/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.31 requests/2.18.4 CPython/2.7.14 Linux/3.12.74-60.64.40-default SUSE Linux Enterprise Server /12 glibc/2.19    
                UID:GID : 3564:250