Fresh install v2.14.2 issue with command core

[troz18]

Hi,

We recently installed a fresh OS (Centos7) on our GPU workstation. We are using CUDA10.2 and have successfully used RELION. However, our install of Cryosparc v2.14.2 seems to keep failing. It seems that the issue arises after the successful installation of the “Cryosparc Master installation”. I have tried to check for orphan processes and have deleted any tmp sock files in between installs with no luck. Please let me know if I have done anything wrong or if more information is necessary to address the problem.

Install Log:

./install.sh --standalone --license $LICENSE_ID --worker_path /home/tyler/cryosparc_server/software/cryosparc/cryosparc2_worker --cudapath /usr/local/cuda --ssdpath /drives/scratch/cryosparc_cache --initial_email "tcr54@institution.edu" --initial_password changed_password --initial_name "Tyler the Creator"

Install log:
Are the above settings correct?
1) Yes
2) No
#? 1
Continuing.

******************************************************************

 Setting up hard-coded config.sh environment variables

******************************************************************

 Installing all dependencies.

  Checking dependencies... 
  Dependencies for python have changed - reinstalling...
  ------------------------------------------------------------------------
  Installing anaconda python...
  ------------------------------------------------------------------------
PREFIX=/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda
installing: python-2.7.16-h9bab390_0 ...
Python 2.7.16 :: Anaconda, Inc.
installing: ca-certificates-2019.1.23-0 ...
installing: libgcc-ng-8.2.0-hdf63c60_1 ...
installing: libstdcxx-ng-8.2.0-hdf63c60_1 ...
installing: libffi-3.2.1-hd88cf55_4 ...
installing: ncurses-6.1-he6710b0_1 ...
installing: openssl-1.1.1b-h7b6447c_1 ...
installing: yaml-0.1.7-had09818_2 ...
installing: zlib-1.2.11-h7b6447c_3 ...
installing: libedit-3.1.20181209-hc058e9b_0 ...
installing: readline-7.0-h7b6447c_5 ...
installing: tk-8.6.8-hbc83047_0 ...
installing: sqlite-3.27.2-h7b6447c_0 ...
installing: asn1crypto-0.24.0-py27_0 ...
installing: certifi-2019.3.9-py27_0 ...
installing: chardet-3.0.4-py27_1 ...
installing: enum34-1.1.6-py27_1 ...
installing: futures-3.2.0-py27_0 ...
installing: idna-2.8-py27_0 ...
installing: ipaddress-1.0.22-py27_0 ...
installing: pycosat-0.6.3-py27h14c3975_0 ...
installing: pycparser-2.19-py27_0 ...
installing: pysocks-1.6.8-py27_0 ...
installing: ruamel_yaml-0.15.46-py27h14c3975_0 ...
installing: six-1.12.0-py27_0 ...
installing: cffi-1.12.2-py27h2e261b9_1 ...
installing: setuptools-41.0.0-py27_0 ...
installing: cryptography-2.6.1-py27h1ba5d50_0 ...
installing: wheel-0.33.1-py27_0 ...
installing: pip-19.0.3-py27_0 ...
installing: pyopenssl-19.0.0-py27_0 ...
installing: urllib3-1.24.1-py27_0 ...
installing: requests-2.21.0-py27_0 ...
installing: conda-4.6.14-py27_0 ...
installation finished.
  ------------------------------------------------------------------------
    Done.
    anaconda python installation successful.
  ------------------------------------------------------------------------
  Preparing to install all conda packages...
  ------------------------------------------------------------------------

Downloading and Extracting Packages
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
######################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
  ------------------------------------------------------------------------
    Done.
    conda packages installation successful.
  ------------------------------------------------------------------------
  Preparing to install all pip packages...
  ------------------------------------------------------------------------
Processing ./deps_bundle/python/python_packages/pip_packages/Flask-JSONRPC-0.3.1.tar.gz
Processing ./deps_bundle/python/python_packages/pip_packages/Flask-PyMongo-0.5.1.tar.gz
Processing ./deps_bundle/python/python_packages/pip_packages/libtiff-0.4.2.tar.gz
Requirement already satisfied: Flask>=0.10 in ./deps/anaconda/lib/python2.7/site-packages (from Flask-JSONRPC==0.3.1)
Requirement already satisfied: PyMongo>=2.5 in ./deps/anaconda/lib/python2.7/site-packages (from Flask-PyMongo==0.5.1)
Requirement already satisfied: Werkzeug>=0.7 in ./deps/anaconda/lib/python2.7/site-packages (from Flask>=0.10->Flask-JSONRPC==0.3.1)
Requirement already satisfied: Jinja2>=2.4 in ./deps/anaconda/lib/python2.7/site-packages (from Flask>=0.10->Flask-JSONRPC==0.3.1)
Requirement already satisfied: itsdangerous>=0.21 in ./deps/anaconda/lib/python2.7/site-packages (from Flask>=0.10->Flask-JSONRPC==0.3.1)
Requirement already satisfied: click>=2.0 in ./deps/anaconda/lib/python2.7/site-packages (from Flask>=0.10->Flask-JSONRPC==0.3.1)
Requirement already satisfied: MarkupSafe in ./deps/anaconda/lib/python2.7/site-packages (from Jinja2>=2.4->Flask>=0.10->Flask-JSONRPC==0.3.1)
Skipping bdist_wheel for Flask-JSONRPC, due to binaries being disabled for it.
Skipping bdist_wheel for Flask-PyMongo, due to binaries being disabled for it.
Skipping bdist_wheel for libtiff, due to binaries being disabled for it.
Installing collected packages: Flask-JSONRPC, Flask-PyMongo, libtiff
  Running setup.py install for Flask-JSONRPC ... done
  Running setup.py install for Flask-PyMongo ... done
  Running setup.py install for libtiff ... done
Successfully installed Flask-JSONRPC-0.3.1 Flask-PyMongo-0.5.1 libtiff-0.4.2
You are using pip version 9.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
  ------------------------------------------------------------------------
    Done.
    pip packages installation successful.
  ------------------------------------------------------------------------
  Main dependency installation completed. Continuing...
  ------------------------------------------------------------------------
  Completed.
  Currently checking hash for mongodb
  Dependencies for mongodb have changed - reinstalling...
  ------------------------------------------------------------------------
  mongodb 3.4.10 installation successful.
  ------------------------------------------------------------------------
  Completed.
  Completed dependency check. 

************ CRYOSPARC MASTER INSTALLATION COMPLETE **************

 CryoSPARC System master has now been installed. 
 Add bin directory to your ~/.bashrc ?
1) Yes
2) No
#? 1

 The following lines were appended to your ~/.bashrc:
  export PATH=/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/bin:$PATH

************ CRYOSPARC SYSTEM: STANDALONE INSTALLER **************

  The installer will now download, install and connect the worker
  package on this machine. The worker package will allow cryoSPARC
  to run compute jobs.

******************************************************************

 Starting cryoSPARC...

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

command_vis: started
command_proxy: started
webapp: started
-----------------------------------------------------

CryoSPARC master started. 
 From this machine, access cryoSPARC at
    http://localhost:39000

 From other machines on the network, access cryoSPARC at
    http://C149-B2:39000


Startup can take several minutes. Point your browser to the address
and refresh until you see the cryoSPARC web interface.

******************************************************************

 Creating the initial cryoSPARC user...

Creating user Tyler the Creator with email: tcr54@institution.edu password: changed_password
*** client.py: command (http://C149-B2:39002/api) did not reply within timeout of 300 seconds, attempt 1 of 3
*** client.py: command (http://C149-B2:39002/api) did not reply within timeout of 300 seconds, attempt 2 of 3
*** client.py: command (http://C149-B2:39002/api) did not reply within timeout of 300 seconds, attempt 3 of 3
Traceback (most recent call last):
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 82, in <module>
    cli = CommandClient(host, int(port))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 33, in __init__
    self._reload()
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 61, in _reload
    system = self._get_callable('system.describe')()
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 49, in func
    r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='c149-b2', port=39002): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2003201bd0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Thank you very much for the help,
Tyler

[troz18]

Additional logs:

Command_core log

[tyler@C149-B2 ~]$ cryosparcm log command_core
****** Scheduler Failed **** 
****** Heartbeat check failed ****
[JSONRPC ERROR  2020-03-02 14:30:02.869286  at  get_num_active_licenses ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 114, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 1421, in get_num_active_licenses
    for j in jobs_running:
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 198, in background_worker
    concurrent_job_monitor()
  File "cryosparc2_command/command_core/__init__.py", line 1428, in concurrent_job_monitor
    current_concurrent_licenses_deque.append(get_num_active_licenses())
  File "cryosparc2_command/command_core/__init__.py", line 123, in wrapper
    raise e
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 203, in background_worker
    heartbeat_manager()
  File "cryosparc2_command/command_core/__init__.py", line 1472, in heartbeat_manager
    active_jobs = get_active_licenses()
  File "cryosparc2_command/command_core/__init__.py", line 1437, in get_active_licenses
    for j in jobs_running:
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
****** Concurrent job monitor failed ****
****** Instance heartbeat failed ****
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 187, in background_worker
    scheduler_run_core() # sets last run time
  File "cryosparc2_command/command_core/__init__.py", line 1551, in scheduler_run_core
    prune_scheduler_queued_jobs()
  File "cryosparc2_command/command_core/__init__.py", line 1534, in prune_scheduler_queued_jobs
    all_queued_jobs = list(mongo.db['sched_queued'].find({}, {'queued_job_hash' : 1, 'last_scheduled_at' : 1}))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 192, in background_worker
    check_heartbeats()
  File "cryosparc2_command/command_core/__init__.py", line 1790, in check_heartbeats
    'heartbeat_at' : {'$lt' : deadline} }, {'project_uid' : 1, 'uid' : 1}))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
****** Scheduler Failed **** 
****** Heartbeat check failed ****
[JSONRPC ERROR  2020-03-02 14:32:04.288851  at  get_num_active_licenses ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 114, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 1421, in get_num_active_licenses
    for j in jobs_running:
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 198, in background_worker
    concurrent_job_monitor()
  File "cryosparc2_command/command_core/__init__.py", line 1428, in concurrent_job_monitor
    current_concurrent_licenses_deque.append(get_num_active_licenses())
  File "cryosparc2_command/command_core/__init__.py", line 123, in wrapper
    raise e
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
COMMAND CORE STARTED ===  2020-03-02 14:32:39.907269  ==========================
*** BG WORKER START
[JSONRPC ERROR  2020-03-02 14:33:12.362738  at  set_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 114, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 458, in set_config_var
    mongo.db[colname].update_one({'name' : name}, {op : {'name' : name, 'value' : value}}, upsert=True)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/collection.py", line 890, in update_one
    with self._socket_for_writes() as sock_info:
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
-----------------------------------------------------
[2020-03-02 14:33:12,369] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "cryosparc2_command/command_core/__init__.py", line 263, in startup
    set_config_var('running_version', versionfile.read().strip())
  File "cryosparc2_command/command_core/__init__.py", line 123, in wrapper
    raise e
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 187, in background_worker
    scheduler_run_core() # sets last run time
  File "cryosparc2_command/command_core/__init__.py", line 1551, in scheduler_run_core
    prune_scheduler_queued_jobs()
  File "cryosparc2_command/command_core/__init__.py", line 1534, in prune_scheduler_queued_jobs
    all_queued_jobs = list(mongo.db['sched_queued'].find({}, {'queued_job_hash' : 1, 'last_scheduled_at' : 1}))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 192, in background_worker
    check_heartbeats()
  File "cryosparc2_command/command_core/__init__.py", line 1790, in check_heartbeats
    'heartbeat_at' : {'$lt' : deadline} }, {'project_uid' : 1, 'uid' : 1}))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 873, in __send_message
    **kwargs)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 888, in _send_message_with_response
    server = topology.select_server(selector)
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/tyler/cryosparc_server/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
ServerSelectionTimeoutError: c149-b2:39001: [Errno -2] Name or service not known
****** Scheduler Failed **** 
****** Heartbeat check failed ****

Webapp log

[tyler@C149-B2 ~]$ cryosparcm log webapp
cryoSPARC v2
(node:66711) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Ready to serve GridFS
cryoSPARC v2
(node:14843) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Ready to serve GridFS

Thanks again,
Tyler

[stephan]

Hi @troz18,

Can you double check that your ports are open from 39000 to 39010?

Problems during installation on workstation; [database spawn ERROR]

Hi @Daniel_Castano, Any chance you’re running on CentOS? You need to ensure ports 39000 to 39005 are not blocked by the firewall. Please refer to this: https://stackoverflow.com/questions/24729024/open-firewall-port-on-centos-7 - Suhail

[troz18]

The issue is resolved. It seems that it was an issue with the way the network and firewall were configured. IT helped us resolve this.

Thanks again,
Tyler