Problems during installation on workstation; [database spawn ERROR]

Hi developers!

I’m trying to install cryoSparc as standalone; everything goes smooth at first but I’m stacked when the master process starts to complete the standalone installer.

I paste below the error point… any ideas?


*********** CRYOSPARC SYSTEM: STANDALONE INSTALLER **************

  The installer will now download, install and connect the worker
  package on this machine. The worker package will allow cryoSPARC
  to run compute jobs.

******************************************************************

 Starting cryoSPARC...

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
    "listDatabases")["databases"]]
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
    with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
    with self._get_socket(read_preference) as sock_info:
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/home/casdanie/tools/cryosparc/s5/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused
~/.bashrc has been executed

Hi @Daniel_Castano,

Any chance you’re running on CentOS? You need to ensure ports 39000 to 39005 are not blocked by the firewall. Please refer to this: https://stackoverflow.com/questions/24729024/open-firewall-port-on-centos-7

- Suhail

Hi Suhail,

Thanks! No, my setting is an Ubuntu 16.04…
Actually, if I run
netstat -an | grep 39000 | grep -i listen

I get nothing. Any advice on how to proceed from here?
Daniel

Hi Daniel,

Hmm, interesting. Can you try running cryosparcm status and pasting the output? I’d also recommend cryosparcm restart to stop and start the instance - sometimes that solves the problem.

- Suhail

Hi Suhail,

thanks for your answer… found a weird effect. cryosparc reports that there is a missing folder:

>     casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s5/cryosparc2_master[0]>  bin/cryosparcm status
>     bin/cryosparcm: line 25: cd: /home/casdanie/tools/cryosparc/s5/cryosparc2_master/bin
>     /home/casdanie/tools/cryosparc/s5/cryosparc2_master/bin/../: No such file or directory

… but the folder is actually there:

casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s5/cryosparc2_master[1]> ls /home/casdanie/tools/cryosparc/s5/cryosparc2_master/bin/…/
bin cryosparc2_webapp install.sh
check_install_deps.sh deps run
config.sh deps_bundle setup_client_ipython.py
cryosparc2_command deps_bundle_hashes supervisord.conf
cryosparc2_compute deps_hashes version

Does it mean that I cannot use the “cryosparcm” command directly providing the path from a higher directory (i.e., from cryosparc2_master writing “bin/cryosparcm XXX”)? Is it necessary to explicitely include the “bin” in the PATH?

In any case, after including the bin path, restarting with “cryosparcm restart”, I got the report that I paste below

casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s5/cryosparc2_master[0]>  cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/casdanie/tools/cryosparc/s5/cryosparc2_master
Current cryoSPARC version: v2.9.0
----------------------------------------------------------------------------

cryosparcm process status:

command_core                     STOPPED   Jul 15 09:43 AM
command_proxy                    STOPPED   Jul 15 09:43 AM
command_vis                      STOPPED   Jul 15 09:43 AM
database                         STOPPED   Jul 15 09:43 AM
watchdog_dev                     STOPPED   Not started
webapp                           STOPPED   Jul 15 09:43 AM
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_MASTER_HOSTNAME="bs-dalc03.ethz.ch"
export CRYOSPARC_DB_PATH="/home/casdanie/tools/cryosparc/s5/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

Well… I keep trying to install cryosparc. The error I land on most frequently is the one I paste here:

******* CRYOSPARC WORKER INSTALLATION COMPLETE *******************

 In order to run processing jobs, you will need to connect this
 worker to a cryoSPARC master.

******************************************************************

******************************************************************

 Connecting the cryoSPARC worker to the master...

bin/cryosparcw: line 30: cd: /home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin
/home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin/../: No such file or directory
~/.bashrc has been executed
casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s6/cryosparc2_master[0]> ls /home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin/../
bin		       cryosparc2_compute  deps_bundle_hashes  version
check_install_deps.sh  deps		   deps_hashes
config.sh	       deps_bundle	   install.sh
casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s6/cryosparc2_master[0]> 

I cannot interpret the reported error, as the offending path is clearly in the file system.
In any case; is there a way that I can complete the connection of the worker to the master manually? What I mean is that this problem with the folders uncorrectly seen as missing by cryosparc persists when trying to use “cryosparcw connect”

casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s6/cryosparc2_worker[1]> bin/cryosparcw connect --worker bioem-gpu03 --master bioem-gpu03 --ssdpath /local0/tmp/cryosparc/scratch
bin/cryosparcw: line 30: cd: /home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin
/home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin/../: No such file or directory
casdanie@d.ethz.ch@bs-dalc03:~/tools/cryosparc/s6/cryosparc2_worker[1]> ls /home/casdanie/tools/cryosparc/s6/cryosparc2_worker/bin/../
bin		       cryosparc2_compute  deps_bundle_hashes  version
check_install_deps.sh  deps		   deps_hashes
config.sh	       deps_bundle	   install.sh

Hi @Daniel_Castano,

Are you using bash? If possible, can you switch to that shell and then run the script?

- Suhail

Yep… I always use bash:
casdanie@d.ethz.ch@bs-dalc03:~/.ssh[0]> echo $0
-bash

Hi

I had a similar issue today trying to update to 2.9.0. There may have been someone logged in to the database, but I can’t find any processes from supervisord and the .sock file disappears when you stop cryosparc. Any assistance would be appreciated.

Regards
Christine

CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
    "listDatabases")["databases"]]
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
    with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
    with self._get_socket(read_preference) as sock_info:
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/ssd/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused

actually I found the following presumably orphan process:

root     36241  0.0  0.0  21848  1924 ?        S    Jun08   0:00 bash /ssd/cryosparc2/cryosparc2_worker/bin/cryosparcw run --project P6 --job J6 --master_hostname jlinux9.qb3.berkeley.edu --master_command_core_port 39002

should I kill this or is it important? I am not sure why it is running as root.

Hi @geec,

If an update fails, most of the time its usually an orphan process thats still alive holding resources. If you can kill it before updating, that might fix your problems.

Just to update, it turned out somehow someone had run cryosparc as root, (not sure how) and as well as the orphaned process there were two files with root ownership in the database folder

WiredTiger.turtle
WiredTigerLAS.wt

and also a log file in cryosparc_master/run
command_core.log

Once I changed the ownership back to the user account that was used to set up cryosparc2 then it would start correctly.