Error when reinstall CryoSPARC binaries

Install, Configure and Update
1

Hi all,

My CryoSPARC crashed when I tried to reinstall it in a different location in a cluster.

###CryoSPARC instance information
Type: cluster

$ cryosparcm status

CryoSPARC System master node installed at
/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master
Current cryoSPARC version: v4.3.1

CryoSPARC is not running.

global config variables:
export CRYOSPARC_LICENSE_ID=“XXXXXXXXXXXXXXX”
export CRYOSPARC_MASTER_HOSTNAME=“em504-02.ibex.kaust.edu.sa”
export CRYOSPARC_DB_PATH=“/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/cryosparc2_db”
export CRYOSPARC_BASE_PORT=39162
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX=‘CS-’
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_MONGO_EXTRA_FLAGS=“”

$ uname -a && free -g
Linux em504-02 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 376 37 31 17 307 320
Swap: 29 0 29

###Here are the errors when I use cryosparcm start or cryosparcm restart:
CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Starting cryoSPARC System master process…
CryoSPARC is not already running.
configuring database
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
File “”, line 1, in
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/cryosparc_compute/database_management.py”, line 48, in configure_mongo
initialize_replica_set()
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/cryosparc_compute/database_management.py”, line 87, in initialize_replica_set
admin_db = try_get_pymongo_db(mongo_client)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/cryosparc_compute/database_management.py”, line 250, in try_get_pymongo_db
admin_db.command(({‘serverStatus’: 1}))
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py”, line 827, in command
with self.__client._socket_for_reads(read_preference, session) as (sock_info, secondary_ok):
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/contextlib.py”, line 113, in enter
return next(self.gen)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 1478, in _socket_for_reads
server = self._select_server(read_preference, session)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 1436, in _select_server
server = topology.select_server(server_selector)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 250, in select_server
return random.choice(self.select_servers(selector, server_selection_timeout, address))
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 211, in select_servers
server_descriptions = self._select_servers_loop(selector, server_timeout, address)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 226, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:39163: [Errno 111] Connection refused, Timeout: 20.0s, Topology Description: <TopologyDescription id: 6531156c8a6d67acbb3e01c7, topology_type: Single, servers: [<ServerDescription (‘localhost’, 39163) server_type: Unknown, rtt: None, error=AutoReconnect(‘localhost:39163: [Errno 111] Connection refused’)>]>
[2023-10-19T14:40:35+0300] Error configuring database. Most recent database log lines:
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] allocator: tcmalloc
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] modules: none
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] build environment:
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] distarch: x86_64
2023-10-19T14:39:21.847+0300 I CONTROL [initandlisten] target_arch: x86_64
2023-10-19T14:39:21.848+0300 I CONTROL [initandlisten] options: { net: { port: 39163 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_master/cryosparc2_db”, journal: { enabled: false } } }
2023-10-19T14:39:21.848+0300 E STORAGE [initandlisten] Failed to set up listener: SocketException: Address already in use
2023-10-19T14:39:21.850+0300 I CONTROL [initandlisten] now exiting
2023-10-19T14:39:21.850+0300 I CONTROL [initandlisten] shutting down with code:48

###In first installation in different folder, I install the cryosparm on em504-02.ibex.kaust.edu.sa with CRYOSPARC_BASE_PORT=39162. In the second installation I tried to use the same node and port. I also turn off cryosparcm in the first installation and stopped all the process are running before installed the second one.

2

Problems of cryosparm solve. The error came from slow reaction from the local server, which made cryosparcm didn’t totally shut down in previous installation.

However, I realized that I have another problem with the cryosparcw.

So the cryosparcm actually recognized the cryosparcw in the first installation, not the second one. When I tried to change the cryosparcw, I had these errors:

bin/cryosparcw connect --worker em504-02.ibex.kaust.edu.sa --master em504-02.ibex.kaust.edu.sa --port 39162 --nossd

CRYOSPARC CONNECT

Attempting to register worker em504-02.ibex.kaust.edu.sa to command em504-02.ibex.kaust.edu.sa:39164
Connecting as unix user nguyenqp
Will register using ssh string: nguyenqp@em504-02.ibex.kaust.edu.sa
If this is incorrect, you should re-run this command with the flag --sshstr

*** CommandClient: em504-02.ibex.kaust.edu.sa:39164/api URL Error [Errno 111] Connection refused
Traceback (most recent call last):
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 104, in func
with make_json_request(self, “/api”, data=data) as request:
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/contextlib.py”, line 113, in enter
return next(self.gen)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 191, in make_request
raise CommandClient.Error(client, error_reason, url=url)
cryosparc_tools.cryosparc.command.Error: *** CommandClient: (em504-02.ibex.kaust.edu.sa:39164/api) URL Error [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “bin/connect.py”, line 75, in
cli = client.CommandClient(host=master_hostname, port=command_core_port, service=“command_core”)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_compute/client.py”, line 36, in init
super().init(service, host, port, url, timeout, headers, cls=NumpyEncoder)
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 91, in init
self._reload() # attempt connection immediately to gather methods
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 118, in _reload
system = self._get_callable(“system.describe”)()
File “/ibex/scratch/nguyenqp/CryoSPARC/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 107, in func
raise CommandClient.Error(
cryosparc_tools.cryosparc.command.Error: *** CommandClient: (http://em504-02.ibex.kaust.edu.sa:39164) Did not receive a JSON response from method “system.describe” with params ()

3

Welcome to the forum @nguyenqp .

Is CryoSPARC running (it needs to for cryosparcw connect to work) and are requests to the 10-port range starting at port 39164 not blocked by a firewall?

Please post outputs of these commands:

cryosparcm status | grep -v LICENSE
ps -eopid,ppid,start,cmd | grep cryosparc
host em504-02.ibex.kaust.edu.sa
curl em504-02.ibex.kaust.edu.sa:39164
4

Hi @wtempel, thanks for your suggestion. It was blocked by a firewall so I solved the issue with our local IT team.

1 Like
5