Cryosparc 2.15 master webapp does not start

Troubleshooting
1

I am trying to install cryosparc on a machine which is supposed to serve as a master in a small cluster. Upon apparently successful installation of cryosparc master, initial start procedure hanged after command_core loaded.

% cryosparcm status

cryosparcm process status:

app STOPPED Not started
app_dev STOPPED Not started
command_core RUNNING pid 10960, uptime 0:04:56
command_proxy STOPPED Not started
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 10872, uptime 0:04:59
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started

global config variables:

export CRYOSPARC_MASTER_HOSTNAME=“xxx.yyy.zzz”
export CRYOSPARC_DB_PATH="/home/structbio/cryosparc/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

$ cryosparcm log command_core

Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 205, in background_worker
heartbeat_manager()
File “cryosparc2_command/command_core/init.py”, line 1571, in heartbeat_manager
active_jobs = get_active_licenses()
File “cryosparc2_command/command_core/init.py”, line 1536, in get_active_licenses
for j in jobs_running:
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1114, in next
if len(self.__data) or self._refresh():
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1036, in _refresh
self.__collation))
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 873, in __send_message
**kwargs)
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 888, in _send_message_with_response
server = topology.select_server(selector)
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 214, in select_server
address))
File “/home/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 189, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: sangho-hkl.skku.edu:39001: timed out
****** Concurrent job monitor failed ****
****** Instance heartbeat failed ****

$ cryosparcm log database
2020-09-01T17:34:13.791+0900 I REPL [conn3] This node is localhost:39001 in the config
2020-09-01T17:34:13.791+0900 I REPL [conn3] transition to STARTUP2
2020-09-01T17:34:13.791+0900 I REPL [conn3] Starting replication storage threads
2020-09-01T17:34:13.792+0900 I REPL [conn3] Starting replication fetcher thread
2020-09-01T17:34:13.792+0900 I REPL [conn3] Starting replication applier thread
2020-09-01T17:34:13.792+0900 I REPL [conn3] Starting replication reporter thread
2020-09-01T17:34:13.792+0900 I REPL [rsSync] transition to RECOVERING
2020-09-01T17:34:13.792+0900 I COMMAND [conn3] command local.replset.minvalid appName: “MongoDB Shell” command: replSetInitiate { v: 2, key: { version: 1 }, ns: “admin.system.version”, name: “incompatible_with_version_32” } numYields:0 reslen:22 locks:{ Global: { acquireCount: { r: 14, w: 8, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 63 } }, Database: { acquireCount: { r: 2, w: 2, W: 6 } }, Collection: { acquireCount: { r: 2, w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_command 1375ms
2020-09-01T17:34:13.793+0900 I REPL [rsSync] transition to SECONDARY
2020-09-01T17:34:13.793+0900 I REPL [rsSync] conducting a dry run election to see if we could be elected
2020-09-01T17:34:13.793+0900 I REPL [ReplicationExecutor] dry election run succeeded, running for election
2020-09-01T17:34:13.796+0900 I - [conn3] end connection 127.0.0.1:35468 (1 connection now open)
2020-09-01T17:34:14.507+0900 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 1
2020-09-01T17:34:14.507+0900 I REPL [ReplicationExecutor] transition to PRIMARY
2020-09-01T17:34:14.507+0900 I REPL [ReplicationExecutor] Entering primary catch-up mode.
2020-09-01T17:34:14.507+0900 I REPL [ReplicationExecutor] Exited primary catch-up mode.
2020-09-01T17:34:15.793+0900 I REPL [rsSync] transition to primary complete; database writes are now permitted

Can someone help me resolve this trouble?

2

Hi @sangho,

Can you double check that ports 39000 to 39010 are open?

- Suhail

3

Hi Suhail

Thanks for your help. I already opened the ports 39000-39010 using the directions on the web page you mentioned. However, nothing changed.

4

It seems that port 39001 cannot be connected. Below is a part of database log file:

NETWORK [thread1] waiting for connections on port 39001
NETWORK [thread1] connection accepted from 127.0.0.1:38120 #1 (1 connection now open)
NETWORK [conn1] received client metadata from 127.0.0.1:38120 conn1: { driver: { name: “PyMongo”, version: “3.4.0” }, os: { type: “Linux”, name: “CentOS Linux 7.8.2003 Core”, architecture: “x86_64”, version: “3.10.0-1127.19.1.el7.x86_64” }, platform: “CPython 2.7.15.final.0” }
NETWORK [thread1] connection accepted from 127.0.0.1:38122 #2 (2 connections now open)
NETWORK [conn2] received client metadata from 127.0.0.1:38122 conn2: { driver: { name: “PyMongo”, version: “3.4.0” }, os: { type: “Linux”, name: “CentOS Linux 7.8.2003 Core”, architecture: “x86_64”, version: “3.10.0-1127.19.1.el7.x86_64” }, platform: “CPython 2.7.15.final.0” }
NETWORK [thread1] connection accepted from 127.0.0.1:38124 #3 (1 connection now open)
NETWORK [conn3] received client metadata from 127.0.0.1:38124 conn3: { application: { name: “MongoDB Shell” }, driver: { name: “MongoDB Internal Client”, version: “3.4.10-4-g67ee356c6b” }, os: { type: “Linux”, name: “CentOS Linux release 7.8.2003 (Core)”, architecture: “x86_64”, version: “Kernel 3.10.0-1127.19.1.el7.x86_64” } }
NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
NETWORK [signalProcessingThread] closing listening socket: 6
NETWORK [signalProcessingThread] closing listening socket: 7
NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
NETWORK [signalProcessingThread] shutdown: going to flush diaglog…
NETWORK [thread1] waiting for connections on port 39001

I checked the port opening in firewall-cmd. Below is the output:

public (active)
target: default
icmp-block-inversion: no
interfaces: eno1 enp14s0
sources:
services: dhcpv6-client http mountd nfs ntp rpc-bind samba ssh
ports: 39000/tcp 39001/tcp 39002/tcp 39003/tcp 39004/tcp 39005/tcp 39006/tcp 39007/tcp 39008/tcp 39009/tcp 39010/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

Can somebody help resolve this troublesome issue so that I can use the cryosparc program?