Can't Install Cryosparc 4.0

tim.chang · October 7, 2022, 5:09pm

Hi. I have a system need Cryosparc installed.

I installed on other systems (20+), never see this issue.

After I run installation scripts.

I answer 1st question seems okay.

But When I answer 2nd question

CryoSPARC System master has now been installed.
 Add bin directory to your ~/.bashrc ?
1) Yes
2) No"

Then it stuck on “command_core: started”. And never go through.

I remove whole folder and redownload “cryosparc_master.tar.gz” and “cryosparc_worker.tar.gz”.

And re-extract both then re-run installation scripts.

But issue is still the same.

Any suggestion on this issue?

Thanks.

wtempel · October 7, 2022, 5:21pm

Please post the content of
cryosparc_master/run/command_core.log
when the installation gets stuck.

tim.chang · October 10, 2022, 5:38pm

the content of cryosparc_master/run/command_core.log as follow:

2022-10-07 09:58:43,180 COMMAND.MAIN         start                INFO     |  === STARTED ===
2022-10-07 09:58:43,181 COMMAND.BG_WORKER    background_worker    INFO     |  === STARTED ===
2022-10-07 09:58:43,181 COMMAND.CORE         run                  INFO     | === STARTED TASKS WORKER ===
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1925, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 144, in background_worker
    last_audit_date = get_config_var('audit', fail_notset=False, default={})\
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 185, in wrapper
    return func(*args, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 283, in get_config_var
    res = mongo.db[colname].find_one({'name' : name})
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 1328, in find_one
    for result in cursor.limit(-1):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/cursor.py", line 1238, in next
    if len(self.__data) or self._refresh():
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/cursor.py", line 1130, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1935, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1883, in __start_session
    server_session = self._get_server_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1921, in _get_server_session
    return self._topology.get_server_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 520, in get_server_session
    session_timeout = self._check_session_support()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 502, in _check_session_support
    None)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 220, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: sn4622107454:39001: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 63405ac3412fd9eb6a1cb923, topology_type: Single, servers: [<ServerDescription ('sn4622107454', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('sn4622107454:39001: [Errno -2] Name or service not known')>]>

2022-10-10 09:36:34,612 COMMAND.MAIN         start                INFO     |  === STARTED ===
2022-10-10 09:36:34,612 COMMAND.BG_WORKER    background_worker    INFO     |  === STARTED ===
2022-10-10 09:36:34,613 COMMAND.CORE         run                  INFO     | === STARTED TASKS WORKER ===
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1925, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 144, in background_worker
    last_audit_date = get_config_var('audit', fail_notset=False, default={})\
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 185, in wrapper
    return func(*args, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 283, in get_config_var
    res = mongo.db[colname].find_one({'name' : name})
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 1328, in find_one
    for result in cursor.limit(-1):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/cursor.py", line 1238, in next
    if len(self.__data) or self._refresh():
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/cursor.py", line 1130, in _refresh
    self.__session = self.__collection.database.client._ensure_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1935, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1883, in __start_session
    server_session = self._get_server_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1921, in _get_server_session
    return self._topology.get_server_session()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 520, in get_server_session
    session_timeout = self._check_session_support()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 502, in _check_session_support
    None)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 220, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: sn4622107454:39001: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 63444a12d2f8497d3df4fcfc, topology_type: Single, servers: [<ServerDescription ('sn4622107454', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('sn4622107454:39001: [Errno -2] Name or service not known')>]>

wtempel · October 11, 2022, 9:46pm

What are

the value of CRYSPARC_MASTER_HOSTNAME inside
/home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh
the output of
hostname -f

?
If you wish, you may email us the file produced by the
cryosparcm snaplogs
command.

tim.chang · October 12, 2022, 12:05am

Hi. This system got some issue with booting to OS.

But I’ll try to get it back to OS asap, then I can get the log which needed for this issue. Thanks.

tim.chang · October 13, 2022, 5:07pm

Hi, I cat the file “/home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh”.

It shows as follow:

Instance Configuration

export CRYOSPARC_LICENSE_ID=“xxxxxx-xxxx-xxx-xxx-xxxxxxx”
export CRYOSPARC_MASTER_HOSTNAME=“SN4622107454”
export CRYOSPARC_DB_PATH="/home/cryosparc_user/software/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000

Security

export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true

Cluster Integration

export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000

Project Configuration

export CRYOSPARC_PROJECT_DIR_PREFIX=‘CS-’

Development

export CRYOSPARC_DEVELOP=false

Other

export CRYOSPARC_CLICK_WRAP=true

hostname -f
SN4622107454

By the way, how do I email to log to you which generated by “cryosparcm snaplogs”.

wtempel · October 14, 2022, 7:52pm

Please can you also share with us the outputs of

cat /etc/hosts
host $(hostname -f)

When CryoSPARC is in an (improperly) running state
(stuck at command_core: started), what happens if, in a different terminal window on this host, you run:

curl sn4622107454:39001
curl SN4622107454:39001
in a mongo shell (cryosparcm mongo):
show collections

?

Since there have been changes to the database subsystem recently, it might be worth testing if the problem also occurs with CryoSPARC v3.3.2.

curl -L https://get.cryosparc.com/download/master-v3.3.2/$LICENSE_ID -o cryosparc_master.tar.gz
curl -L https://get.cryosparc.com/download/worker-v3.3.2/$LICENSE_ID -o cryosparc_worker.tar.gz
[...]

tim.chang · October 14, 2022, 9:41pm

Hi wtempel,
After saw your replay, I released the original Hostname I put it is capital SN.

But somehow the cryosparc seems only recognize small letters.

So I rename it as sn***, then reboot the system.

Reintall the Cryosparc again, this time installation went through.

And I can queue the job now. Thanks for your help.