Python issue with cryosparcm

huqi · July 24, 2024, 2:01pm

Dear All,

This is related to the previous topic, Small beta-barrel protein alignment - 3D Refinement - CryoSPARC Discuss.

We installed CryoSPARC on our HPC. If I load Python 3.11.5 or 3.8.1, then run ‘cryosparcm start’, or ‘cryosparcm status’, I got the following error message,

[ID@hpc02 ID]$ cryosparcm status
Python path configuration:
PYTHONHOME = ‘/usr/local/biotools/python/3.11.5/’
PYTHONPATH = (not set)
program name = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
sys.base_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.base_exec_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.platlibdir = ‘lib’
sys.executable = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
sys.prefix = ‘/usr/local/biotools/python/3.11.5’
sys.exec_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.path = [
‘/usr/local/biotools/python/3.11.5/lib/python310.zip’,
‘/usr/local/biotools/python/3.11.5/lib/python3.10’,
‘/usr/local/biotools/python/3.11.5/lib/python3.10/lib-dynload’,
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named ‘encodings’

Current thread 0x00007f0e73366740 (most recent call first):

Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

/usr/local/biotools/cryosparc/cryosparc_master/bin/cryosparcm: line 1258: supervisorctl: command not found
CryoSPARC process status:

Python path configuration:
PYTHONHOME = ‘/usr/local/biotools/python/3.11.5/’
PYTHONPATH = (not set)
program name = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
isolated = 0
environment = 1
user site = 0
import site = 1
sys._base_executable = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
sys.base_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.base_exec_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.platlibdir = ‘lib’
sys.executable = ‘/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/bin/python’
sys.prefix = ‘/usr/local/biotools/python/3.11.5’
sys.exec_prefix = ‘/usr/local/biotools/python/3.11.5’
sys.path = [
‘/usr/local/biotools/python/3.11.5/lib/python310.zip’,
‘/usr/local/biotools/python/3.11.5/lib/python3.10’,
‘/usr/local/biotools/python/3.11.5/lib/python3.10/lib-dynload’,
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named ‘encodings’

Current thread 0x00007f02207f6740 (most recent call first):

Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

If I run ‘cryosparcm status’ under my default Python 3.6.15, I got the following,

[ID@hpc02 ID]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC process status:

app RUNNING pid 1987983, uptime 15:43:49
app_api RUNNING pid 1988003, uptime 15:43:46
app_api_dev STOPPED Not started
command_core RUNNING pid 1987910, uptime 15:44:36
command_rtp RUNNING pid 1987958, uptime 15:44:07
command_vis RUNNING pid 1987949, uptime 15:44:10
database RUNNING pid 1987797, uptime 15:44:46

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

I previously installed Python 3.6.15 under miniconda under my own directory in order to run Topaz.
CryoSPARC can be started under Python 3.6.15. And both CryoSPARC and Topaz run correctly so far.
We recently installed cryosparc tools with Python 3.11.5, but encountered problems when running it (please see the previous topic for details, link is at the top of the post).

We have updated to v4.5.3 and applied patch 240708-gevent.

Looks like it is a python problem, though I do not know the way to fix it. I am also wondering if it is related to the miniconda and Python 3.6.15 I installed. Any suggestions or comments are very much appreciated!

Regards,
Qi

wtempel · July 24, 2024, 3:01pm

The PYTHONHOME environment variable may interfere with CryoSPARC.
What is the output of this command sequence

unset PYTHONHOME
cryosparcm status

huqi · July 24, 2024, 3:03pm

Thank you for your prompt response @wtempel

Following is what I got.

[ID@hpc02 CS-gm146-1]$ unset PYTHONHOME

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC process status:

app RUNNING pid 1987983, uptime 17:03:55
app_api RUNNING pid 1988003, uptime 17:03:52
app_api_dev STOPPED Not started
command_core RUNNING pid 1987910, uptime 17:04:42
command_rtp RUNNING pid 1987958, uptime 17:04:13
command_vis RUNNING pid 1987949, uptime 17:04:16
database RUNNING pid 1987797, uptime 17:04:52>

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?
[ID@hpc02 CS-gm146-1]$ python -V
Python 3.11.5

Qi

wtempel · July 24, 2024, 3:15pm

@huqi You may want to

ensure that PYTHONHOME is not set in the environment where you run CryoSPARC commands like cryosparcm or cryosparcw.
completely stop CryoSPARC
start CryoSPARC again (keeping in mind (1.))
then check the output of
cryosparcm status

huqi · July 24, 2024, 3:56pm

@wtempel I followed your instructions and completed the steps. Looks the same results were obtained.

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC process status:

app RUNNING pid 2049443, uptime 0:00:14
app_api RUNNING pid 2049461, uptime 0:00:13
app_api_dev STOPPED Not started
command_core RUNNING pid 2049396, uptime 0:00:29
command_rtp RUNNING pid 2049413, uptime 0:00:19
command_vis RUNNING pid 2049410, uptime 0:00:21
database RUNNING pid 2049287, uptime 0:00:34

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

I made sure to execute ‘unset PYTHONHOME’. CryoSPARC was shut down completely and restarted correctly without errors.

Qi

wtempel · July 24, 2024, 4:41pm

You may want to additionally ensure that PYTHONHOME is not automatically set in shells opened by the Linux user that runs CryoSPARC processes.
Are any errors shown when you run the command

cryosparcm filterlog -l error command_core

?
In a new shell, what is the output of the second command?

eval $(cryosparcm env) # load the CryoSPARC environment
curl https://get.cryosparc.com/checklicenseexists/$CRYOSPARC_LICENSE_ID
exit # exit the shell

huqi · July 24, 2024, 4:53pm

Looks I got a lot of errors when running

cryosparcm filterlog -l error command_core

Following is only a portion of the entire error message.

[ID@hpc02 ~] 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 12:42:02,959 wrapper 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:19,854 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:21,791 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-19 14:59:23,922 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,605 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,701 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,724 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run 2024-07-22 14:59:29,746 get_gpu_info_run cryosparcm filterlog -l error command_core ERROR | JSONRPC ERROR at get_id_by_email_password ERROR | Traceback (most recent call last): ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper ERROR | res = func(*args, **kwargs) ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/user.py", line 108, in get_id_by_email_password ERROR | assert user and check_pass(password, user['services'].get('password', {}).get('bcrypt', '')), \ ERROR | AssertionError: User with this email and password combination does not exist ERROR | Failed to get GPU info on hpcgpu40 ERROR | Traceback (most recent call last): ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1516, in get_gpu_info_run ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode() ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 421, in check_output ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 526, in run ERROR | raise CalledProcessError(retcode, process.args, ERROR | subprocess.CalledProcessError: Command '['ssh', 'root@hpcgpu40', 'bash -c "eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py"‘]’ returned non-zero exit status 255.
ERROR | Failed to get GPU info on hpcgpu41
ERROR | Traceback (most recent call last):
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1516, in get_gpu_info_run
ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 421, in check_output
ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 526, in run
ERROR | raise CalledProcessError(retcode, process.args,
ERROR | subprocess.CalledProcessError: Command ‘[‘ssh’, ‘root@hpcgpu41’, ‘bash -c “eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py"']' returned non-zero exit status 255. ERROR | Failed to get GPU info on hpcgpu42 ERROR | Traceback (most recent call last): ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1516, in get_gpu_info_run ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode() ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 421, in check_output ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 526, in run ERROR | raise CalledProcessError(retcode, process.args, ERROR | subprocess.CalledProcessError: Command '['ssh', 'root@hpcgpu42', 'bash -c "eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py”’]’ returned non-zero exit status 255.
ERROR | Failed to get GPU info on hpcgpu38
ERROR | Traceback (most recent call last):
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1516, in get_gpu_info_run
ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 421, in check_output
ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 526, in run
ERROR | raise CalledProcessError(retcode, process.args,
ERROR | subprocess.CalledProcessError: Command ‘[‘ssh’, ‘root@hpcgpu38’, ‘bash -c “eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py"']' returned non-zero exit status 255. ERROR | Failed to get GPU info on hpcgpu39 ERROR | Traceback (most recent call last): ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1516, in get_gpu_info_run ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode() ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 421, in check_output ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 526, in run ERROR | raise CalledProcessError(retcode, process.args, ERROR | subprocess.CalledProcessError: Command '['ssh', 'root@hpcgpu39', 'bash -c "eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py”’]’ returned non-zero exit status 255.
ERROR | Failed to get GPU info on hpcgpu40
ERROR | Traceback (most recent call last):
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1516, in get_gpu_info_run
ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 421, in check_output
ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 526, in run
ERROR | raise CalledProcessError(retcode, process.args,
ERROR | subprocess.CalledProcessError: Command ‘[‘ssh’, ‘root@hpcgpu40’, ‘bash -c “eval $(/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py”’]’ returned non-zero exit status 255.
ERROR | Failed to get GPU info on hpcgpu41
ERROR | Traceback (most recent call last):
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 1516, in get_gpu_info_run
ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 421, in check_output
ERROR | return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py”, line 526, in run
ERROR | raise CalledProcessError(retcode, process.args,

Running the commands in a new shell,

eval $(cryosparcm env) # load the CryoSPARC environment
curl https://get.cryosparc.com/checklicenseexists/$CRYOSPARC_LICENSE_ID
exit # exit the shell

I got

[ID@hpc02 ~] eval (cryosparcm env)
[ID@hpc02 ~]$ curl https://get.cryosparc.com/checklicenseexists/$CRYOSPARC_LICENSE_ID
{“success”: true}[ID@hpc02 ~]$ exit
logout

wtempel · July 24, 2024, 5:41pm

Thanks for running these commands.

is an older error message. Are there any error messages that are newer than the the most recent startup of CryoSPARC?
It is possible that components other than PYTHONHOME interfere with CryoSPARC operation. The output of the command

cryosparcm call env | grep -v CRYOSPARC_LICENSE_ID

might provide some clues.

huqi · July 24, 2024, 7:11pm

These are parts of the errors reported in the ‘command_core.log’.

2024-07-24 12:48:45,155 wrapper              ERROR    |     resp = conn.urlopen(
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-07-24 12:48:45,155 wrapper              ERROR    |     httplib_response = self._make_request(
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 404, in _make_request
2024-07-24 12:48:45,155 wrapper              ERROR    |     self._validate_conn(conn)
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
2024-07-24 12:48:45,155 wrapper              ERROR    |     conn.connect()
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py", line 400, in connect
2024-07-24 12:48:45,155 wrapper              ERROR    |     self.ssl_context = create_urllib3_context(
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 312, in create_urllib3_context
2024-07-24 12:48:45,155 wrapper              ERROR    |     context.options |= options
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py", line 620, in options
2024-07-24 12:48:45,155 wrapper              ERROR    |     super(SSLContext, SSLContext).options.__set__(self, value)
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py", line 620, in options
2024-07-24 12:48:45,155 wrapper              ERROR    |     super(SSLContext, SSLContext).options.__set__(self, value)
2024-07-24 12:48:45,155 wrapper              ERROR    |   File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py", line 620, in options
2024-07-24 12:48:45,155 wrapper              ERROR    |     super(SSLContext, SSLContext).options.__set__(self, value)
2024-07-24 12:48:45,155 wrapper              ERROR    |   [Previous line repeated 448 more times]
2024-07-24 12:48:45,155 wrapper              ERROR    | RecursionError: maximum recursion depth exceeded while calling a Python object
2024-07-24 12:49:07,794 export_all_projects_run INFO     | Finished exporting all projects
2024-07-24 12:49:08,727 heartbeat_manager    ERROR    | maximum recursion depth exceeded while calling a Python object
2024-07-24 12:49:08,727 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-24 12:50:09,030 heartbeat_manager    ERROR    | maximum recursion depth exceeded while calling a Python object
2024-07-24 12:50:09,030 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-24 12:51:09,375 heartbeat_manager    ERROR    | maximum recursion depth exceeded while calling a Python object

wtempel · July 24, 2024, 8:43pm

Please can you email us a current tgz log archive that you can create with the command
cryosparcm snaplogs. Please include in your email a link to the forum topic.

huqi · July 24, 2024, 9:22pm

@wtempel Thank you very much for your help. One thing I wanted to mention is that when I restarted CryoSPARC, there were errors related to the GPU information. Although all the GPUs can be accessed and used normally afterwards.

2024-07-24 12:48:33,638 get_gpu_info_run ERROR | Failed to get GPU info on hpcgpu41
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | Traceback (most recent call last):
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/**init**.py”, line 1516, in get_gpu_info_run
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/gevent/subprocess.py”, line 418, in check_output
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | raise CalledProcessError(retcode, process.args, output=output)
2024-07-24 12:48:33,638 get_gpu_info_run ERROR | subprocess.CalledProcessError: Command ‘[‘ssh’, ‘root@hpcgpu41’, ‘bash -c “eval Missing open brace for subscript(/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py”’]’ returned non-zero exit status 255.

wtempel · July 24, 2024, 9:53pm

huqi:

subprocess.CalledProcessError: Command ‘[‘ssh’, ‘root@rohpcgpu41’, ‘bash -c “eval Missing open brace for subscript(/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py”’]’ returned non-zero exit status 255.

There are two problems in this line:

root@rohpcgpu41

the worker was apparently connected under the worker computer’s root account, but CryoSPARC master and worker processes should run under a common, non-privileged Linux account.
If mXXXXXX can be that common Linux account, the old worker configurations that currently use a root@... ssh string should be replaced with worker connections that use a mXXXXXX@... ssh string (by running cryosparcw connect on each worker under the mXXXXXX account (details). After the worker connections have been corrected, you may want to run the following command on the CryoSPARC master and observe its output:
```
ssh mXXXXXX@rohpcgpu41 bash -c "eval $(/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py"
```
The command may fail if the bash shell automatically sets the PYTHONHOME variable (see Python issue with cryosparcm - #2 by wtempel).

huqi · July 26, 2024, 3:17pm

@wtempel Thank you so much for your suggestions. Following are the output of running the commands.

[mxxxxx@rohpc02 ~]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC process status:

app RUNNING pid 2251431, uptime 20:21:29
app_api RUNNING pid 2251449, uptime 20:21:27
app_api_dev STOPPED Not started
command_core RUNNING pid 2251406, uptime 20:21:42
command_rtp RUNNING pid 2251422, uptime 20:21:34
command_vis RUNNING pid 2251419, uptime 20:21:35
database RUNNING pid 2251296, uptime 20:21:47

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

[mxxxxx@rohpc02 ~] ssh mxxxxx@rohpcgpu41 bash -c "eval (/biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw env); python /biotools8/biotools/cryosparc/cryosparc_worker/cryosparc_compute/get_gpu_info.py"

[{“id”: 0, “name”: “NVIDIA A100-SXM4-80GB”, “mem”: 84987740160}, {“id”: 1, “name”: “NVIDIA A100-SXM4-80GB”, “mem”: 84987740160}, {“id”: 2, “name”: “NVIDIA A100-SXM4-80GB”, “mem”: 84987740160}, {“id”: 3, “name”: “NVIDIA A100-SXM4-80GB”, “mem”: 84987740160}]

Looks the worker is connected using the mxxxxx account, but the license error persists.

A similar error as previous ‘root@hpcgpu41’ by running :

cryosparcm filterlog -l error command_core

2024-07-25 12:56:04,373 scheduler_run_core ERROR | subprocess.CalledProcessError: Command ‘[‘ssh’, ‘mxxxxx@rohpcgpu41’, ‘bash -c "nohup /biotools8/biotools/cryosparc/cryosparc_worker/bin/cryosparcw run --project P5 --job J126 --master_hostname rohpc02 --master_command_core_port 39002 > /home/mxxxxx/CS-testagainproject/J126/job.log 2>&1 & "’]’ returned non-zero exit status 255.

wtempel · July 26, 2024, 3:37pm

Please can you post the outputs of the following commands as mxxxxx on rohpc02:

whoami
ls -al /home/mxxxxx/CS-testagainproject/J126/
df -h /home/mxxxxx/CS-testagainproject/J126/
ssh mxxxxx@rohpcgpu41 "hostname; ls -al /home/mxxxxx/CS-testagainproject/J126/; df -h /home/mxxxxx/CS-testagainproject/J126/"

huqi · July 26, 2024, 3:47pm

Apologies. The error was from testing yesterday. After restarting CryoSPARC, this error seems resolved.

[mxxxxx@rohpc02 ~]$ ssh mxxxxx@rohpcgpu41 “hostname; ls -al /home/mxxxxx/CS-testagainproject/J126/; df -h /home/mxxxxx/CS-testagainproject/J126/”
rohpcgpu41
total 144
drwxrwxr-x 3 mxxxxx mxxxxx 84 Jul 25 12:55 .
drwxrwxr-x 131 mxxxxx mxxxxx 2853 Jul 25 12:55 …
-rw-rw-r-- 1 mxxxxx mxxxxx 18 Jul 25 12:55 events.bson
drwxrwxr-x 2 mxxxxx mxxxxx 0 Jul 25 12:55 gridfs_data
-rw-rw-r-- 1 mxxxxx mxxxxx 5157 Jul 25 12:55 job.json
Filesystem Size Used Avail Use% Mounted on
rohpcisix1-cl:/ifs/home 10G 651M 9.4G 7% /home

Below are the errors in command_core.log after restarting CryoSPARC today.

2024-07-26 10:32:07,524 export_all_projects INFO | EXPORTING ALL PROJECTS IN 60s…
2024-07-26 10:32:08,727 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:32:08,727 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:32:17,532 update_all_job_sizes_run INFO | Finished updating all job sizes (0 jobs updated, 0 projects updated)
2024-07-26 10:33:07,562 export_all_projects_run INFO | Finished exporting all projects
2024-07-26 10:33:09,085 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:33:09,085 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:33:16,961 wrapper ERROR | JSONRPC ERROR at validate_license
2024-07-26 10:33:16,961 wrapper ERROR | Traceback (most recent call last):
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 196,
in wrapper
2024-07-26 10:33:16,961 wrapper ERROR | res = func(*args, **kwargs)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, li
ne 8303, in validate_license
2024-07-26 10:33:16,961 wrapper ERROR | is_valid, *_ = core.validate_license()
2024-07-26 10:33:16,961 wrapper ERROR | File “cryosparc_master/cryosparc_command/core.py”, line 1098, in cryosparc_master.cryosparc_command.
core.validate_license
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/pyt
hon3.10/site-packages/requests/api.py”, line 73, in get
2024-07-26 10:33:16,961 wrapper ERROR | return request(“get”, url, params=params, **kwargs)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/pyt
hon3.10/site-packages/requests/api.py”, line 59, in request
2024-07-26 10:33:16,961 wrapper ERROR | return session.request(method=method, url=url, **kwargs)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/pyt
hon3.10/site-packages/requests/sessions.py”, line 587, in request
2024-07-26 10:33:16,961 wrapper ERROR | resp = self.send(prep, **send_kwargs)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/pyt
hon3.10/site-packages/requests/sessions.py”, line 701, in send
2024-07-26 10:33:16,961 wrapper ERROR | r = adapter.send(request, **kwargs)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/pyt
hon3.10/site-packages/requests/adapters.py”, line 487, in send
2024-07-26 10:33:16,961 wrapper ERROR | resp = conn.urlopen(
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py”, line 715, in urlopen
2024-07-26 10:33:16,961 wrapper ERROR | httplib_response = self._make_request(
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py”, line 404, in _make_request
2024-07-26 10:33:16,961 wrapper ERROR | self._validate_conn(conn)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py”, line 1058, in validate_conn
2024-07-26 10:33:16,961 wrapper ERROR | conn.connect()
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py”, line 400, in connect
2024-07-26 10:33:16,961 wrapper ERROR | self.ssl_context = create_urllib3_context(
2024-07-26 10:33:16,961 wrapper ERROR | File "/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/util/ssl.py", line 312, in create_urllib3_context
2024-07-26 10:33:16,961 wrapper ERROR | context.options |= options
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py”, line 620, in options
2024-07-26 10:33:16,961 wrapper ERROR | super(SSLContext, SSLContext).options.set(self, value)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py”, line 620, in options
2024-07-26 10:33:16,961 wrapper ERROR | super(SSLContext, SSLContext).options.set(self, value)
2024-07-26 10:33:16,961 wrapper ERROR | File “/biotools8/biotools/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/ssl.py”, line 620, in options
2024-07-26 10:33:16,961 wrapper ERROR | super(SSLContext, SSLContext).options.set(self, value)
2024-07-26 10:33:16,961 wrapper ERROR | [Previous line repeated 448 more times]
2024-07-26 10:33:16,961 wrapper ERROR | RecursionError: maximum recursion depth exceeded while calling a Python object
2024-07-26 10:34:09,369 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:34:09,369 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:35:09,709 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:35:09,709 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:36:10,011 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:36:10,012 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:37:10,340 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:37:10,340 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:38:10,642 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:38:10,642 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:39:10,918 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object
2024-07-26 10:39:10,919 heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2024-07-26 10:40:11,196 heartbeat_manager ERROR | maximum recursion depth exceeded while calling a Python object

huqi · August 1, 2024, 9:45pm

Hi @wtempel ,

The latest update with the new patch. It looks solved the license problem.

[mxxxxx@hpc CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at

/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240729-gthread

CryoSPARC process status:

app RUNNING pid 2981998, uptime 0:31:05
app_api RUNNING pid 2982016, uptime 0:31:04
app_api_dev STOPPED Not started
command_core RUNNING pid 2981893, uptime 0:31:19
command_rtp RUNNING pid 2981918, uptime 0:31:10
command_vis RUNNING pid 2981915, uptime 0:31:12
database RUNNING pid 2981785, uptime 0:31:24

License is valid

global config variables:
export CRYOSPARC_LICENSE_ID=“xxxxxxxxxxxxxxxxxxxxxxxxxxxx”
export CRYOSPARC_MASTER_HOSTNAME=“hpc”
export CRYOSPARC_DB_PATH=“/biotools8/biotools/cryosparc/cryosparc_database”
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX=‘CS-’
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_FORCE_USER=true

Though the error with cryosparc-tools persists,

[mxxxx@hpc CS-gm146-1]$ python combineParticles.py
Connection succeeded to CryoSPARC command_core at http://hpc:39002
Connection succeeded to CryoSPARC command_vis at http://hpc:39003
Connection succeeded to CryoSPARC command_rtp at http://hpc:39005
/usr/local/biotools/python/3.11.5/lib/python3.11/contextlib.py:137: UserWarning: *** CommandClient: (http://hpc:39003/external/projects/Pxx/jobs/Jxx/outputs/particles/dataset) URL Error [Errno 104] Connection reset by peer, attempt 1 of 3. Retrying in 30 seconds
return next(self.gen)
/usr/local/biotools/python/3.11.5/lib/python3.11/contextlib.py:137: UserWarning: *** CommandClient: (http://hpc:39003/external/projects/Pxx/jobs/Jxx/outputs/particles/dataset) URL Error [Errno 104] Connection reset by peer, attempt 2 of 3. Retrying in 30 seconds
return next(self.gen)
Traceback (most recent call last):
File “/data/cryoem/mxxxx/GM146-1/CS-gm146-1/combineParticles.py”, line 29, in
external_job_uid = project.save_external_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/mxxxx/.local/lib/python3.11/site-packages/cryosparc/project.py”, line 278, in save_external_result
return self.cs.save_external_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/mxxxxx/.local/lib/python3.11/site-packages/cryosparc/tools.py”, line 584, in save_external_result
job.save_output(output, dataset)
File “/home/mxxxxx/.local/lib/python3.11/site-packages/cryosparc/job.py”, line 1520, in save_output
with make_request(self.cs.vis, url=url, data=dataset.stream()) as res:
File “/usr/local/biotools/python/3.11.5/lib/python3.11/contextlib.py”, line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File “/home/mxxxx/.local/lib/python3.11/site-packages/cryosparc/command.py”, line 226, in make_request
raise CommandError(error_reason, url=url, code=code, data=resdata)
cryosparc.errors.CommandError: *** (http://hpc:39003/external/projects/Pxx/jobs/Jxx/outputs/particles/dataset, code 500) URL Error [Errno 104] Connection reset by peer

Any suggestions/comments are appreciated!

Qi

wtempel · August 12, 2024, 3:52pm

@huqi How many particles are being input to and output by the combineParticles.py script?
If you encounter this issue again, please can you run these commands immediately thereafter on the CryoSPARC master computer

date # post the output under this forum topic
cryosparcm snaplogs # email us the tgz file

Python issue with cryosparcm

Current thread 0x00007f0e73366740 (most recent call first):

Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 ID]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

[mxxxxx@rohpc02 ~]$ cryosparcm status

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

[mxxxxx@hpc CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at

License is valid

Python issue with cryosparcm

Current thread 0x00007f0e73366740 (most recent call first): Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress

CryoSPARC System master node installed at /biotools8/biotools/cryosparc/cryosparc_master Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 ID]$ cryosparcm status

CryoSPARC System master node installed at /biotools8/biotools/cryosparc/cryosparc_master Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at /biotools8/biotools/cryosparc/cryosparc_master Current cryoSPARC version: v4.5.3+240708-gevent

[ID@hpc02 CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at /biotools8/biotools/cryosparc/cryosparc_master Current cryoSPARC version: v4.5.3+240708-gevent

[mxxxxx@rohpc02 ~]$ cryosparcm status

CryoSPARC System master node installed at /biotools8/biotools/cryosparc/cryosparc_master Current cryoSPARC version: v4.5.3+240708-gevent

[mxxxxx@hpc CS-gm146-1]$ cryosparcm status

CryoSPARC System master node installed at

License is valid

Current thread 0x00007f0e73366740 (most recent call first):

Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent

CryoSPARC System master node installed at
/biotools8/biotools/cryosparc/cryosparc_master
Current cryoSPARC version: v4.5.3+240708-gevent