Extensive test Culture exposure error

Install, Configure and Update
1

Hi All,

I’m currently running Cryosparc v4.6.2 and I’m getting the following error:


[CPU:   98.1 MB  Avail: 507.81 GB]

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2304, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/workflows/run_extensive_validation.py", line 77, in interactive_exposure_curation
    fieldsres = cli.interactive_post(puid, juids[job_key], 'get_fields_and_thresholds', {})
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 122, in func
    raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://sn4622126305:39002, code 400) Encountered ServerError from JSONRPC function "interactive_post" with params ('P1', 'J7', 'get_fields_and_thresholds', {}):
ServerError: HTTPConnectionPool(host='sn4622126305', port=47559): Max retries exceeded with url: /get_fields_and_thresholds (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fed7194c550>: Failed to establish a new connection: [Errno 22] Invalid argument'))
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fed7194c550>: Failed to establish a new connection: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/urllib3/util/retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='sn4622126305', port=47559): Max retries exceeded with url: /get_fields_and_thresholds (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fed7194c550>: Failed to establish a new connection: [Errno 22] Invalid argument'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
    res = func(*args, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 7425, in interactive_post
    r = requests.post('http://'+job['interactive_hostname']+':'+str(job['interactive_port'])+'/'+endpoint, json = body, timeout=timeout)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/requests/adapters.py", line 520, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='sn4622126305', port=47559): Max retries exceeded with url: /get_fields_and_thresholds (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fed7194c550>: Failed to establish a new connection: [Errno 22] Invalid argument'))

Does anyone know how to fix this issue? Let me know if you need any more information. I’m not sure what other info I need to provide.

2

Oh I should note that this is a single system. I tried reconnecting the worker node via:

cryosparcw connect --worker localhost --master localhost --port 39000

But no luck still. I tried reinstalling Cryosparc but I’m still having the same issue. System is running Ubuntu 22 and it’s up to date.

3

Even though I do not recall this particular error, I would initially hypothesize that this may be a network-related issue.
Has your network team already “registered” your workstation such that it receives the same, “resolvable”, fully qualified domain name on each reboot?

For a single workstation CryoSPARC instance without additional workers, a workaround may exist.

Please can you post the outputs of these commands:

uname -a
cat /etc/hosts
grep -e HOST -e PORT /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh
hostname -f
ip a
host $(hostname -f)
4

Hi wtempel,

Yes the system is registered and gets internet access.

root@sn4622126305:/home/exx# uname -a
Linux sn4622126305 5.15.0-134-generic #145-Ubuntu SMP Wed Feb 12 20:08:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@sn4622126305:/home/exx# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 changeme

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
root@sn4622126305:/home/exx# grep -e HOST -e PORT /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh
export CRYOSPARC_MASTER_HOSTNAME="sn4622126305"
export CRYOSPARC_BASE_PORT=39000
root@sn4622126305:/home/exx# hostname -f
sn4622126305
root@sn4622126305:/home/exx# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 7c:c2:55:6b:7f:32 brd ff:ff:ff:ff:ff:ff
    altname enp4s0f0
    inet 172.25.10.120/24 metric 100 brd 172.25.10.255 scope global dynamic eno1
       valid_lft 60664sec preferred_lft 60664sec
    inet6 fe80::7ec2:55ff:fe6b:7f32/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether da:4f:0f:cb:6d:67 brd ff:ff:ff:ff:ff:ff
    inet 169.254.3.1/24 metric 100 brd 169.254.3.255 scope global dynamic eth1
       valid_lft 693630sec preferred_lft 693630sec
    inet6 fe80::d84f:fff:fecb:6d67/64 scope link
       valid_lft forever preferred_lft forever
4: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 7c:c2:55:6b:7f:33 brd ff:ff:ff:ff:ff:ff
    altname enp4s0f1
root@sn4622126305:/home/exx# host $(hostname -f)
sn4622126305 has address 172.25.10.120
sn4622126305 has address 169.254.3.1
sn4622126305 has IPv6 address fe80::7ec2:55ff:fe6b:7f32
sn4622126305 has IPv6 address fe80::d84f:fff:fecb:6d67
5

Thanks @chris.chau. What are the outputs of these commands:

curl sn4622126305:39001
curl localhost:39001
curl sn4622126305:39002
curl localhost:39002
ps -eo user,pid,ppid,start,command | grep -e cryosparc_ -e mongo
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
6

Hi Wtempel,

Here’s the output:

root@sn4622126305:/home/exx# curl sn4622126305:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
root@sn4622126305:/home/exx# curl localhost:39001
It looks like you are trying to access MongoDB over HTTP on the native driver port.
root@sn4622126305:/home/exx# curl sn4622126305:39002
Hello World from cryosparc command core.
root@sn4622126305:/home/exx# curl localhost:39002
Hello World from cryosparc command core.
root@sn4622126305:/home/exx# ps -eo user,pid,ppid,start,command | grep -e cryosparc_ -e mongo
cryospa+    7453       1   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryosparc_user/software/cryosparc/cryosparc_master/supervisord.conf
cryospa+    7563    7453   Mar 11 mongod --auth --dbpath /home/cryosparc_user/software/cryosparc/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryospa+    7667    7453   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7668    7667   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7700    7453   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7711    7700   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7737    7453   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7738    7737   Mar 11 python /home/cryosparc_user/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/cryosparc_user/software/cryosparc/cryosparc_master/gunicorn.conf.py
cryospa+    7770    7453   Mar 11 /home/cryosparc_user/software/cryosparc/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
cryospa+   12537   11137 21:15:13 vim cryosparc_acctinfo.txt
root       14004   13977 22:01:37 grep --color=auto -e cryosparc_ -e mongo
root@sn4622126305:/home/exx# ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
srwx------ 1 cryosparc_user cryosparc_user 0 Mar 11 22:31 /tmp/cryosparc-supervisor-206773da3c7c06e952eddaffaea9188d.sock
srwx------ 1 cryosparc_user cryosparc_user 0 Mar 11 22:31 /tmp/mongodb-39001.sock
root@sn4622126305:/home/exx#
7

Thanks @chris.chau These outputs look normal to me.
To avoid accidental damage to your CryoSPARC installation, may I suggest that you run CryoSPARC-related commands under the cryosparc_user account.
Please can you let us know

  1. What kind of CryoSPARC jobs have you been able to run on this CryoSPARC installation so far, before you encountered the error.
  2. What is the output of the command
    /home/cryosparc_user/software/cryosparc/cryosparc_master/bin/cryosparcm licensestatus
  3. do the CRYOSPARC_LICENSE_ID definitions inside
    /home/cryosparc_user/software/cryosparc/cryosparc_master/config.sh
    and
    /home/cryosparc_user/software/cryosparc/cryosparc_worker/config.sh
    match exactly?
  4. What action and job type triggered Failed to establish a new connection: [Errno 22] Invalid argument’
  5. In which log file or web app section did you observe the error message?
8
  1. What kind of CryoSPARC jobs have you been able to run on this CryoSPARC installation so far, before you encountered the error.

I was just trying to run the Extensive Validation test. It’s failing at the Curate exposures test.
2. cryosparc_user@sn4622126305:~$ /home/cryosparc_user/software/cryosparc/cryosparc_master/bin/cryosparcm licensestatus License is valid

  1. Yeah they do match

image
image1926×1581 256 KB

It just get stuck here and the J1 Job gives the error and says failed

image
image1831×1076 286 KB

9

Any other suggestions?

10

I tried completely removing the cryosparc user and reinstalling.
I tried removing the cryosparc worker and reinstalling the worker. No luck yet.