Connect a managed worker

satchal · April 4, 2022, 8:57pm

I’m trying to connect a new worker to master (from a standalone installation) and am having the following error. Any advice/suggestions would be greatly appreciated!

./bin/cryosparcw connect --worker 128.135.41.238 --master 128.135.41.107 --port 61000 --ssdpath /mnt/OS --update --gpus 0,1
 ---------------------------------------------------------------
  CRYOSPARC CONNECT --------------------------------------------
 ---------------------------------------------------------------
  Attempting to register worker 128.135.41.238 to command 128.135.41.107:61002
  Connecting as unix user kosslab
  Will register using ssh string: kosslab@128.135.41.238
  If this is incorrect, you should re-run this command with the flag --sshstr <ssh string>
 ---------------------------------------------------------------
*** client.py: command (http://128.135.41.107:61002/api) did not reply within timeout of 300 seconds, attempt 1 of 3
*** client.py: command (http://128.135.41.107:61002/api) did not reply within timeout of 300 seconds, attempt 2 of 3
*** client.py: command (http://128.135.41.107:61002/api) did not reply within timeout of 300 seconds, attempt 3 of 3
Traceback (most recent call last):
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f76a12e7150>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='128.135.41.107', port=61002): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f76a12e7150>: Failed to establish a new connection: [Errno 113] No route to host'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/connect.py", line 75, in <module>
    cli = client.CommandClient(host=master_hostname, port=command_core_port)
  File "/home/kosslab/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 40, in __init__
    self._reload()
  File "/home/kosslab/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 68, in _reload
    system = self._get_callable('system.describe')()
  File "/home/kosslab/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 56, in func
    r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/kosslab/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='128.135.41.107', port=61002): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f76a12e7150>: Failed to establish a new connection: [Errno 113] No route to host'))

RyanFeathers · April 7, 2022, 7:15pm

Hi Satchal,

Have you verified that you can ssh between computers and that the port you want to use is open? The error indicates that the worker cannot connect to the master.

Best,

Ryan

satchal · April 9, 2022, 9:10pm

Hi Ryan,

You are right, it’s a port issue, but the port (61000) I’m using for cryoSPARC appears to be open. When the worker connects to the master, which port does it typically try to use? If it connects by ssh, I would expect it would try 22, but the installation instructions do not suggest this. Maybe I’m missing something.

I ended up requesting another license and installed as a standalone on my new system. Not ideal, but it was the easiest fix to get things up and running here.

Thanks,

Satchal

RyanFeathers · April 10, 2022, 12:09am

Hi Satchal,

I don’t know how it works if using ports other than 39000 but you need a range of ports open i.e. 39000-39010.

Best,

Ryan