Jobs fail because master and worker hostnames have changed

Troubleshooting
1

Hi I do get now problems with launching jobs on remote work stations.

It just hangs for hours in launching and does nothing output from the command log:

Trying to schedule on (hostname)
Launchable: True
Alloc slots : {u’GPU’: [0], u’RAM’: [0, 1, 2], u’CPU’: [0, 1, 2, 3]}
Alloc fixed : {u’SSD’: True}
Alloc licen : True
– Launchable! – Launching.
---- Running project UID P9 job UID J18

Any chance to get more output to see what the problem is.

2

Hello,

Please take a look at the documentation here.

3

Hi thanks for the tip.

Now I found the issue. Our IT changed the hostnames of our workstations and thus the old hostnames dont work anymore. I already thought this might be the issue and I disconnected the worker via
cryosparcm cli ‘remove_scheduler_target_node("‘wronghostname’")’
and reconnected the worker via
cryosparcw connect --master "newhostname
But the error persists and I am getting in the joblog

“/home/david.haselbach/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/requests/adapters.py”, line 508, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host=‘wronghostname’, port=39002): Max retries exceeded with url: /api (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x2b067b2f7310>: Failed to establish a new connection: [Errno -2] Name or service not known’,))

So it seems the old hostname is saved somewhere and I dont really know where. Any idea how I can change this?

4

I reconnected again and now I get the following error:

Traceback (most recent call last):
  File "/home/david.haselbach/bin/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/david.haselbach/bin/cryosparc2/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/david.haselbach/bin/cryosparc2/cryosparc2_master/cryosparc2_compute/client.py", line 57, in <module>
    print eval("cli."+command)
  File "<string>", line 1, in <module>
  File "/home/david.haselbach/bin/cryosparc2/cryosparc2_master/cryosparc2_compute/client.py", line 31, in func
    assert False, res['error']
AssertionError: {u'message': u"OtherError: argument of type 'NoneType' is not iterable", u'code': 500, u'data': None, u'name': u'OtherError'}
5

Hey,

The master hostname is maintained in the config.sh file kept in cryosparc2_package/cryosparc2_master/.

After stopping cryoSPARC, you should be able to edit this file and change the master hostname. Then, after restarting, you should be able to successfully reconnect the worker nodes using the syntax bin/cryosparcw connect --worker <worker_hostname> --master <master_hostname> --port <port_num> --ssdpath <ssd_path>

6

This worked. Thank you!