Yes. We will always use node 8.
When I try and run cyrosparcw connect I get the following errors. I have tried using a couple of different IP addresses for the location of the master.
[cryosparc@gates cryosparc_worker]$ ./bin/cryosparcw connect \
--master gates.shapirolab.zi.columbia.edu \
--port 39000 \
--worker gpu008 \
--ssh cryosparc@gpu008 \
--gpus 0,1,2,3,4,5,6,7
CRYOSPARC CONNECT --------------------------------------------
Attempting to register worker gpu008 to command gates.shapirolab.zi.columbia.edu:39002
Connecting as unix user cryosparc
Will register using ssh string: cryosparc@gpu008
If this is incorrect, you should re-run this command with the flag --sshstr
Connected to master.
Current connected workers:
Worker will be registered with 64 CPUs.
Autodetecting available GPUs…
Traceback (most recent call last):
File “/cm/shared/apps/cryosparc/cryosparc_worker/cryosparc_compute/nvidia_smi_util.py”, line 49, in run_nvidia_smi_query
memory_use_info = output_to_list(subprocess.check_output(command.split(), stderr=subprocess.STDOUT))
File “/cm/shared/apps/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/subprocess.py”, line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File “/cm/shared/apps/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/subprocess.py”, line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘[‘nvidia-smi’, ‘–query-gpu=driver_version’, ‘–format=csv,noheader,nounits’]’ returned non-zero exit status 9.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/cm/shared/apps/cryosparc/cryosparc_worker/bin/connect.py”, line 233, in
gpu_devidxs = check_gpus()
File “/cm/shared/apps/cryosparc/cryosparc_worker/bin/connect.py”, line 95, in check_gpus
driver_version = get_driver_version()
File “/cm/shared/apps/cryosparc/cryosparc_worker/cryosparc_compute/nvidia_smi_util.py”, line 65, in get_driver_version
return run_nvidia_smi_query({“driver_version”: “driver_version”})[0][“driver_version”]
File “/cm/shared/apps/cryosparc/cryosparc_worker/cryosparc_compute/nvidia_smi_util.py”, line 55, in run_nvidia_smi_query
raise RuntimeError(
RuntimeError: command ‘[‘nvidia-smi’, ‘–query-gpu=driver_version’, ‘–format=csv,noheader,nounits’]’ returned with error (code 9): b"NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.\n\n"
[cryosparc@gates cryosparc_worker]$