We’ve been running cryoSPARC with one master and one standalone worker. Yesterday we tried adding another standalone worker, but kept getting a “host key verification failed” message, with jobs refusing to run on the new worker. In an attempt to debug the problem, the PI deleted the cryosparc_user’s .ssh/known_hosts file (subsequently repeated by me on all 3 machines) and now we can’t get either standalone worker to take jobs: both give the “host verification failed” error message, or precisely:
"Command '['ssh', firstname.lastname@example.org', u'bash -c "nohup /data1/local/home/cryosparc_user/cryosparc2_worker/bin/cryosparcw run --project P21 --job J256 --master_hostname kraken.biosci.utexas.edu --master_command_core_port 39002 > /EM/cryosparc/MorganGilman/P21/J256/job.log 2>&1 & "']' returned non-zero exit status 255"
Running job on remote worker node hostname javelina.biosci.utexas.edu Failed to launch! 255 Host key verification failed.
As the cryosparc_user I can ssh freely between both standalone workers and the master, so apparently it’s referencing some other host key?