Failed to launch! 255 Host key verification failed?

solved

(Patrick Goetz) #1

We’ve been running cryoSPARC with one master and one standalone worker. Yesterday we tried adding another standalone worker, but kept getting a “host key verification failed” message, with jobs refusing to run on the new worker. In an attempt to debug the problem, the PI deleted the cryosparc_user’s .ssh/known_hosts file (subsequently repeated by me on all 3 machines) and now we can’t get either standalone worker to take jobs: both give the “host verification failed” error message, or precisely:

"Command '['ssh', u'cryosparc_user@javelina.biosci.utexas.edu', u'bash -c "nohup /data1/local/home/cryosparc_user/cryosparc2_worker/bin/cryosparcw run --project P21 --job J256 --master_hostname kraken.biosci.utexas.edu --master_command_core_port 39002 > /EM/cryosparc/MorganGilman/P21/J256/job.log 2>&1 & "']' returned non-zero exit status 255"

and

Running job on remote worker node hostname javelina.biosci.utexas.edu
Failed to launch! 255
Host key verification failed.

As the cryosparc_user I can ssh freely between both standalone workers and the master, so apparently it’s referencing some other host key?


(Jason McLellan) #2

@spunjani @sarulthasan Can you help us with this issue? We can’t run any jobs at the moment.


(Stephan Arulthasan) #3

Hi @Jason, @pgoetz,

Can you confirm, if you log onto the master node kraken.biosci.utexas.edu as cryosparc_user and try to execute the command ssh cryosparc_user@javelina.biosci.utexas.edu bash -c "whoami", there is no error, or confirmation to verify the host?


(Jason McLellan) #4

Hi Stephan,

Many thanks for the response. Entering that command resulted in the following:

ssh cryosparc_user@javelina.biosci.utexas.edu bash -c “whoami”
The authenticity of host ‘javelina.biosci.utexas.edu (129.116.159.26)’ can’t be established.
ECDSA key fingerprint is <>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘javelina.biosci.utexas.edu’ (ECDSA) to the list of known hosts.
cryosparc_user

Once the javelina.sci.utexas.edu was added to the list of known hosts, cryoSPARC jobs were able to be run on javelina. I ran a similar command for our new worker and now it works as well.

Thanks very much for helping us troubleshoot this issue. If it’s useful to others, perhaps this could be added to the installation instructions?

Best regards,
Jason


(Stephan Arulthasan) #5

Hi @Jason,

Awesome! We will add this to our site.

Thanks,

Stephan