Hi,
This morning I noticed this error while attempting to update a project on my server via git clone (I’m using Ansible’s ssh-agent-forwarding, which worked fine for months)
"Failed to download remote objects and refs: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in /home/ubuntu/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/ubuntu/.ssh/known_hosts:1
remove with:
ssh-keygen -f "/home/ubuntu/.ssh/known_hosts" -R "github.com"
RSA host key for github.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The fix didn’t solve the problem, just changed the error message:
TASK [SOFTWARE - Clone xxx-project (run as non-privileged user)] ********************************************************************************************************************************************************************************
fatal: [server]: FAILED! => {“changed”: false, “cmd”: [“/usr/bin/git”, “fetch”, “–tags”, “–force”, “origin”], “msg”: “Failed to download remote objects and refs: Host key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n”}
I then discovered this blog entry which states, that Github updated their RSA SSH key this morning:
https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/
Now, everything they suggest work somehow. But it doesn’t fix the problem, that I’m unable to clone my project on the target machine.
Anybody having the same issue and maybe a solution?
To be specific: I’m unable to execute this (and maybe many other lines as well) now. I can update the project with manual git command on the target host, but not with Ansible from my machine:
- name: SOFTWARE - Clone xxx-project (run as non-privileged user)
tags: software
git:
repo: “{{ xxx_procect_git_repo_clone_url }}”
dest: xxx-project
force: yes
become: yes
become_user: “{{ non_root_user }}”
OK, maybe a solution (workaround?)
I got another error message later, which clarified the failing command:
FAILED! => {“changed”: false, “cmd”: “/usr/bin/git ls-remote git@github.com:xxxxxxxxxxx.git -h refs/heads/HEAD”, "
So I ssh’ed to the machine and issued exactly this command:
The response lead me to an offending, but not obvious “known_host” entry (which obviously described “github.com”), maybe left by “ssh-agent” (hmm, didn’t know that. Shouldn’t be left there I guess).
Warning: the ECDSA host key for ‘github.com’ differs from the key for the IP address ‘140.82.121.3’
Offending key for IP in /home/ubuntu/.ssh/known_hosts:2
I removed this entry in line 2 and it worked. So the fix is: Remove everything from your .ssh/known_hosts which looks like remains from ssh-agent. It might trap the github auth.
You can find out more about why this happened on this Github blog: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/
Thanks. I quoted this in my original post.
Meanwhile this is the way I found and it made the ssh-agent deployment run on 5+ machines again:
- Remove everything from the target hosts ~/.ssh/known_host
- Follow the suggestions given in the blog entry to add the. new github keys
Errata: The old keys describing github in my ~/.ssh/known_hosts have not been left by ssh-agent. Instead it seems to be a “normal behaviour” of git to add the github key again cryptically once a project is cloned locally… Not sure (just because I blamed ssh-agent)
And *this* is why relying on known_hosts has caused more failures of
working software than prevention of faked host access since SSH was
originally written.
The typical entry to disable it in ~/.ssh/config is:
Host *
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
LogLevel error
Sounds legit and works. But isn’t “StrictHostKeyChecking=no” dangerous? (To not start a religious war here :))
Tested an ansible setup from my server (ssh-agent) w/o having github key in known_hosts and the “default” (empty) ssh config. Doesn’t work. I had to add github’s key on the host in order to make this work again.
Not storing github and having the mentioned config let me do the setup, so I conclude, that now anybody could mock to be github to make me ssh connect to him. I would say this is a bit more scary as to have to replace an official github key once in a lifetime.
There is an increased risk. The risk of needing to clean up from reset
host keys is also a significant one, and
tuning and picking which keys are and are without that filter is a
burden. Tools like ansible can, in theory,
provide just such tuning on a server-by-server and SSH-service by
SSH-service basis. But I've several times
encountered git server setups where the admin copied over the Host's
private keys, but not the exposed
git related SSH service's keys because he *did not understand the
distinction*, and it's seriously screwed up
working setups both for the Ansible server and the clients. Manually
insertinig the options into all the SSH
commands eliminates those checks on a case-by-case basis, but frankly,
I have a day job, not the
time to go implant the workaround into every developer's SSH command
line settings.
SSH is NOT hard to manage. You need a well defined management practice. We have a service account on our machines with populated SSH public key. We tightly control access to the private key.
Walter
That's true. But the return on investment of relying on SSH hostkeys
to be consistent is.... generally not worth the trouble.
Nico Kadel-Garcia