# Panik Github RSH SSH host key change breaks installations **URL:** \ **Category:** Ansible Project\ **Tags:** ubuntu\ **Created:** [March 24, 2023, 12:20pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807 "2023-03-24T12:20:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 24, 2023, 12:20pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/1 "2023-03-24T12:20:26Z")
Hi, This morning I noticed this error while attempting to update a project on my server via `git clone` (I’m using Ansible’s ssh-agent-forwarding, which worked fine for months) "Failed to download remote objects and refs: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s. Please contact your system administrator. Add correct host key in /home/ubuntu/.ssh/known\_hosts to get rid of this message. Offending RSA key in /home/ubuntu/.ssh/known\_hosts:1 remove with: ssh-keygen -f "/home/ubuntu/.ssh/known\_hosts" -R "github.com" RSA host key for [github.com](http://github.com) has changed and you have requested strict checking. Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. The fix didn’t solve the problem, just changed the error message: TASK [SOFTWARE - Clone xxx-project (run as non-privileged user)] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* fatal: [server]: FAILED! =\> {“changed”: false, “cmd”: [“/usr/bin/git”, “fetch”, “–tags”, “–force”, “origin”], “msg”: “Failed to download remote objects and refs: Host key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n”} I then discovered this blog entry which states, that Github updated their RSA SSH key this morning: [https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/](https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/) Now, everything they suggest work somehow. But it doesn’t fix the problem, that I’m unable to clone my project on the target machine. Anybody having the same issue and maybe a solution? ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 24, 2023, 12:28pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/2 "2023-03-24T12:28:10Z")
To be specific: I’m unable to execute this (and maybe many other lines as well) now. I can update the project with manual git command on the target host, but not with Ansible from my machine: - name: SOFTWARE - Clone xxx-project (run as non-privileged user) tags: software git: repo: “{{ xxx\_procect\_git\_repo\_clone\_url }}” dest: xxx-project force: yes become: yes become\_user: “{{ non\_root\_user }}” ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 24, 2023, 12:37pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/3 "2023-03-24T12:37:22Z")
OK, maybe a solution (workaround?) I got another error message later, which clarified the failing command: FAILED! =\> {“changed”: false, “cmd”: “/usr/bin/git ls-remote [git@github.com](mailto:git@github.com):xxxxxxxxxxx.git -h refs/heads/HEAD”, " So I ssh’ed to the machine and issued exactly this command: The response lead me to an offending, but not obvious “known\_host” entry (which obviously described “[github.com](http://github.com)”), maybe left by “ssh-agent” (hmm, didn’t know that. Shouldn’t be left there I guess). Warning: the ECDSA host key for ‘[github.com](http://github.com)’ differs from the key for the IP address ‘140.82.121.3’ Offending key for IP in /home/ubuntu/.ssh/known\_hosts:2 I removed this entry in line 2 and it worked. So the fix is: Remove everything from your .ssh/known\_hosts which looks like remains from ssh-agent. It might trap the github auth. ---
**Author:** ![Simon\_Kelly](https://avatars.discourse-cdn.com/v4/letter/s/90db22/32.png) [@Simon\_Kelly](https://forum.ansible.com/u/Simon_Kelly)\ **Post date:** [March 24, 2023, 12:40pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/4 "2023-03-24T12:40:57Z")
You can find out more about why this happened on this Github blog: [https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/](https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/) ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 24, 2023, 12:43pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/5 "2023-03-24T12:43:43Z")
Thanks. I quoted this in my original post. Meanwhile this is the way I found and it made the ssh-agent deployment run on 5+ machines again: - Remove everything from the target hosts ~/.ssh/known\_host - Follow the suggestions given in the blog entry to add the. new github keys ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 24, 2023, 1:08pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/6 "2023-03-24T13:08:04Z")
Errata: The old keys describing github in my ~/.ssh/known\_hosts have not been left by ssh-agent. Instead it seems to be a “normal behaviour” of git to add the github key again cryptically once a project is cloned locally… Not sure (just because I blamed ssh-agent) ---
**Author:** ![Nico\_Kadel-Garcia](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/nico_kadel-garcia/32/9998_2.png) [@Nico\_Kadel-Garcia](https://forum.ansible.com/u/Nico_Kadel-Garcia)\ **Post date:** [March 24, 2023, 10:43pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/7 "2023-03-24T22:43:38Z")
And \*this\* is why relying on known\_hosts has caused more failures of working software than prevention of faked host access since SSH was originally written. The typical entry to disable it in ~/.ssh/config is:     Host \*         UserKnownHostsFile /dev/null         StrictHostKeyChecking no          LogLevel error ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 25, 2023, 6:24am UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/8 "2023-03-25T06:24:32Z")
Sounds legit and works. But isn’t “StrictHostKeyChecking=no” dangerous? (To not start a religious war here :)) ---
**Author:** ![Neil\_Young](https://avatars.discourse-cdn.com/v4/letter/n/a3d4f5/32.png) [@Neil\_Young](https://forum.ansible.com/u/Neil_Young)\ **Post date:** [March 25, 2023, 6:34am UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/9 "2023-03-25T06:34:47Z")
Tested an ansible setup from my server (ssh-agent) w/o having github key in known\_hosts _and_ the “default” (empty) ssh config. Doesn’t work. I had to add github’s key on the host in order to make this work again. Not storing github _and_ having the mentioned config let me do the setup, so I conclude, that now anybody could mock to be github to make me ssh connect to him. I would say this is a bit more scary as to have to replace an official github key once in a lifetime. ---
**Author:** ![Nico\_Kadel-Garcia](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/nico_kadel-garcia/32/9998_2.png) [@Nico\_Kadel-Garcia](https://forum.ansible.com/u/Nico_Kadel-Garcia)\ **Post date:** [March 25, 2023, 7:49pm UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/10 "2023-03-25T19:49:03Z")
There is an increased risk. The risk of needing to clean up from reset host keys is also a significant one, and tuning and picking which keys are and are without that filter is a burden. Tools like ansible can, in theory, provide just such tuning on a server-by-server and SSH-service by SSH-service basis. But I've several times encountered git server setups where the admin copied over the Host's private keys, but not the exposed git related SSH service's keys because he \*did not understand the distinction\*, and it's seriously screwed up working setups both for the Ansible server and the clients. Manually insertinig the options into all the SSH commands eliminates those checks on a case-by-case basis, but frankly, I have a day job, not the time to go implant the workaround into every developer's SSH command line settings. ---
**Author:** ![Walter\_Rowe](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/walter_rowe/32/4506_2.png) [@Walter\_Rowe](https://forum.ansible.com/u/Walter_Rowe)\ **Post date:** [March 27, 2023, 11:34am UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/11 "2023-03-27T11:34:38Z")
SSH is NOT hard to manage. You need a well defined management practice. We have a service account on our machines with populated SSH public key. We tightly control access to the private key. Walter ---
**Author:** ![Nico\_Kadel-Garcia](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/nico_kadel-garcia/32/9998_2.png) [@Nico\_Kadel-Garcia](https://forum.ansible.com/u/Nico_Kadel-Garcia)\ **Post date:** [March 28, 2023, 10:52am UTC](https://forum.ansible.com/t/panik-github-rsh-ssh-host-key-change-breaks-installations/37807/12 "2023-03-28T10:52:22Z")
That's true. But the return on investment of relying on SSH hostkeys to be consistent is.... generally not worth the trouble. Nico Kadel-Garcia