I have some playbooks running without any issues in AWX 22.214.171.124 and currently I am working on moving all the playbooks to newer version AWX 23.0.0.When I run the playbook in AWX 23.0.0 GUI, it always fails, giving some SSH related errors, the related SSH command used by playbook looks something like
bash-5.1$ ssh -vvv -F ./ssh.cfg -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ‘User=“test”’ -o ConnectTimeout=10 -o ‘ProxyCommand=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p -q firstname.lastname@example.org’ -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ‘ControlPath=“~/.ssh/ansible-%r@%h:%p”’ test.net
When I copied this SSH command above into AWX task container and directly execute this SSH command , it is running success, so clearly the SSH public/private key set up is correct, also I am sure the credential to be configured in AWX has correctly imported the related SSH private key.
So I am confused now what else can be the cause
The key error logs are
` Authenticated to test.net (via proxy) using \"publickey\".\r\n
debug1: pkcs11_del_provider: called, provider_id = (null)\r\n
debug1: setting up multiplex master socket\r\n
debug3: muxserver_listen: temporary control path /email@example.com:22.pqGv0YsDRO3Skadp\r\n
unix_listener: cannot bind to path /firstname.lastname@example.org:22.pqGv0YsDRO3Skadp: No such file or directory",
` It is weird that the user listed above is ‘ansible-test’, but I have never configured this user anywhere in my platform, the real user should be ‘test’
ansible-test string is coming from this:
When you do that.
~/.ssh/ansible-%r@%h:%p will be expanded in the context of your id’s
$HOME. Is your
hi Thanks a lot, really appreciate your time
But it is a bit weird
I have the same .ssh configuration file for AWX 126.96.36.199 and AWX 23.0.0. Everything on AWX 188.8.131.52 is running smoothly, the issue is only in AWX 23.0.0
But anyway I tried to remove ‘ansible-’ prefix in my ssh.cfg file although i am not 100% sure how this control master works, now my ssh.cfg looks like
But it seems still not working, I have committed the change into GIT and sync the project from AWX GUI and I can confirm the ssh.cfg file in AWX task container has also sync the updated file
My ID in the remote host is actually /home/noc, and user name is also ‘noc’, I have no clue where path ‘/runner’ is coming from
Can anyone or you please suggest some readings about what ‘control path’ parameter is about ?
$ man 5 ssh_config
But AWX isn’t running as you. It’s running as… well, I don’t know what it’s running as, but whoever/whatever it’s running as, the
~/ is shorthand for
/runner, so it probably doesn’t matter what your
ssh.cfg has in it.
Caveat: I use AWX, but somebody else set it up and runs it. So maybe listen to someone who knows rather than my guessing. Good luck.
If we need the AWX team, we can ask them! Hey @AWX, do you know what’s going on here?
Thanks a lot
I temporarily made it work by disabling all control master parameter in the ansible.cfg and ssh.cfg file
Actually my project does not that need concurrency, maximum 2 or 3 users will execute the playbook at the same time
But I will figure it out how this control master feature works in Ansible later after fixing more pressing issues
I have more clue now what’s going on. First of all, you made an assumption about AWX’s use of
ControlPath being in order to share a connection to a particular host among multiple jobs. It isn’t. It’s to avoid the overhead of standing up and tearing down ssh connections for every task within the same job. In fact, now that I read your initial post with eyes open, we can see that
But the ssh TOKEN
%p is the port, which accounts for the
22. The rest of the string –
.pqGv0YsDRO3Skadp – is unique to that particular job. AWX adds that to ensure that different jobs in fact don’t share connections.
Now that you’ve removed the
ControlMaster setting, when you run jobs with level 4 verbosity, do you still see
ControlPath still being set? On our AWX, my job’s ssh connection shows
Our appended uniquifier is shaped a little differently from yours, probably because we’re on a different version of AWX.
Practically, though, it looks like your jobs are running now and connecting to the intended remote hosts. Is that correct?
hi @utoddl thanks a lot for your explanation, yes, my Playbook is being able to connect to the remote host successfully and my logs, I don’t see any ’
Hi @mapleos1123! It looks like the post might be solved - could you check to see if the response by @utoddl worked for you?
If so, it would be super helpful if you could click the on their post to accept the solution - it recognises the input of others, helps our volunteers find new issues to answer, and keeps the forum nice and tidy.
(this is template reply, do feel free to reply if I’ve misunderstood the situation!)