Playbook ssh issues, AWX add ansible- prefix into the real user name

Get Help
,
1

I have some playbooks running without any issues in AWX 9.0.1.0 and currently I am working on moving all the playbooks to newer version AWX 23.0.0.When I run the playbook in AWX 23.0.0 GUI, it always fails, giving some SSH related errors, the related SSH command used by playbook looks something like

bash-5.1$ ssh -vvv -F ./ssh.cfg -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ‘User=“test”’ -o ConnectTimeout=10 -o ‘ProxyCommand=ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p -q test@jump.test.net’ -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ‘ControlPath=“~/.ssh/ansible-%r@%h:%p”’ test.net

When I copied this SSH command above into AWX task container and directly execute this SSH command , it is running success, so clearly the SSH public/private key set up is correct, also I am sure the credential to be configured in AWX has correctly imported the related SSH private key.

So I am confused now what else can be the cause

The key error logs are

  `  Authenticated to test.net (via proxy) using \"publickey\".\r\n

    debug1: pkcs11_del_provider: called, provider_id = (null)\r\n

    debug1: setting up multiplex master socket\r\n

    debug3: muxserver_listen: temporary control path /runner/.ssh/ansible-test@test.net:22.pqGv0YsDRO3Skadp\r\n

    unix_listener: cannot bind to path /runner/.ssh/**ansible-test**@test.net:22.pqGv0YsDRO3Skadp: No such file or directory",
    "unreachable": true

` It is weird that the user listed above is ‘ansible-test’, but I have never configured this user anywhere in my platform, the real user should be ‘test’

2

The ansible-test string is coming from this:

3

When you do that.~/.ssh/ansible-%r@%h:%p will be expanded in the context of your id’s $HOME. Is your $HOME /runner?

4

hi Thanks a lot, really appreciate your time

But it is a bit weird

I have the same .ssh configuration file for AWX 9.0.1.0 and AWX 23.0.0. Everything on AWX 9.0.1.0 is running smoothly, the issue is only in AWX 23.0.0

But anyway I tried to remove ‘ansible-’ prefix in my ssh.cfg file although i am not 100% sure how this control master works, now my ssh.cfg looks like

ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ControlPersist 5m

But it seems still not working, I have committed the change into GIT and sync the project from AWX GUI and I can confirm the ssh.cfg file in AWX task container has also sync the updated file

My ID in the remote host is actually /home/noc, and user name is also ‘noc’, I have no clue where path ‘/runner’ is coming from

Can anyone or you please suggest some readings about what ‘control path’ parameter is about ?

5

$ man 5 ssh_config

But AWX isn’t running as you. It’s running as… well, I don’t know what it’s running as, but whoever/whatever it’s running as, the ~/ is shorthand for /runner, so it probably doesn’t matter what your ssh.cfg has in it.

Caveat: I use AWX, but somebody else set it up and runs it. So maybe listen to someone who knows rather than my guessing. Good luck.

6

If we need the AWX team, we can ask them! Hey @AWX, do you know what’s going on here? :slight_smile:

7

thanks for the tip :slight_smile:

8

Thanks a lot

I temporarily made it work by disabling all control master parameter in the ansible.cfg and ssh.cfg file

Actually my project does not that need concurrency, maximum 2 or 3 users will execute the playbook at the same time

But I will figure it out how this control master feature works in Ansible later after fixing more pressing issues

9

I have more clue now what’s going on. First of all, you made an assumption about AWX’s use of ControlPath being in order to share a connection to a particular host among multiple jobs. It isn’t. It’s to avoid the overhead of standing up and tearing down ssh connections for every task within the same job. In fact, now that I read your initial post with eyes open, we can see that
~/.ssh/ansible-%r@%h:%p became /runner/.ssh/ansible-test@test.net:22.pqGv0YsDRO3Skadp
But the ssh TOKEN %p is the port, which accounts for the 22. The rest of the string – .pqGv0YsDRO3Skadp – is unique to that particular job. AWX adds that to ensure that different jobs in fact don’t share connections.

Now that you’ve removed the ControlMaster setting, when you run jobs with level 4 verbosity, do you still see ControlMaster=auto and ControlPath still being set? On our AWX, my job’s ssh connection shows
-o 'ControlPath="/runner/cp/40c3e6afae"'
Our appended uniquifier is shaped a little differently from yours, probably because we’re on a different version of AWX.

Practically, though, it looks like your jobs are running now and connecting to the intended remote hosts. Is that correct?

10

hi @utoddl thanks a lot for your explanation, yes, my Playbook is being able to connect to the remote host successfully and my logs, I don’t see any ’ ControlMaster=auto and ControlPath’

11

Hi @mapleos1123! It looks like the post might be solved - could you check to see if the response by @utoddl worked for you?

If so, it would be super helpful if you could click the :heavy_check_mark: on their post to accept the solution - it recognises the input of others, helps our volunteers find new issues to answer, and keeps the forum nice and tidy.

Thanks!
(this is template reply, do feel free to reply if I’ve misunderstood the situation!)