delegate_to: with a different user from the target hosts

Kesten_Broughton2 · April 14, 2014, 4:05pm

We are operating a private openstack cloud. We would like to host a single local mirror for data and package repositories per cloud, with many tenants for a given cloud.

Each tenant/project is on a private subnet with a single “pivot” vm with a floating ip. All traffic is routed through the tenant’s pivot.

Each time a tenant is created we create an ssh key for it and load it into all vms on creation. The admin gets the private/public keys for their .ssh/identities/ folder.

For rapid prototyping, we are sharing keys as needed. Eventually we would replace this with web-services.

The goal is to use jump host settings in .ssh/config to make a call like this

name: Get the jdk from local_mirror_host
synchronize: src=“/opt/java/jdks/{{jdk_rpm}}”
dest=“{{_ansible_downloads}}/{{jdk_rpm}}”
delegate_to: “{{local_mirror_host.name}}”

Currently, running ansible playbooks from a laptop via the pivot is working fine using settings like this

[mac ansible controller] /Users/kbroughton/.ssh/config

Host 10.x.y.z # tenant pivot
User cloud-user
HostName app-dev1-pivot

IdentityFile ~/.ssh/identities/app_dev1_key

Host app-dev1-*
User cloud-user

ProxyCommand ssh 10.x.y.z -W %h:%p # -W instead of nc

IdentityFile ~/.ssh/identities/app_dev1_key

However, tasks with delegation fail with an error like this

failed: [21ct-dev1-pivot] => {“cmd”: “rsync --delay-updates -FF --compress --timeout=10 --archive --rsh ‘ssh -i /Users/kbroughton/.ssh/identities/app_dev1_key -o StrictHostKeyChecking=no’ --out-format=‘<>%i %n%L’ /opt/java/jdks/testjdk cloud-user@app-dev1-pivot:/tmp/ansible/jdk1.7.0_25.rpm”, “failed”: true, “item”: “”, “rc”: 23}

msg: Warning: Identity file /Users/kbroughton/.ssh/identities/app_dev1_key not accessible: No such file or directory.

rsync: mkstemp “/tmp/ansible/.jdk1.7.0_25.rpm.NYd0r7” failed: Permission denied (13)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

Killed by signal 1.

It appears there are several problems:

For the pivot vm the delegate is trying to find the key in /Users/kbroughton/.ssh rather than /root/.ssh as specified in the delegate’s .ssh/config
For the lynx01 vm, the delegate is trying to ssh as root. Cloud vms have no root user, only cloud-user. Note the connection from delegate to pivot correctly attempts to use cloud-user.

It appears the delegate .ssh/config is not being applied.
[delegate /local_mirror_host] /root/.ssh/config

Host 21ct-dev1-*

ProxyCommand ssh 10.x.y.z -W %h:%p

User cloud-user

IdentityFile /root/.ssh/identities/app_dev1_key

Host 10.1.30.186

User cloud-user

HostName app-dev1-pivot

IdentityFile ~/.ssh/identities/app_dev1_key

Note that with the above settings it is possible to do
[root@dev1-pivot] ssh app-pivot or app-lynx01
just fine.
Also not that “localhost” is among the target hosts of the task and when applied via the delegate, ansible changes the meaning of localhost from the laptop running the task to the delegate. The file is copied locally from the delegate to itself on the localhost play.

I would expect that ansible would attempt to apply the .ssh/config settings on the delegate host, or at least allow that as an option.

COMPLETE ERROR TRACE

TASK: [centos_common | Get the jdk from local_mirror_host] ********************

ESTABLISH CONNECTION FOR USER: root

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993 && echo $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993’”]

ESTABLISH CONNECTION FOR USER: root

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1397489133.68-9986*****848 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-139748*48848 && echo $HOME/.ansible/tmp/ansible-tmp-1397497748848’”]

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-139748924490 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-13974924490 && echo $HOME/.ansible/tmp/ansible-tmp-1397489133.68-74531171924490’”]

PUT /var/folders/t2/h2233*s8_088c/T/tmpqymjL0 TO /root/.ansible/tmp/ansible-tmp-1397748848/synchronize

PUT /var/folders/t2/h223fk7s8_088c/T/tmpZmzUty TO /root/.ansible/tmp/ansible-tmp-1397483993/synchronize

PUT /var/folders/t2/h2233fk7s8_088c/T/tmpUm0OXD TO /root/.ansible/tmp/ansible-tmp-139748924490/synchronize

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974993/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1391203993/ >/dev/null 2>&1’”]

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974***4490/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1397489133.68-74531171924490/ >/dev/null 2>&1’”]

EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, u’dev1-pivot’, “/bin/sh -c ‘/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974748848/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1397448/ >/dev/null 2>&1’”]

ok: [localhost] => {“changed”: false, “cmd”: “rsync --delay-updates -FF --compress --timeout=10 --archive --rsh ‘ssh -o StrictHostKeyChecking=no’ --out-format=‘<>%i %n%L’ /opt/java/jdks/testjdk /tmp/ansible/jdk1.7.0_25.rpm”, “item”: “”, “msg”: “”, “rc”: 0}

failed: [21ct-dev1-lynx01] => {“cmd”: “rsync --delay-updates -FF --compress --timeout=10 --archive --rsh ‘ssh -o StrictHostKeyChecking=no’ --out-format=‘<>%i %n%L’ /opt/java/jdks/testjdk root@21ct-dev1-lynx01:/tmp/ansible/jdk1.7.0_25.rpm”, “failed”: true, “item”: “”, “rc”: 2}

msg: Warning: Permanently added ‘21ct-dev1-lynx01’ (RSA) to the list of known hosts.

protocol version mismatch – is your shell clean?

(see the rsync man page for an explanation)

rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.6]

failed: [21ct-dev1-pivot] => {“cmd”: “rsync --delay-updates -FF --compress --timeout=10 --archive --rsh ‘ssh -i /Users/kbroughton/.ssh/identities/21ct_dev1_key -o StrictHostKeyChecking=no’ --out-format=‘<>%i %n%L’ /opt/java/jdks/testjdk cloud-user@21ct-dev1-pivot:/tmp/ansible/jdk1.7.0_25.rpm”, “failed”: true, “item”: “”, “rc”: 23}

msg: Warning: Identity file /Users/kbroughton/.ssh/identities/21ct_dev1_key not accessible: No such file or directory.

rsync: mkstemp “/tmp/ansible/.jdk1.7.0_25.rpm.NYd0r7” failed: Permission denied (13)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

Killed by signal 1.

kesten_broughton1 · April 14, 2014, 5:16pm

Discovered that i had ansible_ssh_user=cloud-user for the pivot host and not the lynx01 host in my ansible hosts file.
That’s why in the above the delegate tried to connect as cloud-user to pivot and root to lynx01.
I’d rather not have to use the hosts file because we have one per tenant (dozens) and it’s less maintainable than a single .ssh/config.
But the larger problem is the key being in different locations on my ansible controller (/User/kbroughton) vs the delegate (/roor).

Is there any way to force ansible to respect .ssh/config both locally and on delegates?
For the -i path to keys, the ansible_ssh_private_key_file won’t work as the location is different for controller vs delegate.
The only fix would be to place all tenant keys in /etc/ssh/identities/ on all hosts which seems pretty non-standard.

Michael_DeHaan1 · April 15, 2014, 9:18pm

“Is there any way to force ansible to respect .ssh/config both locally and on delegates?”

When ansible is using the SSH connection type (read: not paramiko, the default on CentOS and RHEL) it will/should pick all this up automatically.

If you’re using paramiko, that’s a bit of a different story.

It will be a very nice day when RHEL7 comes out with a newer SSH and everyone can use -c openssh by default.

kesten_broughton1 · April 17, 2014, 6:38pm

After some more digging, it appears the problem is at the rsync level, so no matter what ansible synchronize does to wrap rsyn, it can’t work.

using rsync on the commandline with -e “ssh -F /Users/kbroughton/.ssh/config” to force it to use the config file, it picks up the IdentityFile, but not the jump-host stuff.

I’ve submitted a bug report 10557
https://bugzilla.samba.org/show_bug.cgi?id=10557

And posted on stackexchange
http://unix.stackexchange.com/questions/94421/how-to-use-ssh-config-setting-for-each-server-by-rsync/125251#125251

kesten

Topic		Replies	Views
Rsyncing a directory from a remote host to another remote host, with each remote host requiring separate SSH key,s user and ssh_opts Ansible Project	0	0	June 4, 2018
Delegate a task to a different host on the remote machine? Ansible Project	3	2	August 20, 2014
ansible_ssh_user and delegate_to unexpected behavior Ansible Project	0	0	January 28, 2014
Specifying ssh key for delegate_to target Ansible Project	0	2	July 6, 2017
synchronize / delegate_to / transferring files between hosts Ansible Project	3	4	June 19, 2015

delegate_to: with a different user from the target hosts

Related topics