Problem connecting to host behind jump server when host set dynamically

Archives Ansible Project
1

I have a problem where a playbook using add_host fails unless the host (which sits behind a jump server) has already been connected to by running another playbook.
Can anyone explain to me why the following happens, and hopefully how to fix it?

I have two playbooks and a hosts file:

dynamic-host.yml:

  • hosts: localhost
    tasks:

  • name: add remote host
    add_host: name=remote groups=hosts

  • hosts: hosts
    name: hello world
    tasks:

  • shell: echo “hello world!”

static-host.yml:

  • hosts: remote
    name: hello world
    tasks:
  • shell: echo “hello world!”

hosts:

[remote]
remote ansible_host=10.0.0.1 ansible_port=22 ansible_ssh_common_args=‘-o ProxyCommand=“ssh -W %h:%p -q 10.1.0.1”’

If I run the dynamic playbook, it fails:

ansible-playbook dynamic-host.yml -i hosts -vvvv

…

fatal: [remote]: UNREACHABLE! => {“changed”: false, “msg”: “ERROR! SSH encountered an unknown error. The output was:\nOpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /home/lsunde/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/home/lsunde/.ansible/cp/ansible-ssh-10.0.0.1-22-lsunde" does not exist\r\ndebug2: resolving "10.0.0.1" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to 10.0.0.1 [10.0.0.1] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 10.0.0.1 port 22: Connection timed out\r\nssh: connect to host 10.0.0.1 port 22: Connection timed out\r\n”, “unreachable”: true}

PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0
remote : ok=0 changed=0 unreachable=1 failed=0

If I then run the static playbook, it succeeds:

ansible-playbook static-host.yml -i hosts -vvvv

…

PLAY RECAP *********************************************************************
remote : ok=2 changed=1 unreachable=0 failed=0

If t then run the dynamic playbook again (after running the static one), it also succeeds:

ansible-playbook dynamic-host.yml -i hosts -vvvv

…

PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0
remote : ok=2 changed=1 unreachable=0 failed=0

Any ideas why this is happening?

2

I am not sure if this is related, but apparently the socket, by which
the multiplexing of the ssh connection should work, does not exist.

Maybe try this in ansible.cfg and see if it fixes your problem:

[ssh_connection]
control_path = %(directory)s/%%C

Johannes

3

/home/lsunde/temp/ansible-dynamic-host/ansible.cfg:

[defaults]
timeout = 60

[ssh_connection]
control_path = %(directory)s/%%C

output:

$ ansible-playbook dynamic-host.yml -i hosts -vvvv
Using /home/lsunde/temp/ansible-dynamic-host/ansible.cfg as config file
…

TASK [setup] *******************************************************************
<10.0.0.1> ESTABLISH SSH CONNECTION FOR USER: None
<10.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/lsunde/.ansible/cp/%C -tt 10.0.0.1 ‘( umask 22 && mkdir -p “$( echo $HOME/.ansible/tmp/ansible-tmp-1486395620.34-123337012460348 )” && echo “$( echo $HOME/.ansible/tmp/ansible-tmp-1486395620.34-123337012460348 )” )’
fatal: [remote]: UNREACHABLE! => {“changed”: false, “msg”: “ERROR! SSH encountered an unknown error. The output was:\nOpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /home/lsunde/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/home/lsunde/.ansible/cp/c6f5791e0e44a3064cdd690f21bcbf0f354fc3a4" does not exist\r\ndebug2: resolving "10.0.0.1" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to 10.0.0.1 [10.0.0.1] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 10.0.0.1 port 22: Connection timed out\r\nssh: connect to host 10.0.0.1 port 22: Connection timed out\r\n”, “unreachable”: true}

PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0
remote : ok=0 changed=0 unreachable=1 failed=0

4

Second guess:
By adding the hosts via add_hosts you overwrite your definition in the
hosts file. And thus none of your options are set anymore. Note that
this is just in memory, so your hosts file wont get changed.

If you call the static play, it uses the hosts file with parameters.

Solution:

A) Move the parameters to the host_vars for this host.

B) Call add_hosts with the parameters.

Johannes

5

Ah! That seems to be it. The following works:

ansible-playbook dynamic-host2.yml -i /dev/null -vvvv

dynamic-host2.yml:

  • hosts: localhost
    tasks:

  • name: add remote host
    add_host:
    name: remote
    ansible_host: 10.0.0.1
    ansible_port: 22
    ansible_ssh_common_args: ‘-o ProxyCommand=“ssh -o ForwardAgent=yes -p 22022 -W %h:%p -q 10.1.0.1”’

  • hosts: remote
    name: hello world
    tasks:

  • shell: echo “hello world!”