Ansible bug - IP of servers change to 127.0.0.1 mid way through playbook

Archives Ansible Project
1

⇒ uname -a
Linux caine 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux

⇒ ansible-playbook --version
ansible-playbook 1.7.2

From playbook:

149 - name: PHP | Copy over php-fpm.conf
150 copy: src=/ansible/files/etc-php5/fpm/php-fpm.conf dest=/etc/php5/fpm/ mode=0644 owner=root
151 tags: php
152
153 - name: PHP | Copy over fpm-php pools
154 synchronize: src=/ansible/files/etc-php5/fpm/pool.d dest=/etc/php5/fpm/ recursive=yes delete=yes
155 tags: php

⇒ ansible-playbook -i …/hosts web_config.yml -t php

PLAY [Ansible config for web servers] *****************************************

GATHERING FACTS ***************************************************************
ok: [web4]
ok: [web5]
ok: [web2]
ok: [web1]

TASK: [PHP | Copy over php-fpm.conf] ******************************************
ok: [web4]
ok: [web5]
ok: [web1]
ok: [web2]

TASK: [PHP | Copy over fpm-php pools] *****************************************
The authenticity of host ‘127.0.0.1 (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.
Are you sure you want to continue connecting (yes/no)? no
fatal: [web1] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue
The authenticity of host ‘127.0.0.1 (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.
Are you sure you want to continue connecting (yes/no)? no
fatal: [web2] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue
The authenticity of host ‘127.0.0.1 (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.
Are you sure you want to continue connecting (yes/no)? no
The authenticity of host ‘127.0.0.1 (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.
Are you sure you want to continue connecting (yes/no)? no
fatal: [web5] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue
fatal: [web4] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

FATAL: all hosts have already failed – aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/home/michael/web_config.retry

web1 : ok=2 changed=0 unreachable=1 failed=0
web2 : ok=2 changed=0 unreachable=1 failed=0
web4 : ok=2 changed=0 unreachable=1 failed=0
web5 : ok=2 changed=0 unreachable=1 failed=0

Using -vvvv just verifies that magically the IP of the servers are now all 127.0.0.1:

TASK: [PHP | Copy over fpm-php pools] *****************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: root
<127.0.0.1> EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/home/michael/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘127.0.0.1’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1426710670.95-25632361611852 && echo $HOME/.ansible/tmp/ansible-tmp-1426710670.95-25632361611852’”]
<127.0.0.1> ESTABLISH CONNECTION FOR USER: root
<127.0.0.1> EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/home/michael/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=root’, ‘-o’, ‘ConnectTimeout=10’, ‘127.0.0.1’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1426710670.96-173902150969631 && echo $HOME/.ansible/tmp/ansible-tmp-1426710670.96-173902150969631’”]
The authenticity of host ‘127.0.0.1 (127.0.0.1)’ can’t be established.
ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Any ideas? This script has worked for months. The only thing different is I have been trying to get linode support working and ran “sudo pip install linode-python”

2

It looks like your name resolution has changed, can you show us your
ansible hosts file, your /etc/resolv.conf and your /etc/hosts ?

3

using -vvvv I get this:

…

TASK: [PHP | Copy over php-fpm.conf] ******************************************
ESTABLISH CONNECTION FOR USER: root
ESTABLISH CONNECTION FOR USER: root

…

ESTABLISH CONNECTION FOR USER: root
ESTABLISH CONNECTION FOR USER: root

…

TASK: [PHP | Copy over fpm-php pools] *****************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: root
<127.0.0.1> ESTABLISH CONNECTION FOR USER: root
<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

…
<127.0.0.1> EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’…

[ hangs here ]

4

do you have delegate_to or connection: local set on that task?

5

Hi Brian.

From hosts:
…

[webservers]
web[01:02]
web[04:05]

…

resolv.conf contains the nameserver of my router.

from etc/hosts:

45.56.XXX.XXX web1
104.237.XXX.XXX web2
104.239.XXX.XXX web4
104.130.XXX.XXX web5

Neither the ansible hosts file or /etc/hosts are changing mid run of the ansible script in the same place every time. This is definitely a bug in Ansible. Thanks for your input. :slight_smile:

6

Hi Brian. :slight_smile:

do you have delegate_to or connection: local set on that task?
No. Here’s the top of web_config.yml

1 —
2 - name: Ansible config for web servers
3 hosts: webservers
4 user: root
5 sudo: no
6 vars_files:
7 - password.yml
8 - hosts.yml
9 vars:
…
23 tasks:

…

7

but on the task that is hitting localhost, do you have a
delegate_to/connection: local there?

8

but on the task that is hitting localhost, do you have a delegate_to/connection: local there?

No, that’s the bug I’m reporting. If I comment out

153 # - name: PHP | Copy over fpm-php pools
154 # synchronize: src=/obsidian/ansible/files/etc-php5/fpm/pool.d dest=/etc/php5/fpm/ recursive=yes delete=yes
155 # tags: php

Then the rest of the playbook works correctly. The synchronize command specifically is using 127.0.0.1 instead of the machine IPs.

9

By default synchronize runs as a local action to copy files from the
ansible host to the remote server, do you have localhost or 127.0.0.1
defined in your hosts file and connection variables set?

10

Thank you Brian!!! Following some advice on using the linode module I added this to the ansible hosts file:

20 [local]
21 localhost

This broke only the synchronize command. Strange.