# ansible ec2\_facts returns false data (if there is NAT on the system level; This is ok if You use AWS router interface gateway) **URL:** \ **Category:** Ansible Project\ **Tags:** ubuntu, aws\ **Created:** [July 14, 2015, 12:16pm UTC](https://forum.ansible.com/t/ansible-ec2-facts-returns-false-data-if-there-is-nat-on-the-system-level-this-is-ok-if-you-use-aws-router-interface-gateway/18756 "2015-07-14T12:16:49Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Jakub\_Muszynski](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@Jakub\_Muszynski](https://forum.ansible.com/u/Jakub_Muszynski)\ **Post date:** [July 14, 2015, 12:16pm UTC](https://forum.ansible.com/t/ansible-ec2-facts-returns-false-data-if-there-is-nat-on-the-system-level-this-is-ok-if-you-use-aws-router-interface-gateway/18756/1 "2015-07-14T12:16:49Z")
**THE PROBLEM:** I’ve just realised why sometimes my playbook fills the template with false data This happens, when the instance is in my VPC subnet (with internet gateway), while in configuration there is **NAT route table on the system level** , then **reguest to the internet goes through NAT instance** and the AWS response is **covered.** Then the **NAT\_instance facts** are **returned** , NOT the current\_instance facts about. **THE DEBUGGING:** If You look into the code, the ec2\_facts fetch a bunch of requests to > ‘[http://169.254.169.254/latest/meta-data](http://169.254.169.254/latest/meta-data)’ in Example: > curl [http://169.254.169.254/latest/meta-data/local-ipv4](http://169.254.169.254/latest/meta-data/local-ipv4) > **172.16.0.200** while **real data** is > eth0: \*\*\* > > inet **172.16.0.110** /24 brd 172.16.0.255 scope global eth0 THE INSTANCE CONFIGURATION: > $ ip r > > default via 172.16.0.200 dev eth0 > > 172.16.0.0/24 dev eth0 proto kernel scope link src 172.16.0.110 > > 172.16.0.0/16 via 172.16.0.1 dev eth0 $ ip a > eth0: \*\*\* > > inet **172.16.0.110** /24 brd 172.16.0.255 scope global eth0 If You keep remote files, You can check it Yourself > export ANSIBLE\_KEEP\_REMOTE\_FILES=1 and then > python /home/ubuntu/.ansible/tmp/ansible-tmp-1436872330.49-72199016469620/ec2\_facts will return as one of the facts: “ansible\_ec2\_local\_ipv4”: “172.16.0.200”, (or run a curl) > curl [http://169.254.169.254/latest/meta-data/local-ipv4](http://169.254.169.254/latest/meta-data/local-ipv4) **THE CURRENT WORKAROUND:** 1. do NOT use (in **roles** nor **tasks** ) 2. - action: ec2\_facts 3. DRAWBACKS: 4. You will not have some variables available (**ansible\_ec2\_\* will be unavailable)** 5. You will have only **ec2\_\* facts** from you LOCAL **inventory cache (ec2.py** if I’m correct now) 6. If You add in playbook (“gather\_facts: True”) then You can also use **ansible\_\* facts** gathered by **setup.py** module 7. so instead of **ansible\_ec2\_local\_ipv4** You can use \*\*ansible\_eth0['ipv4][‘address’]\*\*1. **BUT** this can bring some problems when You have a role, that expects some vatiable (example: ansible\_hostname), but in the playbook You have disabled system fact gathering (“gather\_facts: False”) - You will have to be carefull 8. **OR** You would like to access some AWS variable, independent form Your LOCAL cache1. configure you VPC routing tables so it will point to NAT-instance-interface, rather than IP address 9. 0.0.0.0/0 eni-xxx / i-xxx1. instead of: 10. 0.0.0.0/0 igw-zzzzz + system routing tables1. Then You do not have to override the routing table on the system level 11. You rely on AWS Router 12. DRAWBACKS 13. You will have to change the routing table in the VPC, pointing to other phisical interface, when Your NAT instance will shut down 14. vs1. If kept with system routing table, You will lunch new NAT-instance with “old IP address” attached **QUESTIONS / CONCLUSION:** 15. Be aware about ec2\_facts limitation 16. If possible - rely on Amazon Routing Table 17. How You prevent SPOF in Your VPC subnets? 18. What is Your best-practise to configure VPC subnet (private and public), so they have internet outside access (for github, apt), and are still safe without SPOF that is NAT-instance? ---
**Author:** ![igorc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/igorc/32/5491_2.png) [@igorc](https://forum.ansible.com/u/igorc)\ **Post date:** [July 15, 2015, 12:21am UTC](https://forum.ansible.com/t/ansible-ec2-facts-returns-false-data-if-there-is-nat-on-the-system-level-this-is-ok-if-you-use-aws-router-interface-gateway/18756/2 "2015-07-15T00:21:37Z")
I’m using Ansible with AWS VPC’s, where most of them have public and private subnets, and have never had the problem you are seeing. This is definitely a misconfiguration on your side and nothing to do with Ansible. The ec2\_facts is doing the right thing, there is no other way of collecting data except querying the meta-data repository which is what the AWS CLI tools do anyway. Meaning you will get wrong data using AWS CLI as well. Don’t forget you are in the cloud and your networking is configured in the hypervisor/SDN level and NOT on instance level. Meaning you can create as many network interfaces as you want on instance level and set IP’s on those but none of them will work since you have bypassed the SDN and there is no record of those in the meta-data repository. Which finally means that collecting facts on the instance locally really means nothing if those values don’t match what is in the meta-data repository. Now that we have that cleared, lets move to your problem, which looks to me is AWS routing tables. Or more specific the lack of those. For an instance to be in a private subnet it needs separate routing table from the VPC’s default one (which has IGW created for you when the VPC was created) that has the NAT instance as IGW (internet gateway). And that is all you need, you don’t have to set any routing tables on the system level, the SDN will route the traffic for you. Hope this makes sense. Since you haven’t provided any info about your subnets, routing tables, ACL’s etc. this is more of a guess what’s going on so please correct my assumptions if needed. Thanks, Igor ---
**Author:** ![igorc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/igorc/32/5491_2.png) [@igorc](https://forum.ansible.com/u/igorc)\ **Post date:** [July 15, 2015, 12:52am UTC](https://forum.ansible.com/t/ansible-ec2-facts-returns-false-data-if-there-is-nat-on-the-system-level-this-is-ok-if-you-use-aws-router-interface-gateway/18756/3 "2015-07-15T00:52:01Z")
Have to correct myself, you do provide the subnet information. So in answer to you questions/conclusions they way I do it is: - Use private routing table for the private subnets pointing to the NAT as IGW - Use 2 x NAT instances and NAT takeover script that modifies the the private subnets routing table and points the IGW to itself in case the other NAT instance has failed ---
**Author:** ![Jakub\_Muszynski](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@Jakub\_Muszynski](https://forum.ansible.com/u/Jakub_Muszynski)\ **Post date:** [July 15, 2015, 11:14am UTC](https://forum.ansible.com/t/ansible-ec2-facts-returns-false-data-if-there-is-nat-on-the-system-level-this-is-ok-if-you-use-aws-router-interface-gateway/18756/4 "2015-07-15T11:14:22Z")
Thanks Igor. You are right, it is not ansible “bug”, but an configuration-feature, tough it is the “bad one” since it silently provides the false data. I had to dig into the source code to track it down. There could be some warning in ec2\_facts detecting default route, but it would be some work 😕