I have a role that uses the ec2 module and it is working fine, so I’m assuming I’m ok on dependencies like boto… but when I try to run ec2.py --list I get back:
root@ip-1-1-1-1:/etc/ansible/elastirax# ./ec2.py --list
Looks like AWS is down again:
EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
AuthFailureAWS was not able to validate the provided access credentials removed
That host is an ec2 instance itself. I tried it on another host (a rackspace box) after exporting the credentials the exact same way and the problem isn’t there; both the role and ec2.py work there.
I’ve tried comparing installed packages between the two hosts (both Ubuntu 12.04.3, ansible 1.4.3) and using pip to make sure I have latest boto, but I’m not sure how to troubleshoot from here. Do I need to do something to run ec2.py on an ec2 instance…change a url to an internal one or something?
No change specifically with ec2.py… it doesn’t work on the host where I wanted to use it, but it does on another host I tried it on.
However, on the host where ec2.py is not working, I thought I’d mention that I am able to use boto on the python shell:
ubuntu@ip-172-16-20-21:/etc/ansible$ python
Python 2.7.3 (default, Sep 26 2013, 20:03:06)
[GCC 4.6.3] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
Check the version of boto. I ran into the same issue and found that the version of boto on the system was old enough that it
wouldn’t automatically pull in EC2 role credentials.
IIRC the ec2 module expects AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables, but ec2.py expects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. Are you pulling your credentials from the environment? Could this be the problem?
No outbound firewalls (within the OS or at the security group level)
As far as DNS, I assumed the console url is the same as the endpoint and tried digging ec2.us-west-2.amazonaws.com. The response only lists one IP at a time in an alternating fashion, but after running it numerous times I’ve seen a few IPs show up as return values on both the host that is working and the host that is not working, so I am pretty sure they’re resolving to the same list of addresses.
I’ve run ‘sudo pip install boto --upgrade’ and got version 2.23.0
I was actually pretty hopeful about this last one :)… but I’ve confirmed I get a different error (No handler was ready to authenticate. 1 handlers were checked. Check your credentials) whenever the credentials are exported to the wrong var names or are not exported at all. I also made sure and they are exported as the same names on the host that is working.
Finally, I downloaded a new copy of ec2.py and ec2.ini today from GitHub, but it didn’t change anything.
I think boto is throwing a non-fatal exception that ec2.py is interpreting as fatal. If I comment out the sys.exit call on line line 288 I still get the error message, but then the script prints my ec2 inventory as if nothing is wrong.
I discovered this when I made a new test.py script and basically started pulling out pieces of ec2.py to try and find where it was getting lost. I was boiler-plating in the get_instances_by_region function but didn’t have add_instance yet, so I simply changed the call to add_instance to a print statement instead. At that point I got output from the print that showed it was working, but then the exception was still catching and exiting.
It makes sense to me… but I just recently started learning python so if this is in fact what is happening (while I’ll be happy to try things on my end) I don’t think I’m yet comfortable enough with the try/except particulars to do anything about it myself.
As a final thought, the access keys I am using are in fact for an IAM user. I keep wanting to wonder if perhaps my account is only allowed to query certain regions… but again, I don’t have this problem on the other host I’m using to test with. On that host I can use these same access keys work with stock ec2.py and ec2.ini, querying all regions without issue.
bobtheuser@ip-172-20-fake-124:~/jobs/thajob/workspace/bin/devtools/ansible$ ./ec2.py --list --refresh-cache
Looks like AWS is down again:
EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
AuthFailureAWS was not able to validate the provided access credentialscefef… … …
{
“_meta”: {
“hostvars”: {
“54.fake.141.999”: {
“ec2__in_monitoring_element”: false,
“ec2_ami_launch_index”: “0”,
“ec2_architecture”: “x86_64”,
“ec2_client_token”: “test9”,
“ec2_dns_name”: “ec2-54-fake-141-999.us-west-2.compute.amazonaws.com”,
“ec2_ebs_optimized”: false,
For posterity… I downloaded a fresh copy of ec2.py today and this was no longer an issue. In the new copy there was an ‘if’ that changed to an ‘if not’ near the lines of the exception catch I had to comment out before, so I’m assuming that was it.