How to authorise an EC2 instance for S3 access

Archives Ansible Project
1

I am unable to make use of IAM roles in my Ansible playbooks. Specifically, I have authorised an EC2 instance to get from an S3 bucket, but I cannot work out how to make use of this authorisation from within Ansible.

The question

How do I write Ansible task(s) that satisfies all the following :

  1. Runs on an EC2 instance
  2. Uses the IAM role defined on the EC2 instance to obtain authorisation to access an S3 bucket
  3. Gets a file from the S3 bucket

A work around

I can get the EC2 instance to download from S3, only by passing in my credentials as follows:

`

  • name: Download the part archive from S3
    s3:
    aws_access_key: “{{ lookup(‘env’,‘aws_key’) }}”
    aws_secret_key: “{{ lookup(‘env’,‘aws_secret’) }}”
    region: “{{ aws_packages_region }}”
    bucket: “{{ aws_packages_bucket }}”
    object: “/JI79IML/my_part_X86_64_c7.15.tar.gz”
    dest: “/data/parts/JI79IML/my_part_X86_64_c7.15.tar.gz”
    mode: get
    overwrite: no

`

However, I would rather not send my AWS credentials to the instance. Instead I have defined a role with the appropriate permissions to get files from the S3 bucket.

What I’ve tried

The top answer in the stack overflow question linked below, suggests that it is simple matter of leaving the secret access key parameters out, and letting the Boto library take care of assuming the role.

However, when I try this with Ansible 1.8.4 and Boto 2.36.0 I get

`
msg: No handler was ready to authenticate. 1 handlers were checked. [‘HmacAuthV1Handler’] Check your credentials

`

and with Ansible 1.9.1 and Boto 2.38.0 I get:

msg: Failed to connect to S3: 'module' object has no attribute 'connect_to_region'

How I’ve confirmed the IAM role

To confirm that the IAM role is sufficient, I installed awscli on the EC2 instance and performed the download directly. First, I assumed the role

`

aws sts assume-role --role-arn “${ROLE_ARN}” --role-session-name “GettingMyPart”

`

which returns an absolutely baffling error message that the user with the assumed role cannot assume the role?!? But seems to do the trick, because I can then download the part

`

aws s3api get-object --bucket “${BUCKET_NAME}” --key JI79IML/my_part_X86_64_c7.15.tar.gz my_part_X86_64_c7.15.tar.gz

`

To confirm that the IAM role is required, I created another instance that does not enjoy a role and installed awscli on this second EC2 instance and followed the above steps. In each case, I got the message “Unable to locate credentials” as expected

2

Hi Andrew,

instance profiles do work without any issues, from the error msg: Failed to connect to S3: ‘module’ object has no attribute ‘connect_to_region’

seems like boto is not installed properly, how did you install boto ? can you please try reinstalling boto and check.

  • Benno
3

Thanks Benno,

I install Ansible and Boto in a virtualenv using pip, and then add the following to group_vars/localhosts.yml, which is enough to ensure that the cloudformation, s3, and ec2 modules run on the localhost. Do I need to also install Boto on the remote?

`

Do not use the system installed Python when running locally

ansible_python_interpreter: python

`

The exact set of packages is:

`

Jinja2==2.7.3
MarkupSafe==0.23
PyYAML==3.11
ansible==1.9.1
boto==2.38.0
ecdsa==0.13
paramiko==1.15.2
pycrypto==2.6.1
six==1.9.0
wsgiref==0.1.2

`

regards

Andrew

4

Just a follow up. I tried two more scenarios, the second being the boil-the-ocean approach :slight_smile:

  1. I deactivated the virtual environment, and reinstalled Ansible and Boto to /usr/local using Homebrew and Pip as follows:

`
brew install ansible
pip install boto==2.38.0

`

I then reran the playbook, and got the same error message, but again I was able to execute the s3 and cloudformation tasks locally.

  1. I set my path to a minimum, uninstalled all the Homebrew python, and reinstalled Ansible and Boto using pip into the system as follows:

`
PATH=“/usr/local/bin:/usr/bin:/bin”
pip uninstall boto
brew uninstall ansible
brew uninstall python

curl -O https://bootstrap.pypa.io/get-pip.py
sudo python2.7 get-pip.py
sudo pip install six
sudo pip install boto
sudo pip install ansible

`

I then reran the playbook, and got the same error message, but again I was able to execute the s3 and cloudformation tasks locally.

Thanks

Andrew

5

Hi Andrew,

Sorry if i am understanding this wrong, i assume you already have an ec2 instance which has an iam role attached which gives it access to download buckets/files from S3 right ? and in your playbook you have an s3 task which runs on this target server which has boto and python installed , so i am a bit confused as to why you would need to reinstall ansible boto etc… on your local macbook
Probalby if you can attach your playbook it might make things clear.

  • Benno
6

No problems, I think you have the picture right, but might have missed my earlier question: do I need to install Boto on the target server?

So, yes:

  • All playbooks are run on my laptop
  • A playbook aws-start.yml first creates the EC2 instance. It operates on the localhost
  • A playbook provision.yml then attempts to connect to the S3 bucket. It operates on the EC2 instance

Andrew

7

Hi Andrew,

If the s3 task is running on the target server, then the provisioned instance needs to have boto installed. but if boto is not installed you should have got a message like “boto is not installed on this machine etc…” maybe an incomplete boto installation on target server ?

  • Benno
8

Ok. That is helpful. I will add that as an early task in the provision.yml playbook.

My confusion arises, because i have never installed Boto on the server instance, yet the task runs just fine with Ansible 1.8.4 if I also supply the AWS credential. It is a plain Ubuntu AMI, but perhaps it has the AWS tools installed in someway. I note that awscli depends on botocore.

thanks again

Andrew

9

boto is needed on the machine executing the module, for most cloud
operations this is normally your master/controller (local), but if you
are trying to access s3 from a target/remote server, the module will
run there, hence the requirement of boto on that machine.