Dynamic deployment to AWS w/ Groups

Archives Ansible Project
1

I’d like to be able to create an Ansible script that launches (or updates) 1 or more EC2 instances depending on the group (dev, staging, prod) that I’m deploying. In looking through the examples and the AWS guide, the EC2 module and this dynamic inventory stuff, I feel like I’m missing something: How do I execute this script against a particular inventory group if/when I have no inventory file because of the dynamic inventory recommendation?

Hopefully that makes some sense. As an example, I have a playbook with 2 roles: launch and provision. The launch role begins with this code from one of the documented examples:

  • name: Provision a set of instances
    ec2:
    key_name: “{{ aws_keypair }}”
    group: test
    instance_type: t2.micro
    image: “{{ ami_id }}”
    wait: true
    exact_count: 5
    count_tag:
    Name: Demo
    instance_tags:
    Name: Demo
    register: ec2

Using this example, based on the inventory group, the exact_count value will be different, as will the key_name and perhaps other values. How do I read those from the right place since they would be group_vars? In development, exact_count would be 1 while in production it would be at least 2.

Thanks.

2

I thinks this setup will help you.

– production

– group_vars
-- server.yml – inventory
– site.yml
– staging
– group_vars
-- server.yml – inventory
– myrole
– handlers
`-- main.yml
– tasks

– main.yml
– templates
– crontab.j2
-- vars – main.yml

content of each each inventory file is different, but in your case it will be like this for all the files:

[server]
127.0.0.1

and mentioned the value of exact_count variable inside the server.yml of each environment inside their group_vars

Then run this command:

ansible-playbook -i staging site.yml

OR
ansible-playbook -i production site.yml

Thanks,

3

Thanks, Arbab -

This makes sense, except for the inventory file in each “group” directory. My understanding of the documentation, for what little my understanding is worth at this point, is that an inventory file is somewhat discouraged. Are you saying quite literally that my production/inventory, staging/inventory and development/inventory file would all have exactly the same content? If so, does that mean it’s really just acting as a placeholder and there’s no actionable information in there?

Thanks again for your help understanding how this all comes together.

4

Rob, how you are creating the EC2 instances? because EC2 module run as local action. Can you please share your role/playbook so that I’ll look into this and try to explain the things in better way. Thanks,

5

So I may not have a great answer for that. :slight_smile: I’m only JUST getting started on this task and with Ansible itself.

At the moment I have a deploy.yml playbook file from which I’d like to execute 2 roles: launch & configure. The launch will, of course, launch n instances for any given environment. I started that launch role with 1 task - create.yml - that includes nothing except:

  • name: Provision a set of instances
    ec2:
    key_name: “{{ aws.keypair }}”
    group: “{{ aws_security_group }}”
    instance_type: “{{ aws.instance_type }}”
    image: “{{ aws.ami_id }}”
    wait: true
    exact_count: “{{ aws.instance_count }}”
    count_tag:
    Stack: “{{ group_name }}”
    instance_tags:
    Stack: “{{ group_name }}”
    register: ec2

Realizing how many variables I needed and how I’d need them led me to ask this question before I ever got any further. The forthcoming configure role will install and customize the user shell and install other baseline software required on all servers (e.g. AWS CLI).

Does that help shed any light on things?

6

You don’t need any groups at all I would say, just use the --extra-vars parameter when launching the playbook to tell it what kind of instances you want to launch. For example:

– extra-vars ‘{“my_env”:“prod”}’

then in the playbook you evaluate this variable and set some facts accordingly:

  • set-fact:
    var1: |
    {%- if my_env|lower == ‘prod’ -%}
    5
    {%- elif my_env|lower == ‘dev’ -%}
    1
    {%- endif -%}
    var2: |
    …

and use those vars as you wish in the tasks section.

Make sure you have jinja2 extensions enabled in your ansible.cfg file though:

jinja2_extensions = jinja2.ext.do,jinja2.ext.i18n

Now, when you create your ec2 instances you tag them with your environment variable given as input, per your example:

instance_tags:
Name: Demo
Env: “{{ my_env }}”

so you can later use dynamic inventory group Ansible creates for you based on Tags:

  • hosts: tag_Env_prod
    …

  • hosts: tag_Env_dev
    …

  • hosts: tag_Env_stage
    …

in the same or a new playbook depending on your requirements.

Hope I’m getting your question right …

Cheers,
Igor

7

Excellent method but you cannot use dynamic inventory group Ansible creates based on Tags:

  • hosts: tag_Env_prod
    …

  • hosts: tag_Env_dev
    …

  • hosts: tag_Env_stage
    …

in the same playbook because it will give the create that group doesn’t exist or no host inside the group instead you can use the add_host module. Thanks

8

Arbib is right, you will have to add them to a new group and use that in the next play in the same playbook. I went overboard trying to simplify so it does not confuse the beginner folk that might read this post.

Anyway, the point is: in AWS the tags are very powerful in terms of automation so tag everything and tag as much as possible and use that later to your own advantage.

9

I’ve been running this over in my head and I think I’m more confused than ever. I don’t really care whether I specifically use dynamic inventory or any other specific technique. I care that my script is idempotent, reasonably simple to execute for any developer and somewhat consistent with Ansible recommendations. \

If I run the staging deployment, it should create any instances (to meet the exact_count) that don’t exist, update any configuration deltas in those that do exist and then deploy any projects to those servers.

I guess the bottom line question is simply, what’s the appropriate strategy to make this work? Within that context, I do still need variables specific to each group/environment so the question I asked Arbab still stands. I really don’t want to force developers to drop a ton of “extra vars” when the deploy. I’d like to keep that command line execution as dead simple as possible:

Run the deploy playbook against environment X

ansible-playbook -i staging deploy.yml

I really do appreciate the help with this. Learning Ansible has been fairly simple, but figuring out how to organize for a non-trivial infrastructure has definitely been a challenge.

10

Well as I expected this might be little bit confusing to a beginner. So basically the answer to your question was: tag the instances you create based on the environment you are creating them in and then use that tag for the configuration task(s).

Which means you need to base the exact_count on the parameter you want to group on, in this case the environment tag:

exact_count: “5”
count_tag:
Env: “prod”

In this way when ever you run the creation playbook it will check if there are exactly 5 instances with tag named Env and value of prod and if no will create them and if yes will skip the creation task.

There aren’t any “tons” of input vars in my example, it is just the environment tag name you want to run the playbook for. Very simple.

For the configuration playbook as I said you just need to use:

  • hosts: tag_Env_<prod|staging|dev>

to configure the ec2 instances you have created for the environment.

Sorry I really don’t know how to better explain this to a beginner. Don’t get discouraged though keep working and when you come back to this post in couple of weeks when you master Ansible it will all be crystal clear :slight_smile:

11

Thanks, Igor. This has been great. It sounds like you’re saying I can’t/shouldn’t use an inventory file at all, but explicitly pass an extra var for the environment. Rather than dropping conditionals everywhere I need env-specific values, is there any way to get Ansible to use that env value to automatically read variables from a file? Originally, it seemed like group_vars made the most sense, but if an inventory file doesn’t make sense, I’m not sure group_vars make sense either.

Thanks again.

12

Hey Rob,

Well you definitely need a dynamic inventory which in case of AWS is provided by the ec2.py script. I guess you have already read on how to setup this, some helpful links below:

http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script
https://aws.amazon.com/blogs/apn/getting-started-with-ansible-and-dynamic-amazon-ec2-inventory-management/

So basically you download the script from https://raw.github.com/ansible/ansible/devel/contrib/inventory/ec2.py, make it executable and set it as /etc/ansible/hosts (please take backup first of your existing file). Then you just drop https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/ec2.ini to /etc/ansible/ec2.ini and you are good to go.

This is the way I have it set up. In this way I don’t even have to provide inventory file when I’m running some default stuff since Ansible by default reads the /etc/ansible/hots file. So instead:

$ ansible-playbook -i /some/inventory/path some-playbook.yml

I just run:

$ ansible-playbook some-playbook.yml

and Ansible understands by default that I want to run against the dynamic inventory.

So, this dynamic inventory will be the one that will provide you with all the groups you want to work with later. So after running your creation playbook for first time, next time you run a playbook it will have the new instances included in the inventory for you and nicely sorted under appropriate groups. There will be groups available for zones, subnets, security groups, tags etc etc etc. If you run the /etc/ansible/hosts file (which is actually the ec2.py script renamed in our case) manually, it will give you a screen output of the whole inventory including the groups you have on your disposal.

To conclude, you do not need inventory for your instance creation playbook but you DO need it for any other playbook you want to run against already created instances, like the config playbook for example.

Now about the variables. True you can set the variables in the group_vars but the problem is you need them to be dynamic, thus you need to modify them in runtime. So you can have some default valuse lets say in your group_vars file like:

var1: 1

and then change that accordingly depending on the env you are running against:

  • set-fact:
    var1: {%- if my_env|lower == ‘prod’ -%}5{%- endif -%}

OR you can have different set of variables per environment which might become more messy since you will have to maintain different files and possibly directories. That’s my recommendation was to keep it simple to start with and do it all in single playbook using the above logic. The bottom line is you will have to set those variable somewhere and somehow, it is up to you how you want to do it. Then later when you feel more comfortable with Ansible you might start braking it down to roles, dependencies, different inventory dirs per environment, upload it to GitHub etc etc etc.

Just my 2 cents.

Cheers,
Igor

13

Thanks so much, Igor. Tons of great information in there and I really appreciate the jump start. It was feeling a little bit like a paralysis of choice thing for a while there.