# Idiomatic EC2 project structure **URL:** \ **Category:** Ansible Project\ **Created:** [August 11, 2014, 7:21am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907 "2014-08-11T07:21:47Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Dominic\_Bou-Samra](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Dominic\_Bou-Samra](https://forum.ansible.com/u/Dominic_Bou-Samra)\ **Post date:** [August 11, 2014, 7:21am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/1 "2014-08-11T07:21:47Z")
Hi all, I’ve been playing with Ansible as a solution to our deployment infrastructure. Our stack is: - Dozens of “transporters” running on EC2. - Dozens of “detectors” running on EC2 - Production and staging environments - Multiple regions (atm US East and Sydney) What I require from Ansible is: - A dynamic inventory, MIXED IN with a static group declaration. I would like to be able to refer to my “Australian transporters in production”, as easily as possible. I would like no IP’s/hostnames stored locally (i.e, use EC2.py as a dynamic inventory). - Ability to provision new EC2 instances in any region, in any environment. - As little duplication as possible. I am struggling with trying to lay my project out so that I can achieve these goals. Could anyone provide me with some suggestions or example layouts, ideally with sample command? Let me know if you need more information and what I have so far ---
**Author:** ![Lorin\_Hochstein](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@Lorin\_Hochstein](https://forum.ansible.com/u/Lorin_Hochstein)\ **Post date:** [August 12, 2014, 2:39am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/2 "2014-08-12T02:39:26Z")
Hi Dominic: To support mixing static with dynamic inventory, have the “hostfile” entry in your ansible.cfg point to a directory instead of a file. For example, my ansible.cfg contains: [defaults] hostfile = inventory And my inventory directory looks like this: inventory/hosts inventory/ec2.py inventory/ec2.ini The inventory/hosts file is a static Ansible inventory file. The inventory/ec2.py and inventory/ec2.ini are the dynamic inventory parts. You can edit ec2.ini to specify which regions you want, the example ec2.ini file that ships with ansible is pretty clear on how to do this. I gave a talk on this topic a few months back, the slides may not mean too much without me talking (they’re pretty sparse) but here they are: [http://go-talks.appspot.com/github.com/lorin/camp-devops-talk/talk.slide](http://go-talks.appspot.com/github.com/lorin/camp-devops-talk/talk.slide) You can provision instances using the ec2 module. I recommend that you use tags when you do provisioning because ec2.py will automatically create groups based on tags. One gotcha here is that ec2.py will cache by default, so if you want to launch an instance and then configure it immediately after it comes up, you’ll need to disable caching before in ec2.ini (set cache\_max\_age=0). If you want to do “Australian transporters in production”, I don’t think you can actually define a new group as an intersection of groups, so you’d have to specify the hosts by explicitly doing the intersection of groups: hosts: australia:&transports:&production (where “australia”, “transports” and “production” are all groups that I have assumed you have created via tags). ---
**Author:** ![Dominic\_Bou-Samra](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Dominic\_Bou-Samra](https://forum.ansible.com/u/Dominic_Bou-Samra)\ **Post date:** [August 13, 2014, 7:12am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/3 "2014-08-13T07:12:00Z")
Hi Lorin, Thanks for replying. So the above presentation is exactly the solution we had arrived at in our Ansible spikes. I wish I had seen that 1 week ago :P. You are right about the intersection of groups - it’s a shame you cannot declare hosts like (excuse any syntax issues - code looks funny on a phone): > [staging] [staging:children] tag\_env\_staging [transporters] [transporters:children] tag\_type\_transporter [staging\_transporters] [staging\_transporters:children] staging transporters And then in your playbook, declare: hosts: staging\_transporters But perhaps it’s better to pass in variables like: // configure all staging transporters in us-east-1. Staging inventory declares group\_var with staging param ansible-playbook -i staging -e zone=us-east-1 transporter.yml With transporter.yml looking like: - name: Config transporters hosts: {{environment}}:&{{zone}}:&transporters What do you think of that? ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [August 14, 2014, 11:42am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/4 "2014-08-14T11:42:53Z")
“You are right about the intersection of groups - it’s a shame you cannot declare hosts like (excuse any syntax issues - code looks funny on a phone):” Can you elaborate on what you think is missing? “:children” to define child groups is how that part works, so I’m missing the understanding of what part you find lacking there. Thanks! ---
**Author:** ![Lorin\_Hochstein](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@Lorin\_Hochstein](https://forum.ansible.com/u/Lorin_Hochstein)\ **Post date:** [August 14, 2014, 1:42pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/5 "2014-08-14T13:42:32Z")
> "You are right about the intersection of groups - it's a shame you cannot > declare hosts like (excuse any syntax issues - code looks funny on a > phone):" > > Can you elaborate on what you think is missing? ":children" to define > child groups is how that part works, so I'm missing the understanding of > what part you find lacking there. I can create a new a group as a union of other groups (groups of groups), but AFAIK there's no way to create a new group as the intersection of other groups. For instance, if I have a group named "atlanta" and a group named "webservers", there's no way to define a group called "atlanta\_webservers" that refers to the intersection of those two groups. You have to do "hosts: atlanta:&webservers" in plays. Intersections would be really useful because I want to tag my EC2 instances like: type=webserver env=production And then define a "production\_webservers" group as the intersection of these. Lorin ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [August 14, 2014, 5:25pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/6 "2014-08-14T17:25:29Z")
“I can create a new a group as a union of other groups (groups of groups), but AFAIK there’s no way to create a new group as the intersection of other groups.” Shouldn’t be needed in the inventory file, as this is something you can address using the host spec: - hosts: alpha:&beta or also - hosts: alpha with ansible-playbook foo.yml --limit beta Either of these could be used to apply tags. ---
**Author:** ![Dominic\_Bou-Samra](https://avatars.discourse-cdn.com/v4/letter/d/e99b99/32.png) [@Dominic\_Bou-Samra](https://forum.ansible.com/u/Dominic_Bou-Samra)\ **Post date:** [August 14, 2014, 9:54pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/7 "2014-08-14T21:54:54Z")
Perhaps I am doing it wrongly (thus the idiomatic keyword), but I find that less declarative then being able to define intersections in groups. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [August 14, 2014, 10:32pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/8 "2014-08-14T22:32:06Z")
It’s not an issue of declarative vs imperative here, but saying “talk to these groups”. The children are not technically unions either, they are child relationships. ---
**Author:** ![Lorin\_Hochstein](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@Lorin\_Hochstein](https://forum.ansible.com/u/Lorin_Hochstein)\ **Post date:** [August 15, 2014, 2:05am UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/9 "2014-08-15T02:05:46Z")
I would also find it useful to be able to define a group as an intersection of two groups. It’s something I’ve wanted for a while but just haven’t gotten around to submitting a feature request for. If I use the “hosts: dbservers:&production:&northeast” syntax in N playbooks, then I have N opportunities to make a typo that will have very bad consequences (leaving out the & by mistake). Also, if I was able to define a group called dbservers\_production\_northeast, and then I decided to change the definition of the group, I’d only have to change it in one place. If I use it in N places, I’d have to go and change dbservers:&production:&northeast in N places. Yuck. Relying on the “–limit” argument of ansible-playbook is even more error-prone: there’s a risk of applying the playbook to the wrong hosts _every time_ the playbook is invoked by forgetting the --limit argument. It would be useful if ansible supported pattern syntax for defining children groups, so users could do something like this: [dbservers\_production\_northeast:children] dbservers:&production:&northeast ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [August 16, 2014, 2:33pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/10 "2014-08-16T14:33:27Z")
" would also find it useful to be able to define a group as an intersection of two groups. It’s something I’ve wanted for a while but just haven’t gotten around to submitting a feature request for." This is not something we’d be interested in, most likely, as it’s already doable from the host spec. As a matter of saving time with the ticket system, always a good idea to discuss possible features on ansible-devel first to see if we have good solutions for them, and if we agree, maybe then it becomes a ticket. ---
**Author:** ![Brian\_Jones1](https://avatars.discourse-cdn.com/v4/letter/b/91b2a8/32.png) [@Brian\_Jones1](https://forum.ansible.com/u/Brian_Jones1)\ **Post date:** [September 25, 2014, 7:43pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/11 "2014-09-25T19:43:56Z")
Agree with this actually being useful. The hosts syntax works fine, but makes the playbook less reusable. Why would I want to create two copies of the same playbook, one for ec2 and one for local vms where the only difference is the hosts: specification? Why would I start off making a vm and provisioning it using ec2 terminology in the hosts line? It doesn’t make sense. Obviously you could just use the ec2 names as groups for your local vms and all is well, just more typing with lots of underscores. I have little desire to put hosts: tag\_Type\_Web all over playbooks. I could execute a group\_by: ec2\_tag\_Type but that doesn’t make the group available to --limit. And apparently as discussed earlier in this thread, creating a static inventory file referencing a dynamic group as a ‘child’ doesn’t work. Assume you do the following, and set hostfile to ./inventory ./inventory/vagrant ./inventory/ec2.py ./inventory/production # inventory/vagrant [web] vm1 [db] vm2 [vagrant] vm1 vm2 # inventory/production [web:children] tag\_Type\_Web [production:children] tag\_Type\_Web tag\_Type\_Db ERROR: child group is not defined: (tag\_Type\_Web) The example given previously is just a more complicated version of this. [dbservers\_production\_northeast:children] dbservers:&production:&northeast Goal is something like ansible -l ‘vagrant&web’ ansible -l ‘production&web’ With playbooks referencing - hosts: web Brian ---
**Author:** ![Timothy\_Gerla](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Timothy\_Gerla](https://forum.ansible.com/u/Timothy_Gerla)\ **Post date:** [September 25, 2014, 10:32pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/12 "2014-09-25T22:32:48Z")
> And apparently as discussed earlier in this thread, creating a static > inventory file referencing a dynamic group as a 'child' doesn't work. > > \ > ERROR: child group is not defined: (tag\_Type\_Web) > The example given previously is just a more complicated version of this. It's actually possible to do this easily enough if you also define the tag\_Type child groups in the static inventory as well: # inventory/production [tag\_Type\_Web] [tag\_Type\_Db] [web:children] tag\_Type\_Web [production:children] tag\_Type\_Web tag\_Type\_Db -Tim ---
**Author:** ![Brian\_Jones1](https://avatars.discourse-cdn.com/v4/letter/b/91b2a8/32.png) [@Brian\_Jones1](https://forum.ansible.com/u/Brian_Jones1)\ **Post date:** [September 25, 2014, 11:56pm UTC](https://forum.ansible.com/t/idiomatic-ec2-project-structure/15907/13 "2014-09-25T23:56:38Z")
I decided to just edit ec2.py to create the groups I wanted. Good to know about the empty group trick though! Thanks, Brian