If I work with many projects, each project we use different role. Are there any good ways to arrange structure for this case?
Hi @lanhnguyen , and welcome to the Ansible forum!
Thanks for asking this question, because I’ve recently started to question how we’ve been arranging our projects for the last 7 years. It’s been working for us, but maybe that’s because we started with some fundamentally wrong assumptions. Anyway, ignoring what’s weird about our practices, here’s the “normal” part of how we’ve arranged our projects.
We’re responsible for about two dozen “service lines”, and for each of those we have one Ansible “project”. And by “project” I mean a git repo with a corresponding “SCM project” (that word is over-used!) in our own GitLab instance, as well as one (usually one, but sometimes more, and sometimes none) “AWX project(s)” (see?) in our AWX instance. These “service line projects” (git repositories) typically have a dozen or so Ansible playbooks at the top, as well as a “roles” directory that contains roles specific to that service line. We rarely have any task files, templates, plugins, etc. at the project level; such assets are almost always part of a role. Inside the “role” directory we also have one of the two “requirements.yml” files; the other one is in the top-level “collections” directory, which also contains “ansible_collections/mw/<local_collection_names>/”. The latter contains any local, project-specific collections; we’re unlikely to create any new local roles outside of these local collections.
Besides these “service line” projects, we also have developed a handful of what we call “mw_common_*” roles. These include roles to install/configure:
- locally-built packages,
- apache web servers to our group standards and expectations,
- tomcat / wildfly services
- log rotation
- monitoring
These are all listed as required in each service-line project’s “roles/requirements.yml” file. Similarly, collections that aren’t local to a service-line project are listed in the “collections/requirements.yml” file.
Production jobs for all this happen in our AWX instance. But development/testing work can happen either on our primary workstations or on our “bastion” hosts. Typically any of the projects we’re interested in are "git clone"d side-by-side in some work directory of our choosing. We have about 8 worker bees in our “Middleware” group (hence the “mw” popping up in our project names), so we have about 8 different editors in play. (Our “my editor is better than your editor” discussions are always fun!)
Besides these service-line projects and mw_common_* projects, we have a few more. One is a skeleton project that we use as a template for new projects. Another, “defaults,” contains our agreed upon ansible-lint config, some “extremely global” variables that all projects incorporate, and our group-wide static inventory which works for us because we have almost all static hosts. Plus there are a few scripts for housekeeping duties like running ansible-galaxy commands when needed.
Finally, certain operations on our GitLab projects trigger Jenkins jobs via web hooks, like linting, and causing SCM updates in AWX when relevant branches are updated. And that’s about it.
Maybe something in here answers your question, or inspires more questions. I hope so.
2 Likes
So the content in “roles/requirements.yml” and “collections/requirements.yml” are same?
For example, I have project1 and project2 and project3. I need to install httpd, php, mysql for project 1, and install nodejs and mongodb for project 2, and install ngx, php, mariadb for project 3. For your idea, I create each ansible folder for each project, right? In this case how do I run playbook file for each project on the server that ansible is installed or on the local machine?
current I’m using Windows 11 Pro which has ubuntu 20.04.6 LTS APP. and I install ansible into ubuntu 20.04.6 LTS APP. If I create an ansible folder on projects, these projects are on a folder of windows. How do I run playbook files for each project from ubuntu 20.04.6 LTS APP?
No, they are different. Confusion is understandable though, especially if you’re coming to Ansible at this “interesting time.” There are these roles in the roles/ directory, but there are also collections which may or may not contain roles… Why is it split up this way?
Ansible has evolved a lot in the last few years. Before collections became a thing, roles were the only sharable units of work. The roles/requirements.yml file allowed you to pull shared resources into a project in the form of individual roles. That worked okay-ish for simple stand-alone roles, but complex solutions couldn’t be easily stuffed into such a simple box.
For example, suppose a network vendor published a bunch of roles that work together to configure their suite of networking equipment. Most of those roles use some special custom plugins. Which of those roles is the right one to contain one of those plugins? Well, because any customer may only have a small fraction of the vendor’s devices, they may only use the small subset of those roles that they need to configure the devices they own. So each of those roles will have to contain a copy of all the same plugins. Only, they aren’t all the same, because the vendor doesn’t do a great job of keeping them in sync, or the customers only upgrade individual roles when they hit a problem so they have different possibly conflicting versions installed, or whatever. But in the end every customer has to do version management on a set of roles they didn’t write, all because, fundamentally, individual roles provide an inadequate unit for sharing work at scale. What we need is a sharable unit where cross-role redundancies can be eliminated, and cross-role dependencies can be collected, tested, managed, and maintained as a collection.
And thus collections were born. A collection can contain any number of roles - even zero. Unlike individually distributed roles, if the roles in a collection all need some plugin, they can share a single copy. A collection can be tested and distributed as a unit, so users don’t have to track down the right version of every asset. All the pieces of a collection should work together.
-
The old style of distributing roles still works, though. Thus, the older simple shared roles that we wrote years ago and still use are listed in projects’
roles/requirements.ymlfiles. -
Newer work - shared roles and associated plugins - have been put into collections, so they are listed in projects’
collections/requirements.ymlfiles. -
Roles and collections that aren’t distributed or shared among projects, but are instead specific to a project can live just fine in that project’s
roles/andcollections/ansible_collections/directories. They don’t need to be listed in anyrequirements.ymlfiles.
4 Likes
| project1: | httpd | php | mysql |
| project2: | nodejs | mongodb | |
| project3: | ngx | php | mariadb |
There are no hard and fast rules, but in our shop each of project1, project2, and project3 would be its own git repo / Ansible project. Say we do project1 first. We’d end up with three roles: roles/httpd/, roles/php/, and roles/mysql/.
Skip project2 for now and move on to project3. httpd and ngx are both web servers, but not very similar in their configs, so it’s probably more work than it’s worth to abstract away their differences in some common role. So we’ll end up with a role/ngx/. However, both project1 and project3 need php, so we’d probably turn project1’s role/php/ into a shared role (so it becomes its own git repo / Ansible project). Likewise, mysql and mariadb are similar enough, or used to be, that we could probably turn project1’s role/mysql/ into a shared mymariasqldb role (just kidding about that name).
Back to project2, nodejs and mongodb probably end up being their own roles in that project.
Any of these could be turned into shared roles later if another project comes along that needs one of them.
I don’t understand your question about Windows. But it’s 1:30am local, and the last Windows I ran was WinME, so I’m not your best source for that info. Sorry.
3 Likes
I mean I’m using windows os not linux or ubuntu machine.
On project1 folder, I create application folder and ansible folder. application folder contains source code. ansible folder contains ansible files like as host, playbook, roles…
project 2 and and project 3 are same.
these folder project1, project2, project3 are in folder of the windows os. But to run playbook file , I run in ubuntu 20.04.6 LTS APP
how do I run the playbook file in the folder project1 project2 project3?
Hi,
That’s called WSL FYI.
And to answer your question: You should be able to access Windows filesystem from your WSL machine; IIRC, Windows C: drive is mounted on /mnt/c.
Another way to do it would be to use git; put your files in a repo and git clone where you need to work on / use your files. Though having two or more clones lying around on your machine could become untidy.
I suggest you use git to version your files, then clone once wherever you want and work from the same path on both machines (Windows, WSL).
3 Likes
This is exactly what we have, with the roles specifically written so they can be open and shared while the project repos are private, FWIW I have written roles for almost these applications, some more complex and up to date than others, some not very polished but all GPL’d:
- webarch / apache · GitLab
- webarch / mariadb · GitLab
- webarch / mongodb · GitLab
- webarch / nginx · GitLab
- webarch / nodejs · GitLab
- webarch / php · GitLab
And here is a (rather messy!) example repo that uses them to build a dev server:
3 Likes
Thanks for your idea so much.
Do you think this is the good way for managing multiple project with using ansible to deploy?
Project1
application source
.
.
.
ansible
playbook.yaml
hosts
ansible.cfg
roles/
common/
tasks/
main.yml
handlers/
main.yml
templates/
ntp.conf.j2
files/
bar.txt
foo.sh
vars/
main.yml
defaults/
main.yml
meta/
main.yml
Project2
application source
.
.
.
ansible
playbook.yaml
hosts
ansible.cfg
roles/
common/
tasks/
main.yml
handlers/
main.yml
templates/
ntp.conf.j2
files/
bar.txt
foo.sh
vars/
main.yml
defaults/
main.yml
meta/
main.yml
Do you have any good way to arrange structure of ansible for many projects
Is ‘roles’ folder on each project the roles install path or would it effectively be committed to the same repo as your playbook.s, inventor.y.ies and config files ?
Generally (in case you don’t write collections that is), each role has its own repo. Then you install each of them you need (be it with ansible-galaxy using a requirements.yml file or not, direct git clone, or else) to a location of your choosing (either by defining roles install paths in your config or using ansible-galaxy -p parameter) for Ansible to find and use when you run your playbooks.
Then each of your project would only need to have its playbook.s, inventor.y.ies, requirements.yml file.s, config files, …
Here is how I usually do it:
-
One role per repo (same structure as @chris provided examples)
-
Either one repo per project (mostly deployment projects linked to a specific project -as in thing we build-), or one repo containing multiple ‘projects’, mostly for configuration management; here is what I mean:
# Single project repo structure example /my/project/clone/path/ ├── ansible.cfg ├── .git ├── .gitignore ├── .gitlab-ci.yml ├── inventories │ ├── hosts_production │ ├── hosts_staging │ └── host_vars │ ├── srv_production.yml │ └── srv_staging.yml ├── main.yml ├── README.md └── requirements.yml # Multi-project repo structure example /my/multi/project/clone/path/ ├── .git ├── .gitignore ├── .pre-commit-config.yaml ├── ansible.cfg ├── inventories │ ├── group_vars │ │ ├── all.yml │ │ ├── group1.yml │ │ └── group2.yml │ ├── hosts_production │ ├── hosts_staging │ └── host_vars │ ├── host1.yml │ └── host2.yml ├── ssh │ ├── main.yml │ └── requirements.yml ├── nginx │ ├── main.yml │ └── requirements.yml ├── packages │ ├── main.yml │ ├── requirements.yml │ └── update_nodes.yml ├── playbooks_common │ ├── assert_roles_collections_paths.yml │ ├── setup_update_roles_collections.yml │ └── site.yml └──README.md
4 Likes
@ptn We don’t put project’s source code and ansible in same repo?
@ptn Execute me, What we write in requirements.yml file ?
In the case each project we need to clone difference source code from github and run some commands. For example after clone laravel source we need to move to the folder that contains source to run the commands like as composer install, php artisan optimize, php artisan migrate… or another project, we need to clone nodejs source them, after clone we run the command npm install… So Where we push role for this part of each project?
We don’t put project’s source code and ansible in same repo?
Depends. But roles should be in their own repo. You can picture a role as reusable content, like a library, and most of the time, it doesn’t make sense to have this kind of shared / reusable content in the same spot your project resides.
Please also note that ansible-playbook only needs to know where your roles are on your filesystem; it doesn’t care about where it comes from and how it got there.
But if you use ansible-galaxy to install your roles (= creates an archive from your repo, be it your SCM or a Galaxy repo, then unarchive its content on your ansible config defined roles path (overridable)), it won’t be able to if your repo structure is not the expected one, which usually means one role per repo. I think you could technically do some magic in a parent meta/main.yml file to share a repo between roles, but AFAIK, it’s not the intended way to do so; for this usecase, better use a collection.
Now your question is a bit vague; if you are talking about playbooks, inventories, host/group vars or ansible.cfg files, then yes, you could put them next to your project files. It depends on your needs, really. For instance, having a playbook that build or deploy your project in the same repo is totally fine.
And just to be clear: There is no playbook in role, only yaml files defining resources. You can’t use a role by itself, it has to be imported/included from a playbook. Hope it helps to clear things up.
What we write in requirements.yml file ?
The requirements 
. Joke aside, check this out: Galaxy User Guide — Ansible Documentation.
The idea is to list in a declarative way the resources to install. These are the resources (here collections and/or roles) you’ll need for your tasks, like a custom collection or role that is not shipped with ansible package.
2 Likes
That’s a rather broad characterization. It’s certainly possible to have project-specific roles that will never be shared with other projects. We have one mega-project that has 49 local roles, almost none of which even makes sense outside of the context of that project. Splitting it up into internal roles like that gives an organizing framework without which the [goes to count…] 858 files therein (2979 if you include group_vars and host_vars) would be unmanageable.
Also, there’s some overhead with each step of moving assets through
“local files"→"local roles"→"shared roles"→"collections”.
You gain something with each step also, but whether increasing the footprint is worth it is a judgement call every time. (I have a pet argument from my early days of learning Ansible that there should be a way to roll up an entire project into a single .yaml file, so, going off the left edge of the little map up there. The proliferation of files still bothers me, but I digress.)
My point, if I have one, is that there are lots of ways to do it that will work. What’s best is up to whoever has to maintain it. And it’s okay to change your mind about that later if you want to try it a different way.
Otherwise, yeah, @ptn is giving you good stuff to think about.
3 Likes
@chris In users.yaml file , roles is users, where do you define this role?
In requirements.yml:
- name: users
src: https://git.coop/webarch/users.git
version: master # users
scm: git
And ansible.cfg specifies where it’ll be checked out to:
[defaults]
roles_path = galaxy/roles
3 Likes
@chris
I did same you, I defined roles_path in config file.
The I run the command to pull role source from github but it didn’t pull. Do you know why?
I used your git but It also can not pull
It always load config file from /etc/ansible/ansible.cfg. How do we set config file in another path?