Working in an environment where we hope to reuse common playbooks and roles across the organisation, I’ve been thinking a lot on how to manage updates to roles and playbooks without breaking repeatability (running the same playbook against the same environment should have the same result, even if the two runs are separated by months).
and I plan to add some more rules for ansible-lint to allow checking that roles fit the techniques (I’m not sure even whether to publish the rules, but they certainly won’t be core rules as they may well be entirely specific to my environment)
Anyway thoughts are welcome on whether there are better ways to do it! (Particularly if there’s a pure DVCS way that achieves a similar outcome)
So, let’s not go down that lint-discussion road again. We know where it leads.
Rather, let’s once again discuss how we can improve roles to do what we need.
As for role versioning, there have been a few who have liked the things that chef did with their library tool (I haven’t used it), and we’ve posted quite a few times that we’re open to making the ansible-galaxy CLI work better with raw SCM repos as well as versioning deps.
There’s also been the suggestion that ansible have a tag to assert the required ansible version, or perhaps it’s a module.
All of this seems like a good thing to do.
I don’t particularly care for the idea of requiring a version in the role name, as that breaks the ability to cleanly branch the role in Galaxy, which is handled via git tags presently.
I know your thoughts on ansible-lint - that the behaviour should be integrated into core. But my pull request to do something along those lines has been open for 8 months https://github.com/ansible/ansible/pull/5123
ansible-lint provides a way of enforcing locally applicable standards - for example, we use it to warn against using Ansible as a build tool, among other things.
Anyway, to focus on the main points you make.
My understanding is that ansible galaxy role versions matter at install time. I guess that means that each playbook installs the roles locally. This means that the problem of using one role version for uat and one role version for prod is not necessarily solved is it?
I assume there are existing discussions on using repos other than github - I’ll have to look into this. I’m definitely keen to avoid reinventing any wheels if I can.
To be clear, I’m not particularly concerned about the versioning of Ansible itself, just that the same playbook can reference different roles.
I know your thoughts on ansible-lint - that the behaviour should be
integrated into core. But my pull request to do something along those lines
has been open for 8 months https://github.com/ansible/ansible/pull/5123
I think it's no-secret that we've had a large queue lately.
A lot of our prioritzation has been around hardening of core functionality,
and also keeping with EC2 and other hoping things moving. This is still
interesting to me just not something our heuristics have bubbled to the top.
Part of the problem is having absolutely ridiculous levels of contribution
at this point, which we are thankful for.
Anyway, to focus on the main points you make.
My understanding is that ansible galaxy role versions matter at install
time. I guess that means that each playbook installs the roles locally.
This means that the problem of using one role version for uat and one role
version for prod is not necessarily solved is it?
playbooks don't actually install the roles, but ansible-galaxy CLI calls do.
In the ansible-galaxy requirements file you can in fact say
username.rolename,v1.00
username.rolename2,v1.05
And if you had them so configured to check out in locations (ideally using
a different role path for each) that would be easy to do. You might also
have an ansible.cfg that selects this rolepath and also adds it to
.gitignore
One of the things missing though is the ability to (right now) specify
git:// and ssh:// git locations so it doesn't just have to download from
Ansible galaxy proper.
I think we're also open to new formats for the requirements file if needed,
as long as it can continue to support the old format.
I think I can make good use of this - I’ll have to rethink our workflow but the final result should be better and meet my requirements. Hopefully I’ll have a new blog post and I’ll mark the old one as deprecated.
I would love to have this functionality added to ansible-galaxy. I’m currently using librarian-ansible because it allows me to specify my own git repos for sourcing roles. I don’t care for how heavy-handed librarian-ansible is, though: each time I run any librarian-ansible command it wipes out all changes in the librarian_roles directory, causing all my customizations to be lost. I much prefer the behavior of ansible-galaxy but can’t use it because I host my roles on a private git repository.
One thing I do like very much about librarian-ansible is the syntax/capability of the Ansiblefile:
`
This will pull a role using the Ansible Galaxy API
role “kunik.deploy-upstart-scripts”
This allows me to specify my own git repository as a source as well as a version number
role “disa-stig-rhel6”, git: “git@git.acme.com:ansible/role-disa-stig-rhel6.git”, “1.0”
I can also point to github repos
role "role “pgolm.ansible-playbook-monit”, github: “pgolm/ansible-playbook-monit”
`
There are philosophical differences in how the two tools operate as well. With librarian-ansible, you are not meant to ever customize roles under the control of librarian-ansible. This makes keeping the roles up to date rather easy but customizing them is not recommended since any changes will be lost (unless you copy the role out of the librarian_roles folder, essentially forking the role).
ansible-galaxy, on the other hand, lends itself more to using shared roles as a starting point and not as an ongoing source of truth. I could be using ansible-galaxy wrong, but that’s how I see it (or maybe it’s flexible enough to be used either way). I run ansible-galaxy once to get the role, then customize it to my liking. Running ansible-galaxy again does not overwrite any customizations I have made to the role, which is much friendlier and more in line with how I like to use the tool.
Good intentions aside, that librarian project should not needed to be created - there’s somewhat of a tragedy of the commons effect from github where not everyone joins the lists and knows about the work we are all trying to build here, coupled with a bit of a simple name problem that led people to assume ansible-galaxy was only for galaxy.ansible.com, when that wasn’t the case - it’s really the role tool for the whole “galaxy” of ansible modules, or whatever.
Ansible’s always been a “batteries included” kind of ecosystem, where we want everybody to collaborate on shared tooling, rather than have 5 ecosystem projects spring up that all work differently, and lack the critical mass.
It’s the same reason we have the 235-ish modules in core, rather than the 40-60 of some other projects - so you have to hunt less around distributed projects and find that some do some things and others do other things.
I particularly don’t like how that tool creates another syntax for describing things, worse yet one that looks like Ruby in a system of software where that doesn’t /quite/ make sense, but it also didn’t quite make sense that a simple list of roles needed to be in YAML just yet either.
So it may be in the future we support both a YAML format and the existing string format, TBD.
In any event, I regret that no list discussion occured when that tool came onto the scene, as I think a lot of those contributions could have been funneled into the shared project.
That seems reasonable, especially if skipping hg/git if not installed, though I’m guessing that’s maybe not as easy for ssh:// git?
Just trying to think aloud whether specifying the protocol is a good idea. I’m thinking it might be, and it would save some network activity on long resource lists as well.
I agree with you completely. I don’t particularly like using librarian-ansible but it’s all I have at the moment. I really just want to be able to specify custom sources using ansible-galaxy.
The syntax is indeed Ruby because librarian-ansible is written in Ruby. What I was trying to explain, which I don’t think I did very well, was that the ability to be explicit rather than implicit in specifying role sources is what I like best about the configuration file. I have no affection of the exact syntax used.
Will’s PR uses logic in the code to determine the type of remote server: I just prefer to be more explicit rather than relying on the program to figure it out. But that’s my preference and could easily be argued either way.
“The syntax is indeed Ruby because librarian-ansible is written in Ruby.”
Yeah, that didn’t make sense to me at all either.
I’m not sure what features people need out of the tool, but it would be easier for us to talk about what tools ansible-galaxy needs here, rather than referencing the tools i n the Chef ecosystem I’m unfamiliar with.
(Also see previous comment on being explicit, I think that’s what we should do…but I like where this is going)
We can keep simple syntax for now and move on from there if needed (and if so, supporting both as best we can). Might not be needed.
Our initial goal was to make something more or less like a pip requirements file, something pretty basic.
This works with the ansible-galaxy in the pull request but would not as it stands without some form of scm detection.
On reflection, I think I’d be happiest with the scm+url suggestion - this would eliminate the need for scm detection and keep the role_name/url, role_version format of the rolesfile
role_name would continue to be derived from the repo name.
From Sam’s example, this would then look more like this (not 100% happy with git+git but it’s nicer than handling the special case).
# Custom roles using various protocols git+ssh://git@git.acme.com:ansible/role-disa-stig-rhel6.git,1.0 git+https://git.acme.com/ansible/role-kibana.git git+git://git@git.acme.com:ansible/role-logstash.git
This would end up with roles called e.g. role-logstash, which might not be what you want, but I would prefer to keep the rolesfile simple.
I am currently investigating different options for managing roles and looking forward to having this in Ansible itself. My requirements are similar - having roles shared (and versioned) in private git repos, referenced from playbooks in other repos.
The solution you are discussing here sounds like in the right direction. I would just add one more requirement - being able to overwrite the role-name and rely on the git repo name exclusively. Idea for syntax if you want to keep the role_name/url, role_version format:
git+https://git.acme.com/ansible/role-logstash.git#alias=logstash
This would reference the role from the private repo but use the name ‘logstash’ instead of ‘role-logstash’.
Also, what is the intended mechanism for pinning versions in dependencies in meta/main.yml?
dependencies:
git+http://git.example.com/repos/role-abc,v1.0
There’s probably some work to trim the .git off the role_name too (I just don’t bother putting the .git in the repo url but maybe not all git servers cope with that)
I’ve solved the trimming the .git off the end and also the ability to specify role_version in meta/main.yml
It doesn’t work with an upgrade of a dependency because the check is currently for if the role exists, not if it’s a particular version.
I’ve added an integration test for all of this too - so I know ansible-galaxy and ansible-playbook work with the changes against SCM roles but not if the changes break existing roles inside or outside of galaxy - be good to get some additional testing around this.
Yeah I think the above syntax, and also in dependencies, would be fine.
Dependency downloads are all in the ansible-galaxy CLI, so I think that would cover everything. We would just need the core change that would get the role name out correctly too.
" would just add one more requirement - being able to overwrite the role-name and rely on the git repo name exclusively. Idea for syntax if you want to keep the role_name/url, role_version format:"
While interesting, we don’t want to do this because we already have “,version” used for describing roles in a way that is not specific. Further, I don’t think those URLs technically support comments in the case of ssh://, so that seems like something we’d want to avoid.