Better galaxy integration with ansible-playbook

Archives Ansible Developer
1

I would like to suggest some improvements to ansible-galaxy and ansible-playbook that make the latter less dangerous.

My particular concern is where someone has checked out the latest version of a playbook, but then forgets to run ansible-galaxy to update their roles.

Your first thought might be “This is terrible! I should improve the documentation and educate the user to use the tools properly”. Your second thought should then be “Hang on, this is an automation tool that I use to avoid writing documentation other than ‘run this playbook against these hosts’ - why do I need to document setting it up?”

So, on that basis, these are the improvements I see as necessary.

  • ansible-galaxy should be more idempotent (i.e. no attempt is made to install a role that is already installed) https://github.com/ansible/ansible/pull/12904
  • ansible-galaxy should work with tree-ish references (not just tags and commits) https://github.com/ansible/ansible/pull/13203
  • ansible-playbook should install roles needed for a playbook. If ansible-galaxy has already been run, this would be a no-op if ansible-galaxy is idempotent. I can see that this might need more playbook parameters (e.g. rolesfile and rolesdir) but could default to playbook_dir/roles, playbook_dir/rolesfile.yml and be overrideable in ansible.cfg. I suspect more code will need to be moved from bin/ansible-galaxy and lib/ansible/cli/galaxy to lib/ansible/galaxy/roles

Now that Ansible is less of a moving target, I could attempt to work on the above, but only if there is any likelihood of the work being accepted.

Will

2

This is a feature i was planing, but with one other requirement: being
able to whitelist where you install from, otherwise you can wind up
executing anything.

3

Fair enough - seems like a sensible precaution.

Let me know if there’s anything I can do to help with this.

Will

4

Well, there is another feature that would make it a lot easier to deal
with the rest, specially with multiple role versions, being able to
install a role with it's versioned name and symlink the 'lastest' to
the 'common name':

roles/
   briancoca.oracle_java7.v1.0
   briancoca.oracle_java7.v2.2
   briancoca.oracle_java7.qs3ih6x
   briancoca.oracle_java7 => briancoca.oracle_java7.qs3ih6x

Or something similar, so now they can be referenced by the generic
name or to a specific version, all living together and not disrupting
the different plays. Hopefully being able to move a 'common name'
existing role out of the way into it's own 'versioned name' if
possible.

5

Well, there is another feature that would make it a lot easier to deal
with the rest, specially with multiple role versions, being able to
install a role with it’s versioned name and symlink the ‘lastest’ to
the ‘common name’:

Having multiple versions of a role in a roles directory would make this suggestion necessary, not just easier to deal with - otherwise for a given playbook, there would be no way of knowing what version of a role to use, unless you’re going to move version pinning from the rolesfile to playbook.

roles/
briancoca.oracle_java7.v1.0
briancoca.oracle_java7.v2.2
briancoca.oracle_java7.qs3ih6x
briancoca.oracle_java7 => briancoca.oracle_java7.qs3ih6x

Or something similar, so now they can be referenced by the generic
name or to a specific version, all living together and not disrupting
the different plays. Hopefully being able to move a ‘common name’
existing role out of the way into it’s own ‘versioned name’ if
possible.

This is very different to how we do things, where we give each application/environment combination its own rolesfile and roles directory, and have deprecated the use of meta/main.yml for role dependencies to avoid diamond dependency version conflicts.

Perhaps if ansible-galaxy (the tool) and rolesfiles went away entirely, the integration with Tower might be a little easier (I think the problem is having different versions of a role for different environments for the same lifecycle)

Will

6

That is one way to work around the issue. I propose this because it is
something we get asked for a lot, as people seem to want to keep all
there roles in the same place, across different playbooks and
environments. As much as I agree that the problem is on the role
versioning side, this happens often enough in many environments.

I hate to have ansible-galaxy become a 'package manager' but that is
mostly what it is, I don't see how it can 'go away' and still keep all
the current functionality. The roles files should be easier to do away
with, it would require changing how roles are referenced and a
transition period in which we support all the forms.

7

I quite like having a separate roles file as it does separate versioning from the playbook.

However, the mismatched versioning problem doesn’t really go away with your proposal

Suppose we have
roleA, v1: defaults/main.yml: unsafe_behaviour: True
roleA, v2: defaults/main.yml: unsafe_behaviour: False

roleB, meta/main.yml includes roleA v1

playbook includes both roleA and roleB. The rolefile specifies roleA v2

What should happen in that scenario, what should the value of unsafe_behaviour be?

This is based on our real world experience. My patch for ansible-galaxy does warn when different versions of the same role attempt to be installed.

Will

8

Hi,

9

I would shy away from treating upstream roles and personal roles that differently.

We just version all of our roles so that development of roles by one application doesn’t accidentally (or purposefully, for that matter - we do introduce breaking changes) break roles depended upon by another application.

We have used symlinks to mimic this (which is a very low-tech fragile approach, as the symlink gets blatted when rerunning ansible-galaxy install), but I wonder if

src: /path/to/development/roledirectory

would work (in the same way that I point roles to HEAD of my fork when developing roles and then repoint to a tagged version of the role from our shared repository when finished)

10

One thing I would add to allow role development to still work would be to test if the target checked out role directory is a symlink and leave it alone if it is.

I’ve been using a environment specific tasks called from the other playbook to achieve something similar, using galaxy role to store env specific variables:

- name: check if project secret repo is a link
stat: path={{ playbook_dir }}/roles/env-stage
register: env_stage_checkout
always_run: true
run_once: true
become: false
delegate_to: localhost
tags:
- always

- name: update secrets checkout
command: "/usr/bin/ansible-galaxy install -f -p {{ playbook_dir }}/roles git+ssh://git@my.git.lab/shared-access/env-stage.git"
when: not env_stage_checkout.stat.islnk
changed_when: false
always_run: true
run_once: true
become: false
delegate_to: localhost
tags:
- always
11

Just a reminder that Will contributed a proposal for improving role installation. Brian and I added some thoughts as well. Would love to get feedback and begin nailing down a solution. The target for this is the 2.2 release.

–Chris
@chouseknecht

12

Thanks for this reminder, I kind of forgot about it.

Just a reminder that Will contributed a proposal for improving role
installation. Brian and I added some thoughts as well. Would love to get
feedback and begin nailing down a solution. The target for this is the 2.2
release.

I have one point of discussion I'd like to add in this loop. I'm not
sure if this is possibly covered by the current proposal.

I have often encountered use case where I'd like to use a different
role, depending on the instance of the application.

Say I provide services to manage Gitlab installations for several
customers. Each customer has it's specific version of Gitlab that is
deployed, with a specific version of the Gitlab role. When a new
Gitlab version comes out, there is the extra possibility that the role
needs an update. Now, I do not want to suddenly use that updated role
for all my customers. I need to be able test the deploy/upgrade of a
particular Gitlab version, with a particular Gitlab role, for a
specific customer, e.g. in a testing environment, and only after
acceptance of that, go forth in production.

Possibly, this might mean I need to keep track of the needed version
of the role in the inventory, for a particular host/environment.
As a role is targetted to a particular group in a playbook, I'd like
also to avoid having to manage lots of groups for different playbooks
that use those different role versions.

This could possibly mean, having dynamic role inclusion based on a
role defined in the playbook, and it's version managed as a per host
variable.

Thoughts?

Serge

13

I prefer Option 3 -- but it starts to make me wonder if we're
conflating "roles" and "galaxy" too much. It's always been a bit
tricky to distinguish between "galaxy server" and "galaxy client" in
discussion, and this will blur the lines even more.

If we're bringing more role management logic into ansible itself, then
maybe the galaxy client should be strictly for galaxy contributors to
interact with galaxy itself. Because this change will make
galaxy-client functionally invisible to people who don't intend to
upload roles to galaxy, right?

ansible-galaxy has the following functions that are strictly server side:

delete/import/login/setup, which are for galaxy-server contributors
search, which is for finding content on galaxy-server

But then galaxy also has local role functions, which don't actually
pertain to galaxy-server at all:
init, which is actually a way to stub out a role locally
list, which lists locally installed roles
remove, which removes a role locally (different from delete, which
deletes a role from galaxy itself)

If we're moving to auto-install of roles in ansible, I wonder if these
"local role" based functions should also be moved into ansible.

Just my $0.02.

--g

14

I would keep ansible-galaxy as the package manager, just because it is explicit. The autodownload makes sense from an execution standpoint in ansible-playbook, but not the rest of the management.

15

Hi,

I mostly like option 3. But I would prefer to not keep all versions of a given role in my role path. I would rather see ansible-galaxy download roles in to ~/.ansible/galaxy and then just runtime deploy given version of roles into role path.