Ansigenome: A tool to help you gather information and manage your roles

Archives Ansible Project
1

What do you guys think about https://github.com/nickjj/ansigenome. I released it yesterday and would appreciate some feedback.

Here’s a snippet from the readme:

Ansigenome is a command line tool designed to help you manage your Ansible roles. It does 6 things:

**scan** a path containing Ansible roles and report back useful stats
**rebuild** a path of roles which auto generates a `README.md` and `meta/main.yml` from templates
**run** shell commands inside of each role's directory
**init** new roles with a travis-ci test already made for you
**export** a path of roles to a file to be consumed by `ansible-galaxy install -r`
**dump** a json file containing every stat it gathers from a directory

Also do you think something like this belongs as a core tool in ansible? I imagine the json dump could be useful for generating stats on the galaxy. I also feel like it’s a good idea to push people into best practices for releasing ansible roles. It would be great if we can all agree on some standard and then stick to it. I tried to help with this situation by adding automatic travis tests to each new role you generate with Ansigenome. It also exposes a git branch in the meta file to hint that you should really be tagging your role versions.

2

No offense intended, I saw this on twitter yesterday, and honestly, I didn’t get it.

First off, ansible-galaxy is already a command line tool help manage (create, stub out, and download ansible roles). ansible-galaxy is already open source as a command line tool, and it’s not galaxy specific (just named as such). We do welcome contributions, but would like to talk about syntax and usage first a bit. Some of the things we’ve talked about was supporting private git repos. Contributions have been slight, however.

Things like “run a shell command inside each role directory” seem really abstract to me, and I had difficulty following the documentation to understand why I would want to do the above things listed on the PyPi page.

I don’t know what “rebuild a role” would mean. Roles don’t have to be built.

Travis integration doesn’t seem to make sense because it’s a very limited environment for running integration tests.

So, yeah, I’d rather have seen this as a discussion for what facilities are lacking in the /usr/bin/ansible-galaxy tool, and perhaps one of those might have been a “–list-roles” subcommand, that could use something like your stats output.

I’m still not understanding things like “run” or “rebuild”.

3

Thanks for the feedback, I’ll try to address some of your concerns:

Re: “run”:
The stemmed from the ginas project. It’s currently 50 roles inside of a single git repo. The author of ginas recently decided he’s going to try and break up each role into stand alone git repos and release whatever roles he can on the galaxy. Now… he is a unix beast and could write a script to walk each directory, create a new git repo, add a remote origin to it, etc… but it’s annoying to write all of that stuff. Wouldn’t it be cool to just type:

ansigenome run -m 'git init && git remote add origin https://github.com/*yourname*/%role_name.git

That’s all you would need to write to make 50 git dirs and add a custom remote to each one. %role_name gets replaced with the current role in the loop.

Re: “rebuild”:
Also he wanted to make a readme standard for all his roles. He wanted an easy way to adjust a readme template for all his roles at once. By providing just a tiny bit of information in your meta file it will create an interesting readme file for you based on the stats it found by inspecting your role.

Since last night I’ve added a bit more documentation on rebuilding:
https://github.com/nickjj/ansigenome#example-rebuild-output

You can see an example output there and how you would use it.

Re: travis:
There’s a lot of low quality roles on the galaxy that aren’t idempotent. The init command of ansigenome uses the standard ansible-galaxy init structure but it adds a travis test that runs your role twice and checks to make sure there are 0 changes. That’s pretty useful to have for free. If you’re not a unix wizard (I’m not) and fairly new to ansible then you might not be sure how to test ansible roles either. I just wanted to give people a stepping stone.

4

Thanks for the feedback, I'll try to address some of your concerns:

*Re: "run":*
The stemmed from the ginas project. It's currently 50 roles inside of a
single git repo. The author of ginas recently decided he's going to try and
break up each role into stand alone git repos and release whatever roles he
can on the galaxy. Now... he is a unix beast and could write a script to
walk each directory, create a new git repo, add a remote origin to it,
etc.. but it's annoying to write all of that stuff. Wouldn't it be cool to
just type:

`ansigenome run -m 'git init && git remote add origin https://github.com/

*yourname*/%role_name.git`

I'd probably go "find . | xargs" myself, maybe bash alias it.

For most internal projects, keeping one role per repo is really
complicated. It works great for galaxy because it allows for seperate
versions and forking, and easier pull requests.

Also I may point out it appears you are using a new templating system :slight_smile:

That's all you would need to write to make 50 git dirs and add a custom
remote to each one. %role_name gets replaced with the current role in the
loop.

*Re: "rebuild":*
Also he wanted to make a readme standard for all his roles. He wanted an
easy way to adjust a readme template for all his roles at once. By
providing just a tiny bit of information in your meta file it will create
an interesting readme file for you based on the stats it found by
inspecting your role.

This seems a bit specialized at first glance. It is true that
ansible-galaxy init does instantiate a readme template for a role,
contributions to make that configurable would be welcome, I'm not sure
"stats based" are the ideal way to go about it.

Since last night I've added a bit more documentation on rebuilding:
https://github.com/nickjj/ansigenome#example-rebuild-output

You can see an example output there and how you would use it.

*Re: travis:*There's a lot of low quality roles on the galaxy that aren't
idempotent. The init command of ansigenome uses the standard ansible-galaxy
init structure but it adds a travis test that runs your role twice and
checks to make sure there are 0 changes. That's pretty useful to have for
free. If you're not a unix wizard (I'm not) and fairly new to ansible then
you might not be sure how to test ansible roles either. I just wanted to
give people a stepping stone.

What about roles that require authentication to external sources, or spin
up new cloud instances?

This is why I'm saying public integration tests is not feasible for many
roles.

See our testing guide at http://docs.ansible.com/test_strategies.html for
some pointers on testing in general. I don't think people would need to
adopt an extra ecosystem tool in order to test things.

---

I tried to stick to the ansible-style way of making command line tools so
that maybe merging certain things into ansible-galaxy would be easier. What
was wrong with the syntax btw? I went for a stock pep8 setup.

Which syntax?

Thanks again for ansible Michael.

You are welcome. I don't mean to be harsh above, but I do question when
"ecosystem" tools instantly show up in PyPi without some attempt to discuss
them first, or an attempt to build a stronger core.

We were built on "batteries included" to a great degree. I agree we move
slower now as we must be more conservative about making sure things are
simple and robust than two years ago, and because we have thousands of
people trying to throw code at Ansible all at once. But I generally don't
believe recreating the problems, of say, other ecosystems with
metaphor-filled tool names of varying utility, are ultimately good in the
long run, because someone feels they need to use everything.

There's a desire for minimalism, and making sure everything fits in.
Right down to making sure we use the same ways of replacing variables.

I don't mean to be a control freak - this may very well serve a purpose,
but I think the needs of it are not the needs of many, per se, because most
large internal projects will have roles in all in one repo, and in many
cases, Travis doesn't apply, etc.

So some decisions made are very specific to a few very specific use cases.

That is not to say I think it should not exist. I'm just questioning how
to make something more generally accessible to everyone, and also how to
continue to evolve those tools around a hub everybody can collaborate
around.

Hope this makes sense.

5

Wanted to address this, BTW. This is not /always/ the case.

Ansible is also designed to model a wide variety of actions. "reboot" is
a valid role. "restart_caching_layer" could be a valid role.
"rotate_ssh_keys" could be one as well, as could even
"erase_disks_to_government_standards".

Not all operations in ansible have to be idempotent. We achieve this
easily by having modules that (usually, in their most commonly used states)
have declarative properties.

But one of the great powers of ansible is the ability to break out of that
model, and use it for more than config management, and for things that
actually are also stepwise.

Galaxy is naturally a commons, and content there can be of varying nature,
but that's why the review system is in place. I'd recommend people use
this to comment on roles they like, as this helps folks find better content.

Ansible's niceness is that you can be declarative when you want, but it
also lets you be imperative when you want. (As such, with great power
comes great responsibility... except when you are doing "cloud like cloud"
and are just throwing images/instances away on upgrade, in which case, not
so much).

6

Also I may point out it appears you are using a new templating system :slight_smile:

I’m not. I’m using jinja2 and the tool itself has no extra dependencies other than what ansible requires. The %foo syntax is only used on the command line so you as the scripter have access to the current role name in your script. It can’t just read the folder name because it might be prefixed with your git username to conform to the galaxy’s way of naming roles.

In the above example you might have your role at yourname/ansible-myrole on github but it’s “yourname.myrole” on the galaxy. %role_name returns just “myrole” as a convenience.

It is true that ansible-galaxy init does instantiate a readme template for a role, contributions to make that configurable would be welcome, I’m not sure “stats based” are the ideal way to go about it.

I think there’s a misunderstanding about what the rebuild command does because you looked at the docs before I had documentation written about it. It’s not stats based really.

Look, the default ansible-galaxy readme template expects you to enter:

  1. your default variables

This means you have your defaults listed in both your meta file AND your readme and it’s annoying to keep both in sync by copy/pasting. Ansigenome solves this by just reading in your defaults from the meta file and dumps it under a defaults header in the readme. You don’t have to configure anything to get this functionality but optionally you can overwrite the defaults output in the readme by changing your meta file. It also doesn’t clutter your meta file with the automated readme contents either.

  1. a list of dependencies

Yikes, here’s another thing I need to copy/paste from my meta file without Ansigeome. Ansigenome solves this by just looking at your dependencies list in the meta file and outputs them cleanly in the readme without configuration.

  1. example playbook

Ansigenome addresses this in a different way. By default it doesn’t output anything like this to the readme since it’s very custom and impossible to guess from just looking at your role. So instead all you have to do is goto the meta file and fill out the “inventory” field and it’ll automatically create a quick start guide in your readme listing out whatever you wrote there.

  1. requirements

Ansignome takes a peek at your platforms in the meta file and automatically builds a list of platforms for you in the readme. You don’t need to copy/paste them into your readme anymore.

You guys clearly thought some of those things were annoying so you built that into the galaxy role page, like the platforms, etc… The problem is I don’t want to have to look at a git repo AND the galaxy to figure out what a role does or needs to work.

As a role user looking to use a public role I want to see:

  1. What defaults are available?
  2. What facts are set?
  3. What platforms does it work on?
  4. What dependencies do I need?
  5. Is it idempotent?
  6. Is it well tested and in working condition?

As an aside I also don’t want to have to look at both a galaxy profile and a git repo. I want a single point of truth that’s close to home (git repo).

As an open source roler build I want to:

  1. Not have to copy/paste a ton of stuff into a readme file.
  2. Easily be able to create unique content in my readme when needed.
  3. Have as much documentation written automatically as possible.

Ansigenome addresses all of the above issues and more with the rebuild/init commands and does it in a way that requires no configuration at all from the user for both role consumers and authors.

What about roles that require authentication to external sources, or spin up new cloud instances? This is why I’m saying public integration tests is not feasible for many roles.

Ok, if you don’t want an idempotency test added by default then just delete the .travis.yaml file in the role’s directory. It also won’t even run unless you specifically login to travis and turn the repo on. Having tests like this is a benefit to the community IMO and adding your own tests that ignore idempotency is super easy and standard to do. It shows how to use the role since you need to set it up on travis. It acts as fool proof documentation.

because most large internal projects will have roles in all in one repo, and in many cases, Travis doesn’t apply, etc.

Then just don’t use travis then, problem solved heh? Also you can have roles in 1 repo and benefit from Ansignome in the same way as having separate roles. It gathers stats on all roles it finds in a directory you give it. It doesn’t care about any git directories. I used that run example of adding git directories as an example of the type of things you can do with the run commands. By no means do you need git dirs for each role. I want to make that super clear.

7

Oops, in that last reply I meant you have your defaults listed in both your defaults/main.yml file and the readme. It also reads it from the defaults/main.yml file to generate the readme, or if you have something in the “defaults” field in the meta file it uses that instead.

8

Also I may point out it appears you are using a new templating system :slight_smile:

I'm not. I'm using jinja2 and the tool itself has no extra dependencies
other than what ansible requires. The %foo syntax is only used on the
command line so you as the scripter have access to the current role name in
your script. It can't just read the folder name because it might be
prefixed with your git username to conform to the galaxy's way of naming
roles.

Yep, I was referring to "%foo"

In the above example you might have your role at yourname/ansible-myrole
on github but it's "yourname.myrole" on the galaxy. %role_name returns just
"myrole" as a convenience.

It is true that ansible-galaxy init does instantiate a readme template for

a role, contributions to make that configurable would be welcome, I'm not
sure "stats based" are the ideal way to go about it.

I think there's a misunderstanding about what the rebuild command does
because you looked at the docs before I had documentation written about it.
It's not stats based really.

Look, the default ansible-galaxy readme template expects you to enter:

*1. your default variables*

This means you have your defaults listed in both your meta file AND your
readme and it's annoying to keep both in sync by copy/pasting. Ansigenome
solves this by just reading in your defaults from the meta file and dumps
it under a defaults header in the readme. You don't have to configure
anything to get this functionality but optionally you can overwrite the
defaults output in the readme by changing your meta file. It also doesn't
clutter your meta file with the automated readme contents either.

With regard to the default variable entry (which is so far unused in
galaxy, and can be left blank), it seems it would be better to just improve
the ansible-galaxy CLI in this place so the usage of two tools together was
not required.

*2. a list of dependencies*

Yikes, here's another thing I need to copy/paste from my meta file without
Ansigeome. Ansigenome solves this by just looking at your dependencies list
in the meta file and outputs them cleanly in the readme without
configuration.

The galaxy dependency field here is free form, and can include things like
packages that are required. It is not role specific.

*3. example playbook*

Ansigenome addresses this in a different way. By default it doesn't output
anything like this to the readme since it's very custom and impossible to
guess from just looking at your role. So instead all you have to do is goto
the meta file and fill out the "inventory" field and it'll automatically
create a quick start guide in your readme listing out whatever you wrote
there.

Unclear of how inventory and example playbooks are related.

*4. requirements*

Ansignome takes a peek at your platforms in the meta file and
automatically builds a list of platforms for you in the readme. You don't
need to copy/paste them into your readme anymore.

You guys clearly thought some of those things were annoying so you built
that into the galaxy role page, like the platforms, etc.. The problem is I
don't want to have to look at a git repo AND the galaxy to figure out what
a role does or needs to work.

I'd like to see an attempt at not bifurcating the tooling here.

*As a role user looking to use a public role I want to see:*

1. What defaults are available?
2. What facts are set?
3. What platforms does it work on?
4. What dependencies do I need?

So far the webpage handles all of this.

5. Is it idempotent?

This is not something a machine can really test. It is something a
machine can approximate.

6. Is it well tested and in working condition?

Even if you have a green light or not, it doesn't indicate the quality of
said tests.

Galaxy is in many cases just a starting point and set of examples, it's
fully expected that people will need to modify roles from time to time to
make use of them.

What about roles that require authentication to external sources, or spin

up new cloud instances? This is why I'm saying public integration tests is
not feasible for many roles.

Ok, if you don't want an idempotency test added by default then just
delete the .travis.yaml file in the role's directory. It also won't even
run unless you specifically login to travis and turn the repo on. Having
tests like this is a benefit to the community IMO and adding your own tests
that ignore idempotency is super easy and standard to do. It shows how to
use the role since you need to set it up on travis. It acts as fool proof
documentation.

To be fair, tests aren't documentation either.

9

Unclear of how inventory and example playbooks are related.

In a lot of cases the playbook itself is always the same except for the name of the role. I have around 20 roles on the galaxy and pretty much all of them have an identical playbook in the example.

The thing that changes is overwriting or defining a few variables in your inventory. Like “here’s the bare minimum to get rolling with the role”, rather than just show someone 50 default variables up front. That is where the inventory name comes into play. I consider group_vars/etc. as part of my inventory.

I’d like to see an attempt at not bifurcating the tooling here.

I wanted to merge some of this into core if you guys felt it was reasonable but the ansible-galaxy script is untouchable IMO. It is already massive with a bunch of mixed concerns. I don’t think it can sustain me dropping in 1k+ lines of python without it exploding and to keep it all broken up would have required coming up with my own directory structure which is fine for my project but I wasn’t sure that would fly over well with ansible.

What parts do you like and don’t like?
How much of it wouldn’t ever get into core?
Would ansible-galaxy need to be renamed?

10

Unclear of how inventory and example playbooks are related.

In a lot of cases the playbook itself is always the same except for the
name of the role. I have around 20 roles on the galaxy and pretty much all
of them have an identical playbook in the example.

This is probably a sign they might need better examples :slight_smile:

I'd like to see an attempt at not bifurcating the tooling here.

I wanted to merge some of this into core if you guys felt it was
reasonable but the ansible-galaxy script is untouchable IMO. It is already
massive with a bunch of mixed concerns.

Strongly disagree with this statement. It's quite easy to add new
functions, and it doesn't matter what functions or files things are in.
It's also easy enough to make a galaxy_utils/ subpackage in there too.

I

I don't think it can sustain me dropping in 1k+ lines of python without it
exploding and to keep it all broken up would have required coming up with
my own directory structure which is fine for my project but I wasn't sure
that would fly over well with ansible.

What parts do you like and don't like?
How much of it wouldn't ever get into core?

We are going to have editorial views about what Ansible should be. All
being said, we do appreicate and welcome attempts to contribute and work
towards good combined tooling. Your cases seem a bit niche for the
mainstream, and that's fine, you can absolutely keep using it - but we seem
to be at an impasse and should call it as such, I think.

Would ansible-galaxy need to be renamed?

It has no reason to be.

11

I try to set sane defaults. Almost all of my playbooks look like:

`

  • name: Do something
    hosts: some_group
    sudo: true

roles:

  • { role: my.role, tags: }

`

I have about 15 roles that work with that identical setup minus the name. What changes between them are a few inventory variables.

So I guess you’re saying it’s not worth merging anything then?

12

Perhaps some of the scan/report stuff might be interesting.

I’m more interesting in seeing support for non-github git repos (git:// and ssh://) and a possibly some refinements in versioning role downloads (some people ask about multiple versions of roles and workflow, and things like cook-librarian type features) a bit first.