Changing the way we organize the module repos

Archives Ansible Developer
1

Hi all!

As ansible has gotten super big lately, and also because people like virtualenv, it’s time to change the way ansible modules are organized and packaged.

The proposal we have is to keep them in lib/ansible/modules/{core,extras,user}/ and use the python subsystem to find this module path. --module-path will continue to work.

For the project, core and extras will be git submodules, requiring a recursive git checkout to fetch them.

Most existing ansible modules will go to core, others will go to extras - with the docs denoting which are which - the main thing is core modules will have slightly higher ticket priority. (The main application, ansible/ansible, will have the highest).

As such, this may eventually allow lieutenant level merging on extras, which occasional updates to core to merge in submodules.

The individual docs pages for core/extras would link to which repo to file bugs on, and a link to the code for the module in question.

In implementing this, we intend to make a short term feature flag USE_NEW_STYLE_LOADER for about a week, so we can make these changes in the open, and replace them all by working in the upstream. Nobody should notice anything but existing users MAY need to make a checkout of the git submodules in the worst case.

This will require some minor changes to core and the documentation formatting system, and when we are done we should have two more github repos in the ansible/ github organization.

The release cycle, as well as packaging, will be the same – ansible will be one pip package, not 3, and there will be one RPM/deb, not three.

Virtualenv users should be instantly happier, and this will also help us immensely in managing what github issues/PRs go with what content, and making it easier to find things, as, let’s face it, GitHub’s issue tracker is good for small projects, but not big ones, but we like the lower barrier to entry.

Also over the next few months, we’ll also be taking a bit more time to attack core refactoring items, which is further made easier by the seperate queues. (One of these items is sending JSON all the way down - possibly modules all the way down RAW with JSON inputs (TBD), and simplifying the way module templating is done).

Hope this makes sense - shouldn’t really affect anyone at user level - but happy to answer any questions!

–Michael

2

This is now done, see my recent post to ansible-project and -announce, and let me know if you see any problems with this.

I suspect there might be a few lurking things, wasn’t quite intended to go out this early, but kind of had to happen :slight_smile:

3

Awesome. Thanks Michael! This already is making it easier to reason about the giant “update OpenStack modules” PR I’ve had floating around. It seems that there is a nice path forward for that which this gives me - lemme see real quick if it makes sense to you:

  • new very small PR to ansible/ansible with updates to core things like module_utils / module_doc_fragments
  • new very small PR to ansible/ansible-modules-core with a patch consuming the above
  • nothing else to ansible itself for this release

Then - on the side, take the giant patch and put it into its own installable repo and point folks at it as an updated set of modules that I’d love feedback on that depends on ansible 1.8. Once I’ve gotten a few months of real usage feedback from folks and we’ve ironed out the kinks, propose those modules to modules core.

I really like that because it gives me a sane way to use some new stuff in production and have other people test it before telling you that I think it’s solid for ansible core.

4

Glad that is making sense.

I was initially worried a bit about the submodules workflow adding to things, but we definitely hit that point. To an extent, I think it makes it psychologically easier in most places, because if you want to add a module or tweak a module, they are sitting there at the top of a much smaller repo, so it should pay off.

Yeah just setting the ANSIBLE_LIBRARY path to your copy of core is a real easy way to keep a fork of the modules and have people submit patches to it and quickly go back and forth. Not to say this couldn’t be done before, but it would involve pointing it to a subdir of a copy of ansible’s whole tree, which is at least a little bit harder to keep straight - use ansible from here, but use the libraries from another ansible, that’s a little goofier.

I think in the end, this may eventually enable more lieutenant-based merging on core/extras.

The core/extras split right now is pretty arbitrary - it’s only mostly to get a seperate issue tracker.

I think we recognize that eventually we should be able to keep a module_utils and module_docs_fragments piece in core/extras repos as well, that’s just not there now. Wanted to get the plumbing going first.

so the one thing we’ll have to watch for is just not merging an update to module_docs_fragments for something that’s not consistently already implemented for the HEAD of ansible-modules-core/extras, but that shouldn’t come up too much, and we’ll deal with it when it comes.

Thanks!

5

Funny you should mention module_doc_fragments. :slight_smile: I just made:

https://github.com/ansible/ansible-modules-core/pull/8
and
https://github.com/ansible/ansible/pull/9191

and now I wish that github PR referencing (#9191) worked across repos. :slight_smile: But in any case, I think that changes requiring both a module_utils/module_doc_fragments AND a change to the modules themselves are probably an edge case.

6

Got it!

Yeah since module_docs are basically there for the website, it shouldn’t be too bad for them to be non-atomic.

7

PR referencing works across repos

ansible/ansible#9191

Will

8

I was wondering what this might also mean for managing the common code in lib/ansible/module_utils/*

Depending how things evolve at $CUST, I might start working on a series of modules to manage cloud stuff, which will probably have different parts of shared code.
AFAIK, there is no support for that outside of having the shared code in the main repo?

1/ Will this stay core code only, or should we allow for a custom location as with other plugins, possibly also at submodule level?

2/ Shouldn’t we at least support a custom location for this, to allow people to easily develop their own modules with this system in place?

Serge

9

yeah I was alluding to that in another reply - at some point, we’re going to want to modify module_utils so things can move into each repo, and also do this for the module doc stubs.

Ansible imports module_utils, IIRC, though, so I’d want to just make sure any imports of that from runner are guarded and suggest how to check out submodules if missing.

I don’t think we need to contend with other types of plugins at this time - since the existing system was working out fine really.

The goal of this wasn’t really to make multiple “bundles” of trees seperate, so much as to organize the chaos of the GitHub issue tracker.

Module paths remain configurable as they always have via ANSIBLE_LIBRARY and ansible.cfg, so that hasn’t gone away either.

This wasn’t really to enable mixing and matching versions so much as wrangling the GitHub issues system and making it more maintainable.

10

hi Michael,

I’m not sure I understand your reply here.

My question was basically, what workflow is suggested (or might be implemented later) when developing and using modules which need custom common code as e.g. openstack modules that is not or not yet upstream?

Serge

11

Add to common code first, for now.

Longer term, we’ll have a way for each repo to carry it’s own bits.