IMPORTANT HEADS UP: If running from devel branch, now using submodules

Archives Ansible Project
1

Hi everyone,

This email is to developer-types who are not subscribed to ansible-devel and did not read this already. If you consume stable releases of ansible only, and do not submit pull requests, you can MOSTLY skip this email.

Well, not quite - the way we file bugs has also changed.

Basics? I’ve changed the main ansible project to now be split into three:

ansible
ansible-modules-core
ansible-modules-extra

ansible will still be shipped as 1 package in pip, as well as in RPM/deb, etc, and I would like all distros to keep doing exactly that.

However, splitting the repos will, soon, allow us to use seperate issue trackers and make it easier to manage the project.

Since ansible modules now travel inside the python package, certain virtualenv users should also be much much happier, diminishing questions about why we want to write to LSB locations like /usr/share/ansible when installing in python-esque ways.

A few important things to note is when submitting a bug on a module, check the docs page for that module and see if it’s in “core” or “extras” and file the bug in the appropriate github repo. Language and central issues should go to core, and I suspect it will be a while before this becomes second nature.

Should anyone hit any problems with this, let us know ASAP.

If you have existing checkouts:

git pull --rebase
git submodule update

If cloning the repo, just add “–recursive” to the end of your clone.

If using the devel branch, you can also delete /usr/share/ansible library content, as this may cause problems with using “stale” modules. Packaging will take care of this automatically on upgrade.

If doing development on modules, simply check out one of these two new repos to whatever location, and configure it ahead in your “ANSIBLE_LIBRARY” path to make sure you’re running off those versions instead of core.

In many ways, this makes it really easy to run any version of Ansible with newer versions of the library, but don’t do that, if you are to report issues here, as that will greatly confuse us and may not always work :slight_smile:

When filing bugs, please be sure to file the ticket on the right repo – that’s the main thing, though nothing really changes in this capacity. We’ll have ansibot tracking all three repos in short order.

Thank you and let me know if there are any questions!

2

What happens with existing pull requests against modules? Do we need to resubmit them to the new repos (as presumably they’ll no longer merge)?

I’ve just handed over how we maintain our internal bleeding edge fork (devel + a couple of as-yet-unaccepeted pull requests), so this will make Monday much more fun.

Will

3

Ok, so the instructions are wrong.

git pull --rebase
git submodule init
git submodule update (without the init, this does nothing)

Also, renaming the modules to *.py without an intervening commit does not at all help recreating commits through patching

To recreate an old pull request on ansible-modules-core after following the above instructions, do this:
git checkout your_branch
git rebase f35ed8a6c0dc81 # last commit before the removal of library modules
git format-patch f35ed8a6c0dc81 --stdout | sed ‘s:library/[^ \n]*:&.py:g’ > /tmp/patchname.patch # the sed adds .py to the module file names
git checkout devel
cd lib/ansible/modules/core/
git checkout -b your_branch
git apply --check -p2 /tmp/patchname.patch # -p2 here strips off the leading directory (in most cases, library)

If the above step works, use
git apply -p2 /tmp/patchname.patch
git commit -a
git push your_fork_remote your_branch

This only works if your patch only affects modules - if it affects more than just modules (e.g. tests) you can use the --exclude to exclude files

e.g. for a patch containing both test/integration and library stuff

git apply -p2 /tmp/patchname.patch --exclude integration/*
and in the ansible repo
git apply /tmp/patchname.patch --exclude library/*
git commit -a
git push your_fork_remote your_branch

Will

4

Some mistakes by me in the above instructions (not least the invalid assertion that the renaming of the modules didn’t have an intermediate commit - git log on the file suggested it didn’t but I probably got that wrong)

Anyway to recreate an old pull request on ansible-modules-core after following the above instructions, do this from the root of your ansible repo

git checkout branch_containing_your_commit
git rebase f35ed8a6c0dc81 # last commit before the removal of library modules
git format-patch f35ed8a6c0dc81 --stdout | > /tmp/patchname.patch # the sed adds .py to the module file names
git checkout devel

cd lib/ansible/modules/core/
git reset --hard 417309a626 # this is before the module renames
git checkout -b branch_for_new_pull_request
git am -p2 /tmp/patchname.patch # -p2 here strips off the leading directory (in most cases, library)
git rebase devel
git push your_remote_for_your_fork branch_for_new_pull_request

The earlier note about exclusions still applies.

Will

5

Still mistakes. I’m moving my instructions to a gist!

https://gist.github.com/willthames/afbaaab0c9681ed45619

Will

6

simple commands for existing checkouts:

git pull --rebase
git submodule update --init --recursive

That should be enough

7

Yeah I think we can keep it simple.

If I’ve missed an “–init” somewhere the easiest thing is to link to a minimal pull request.

Another thing I thought of

(A) is making the “–version” command output the SHA’s of the submodules when running from source.
(B) making runner arbitrarily look for the “ping” module, and if can’t find one, if running from git, have the CLI suggest the git commands to run.

We are likely going to be needing to close out module-tagged pull requests and suggest that they be resubmitted, but have not done so yet.
Having the instructions of how to do that in the message (like our usual ticket_stubs/ responses) I think would be super helpful. That being said, a lot has changed since, and many pull requests may be out of date - having some changes reapplied may be desirable.

There are import scripts for moving bug tickets that I think we can use, though it’s slighty problematic in that we have two repos and the system can’t tell which is which. As such, we may ask these be trivially resubmitted.

I’d say hold off on this now until we take some action on that ticket queue, which is the plan for next week.

8

Raised https://github.com/ansible/ansible/pull/9185 for the trivial doc fix (thanks Brian for the tip)

I’ve managed to update all my module based PRs - it’s definitely not easy to reproduce a set of commits from one repo to another (to be honest
it’s probably just easier to copy the fix and commit log if it’s only a single commit which most PRs will typically be) - I only had two PRs which were module affecting though.

The instructions in my gist do include rebasing against a currently quite recent version of devel (i.e. the last before the modules were removed from core) and if they look like something worth adding to a ticket stub I can create such a stub

Also raised https://github.com/ansible/ansible/pull/9186 to improve ansible version information. It now looks like:

ansible 1.8 (submodule_ansible_version c621f54538) last updated 2014/09/28 11:05:48 (GMT +1000)
  lib/ansible/modules/core: (ec2_snapshot_remove 3a77c31ecb) last updated 2014/09/27 18:23:31 (GMT +1000)
  lib/ansible/modules/extras: (detached HEAD 110250d344) last updated 2014/09/27 14:33:42 (GMT +1000)

Will

9

Awesome, digging in on these now.

10

Ok, a few things done just now:

(A) merged Will’s patch to docs (thanks!)

(B) merged Will’s patch to add --version details for the submodules, which will be huge. I also tweaked it to understand what to do if there were no submodules.

(C) I’ve tweaked the runtime code if a module is not found to look for the ‘ping’ module. If the ‘ping’ module can’t be found, we know that submodule updates never happened. This should prevent development branch users who use GitHub, but don’t know about this list, from getting confused about module paths not working – which should make things rather self documenting.

11

(D) I’ve added the configured module path to the output of ansible --version, in case someone has configured modules in their local checkout to override modules in the checkouts, we’ll be able to see what those paths are in the bug reports.

12

Regarding the gist, I’m going to talk through the helpful gist thing with James and Toshio Monday.

Should that work out - and I’m sure it will - we can definitely include that in a “stub” thing, and then provide comments on all the tickets.

It’s pretty cool that it’s been possible to get that as standardized.

Still have yet to determine what to do with bug tickets, but that will probably be to look at the docs page, figure out if it’s core or extras, and open it on the right repo.

We also have to get ansibot also working on those repos - easy enough - to provide the usual boilerplate, and also to suggest to those opening tickets on the main project that they MAY want to open one on the others instead.

13

And I’ve just pushed a few additional fixes

(E) Ansible will be happy if there is no configured module path, before at least an empty one was required, but if there was no ansible.cfg, it got confused

(F) I’ve fixed a problem where the powershell module ‘setup’ got found before the Python one. Now ‘.py’ modules are searched unless it’s a Windows module, then ‘.ps1’ are searched, but the streams are never crossed.

14

I’ve just made nodes and retired a reasonably large set of tickets. You should have GitHub notifications if this affects you.

Help moving things to new repos is greatly appreciated so let us know if there are questions!

Thanks!

15

Bother. I saw those emails(one request, 2 emails). However, I was rather surprised that my 6026 pull request hadn't yet been merged. I thought it would have been rather simple. It's been over 6 months, with no updates at all, and suddenly this is closed. How do I know if I go and do even *more* work, to rebase-onto(or email-patch/import) with git, that this change won't still be completely ignored.

For reference, this updated the mount task so that swap entries are treated like other filesystems, in that swapon/swapoff are called.

16

Hi Adam,

I’m sorry you feel that way, naturally we’ve tried to communicate this as much as possible along the path, as well as the comments on all the tickets (and yes, my fingers are sore from doing it – as I didn’t want to lose time with the github script) are a big part of that.

I think we’ve been pretty up-front with the queue, our priority system and that this approach is going to allow faster merging.

Ansible has had over 880 contributors in about 2.5 years - if you can imagine ANY employer working like that, you’d imagine how crazy it might be.

All of this has happened with a lot of ticket prioritization, making sure we work on the most pressing issues, not “insta-merging” things that are dangerous to include the most stability to ansible. This has occasionally meant small tickets have had to wait as we work on what get (not always perfectly) prioritized as more important things, so we can make a dent as we scale things up.

If you would like your feature merged, you are welcome to bring it forward to the repo structure.

Complaints about the system really don’t really help things any, alas, so we’d ask your understanding.

By no means are we holding requests infinitely for kicks, and I believe your github commit history - and release history shows heavy progress is constantly being made.

Ansible is, right now, basically in the Top 100 projects on GitHub for forks - and we have about 38% of forks resulting in contributions merged in, with a lot more pending - for some other projects, those forks result in contributions at about a 1% rate, so we’re actually doing pretty darn well here.

I recognize some people are frustrated with this move, however we’ve already had roughly 60 or 70 people move things over in just a few hours, so I think it’s being recieved pretty well, and we’ve also been able to accelerate some merging (however little, as it’s early yet - or rather late at night) - already.

Stay tuned and you’ll see a lot more to come.

–Michael

17

If following this path, future updates are just"?
git pull

Or will need the rebase/init every time?

18

Well, you should never do git pull. It should always be “git rebase” or “git pull --rebase”

Now, for submodules, you will need to add:

git submodule update --recursive