# IMPORTANT HEADS UP: If running from devel branch, now using submodules **URL:** \ **Category:** Ansible Project\ **Created:** [September 26, 2014, 10:34pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362 "2014-09-26T22:34:01Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 26, 2014, 10:34pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/1 "2014-09-26T22:34:01Z")
Hi everyone, This email is to developer-types who are not subscribed to ansible-devel and did not read this already. If you consume stable releases of ansible only, and do not submit pull requests, you can MOSTLY skip this email. Well, not quite - the way we file bugs has also changed. Basics? I’ve changed the main ansible project to now be split into three: ansible ansible-modules-core ansible-modules-extra ansible will still be shipped as _1_ package in pip, as well as in RPM/deb, etc, and I would like all distros to keep doing exactly that. However, splitting the repos will, soon, allow us to use seperate issue trackers and make it easier to manage the project. Since ansible modules now travel _inside_ the python package, certain virtualenv users should also be much much happier, diminishing questions about why we want to write to LSB locations like /usr/share/ansible when installing in python-esque ways. A few important things to note is when submitting a bug on a module, check the docs page for that module and see if it’s in “core” or “extras” and file the bug in the appropriate github repo. Language and central issues should go to core, and I suspect it will be a while before this becomes second nature. Should anyone hit any problems with this, let us know ASAP. If you have existing checkouts: git pull --rebase git submodule update If cloning the repo, just add “–recursive” to the end of your clone. If using the devel branch, you can also delete /usr/share/ansible library content, as this may cause problems with using “stale” modules. Packaging will take care of this automatically on upgrade. If doing development on modules, simply check out one of these two new repos to whatever location, and configure it ahead in your “ANSIBLE\_LIBRARY” path to make sure you’re running off those versions instead of core. In many ways, this makes it really easy to run any version of Ansible with newer versions of the library, but don’t do that, if you are to report issues here, as that will greatly confuse us and may not always work 🙂 When filing bugs, please be sure to file the ticket on the right repo – that’s the main thing, though nothing really changes in this capacity. We’ll have ansibot tracking all three repos in short order. Thank you and let me know if there are any questions! ---
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [September 27, 2014, 4:22am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/2 "2014-09-27T04:22:43Z")
What happens with existing pull requests against modules? Do we need to resubmit them to the new repos (as presumably they’ll no longer merge)? I’ve just handed over how we maintain our internal bleeding edge fork (devel + a couple of as-yet-unaccepeted pull requests), so this will make Monday much more fun. Will ---
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [September 27, 2014, 5:03am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/3 "2014-09-27T05:03:48Z")
Ok, so the instructions are wrong. git pull --rebase git submodule init git submodule update (without the init, this does nothing) Also, renaming the modules to \*.py without an intervening commit does not at all help recreating commits through patching To recreate an old pull request on ansible-modules-core after following the above instructions, do this: git checkout your\_branch git rebase f35ed8a6c0dc81 # last commit before the removal of library modules git format-patch f35ed8a6c0dc81 --stdout | sed ‘s:library/[^ \n]\*:&.py:g’ \> /tmp/patchname.patch # the sed adds .py to the module file names git checkout devel cd lib/ansible/modules/core/ git checkout -b your\_branch git apply --check -p2 /tmp/patchname.patch # -p2 here strips off the leading directory (in most cases, library) If the above step works, use git apply -p2 /tmp/patchname.patch git commit -a git push your\_fork\_remote your\_branch This only works if your patch only affects modules - if it affects more than just modules (e.g. tests) you can use the --exclude to exclude files e.g. for a patch containing both test/integration and library stuff git apply -p2 /tmp/patchname.patch --exclude integration/\* and in the ansible repo git apply /tmp/patchname.patch --exclude library/\* git commit -a git push your\_fork\_remote your\_branch Will ---
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [September 27, 2014, 8:13am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/4 "2014-09-27T08:13:58Z")
Some mistakes by me in the above instructions (not least the invalid assertion that the renaming of the modules didn’t have an intermediate commit - git log on the file suggested it didn’t but I probably got that wrong) Anyway to recreate an old pull request on ansible-modules-core after following the above instructions, do this from the root of your ansible repo git checkout branch\_containing\_your\_commit git rebase f35ed8a6c0dc81 # last commit before the removal of library modules git format-patch f35ed8a6c0dc81 --stdout | \> /tmp/patchname.patch # the sed adds .py to the module file names git checkout devel cd lib/ansible/modules/core/ git reset --hard 417309a626 # this is before the module renames git checkout -b branch\_for\_new\_pull\_request git am -p2 /tmp/patchname.patch # -p2 here strips off the leading directory (in most cases, library) git rebase devel git push your\_remote\_for\_your\_fork branch\_for\_new\_pull\_request The earlier note about exclusions still applies. Will ---
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [September 27, 2014, 8:25am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/5 "2014-09-27T08:25:46Z")
Still mistakes. I’m moving my instructions to a gist! [https://gist.github.com/willthames/afbaaab0c9681ed45619](https://gist.github.com/willthames/afbaaab0c9681ed45619) Will ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [September 27, 2014, 11:36am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/6 "2014-09-27T11:36:43Z")
simple commands for existing checkouts: git pull --rebase git submodule update --init --recursive That should be enough ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 27, 2014, 12:40pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/7 "2014-09-27T12:40:37Z")
Yeah I think we can keep it simple. If I’ve missed an “–init” somewhere the easiest thing is to link to a minimal pull request. Another thing I thought of (A) is making the “–version” command output the SHA’s of the submodules when running from source. (B) making runner arbitrarily look for the “ping” module, and if can’t find one, if running from git, have the CLI suggest the git commands to run. We are likely going to be needing to close out module-tagged pull requests and suggest that they be resubmitted, but have not done so yet. Having the instructions of how to do that in the message (like our usual ticket\_stubs/ responses) I think would be super helpful. That being said, a lot has changed since, and many pull requests may be out of date - having some changes reapplied may be desirable. There are import scripts for moving bug tickets that I think we can use, though it’s _slighty_ problematic in that we have two repos and the system can’t tell which is which. As such, we may ask these be trivially resubmitted. I’d say hold off on this now until we take some action on that ticket queue, which is the plan for next week. ---
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [September 28, 2014, 1:10am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/8 "2014-09-28T01:10:54Z")
Raised [https://github.com/ansible/ansible/pull/9185](https://github.com/ansible/ansible/pull/9185) for the trivial doc fix (thanks Brian for the tip) I’ve managed to update all my module based PRs - it’s definitely not easy to reproduce a set of commits from one repo to another (to be honest it’s probably just easier to copy the fix and commit log if it’s only a single commit which most PRs will typically be) - I only had two PRs which were module affecting though. The instructions in my gist do include rebasing against a currently quite recent version of devel (i.e. the last before the modules were removed from core) and if they look like something worth adding to a ticket stub I can create such a stub Also raised [https://github.com/ansible/ansible/pull/9186](https://github.com/ansible/ansible/pull/9186) to improve ansible version information. It now looks like: ansible 1.8 (submodule\_ansible\_version c621f54538) last updated 2014/09/28 11:05:48 (GMT +1000)   lib/ansible/modules/core: (ec2\_snapshot\_remove 3a77c31ecb) last updated 2014/09/27 18:23:31 (GMT +1000)   lib/ansible/modules/extras: (detached HEAD 110250d344) last updated 2014/09/27 14:33:42 (GMT +1000) Will ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 28, 2014, 3:18pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/9 "2014-09-28T15:18:39Z")
Awesome, digging in on these now. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 28, 2014, 3:36pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/10 "2014-09-28T15:36:25Z")
Ok, a few things done just now: (A) merged Will’s patch to docs (thanks!) (B) merged Will’s patch to add --version details for the submodules, which will be huge. I also tweaked it to understand what to do if there were no submodules. (C) I’ve tweaked the runtime code if a module is not found to look for the ‘ping’ module. If the ‘ping’ module can’t be found, we know that submodule updates never happened. This should prevent development branch users who use GitHub, but don’t know about this list, from getting confused about module paths not working – which should make things rather self documenting. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 28, 2014, 3:41pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/11 "2014-09-28T15:41:33Z")
(D) I’ve added the configured module path to the output of ansible --version, in case someone has configured modules in their local checkout to override modules in the checkouts, we’ll be able to see what those paths are in the bug reports. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 28, 2014, 3:46pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/12 "2014-09-28T15:46:34Z")
Regarding the gist, I’m going to talk through the helpful gist thing with James and Toshio Monday. Should that work out - and I’m sure it will - we can definitely include that in a “stub” thing, and then provide comments on all the tickets. It’s pretty cool that it’s been possible to get that as standardized. Still have yet to determine what to do with bug tickets, but that will probably be to look at the docs page, figure out if it’s core or extras, and open it on the right repo. We also have to get ansibot also working on those repos - easy enough - to provide the usual boilerplate, and also to suggest to those opening tickets on the main project that they MAY want to open one on the others instead. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 28, 2014, 4:32pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/13 "2014-09-28T16:32:29Z")
And I’ve just pushed a few additional fixes (E) Ansible will be happy if there is no configured module path, before at least an empty one was required, but if there was no ansible.cfg, it got confused (F) I’ve fixed a problem where the powershell module ‘setup’ got found before the Python one. Now ‘.py’ modules are searched unless it’s a Windows module, then ‘.ps1’ are searched, but the streams are never crossed. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 29, 2014, 9:15pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/14 "2014-09-29T21:15:28Z")
I’ve just made nodes and retired a reasonably large set of tickets. You should have GitHub notifications if this affects you. Help moving things to new repos is greatly appreciated so let us know if there are questions! Thanks! ---
**Author:** ![adam2](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@adam2](https://forum.ansible.com/u/adam2)\ **Post date:** [September 29, 2014, 9:23pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/15 "2014-09-29T21:23:14Z")
Bother. I saw those emails(one request, 2 emails). However, I was rather surprised that my 6026 pull request hadn't yet been merged. I thought it would have been rather simple. It's been over 6 months, with no updates at all, and suddenly this is closed. How do I know if I go and do even \*more\* work, to rebase-onto(or email-patch/import) with git, that this change won't still be completely ignored. For reference, this updated the mount task so that swap entries are treated like other filesystems, in that swapon/swapoff are called. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 30, 2014, 12:11am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/16 "2014-09-30T00:11:02Z")
Hi Adam, I’m sorry you feel that way, naturally we’ve tried to communicate this as much as possible along the path, as well as the comments on all the tickets (and yes, my fingers are sore from doing it – as I didn’t want to lose time with the github script) are a big part of that. I think we’ve been pretty up-front with the queue, our priority system and that this approach is going to allow faster merging. Ansible has had over 880 contributors in about 2.5 years - if you can imagine _ANY_ employer working like that, you’d imagine how crazy it might be. All of this has happened with a lot of ticket prioritization, making sure we work on the most pressing issues, not “insta-merging” things that are dangerous to include the most stability to ansible. This has occasionally meant small tickets have had to wait as we work on what get (not always perfectly) prioritized as more important things, so we can make a dent as we scale things up. If you would like your feature merged, you are welcome to bring it forward to the repo structure. Complaints about the system really don’t really help things any, alas, so we’d ask your understanding. By no means are we holding requests infinitely for kicks, and I believe your github commit history - _and_ release history shows heavy progress is constantly being made. Ansible is, right now, basically in the Top 100 projects on GitHub for forks - and we have about 38% of forks resulting in contributions merged in, with a lot more pending - for some other projects, those forks result in contributions at about a 1% rate, so we’re actually doing pretty darn well here. I recognize some people are frustrated with this move, however we’ve already had roughly 60 or 70 people move things over in just a few hours, so I think it’s being recieved pretty well, and we’ve also been able to accelerate some merging (however little, as it’s early yet - or rather late at night) - already. Stay tuned and you’ll see a lot more to come. –Michael ---
**Author:** ![Francisco\_Reyes](https://avatars.discourse-cdn.com/v4/letter/f/b77776/32.png) [@Francisco\_Reyes](https://forum.ansible.com/u/Francisco_Reyes)\ **Post date:** [October 5, 2014, 1:54am UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/17 "2014-10-05T01:54:53Z")
If following this path, future updates are just"? git pull Or will need the rebase/init every time? ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [October 5, 2014, 6:00pm UTC](https://forum.ansible.com/t/important-heads-up-if-running-from-devel-branch-now-using-submodules/16362/18 "2014-10-05T18:00:42Z")
Well, you should never do git pull. It should always be “git rebase” or “git pull --rebase” Now, for submodules, you will need to add: git submodule update --recursive