IMPORTANT - Ansible Repository Merge

Archives Ansible Project
1

Hi all, just wanted to announce/remind people about the upcoming repo merge.

The Plan:

2

Greetings all, just writing to let you all know the present status of repomerge and a problem that we have encountered that you’re likely to run into later as well.

== Executive summary ==

  • We probably have to rewrite git history to fix problems caused by the repomerge script. This will break existing checkouts.

  • Contributors: Please hold off on moving PRs.
    The steps we have to take to correct the repository may mean that the PRs will need to be migrated again (or they may just need to be rebased. We’ll test and let you know). We’ve disabled prmover.pythonanywhere.com for now so no more prs are migrated.

  • git log and git bisect are currently broken in the ansible/ansible devel branch.

  • Checkouts since repomerge (Yesterday, December 6, 2016 UTC) may need to use some extra git flags to pull afterwards. We’ll test and let you know once we’ve fixed the history.

  • The Ansible Core Team is working through the fixed merging strategies to figure out which of them is needed to get things working…

== Detail ==

Yesterday (Tuesday, December 6, 2016) we ran a script which merged the repositories, we performed some minimal checking, and then pushed the branch to the devel branch on github.

Overnight someone commented that the history (via git log) didn’t appear to work quite right (the history of commits existed as shown by git blame but git log --follow FILENAME was unable to find most of the commits).
Further testing showed that git bisect was also having difficulties with the migrated repository: git bisect seemed to get stuck on some commits instead of jumping to a new commit after tagging something as good or bad. It also jumped far back into history instead of another commit within the initial good/bad range.

Today (Wednesday, December 7, 2016) we’ve diagnosed the following problems with the migrated repository:

  • The log problems are caused because the merge script imported the files at locations relative to the top of the module repos. At those locations, git was unable to determine that the files should share a common history with the files in their final locations (even when --follow was used). We prototyped a fix for that which rewrote the file path prior to importing the modules and found that we could fix git log --follow with that.
  • The bisect problem was more mysterious for a while. Eventually we realized that the merge script was interleaving the state of the ansible/ansible tree with the state of the ansible-modules-core and ansible-modules-extras tree. That means that at any given checkout, we might have all of the ansible/ansible files and none of the modules or all of ansible-modules-core and none of ansible/ansible or the same for ansible-modules-extras. When trying to git bisect this tree, we then would run across commits that we couldn’t test accurately because it didn’t have all of the files necessary to test.

The latter problem doesn’t appear to be fixable without removing these commits. So we’re probably going to need to reset the tree to before the repomerge apply a fixed method of merging, and then git push --force the rewritten tree to github. This will invalidate checkouts of the ansible/ansible repository made after the repomerge:

“”"
commit 38b621590311643a6c5867249415cddffc2400b9
Author: James Cammarata <jimi@sngx.net>

Das merge
“”"

At the moment we’re exploring two options as a fixed merge strategy. Both start by resetting to the commit prior to the repomerge commit.

  • The first option will be to simply copy the module files into the new location in the ansible/ansible tree at this point. One commit will then commit all of hte current modules. This strategy will mean that trying to find history for the modules will require going back to the ansible-modules-core and ansible-modules-extras repositories from time to time.
  • The second option is to try exporting patches from the ansible-modules-core and ansible-modules-extras repositories (either via git format-patch or git cherry-pick) and then reapplying those to a new tree inside of ansible/ansible. This would preserve history but is more work to implement. We’re currently trying to find out if this is viable or not.
3

If you’re just on ansible-project list and need more background information look at this mailing list thread: https://groups.google.com/forum/#!topic/ansible-devel/mIxqxXRsmCI

= Repomerge on repomerge, check this thread: Update

Following the outline of a plan by willthames and a lot of of manual merging by jimi-c, the repo merge has been redone and appears to work much better. PRMover, and merging to ansible/ansible is now happening again. The update required removing the original repomerge commits and force pushing the corrected repository to github. We are sorry that this causes a few inconveniences for users and contributors outlined below.

== How to sync your checkout against the new branch

In most cases, this should be sufficient:

$ git pull --rebase origin devel

But in case you have any problems:
IMPORTANT Make sure to do step 1. If you don’t make a backup you MAY LOSE ANY WORK that you’ve done on the devel branch.

  1. Clone your current devel to a new branch, ie. git checkout devel; git checkout -b devel_backup
  2. Reset git to a commit prior to the repo merge: git reset --hard 1a4484be611bdbcaea5f14d1105b7af1ce42552a
  3. Finally, git pull --rebase and you should be on the most up to date devel,
  4. cherry-pick any commits you may have had on devel_backup to the new devel branch.

== Re-migrating PRs

We are extremely sorry but since we had to force push to the repository the PRs which have already been migrated will have to be migrated again. You will receive a dedicated comment on the PR with a link to instructions if your PR falls into this category. We apologize profusely.

== Fixes to PRMover

  • PRMover now properly checks that the user has a clone of ansible/ansible to migrate the PR to. This will give them a more informative error message to fix the problem.
  • PRMover now does a three-way diff which makes some PRs apply cleanly when they wouldn’t apply cleanly with the default merge strategy.

== Why did we have to force push?

The original repomerge script merged the commits from the ansible-modues-core and ansible-modules-extras repositories by using read-tree to push the actual state of the modules repositories into the ansible/ansible repository. The drawback of this was that trying to git bisect would frequently end up on commits where there was only code from the modules repository. Without the ansible/ansible code, it was very hard to test that the commit was valid or not. Rather than subject users trying to diagnose any bugs via bisect to this problem we decided that we needed to rewrite git history to get rid of these commits and do the merge a different way.

Thanks for bearing with us,
Toshio