Ansible 2.10 and Project Restructuring

Archives Ansible Developer
1

Greetings Ansible Community,

For the 2.10 release of Ansible, we plan to restructure the ansible/ansible repository. The majority of modules and plugins will migrate to collection repositories to better enable the community to build, test, and release modules separately from Ansible releases. This will remove the Core maintainers as a bottleneck to community module and plugin development.

Timeline (subject to change):

  • 2020-03-02: we will freeze the devel branch using protected branches, and we will create the temp-2.10-devel branch from devel. This date marks the end of merging non-base plugin/module PRs into ansible/ansible.

  • 2020-03-09: we will perform the initial migration against temp-2.10-devel, and we will do our initial testing of the components.

  • 2020-03-23: we intend to unfreeze devel and merge temp-2.10-devel back into devel. From that point on, devel for ansible/ansible will be for the ansible-base project only.

Migration details:

  • All components of the Ansible project are being evaluated, and tagged for future upstream homes.
  • The mapping of content between the current ansible/ansible directory structure and each collection is being maintained via scenario files. The scenario files define what each collection will contain after the migration takes place.
  • Everything under _core in the ansible.yml migration scenario file will become Ansible 2.10 (ansible-base).
  • Once the migration is run, which will be on or around March 9, 2020, all content will be copied to a temporary location (Ansible Migration GitHub Organization) where it can then be moved to the final location of the content owner’s choosing. All content migrated will then be deleted from the ansible/ansible GitHub repository.
  • By the end of March, the current ansible/ansible GitHub repository will only contain the contents of ansible-base (see definition below), only foundational Ansible components. All content will have been migrated.

What will be in Ansible 2.10?
Ansible 2.10 will contain a minimal amount of Core supported modules and plugins, allowing for additional plugins and modules to be installed separately via collections. The proposed contents of ansible-base are listed under _core here:

https://github.com/ansible-community/collection_migration/blob/master/scenarios/nwo/ansible.yml

To test out a pre-release version of ansible-base today, use the following source:
https://github.com/ansible-collection-migration/ansible-base

The pip install ansible-base command will install Ansible 2.10 as described above (not currently available).
The pip install ansible command will install the Ansible Community Distribution (ACD) that includes Ansible 2.10 and all additional module and plugin content via Collections.

Please note that ACD is still currently in development and not available for testing yet at this time.

What does a “migration” of content actually mean?
It effectively means that anything not part of Ansible Base, or Ansible 2.10, will be automatically relocated to an alternate GitHub organization and repository.

How is content “claimed”?
If content isn’t claimed, it will be migrated to the community.general collection and new GitHub repository (see below, “What happens to unclaimed content?”) and subject to all rules and policies of the Community Collection.

Here is the process for claiming content:

  • Source content is moved to a GitHub organization and repository of your choosing
  • A namespace is ready for use on Ansible Galaxy in order to publish content
  • An Ansible Collection is created, built, and published to Ansible Galaxy
  • Existing content in the Ansible GitHub repository is flagged for deletion
  • BOTMETA is updated to ensure future Ansible community distributions carry the content going forward. Please refer to the Collections Developer Guide (migration section) for details on how to accomplish this.

If certain content isn’t part of Ansible 2.10, how will it get distributed?
Please note that Ansible 2.10 is just one piece of the larger Ansible Community Distribution (ACD) effort in the works. This includes a recombined set of all content similar to previous Ansible releases, but instead pulls content from various upstream repositories to form a final installation of Ansible Base plus Ansible Content. Full details on this effort will be made available soon.

What if I don’t see certain content as part of a scenario file?
It may be that the content has not been merged into the ansible/ansible project on GitHub already and therefore no migration is needed, and therefore no scenario file is needed. If some content is in fact part of Ansible 2.9 and not part of a scenario file, please let us know to ensure the migration goes as expected.

What happens to unclaimed content on March 9, 2020?
Any content in the ansible/ansible project on GitHub that is not claimed and migrated elsewhere will automatically be migrated to the community.general collection under the ansible-collections GitHub organization.

Where do I file issues against Ansible 2.10?
It depends on when you file the issue due to where we are in the migration process:
Before 2020-03-02:
All issues: ansible/ansible project on GitHub
Between 2020-03-02 and 2020-03-23: Please ask. The project repositories will be constantly changing
After 2020-03-23:
ansible-base issues: ansible/ansible project on GitHub
Collections specific issues: ansible-collections/{namespace}.{collection} or in the partner owned GitHub repository

Please note: once content is migrated out of the ansible/ansible repository, any remaining open issues and pull requests will no longer be valid. It will be up to the original reporter to move these to the new GitHub organization and repository. More details to follow closer to the freeze date.

What other details are currently in process and will be detailed in the near future?
Topics such as collection versioning, deprecation, and documentation are being worked on and will be detailed soon. Please refer to the Ansible Collections Community Overview at the following location: https://github.com/ansible-collections/overview/blob/master/README.rst

What if I have more questions?
Please respond to this email or reach out on Freenode IRC in ansible or ansible-devel.

2

I didn’t notice this post so emailed my questions to ansible-project instead, sorry for duplication.

aws modules have been redirected/moved to two different repositories, some to community.aws and some to amazon.aws

https://github.com/ansible-collections/amazon.aws

https://github.com/ansible-collections/community.aws

My first question is, what was the criteria for moving each modules to different repositories ? so we know the correct place to submit future aws modules !!! is there a change log for this move ?

My second question is for those with more experience with Ansible and github:
isn’t better to make the amazon.aws plugin directory be the sub repository/sub module/fork or mirror of community.aws plugin directory ? this way modules and plugins version and code stay the same in both, while one will be used in collection and one as stand alone module !!!

Thanks

3

Hi Sam,

What is the new process for testing changes to the modules that have moved to a collection?

The top-level README of both AWS collection repos points to [how to *use* collections](https://docs.ansible.com/ansible/latest/user_guide/collections_using.html), but that doesn't tell me how to *develop* collections.

Do I need to clone both the collection repo, and the main ansible/ansible repo?
The whole point of this split is that they get developed separately. So do I not need the main repo?
But if I don’t have the main repo, how do I do ``. hacking/env-setup? Do I still need that?

If I want to [run my code locally](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#exercising-module-code-locally) what do I pass as the -m argument? What folder do I need to be in?

What about [unit tests](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#unit-tests). Is the command the same? What folder do I need to be in?

How do I run integration tests? The new AWS collection repo does not have [the policy file listed in the docs](https://docs.ansible.com/ansible/latest/dev_guide/testing_integration.html#testing-policies).

Is the development process documented anywhere? I can't find it in [the FAQ](https://github.com/ansible-collections/overview#contributors-to-ansible). Please point me in the right direction.

Thanks,
Matt

4

Hi,

you'll find some basic information on that in
https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html
(in the devel docs). See in particular the last and second-to-last sections.

Best,
Felix

5

Matt,

Hopefully the documentation provided by Felix answered some of your questions. I’ll try to add a little more clarity.

Do I need to clone both the collection repo, and the main ansible/ansible repo?

No, but with some short term caveats.

If you are only interested in working on collection content, then there is no need to clone the ansible/ansible repo or run the hacking setup script. You only need to clone the collection repo and make sure it is in your collection path.

The caveat is that Ansible 2.9, the current released version, has basic support for collections. This was improved greatly in 2.10, which has not been released.

Start by using the latest version of Ansible 2.9 rather than a source checkout of ansible/ansible. If you run into any issues, try using 2.10 which you can install via pip.

pip uninstall ansible && pip install ansible --pre

If you want to run the ansible source as well as the collection source, you certainly can do that (that’s how I work). But it is not required to use a source checkout of ansible/ansible in order to develop collections.

What is the new process for testing changes to the modules that have moved to a collection?

If I want to [run my code locally](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#exercising-module-code-locally) what do I pass as the -m argument? What folder do I need to be in?

If you have the module in your collection path, you can test the module using its fully qualified collection name (FQCN). For example, if you module is located at:

~/.ansible/collection/ansible_collections/matt/my_collection/plugins/modules/my_module.py

You can run this module with this command:

ansible all -m matt.my_collection.my_module -a ‘[args]’

What about [unit tests](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#unit-tests). Is the command the same? What folder do I need to be in?

How do I run integration tests? The new AWS collection repo does not have the policy file listed in the docs.

Per the docs, unit and integration tests must be run from the root directory of the collection. The command is the same.

Regarding the AWS test questions, I am not sure about the policy file. It would be best to ask in #ansible-aws on Freenode IRC

6

Thanks Sam and Felix.

Unfortunately I’m still very lost.

(Which means there’s probably lots of other potential contributors who are lost too.)

If you have the module in your collection path

What’s a collection path?
Do you mean that iam_role.py needs to be in a directory within my OS $PATH variable?

Or do you mean this:

https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths

If the latter, which folder do I put in that path?
https://github.com/ansible-collections/community.aws/tree/master/plugins/modules
or
https://github.com/ansible-collections/community.aws
or a folder that contains the clone of https://github.com/ansible-collections/community.aws ?

I tried a -m command and got:

ERROR! this task 'community.aws/iam_role.py' has extra params, which is only allowed in the following modules: import_tasks, raw, ansible.windows.win_command, win_command, add_host, meta, include_role, command, include_vars, include, win_shell, import_role, ansible.windows.win_shell, shell, include_tasks, group_by, set_fact, script

That’s a very confusing error message. Definitely not what I’d expect to see if I put the wrong value in ANSIBLE_COLLECTIONS_PATH.
I passed in an empty dict as my argument. So how does it have too many params?

How long do you expect it will take a newcomer to start contributing under this new split repo setup?
Is it supposed to take 20 minutes, 3 hours? What’s a realistic timeframe you’re aiming for?

I’ve contributed to Ansible before, and I’m still struggling. It was a struggle before, and even more of a struggle now.
The relevant documentation just seems spread over too many pages.

I think someone who knows how this works should make a quick start guide, like this. Just 4 paragraphs and 4 code blocks of commands.

Thanks,
Matt

7

Hi Matt,

Unfortunately I'm still very lost.
(Which means there's probably lots of other potential contributors
who are lost too.)

> If you have the module in your collection path

What's a collection path?
Do you mean that iam_role.py needs to be in a directory within my OS
$PATH variable?

Or do you mean this:
https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths

yes, that's meant.

If the latter, which folder do I put in that path?
https://github.com/ansible-collections/community.aws/tree/master/plugins/modules
or
https://github.com/ansible-collections/community.aws
or a folder that contains the clone of
https://github.com/ansible-collections/community.aws ?

The folder which contains ansible_collections. That's a folder
containing the folder community/aws (which finally is the checkout of
https://github.com/ansible-collections/community.aws).

This is all described in
https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#contributing-to-collections

I tried a -m command and got:

ERROR! this task 'community.aws/iam_role.py' has extra params, which
is only allowed in the following modules: import_tasks, raw,
ansible.windows. win_command, win_command, add_host, meta,
include_role, command, include_vars, include, win_shell, import_role,
ansible.windows.win_shell, shell, include_tasks, group_by, set_fact,
script

So you did `ansible localhost -m command`? Or what exactly did you do
to receive this error?

How long do you expect it will take a newcomer to start contributing
under this new split repo setup?
Is it supposed to take 20 minutes, 3 hours? What's a realistic
timeframe you're aiming for?

I would expect that with the instructions in
https://docs.ansible.com/ansible/devel/dev_guide/developing_collections.html#contributing-to-collections
it should go pretty fast (assuming you already know Ansible).

I'm sure this section can be improved, but I think it already contains
everything important. Which parts of it were confusing to you that you
didn't succeed in setting up your dev environment?

Thanks,
Felix

8

Hi Felix,

it should go pretty fast

I have literally spent hours on this, and there are just so many obstacles to a new person trying to run existing tests.
Do you expect it to take hours? Or less than one hour?

Please give me an actual numeric estimate of how long you expect it would take someone to test a one-line patch, if they:

  • already know Python

  • already know how to use Ansible

  • already know the line they want to change

  • already know the Ansible task they want to add as a test

  • already know generic git + GitHub workflows

Are there any other newcomers on this mailing list? If so please let me know how long it took you to set up your environment, and figure out how to run tests in this new world. I would like to know if my experience is typical, or if I’m just a foolish developer.

I’ve been documenting the issues and some documentation additions here . I also submitted a PR to modify documentation, but that was closed without explanation. (As was another of my PRs. May I suggest that when a PR is closed the closer should add a comment? I cannot overstate how much the lack of explanation makes me want to give up and stop contributing.)

As an example of an issue with tests, when I do finally manage to start executing the sanity checks, I get a lot of errors.

ERROR: plugins/modules/rds_param_group.py:0:0: traceback: ModuleNotFoundError: No module named ‘ansible_collections.community.community’
ERROR: plugins/modules/rds_snapshot.py:0:0: traceback: ModuleNotFoundError: No module named ‘ansible_collections.community.community’

Does that mean I’m meant to somehow install a collection called community.community as a dependency? That doesn’t seem to exist. Only community.general. Either way, am I supposed to manually install other collections as dependencies? Isn’t the point of running tests in docker that the dependencies are all there in the images? If I am meant to install them, how do I ensure that the files installed somewhere in the system by ansible-galaxy collection install my_namespace.my_collection (which does not install into the PWD) get mounted into the docker container somehow?

What’s the status of the split into many repos? Should I expect each collection to have sanity tests that pass when run in the master branch?

Thanks,
Matt

9

Matt,

I think there is valid stuff in what you’re saying. Let me write out a formula:

cd ~/Documents/test

git clone https://github.com/ansible-collections/community.aws.git

ansible-galaxy collection build

rm -rf ~/.ansible/collections/ansible_collections/community/aws/

ansible-galaxy collection install community-aws-1.0.0.tar.gz

cd ~/.ansible/collections/ansible_collections/community/aws/

ansible-test sanity

Disclosure: I have worked a lot with other collections. I have never ran the community.aws collection tests before, to my knowledge.

Importantly - is there a good software development reason for all the steps here? Absolutely. Testing the build as opposed to the source is valuable, and at this point with collections, only supporting testing the built version makes sense. Everyone developing needs to know the build-install process, and people who know it tend to be thankful that we have it. I think we need more advocacy of it.

The errors that come from ansible-test in particular are probably the highest impact pain point. There is a very specific practice that it works well for, but it’s hard for people to get there with the error messages that it gives.

However, that still winds me up with

AttributeError: ‘TypeError’ object has no attribute ‘errors’

Maybe I still need to upgrade something. This does work for other collections, so it might be something specific to this content.

Alan
github: AlanCoding

10

That’s not the full error (or the most relevant part of the error), but it matches what I saw without https://github.com/ansible-collections/community.aws/pull/128

11

Gotcha, I had not read close enough. Your fix looks correct.

I proposed a fix on the ansible-test side. Both are valid in my book, and your discovery argues strongly that this should be a failure (as opposed to either a traceback or a pass).

https://github.com/ansible/ansible/pull/70507

Thanks.