We should merge OpenStack neutron_* modules from openstack-ansible-modules to core

Archives Ansible Developer
1

Hi all,

We would like to contribute on OpenStack modules but the current situation is messy as there are unofficial modules under openstack-ansible project and official modules under core.

To get things moving I propose that we merge neutron_* modules from
openstack-ansible-modules (https://github.com/openstack-ansible/openstack-ansible-modules)
to
ansible-modules-core/cloud/openstack (https://github.com/ansible/ansible-modules-core/tree/devel/cloud/openstack)
In core we already have quantum_* modules that are out of sync with neutron_* modules.

Overview:
- 6 files to merge: neutron_floating_ip, neutron_network, neutron_router, neutron_router_gateway, neutron_router_interface, neutron_subnet
- neutron_sec_group is a new module and that is a different story
- quantum_floating_ip_asssociate doesn't have corresponding Neutron module

Changes:
- neutron_floating_ip: conflict between port_network_name and internal_network_name options
- neutron_network: vxvlan support in openstack-ansible
- neutron_router: -
- neutron_router_gateway: -
- neutron_router_interface: -
- neutron_subnet: core module supports subnet name, openstack-ansible module supports host_routes

I can make a crude merge without preserving the commit history. There are at least two ways to do it
1. Copy the neutron_* files beside the quantum_* files and deprecate the quantum_* files in the future
2. Deprecate quantum_* modules right away so that we have only neutron_* files in place

So the question is, should I proceed with the plan?
I'm experienced with Python, but not experienced with Ansible so I don't mind if someone else does the merge.

Thanks,

2

You can deprecate modules without removing them just rename them to
start with an underscore (i.e. _quantum_network.py). This will
document them as deprecated but still make them available/usable for
existing playbooks.

You should also update the documentation in those modules to point to
the new ones that should be used instead to give users a migration
path.

3

I’m willing to work on this. I’d like to see them merged as well, and we use some of them at work, so very relevant to my interests.

ccing Monty as the openstack czar. He already has some plans for what to do with the openstack modules.

-jlk

4

Hey!

Sorry I’ve been slow on the draw since the OpenStack summit - I am finally on the plane on the way home right now - haven’t been there in about 6 weeks. So I owe everyone a write up of what we discussed there, what was agreed on and the overall strategy.

Short version:

  • all modules should go into upstream ansible repo

  • roles and similar should be hacked on in OpenStack/stackforge repos, and we’ll add some automation to publish them to galaxy

  • both want to be fully integration tested in OpenStack Infra CI - as well as as much as possible in ansible’s CI (we should get double coverage that way)

Module org thoughts:

  • modules tend to fall into two categories:

  • modules that are for people deploying/managing an OpenStack Cloud

  • modules that are for people deploying/managing workloads ON an OpenStack cloud

All of them want a prefix - so os_ or openstack_ similar to rax_ and gce_ (I think most have been more interested in os_)

Things for deployers, on the other hand, should be specific - so managing a keystone tenant should really be os_keystone_tenant - because guess what, you’re the deployer, you DO need to know what implementation choices you’ve made. There are currently three sets of modules in three different places that are working on this:

Before we suck any of these into ansible itself, we need to align on what we want the strategy to be, if any, in terms of convergence (there isn’t consensus on the topic, so blessing one in the ansible tree itself seems rather premature - but everyone in Paris seemed keen on reaching consensus) There’s going to be at least some pain all the way around no matter which direction we go because of the intent to prefix openstack things in the ansible tree - so ursula’s cinder_volume_group will want to become os_cinder_volume_group.

Things for end-consumers should not use code names, but should reference the concept. So, to spin up a vm, one would expect to use os_compute rather than os_nova. This allows for some richer implementation flexibility for things like images that might be able to use glance directly or might have to go through nova and that a user should not really need to care about. It also allows us to avoid getting screwed by name changes brought on by trademark issues. So far the list of openstack projects where this has happened is quantum/neutron, moniker/designate, reddwarf/trove and savana/sahara. We can also take advantage of the new aliasing feature in 1.8, but let’s just do it cleanly in the first place since these are the most likely to be used by sets of people not following the internal politics of OpenStack.

Initial reworkings with that in mind are up for review on extras, but it would be great to get a few more people to bang on them before we merge them:

https://github.com/ansible/ansible-modules-extras/pull/92

There were a couple of attempts to do this by morphing the old ones, but the structure of them was such that the code providing compatability with an old model (no keystone v3, no support for multi-cloud, etc) got so hariy so quickly that a clean break and a deprecation of the existing modules was discussed as the cleanest way forward. SO - the new modules should keep in mind that ansible has much stronger backwards compat stances than OpenStack upstream and need to be done with flexibility and the future in mind. We’ll carry the cost of maintenance of the old deprecated ones for a good while so that no one gets screwed.

I will be actively hacking on this again this weekend now that I’ll finally be home - but I’d REALLY love to crank on this with other folks. There is an #openstack-ansible channel where folks are hacking on stackforge/os-ansible-deployment

When I’m not crammed in a tiny seat on a plane, I’ll write this all up on a wiki to make sure we’re all good to go.

Monty

5

Thanks Monty! I’ll be part of the fun.

  • jlk
6

For reminders to those new to the thread, I have given Monty commit on the module repos and he’s “owner” on these puppies.

If we do develop alternative modules we need to deprecate the old ones by renaming them to start with “_” and adding the “deprecated: this is why this is deprecated” message to the module DOCUMENTATION doctoring in each module.

I’m happy to do whatever, and then we would plan on removing the deprecated versions in 1 or 2 releases.

We could also move the stuff in “/openstack” to “/openstack_legacy” to make it even more clear in the “cloud” module lists in the doc sections.

7

Good to hear that there is a plan. As a newcomer I will step aside and watch the progress, but I'm glad to help if you see my input valuable. I’m more interested in os_* modules rather than generic modules.

Next question, If I have something to contribute on a module that exist in all of the two/three/four repositories, should I 1. push to one of these or 2. wait for the refactoring?

Thanks,

8

I believe in OpenStack land the best thing to do would be to wait just a bit, as investment in the existing tree might not be applied going forward.

If you ever need to submit something to module_utils, I would submit that first and then link in the other PRs (in the other repo) to the dependent pull requests.

9

Hey Monty:

I’m a maintainer of the https://github.com/openstack-ansible/openstack-ansible-modules repo. I don’t actively hack on it these days, but I do merge in pull requests, and we still get some coming in. Happy to help in any way I can in this effort.

Take care,

Lorin

10

Hi,

Is there something I can help you with? I would like to commit the changes to upstream and I have three modules waiting already: neutron_port, keystone_group and nova_quota.

I see that there is an ambitious plan to make the code better and I agree that codebase needs major rework, but is there a small step where we could start? Waiting will rot code in all the repositories. For example, can you provide an example of ideal module. Using the example file we can refactor modules to follow the new guidelines and push them to core.

Thanks,

11

Absolutely! The second two definitely sound like deployer/admin modules, right? So first step I’d rename them to os_keystone_group and os_nova_quota. The first one, neutron_port, seems like something that a cloud end user would use, so let’s call that os_port or os_network_port. (see below for a question)

There is a pull with a bunch of modules in it:

https://github.com/ansible/ansible-modules-extras/pull/92

You’ll see that they share various amount of things.such as using the openstack documentation fragment, openstack_argument_spec and the get the cloud connections and client objects through the shade library: cloud = shade.openstack_cloud(**module.params)

I’m using these modules right now myself, so in some senses they’re “ready” - but what would be really useful is a few more eyes or hands on them. j2sol, for instance, just made a great point about server name being the idempotence key - so banging against these in anger would be fantastic so that we can make sure we’re not painting ourselves into a corner with them.

FOR INSTANCE: these modules have os_network, os_router_interface, os_router_gateway and os_subnet. Looking at neutron_port - do we think that should be os_port? or os_network_port. If it’s os_network_port, would that mean we think that os_subnet needs to be os_network_subnet? (that seems silly, but os_port seems a bit overly generic maybe) I don’t think we need a taxonomy per-se, but I’d love opinions from humans who are not me.

Sorry for the long delay on this one - I agree, let’s get it tied off and move forward.