Openstack modules

Archives Ansible Developer
1

I’d like to overhaul the openstack modules, fixing bugs, adding features and making them easier to use. Before I do so, however, I’d like to discuss my proposed changes so they don’t conflict with existing roadmaps (if any).

All modules:

  • Get default authentication info from environment, just like the openstack cli tools do
  • Allow auth via token+endpoint.
  • Remove dependencies on openstack clients libraries, and use REST instead.
  • Allow referencing objects by id or by tenant+name.

keystone_(token|role|tenant|trust|domain|endpoint|service):

  • implement

nova_compute:

  • make do_wait a boolean
  • make do_wait also functional for boots that happened outside this task.
  • malke securitygroups a list
  • implement rebuild
  • allow image_name as an alternative for image_id
  • allow network_name as an alternative for network_id
  • allow brlockdevice_mapping

nova_keypair

  • fix syntax errors

quantum_*

  • make it work for non-admins (e.g. for public cloud users)

quantum_router:

  • allow specifying external gateway
  • allow specifying attached networks

cinder_(volume|snapshot|backup):

  • implement

swift_(container|object):

  • implement
2

So this all sounds great and we’d definitely welcome upgrades to the OpenStack content.

A few comments:

  • I’m fine with auth standardization but existing methods should continue to work as is when doing this – and things should go into module_utils like we have done to standardize AWS and Rackspace auth – so code isn’t repeated between modules.

  • I don’t understand why you would want to not use the provided OpenStack client libs, that seems to be a step back, and a giant maintaince problem. Is there a deprecation of these that I’m not aware of? I’m open to discussing this but it doesn’t seem like the right thing to do if the libraries are well maintained.

  • I don’t see a “do_wait” parameter on http://docs.ansible.com/nova_compute_module.html, I’ll assume you mean “wait”, and in which case type=bool is reasonable in the argument spec.

  • +1 to security groups taking a list, it should also take a string. The “type=list” in the module argument_spec will allow for this as it splits on commas if given a string.

  • +1 to having name options in addition to id options. Be sure to use “mutually_exclusive” in module utils to check that a confusing combination isn’t used.

3

Actually Openstack suggest developing against the REST API rather than using the client libs, presumably because they try and keep the REST API as stable as possible so there shouldn’t be as many changes required as the openstack versions change. (see https://wiki.openstack.org/wiki/SDKs definition 5 and https://wiki.openstack.org/wiki/APIChangeGuidelines last argument in “No we really mean it!”)

I would actually like to see a bunch of tasks for deploying OpenStack itself (rather than deploying clients onto OpenStack). Unfortunately not something I am currently working on at work at the moment but if I can get the time I might start working on that…
There are some interesting dependencies between different pieces that would give me some experience working with Ansible that I might not get otherwise.

Of course, once you have your OpenStack deployment in place you do need to be able to manipulate it and I am not saying that the OpenStack modules do not have a place, just not at the level that I currently have had to deal with OpenStack.

Adam

4

"Actually Openstack suggest developing against the REST API rather than using the client libs, presumably because they try and keep the REST API as stable as possible so there shouldn’t be as many changes required as the openstack versions change. "

I see where that says the REST API is supposed to be stable (we say that too), but where does this say “don’t use our client libraries”?

“I would actually like to see a bunch of tasks for deploying OpenStack itself”

Modules are going to make sense for using OpenStack, I don’t think modules are needed for deploying it.

There are several example roles for that sort of thing. I will freely admit ours in github.com/ansible/ansible-examples is not super well updated, but patches are welcome.

5

"Actually Openstack suggest developing against the REST API rather than using the client libs, presumably because they try and keep the REST API as stable as possible so there shouldn’t be as many changes required as the openstack versions change. "

I see where that says the REST API is supposed to be stable (we say that too), but where does this say “don’t use our client libraries”?

They don’t say it exactly, they do say… “We encourage developers to develop against OpenStack REST API. There will be many tools and applications which favor the REST API over our libraries or command line interfaces.” My reading of “encourage developers to use the REST API” is that they would suggest that rather than using the python client library. However, I admit I could easily be reading nuances into their writing that aren’t intended.

“I would actually like to see a bunch of tasks for deploying OpenStack itself”

Modules are going to make sense for using OpenStack, I don’t think modules are needed for deploying it.

There are several example roles for that sort of thing. I will freely admit ours in github.com/ansible/ansible-examples is not super well updated, but patches are welcome.

And no I wasn’t thinking that we needed an OpenStack deployment module, merely that it would be nice if some roles were available in Galaxy to make it easier for people to deploy OpenStack…

I’ll have a look at the examples and see if I can update/rewrite them as needed…

Adam

6

Yeah I don’t think they would do that, coding against the lib should make things much cleaner to develop, plus code is already there.

– Michael

7

I just had a quick look at the deployment stuff at https://github.com/ansible/ansible-redhat-openstack and I think I can use that as a good starting point…

Thanks for the pointer (sorry I’m now WAY off topic).

Adam.

8

Thanks, I’ll have a look at those modules. My main motive is is nova-network^Wquantum^Wneutron^Wopenstackclient. There are now 4 different iterations of that lib around, each with a disjunct install base, and a somewhat incompatible API. The ReST API is far more stable than the python APIs. Also, as mentioned elsewhere in this thread, OpenStack actually encourages developers to use the ReST APIs. And while they don’t recommend against using the python client libs, all examples, documentation, mailinglists and other sources work against the ReST API. Perhaps, some day in the future, the openstackclient lib settles down, and achieves reasonable stability and code quality. Unfortunately that day hasn’t come yet. Oviously it should be backward compatible. This goes for all of the features. I wasn’t aware of that option. Thanks for pointing it out.

9

“My main motive is is nova-network^Wquantum^Wneutron^Wopenstackclient. There are now 4 different iterations of that lib around, each with a disjunct install base, and a somewhat incompatible API. The ReST API is far more stable than the python APIs.”

Is it unstable, or is it just going through namechanges.

I don’t think the code should have to suffer through a more complex REST-layer implementation when this can be solved with some conditional “try/except” and “import as” kind of logic.

Further, switching to all REST API invalidates pretty much all of the work that was done previously, which I’m not quite ok with as there are people using these modules and it has a very high chance of breaking them.

It’s ok to encourage people to use REST APIs – that’s great – we do this to, but I don’t see anyone being discouraged from using the client libs.

They exist for good reason – to prevent a library from having to all re-invent the logic that wraps the REST API, handles error codes, and does the right thing.

This is really the great tradgedy of REST, for multiple-API calls, worklows, auth, every consumer has to reimplement something for themselves… thus having a library that uses the REST API is entirely what you want.

And that’s ok.

10

Is it unstable, or is it just going through namechanges.

Unfortunately both are happening; though it's mostly the latter.

I don't think the code should have to suffer through a more complex
REST-layer implementation when this can be solved with some
conditional "try/except" and "import as" kind of logic.

[snip even more valid points]

I guess you are right;
I'll just deal with the incompatibilities using an
'ask-for-forgiveness' approach.

But note that a new openstackclient lib is coming, which is entirely
incompatible with existing libs. When that's released, we'll have to do
a rewrite at some point, as the existing libs will no longer be
maintained.

11

There is work being done to create python-openstacksdk. There is supposed to be some discussion this week to get things kicked off. Looks like this will be held in #openstack-sdks on Wednesday at 1pm America/Central.

The idea is that the python-*client packages will no longer be the SDKs or python APIs. python-openstacksdk would become the dependency for all of the clients.

https://github.com/stackforge/python-openstacksdk
https://wiki.openstack.org/wiki/Meetings/PythonOpenStackSDK
https://wiki.openstack.org/wiki/SDK-Development

I won’t go into much more, but the rax modules will transition to this and away from pyrax once it is available.

12

At this point I’d like see our openstack modules use the new libs also (not the REST API).

This is entirely a code maintaince issue.

13

I've booked quite some progress, an I'd really like some feedback. This
isn't a merge request just yet, but I'm getting closer. My work can be
found here: https://github.com/CloudVPS/ansible

To do:
- keystone (trust|domain|endpoint|service)
- nova_compute blockdevice_mapping
- cinder (snapshot|backup)
- swift (*)
- documentation docstrings

Rejected:
- Fall back to REST

14

The link is just to a (temporary) fork of ansible. Can you add release notes of some sort to indicate what you HAVE added/modified?

thanks,

kesten

ps. if you are taking requests/suggestions, I was just about to integrate https://github.com/ansible/ansible-redhat-openstack/blob/master/playbooks/tenant.yml
into my deployment scripts. The first 5 steps use the ansible openstack modules. The last three tasks require shell or command modules to directly access the openstack controllers to setup security group rules etc.
It would be nice if that weren’t the case, so we wouldn’t have to delegate_to and include controllers in our host files etc.

15

I’m not sure how to add such notes to github, without also adding those to a future merge request. Suggestions are welcome

My changes so far:

  • Al modules now get their authentication information from the environment, just like the openstack cli tools do.
  • Where possible, objects (volumes, networks, routers, etc.) can be referenced both by name and by ID.
  • nova_compute has received a general cleanup, and now accepts its parameters in much more intuitive forms.
  • nova_compute now allows vm rebuild
  • nova_compute now allows volume attachments.
  • cinder_volume allows creation and deletion of volumes.
  • quantum is now usable for non-admins (e.g. public cloud users)

Your suggestion to allow security group management is a good one. I’ll add it to the list.

16

I've made a merge request for easy review:
https://github.com/ansible/ansible/pull/6445

17

I’ve been using Koert’s cinder_volume and nova_compute improvements in daily work now and would recommend them.

I think some care should be taken to standardize between rax and non-rax openstack modules for naming of things. Do i report doc issues the same way as regular code issues via github?
I believe believe the nova_compute needs some work to make it idempotent. If a vm is already existing then a volume will not get attached to it even if it is in the arg_spec. Generally, quite useable.

kesten