Hi all,
I submitted a pull request to add more openstack support here: https://github.com/ansible/ansible/pull/8252
I was asked to open the topic to more discussion so… discuss I suppose!
Please feel free to contact me with any questions or concerns tylern@pistoncloud.com
Thanks
Tyler
Awesome! I’ve flagged this email to take a look at the PR!
-jlk
Definitely interested in upgrades, though I’d appreciate more motivations on changes outside of the code if possible, which was my comment about the larger patch set if you can elaborate a bit.
It probably isn’t neccessary to split them into billions of parts, but I would be interested in commentary on underlying bits and a little more about what you are doing with things. Going forward, splitting things into parts might make some things easier, should we decide to not take them all together.
I’ll let Jesse to review as I greatly trust his input on this.
Hi again everyone!
Has anyone had any time to review the pull? Please let me know if you have any questions or concerns.
Thanks
Tyler
Still waiting on info:
"Definitely interested in upgrades, though I’d appreciate more motivations on changes outside of the code if possible, which was my comment about the larger patch set if you can elaborate a bit. "
Not sure what you mean here can you elaborate? What do you mean by motivations to change outside code? Do you need me to explain things further?
Yep!
What I meant was I was hoping for an written explanation of your changes, rather than just trying to pull it all from the patch set code review, which seemed to have a lot of things all thrown together.
Without comments, that makes it very unlikely to be something we’d merge.
Even so, we may still request things broken out.
Ahh got ya, ok here goes:
Not sure who wrote the orginal but you’ll NEVER see this variable called login_username in ANY OpenStack documentation. So I changed it to the accepted os_username, os_password, os_tenant_name, os_auth_url
Just some general bugs, as well as adding boot from volume to nova compute
A LOT of quantum bugs, including very uneccessary use of the keystone module in most cases.
I tried to use the keystone module with no success, so I remade into three modules that work
Add cinder support
Add in volume attachments, security groups and other things
Add in basic swift support
Ahh got ya, ok here goes:
1) Fix login_username --> os_username.
Not sure who wrote the orginal but you'll NEVER see this variable called
login_username in ANY OpenStack documentation. So I changed it to the
accepted os_username, os_password, os_tenant_name, os_auth_url
So you're changing an internal variable or the name or the Ansible
parameter?
The change of the internal variable would be a refactor that just
complicates the review, any bugfixes should be applied first, followed by
any feature changes.
2) Nova keypair/ nova compute fixes
Just some general bugs, as well as adding boot from volume to nova compute
Please be specific.
3) Fixes to quantum
A LOT of quantum bugs, including very uneccessary use of the keystone
module in most cases.
Please be specific, these probably should be in a quantum specific pull
request with examples of how to trigger each of them and the nature of each.
4) Keystone moved into 3 modules
I tried to use the keystone module with no success, so I remade into three
modules that work
Pretty sure folks are using this one, do need specifics on how to trigger.
5) Add in cinder
Add cinder support
This should be a seperate pull request.
7) Add in many other nova features
Add in volume attachments, security groups and other things
This should be a seperate pull request, with documentation added for new
parameters and each parameter noting where it was added.
8) Add swift support
Add in basic swift support
As above...
Hi everyone,
Giving this another go, adding stuff in much smaller steps this time
https://github.com/ansible/ansible/pull/9075
Please let me know what you think.
Thanks
Tyler
Hi everyone,
I’m back!
https://github.com/ansible/ansible-modules-core/pull/11
Please let me know what you think.
Thanks
Tyler
Hi everyone,
I’m trying to get some of my changes through but I dont know if they’re being looked at or not. In the meantime please take a look at
https://github.com/tylernorth/ansible-modules-core
https://github.com/tylernorth/Openstack-Ansible-Test-Suite
Thanks!
Tyler
Hey Tyler,
I'm by no means an expert or authority in the Ansible community, but I
believe that one issue here is that you need to submit to extras first
before trying to get this in core. I'll certainly defer to others on that
point.
Also, on a side note, it appears there are three or four of us working on
this same sort of task 
Might be good to see if we can rally and do a
group effort on this.
John
Updates to existing modules should go where the modules actually lie, but we will leave what is a “better” OpenStack module to a bit of community consensus.
We would rather see seperate pull requests for specific fixes rather than wholesale roll-ups.
“Might be good to see if we can rally and do a group effort on this.”
Any activity on this front should be discussed on this list.
This is the place for it.
Curious to see if others on list are interested in this as well. Some thoughts/responses inline below.
Hi everyone,
Thanks for all of the comments, much appreciated. I’m going to continue to work on our testing infrastructure here, and I’ll be updating my branch of the core modules when possible. If anyone can find any bugs/any additions they can think of please let me know.
I plan on putting in separate pull requests for each commit I have once the ones I have open are merged or looked at.
Thanks!
Tyler
Hey everybody!
As John mentioned, I’ve been working on an updated set of OpenStack modules to support all of the new goodness - and also to be resilient enough to handle all of the node launching and management needs of the OpenStack Infra project (for context if you don’t know us, we manage somewhere between 10k and 20k dynamic hosts as well as a control plane that’s about 100 hosts - each of these are spread across 4 regions of 2 public clouds - so we’re kind of a pathological case.
In any case, that led to a lot of logic landing in the shade-ansible repo that really shouldn’t be in ansible because it’s not specific to ansible, but instead should be upstream. That led to the creation of the libraries shade and os-client-config. Dean is working on adding os-client-config support to python-openstackclient, and over the next cycle I’m going to take a stab at landing patches to all of the other python-*client libraries for it. I originally thought shade was just going to be a temporary landing place and all of it wants to go into python-openstacksdk, but actually I’ve come to believe that while we’ll eventually port shade to use openstacksdk on the backend when it’s ready, shade is really about being a task-oriented library “I want a VM with an IP address please” and less about being a general SDK. Anywho - the new modules are based on shade being able to handle all of the differences in openstack deployments.
At this point, the input signatures are pretty solid, and the new modules now support the clouds I have access to, as well as full end to end workflows like “give me a VM, with an IP address, and also some volumes, and please attach the volumes and then also it would be great to format and mount those in the VM” So it’s about time to start proposing things in to ansible-modules-extra.
https://github.com/ansible/ansible-modules-extras/pull/92
Main design points:
Main thoughts:
New modules should be named with a prefix and the common name for anything that involves consuming a cloud. For instance, os_compute rather than nova_compute. There are a few reasons for this - one is that there are several objects in an OpenStack cloud that might come from different places depending on cloud configuration, but the end user doesn’t really care about that. Images might be via nova api proxy or glance directly. Floating ips might come from nova or neutron. Security groups may be nova or neutron, etc. What you want as a consumer is “give me a floating ip”
You may also want to deploy OpenStack with ansible or to manage OpenStack with ansible. In those cases, you definitely do know that you’re deploying nova, as opposed to consuming the OpenStack Compute service. So for those, modules such as “os_cinder_quota” make perfect sense and splitting them out based on intended audience can allow for a clean experience.
New modules with deprecation of the old modules means we can properly and cleanly support things like keystone v3 without a crapton of compatability shims - and we can design for the future.
Same thing with inventory - unlike ec2 - the things you need from an OpenStack inventory span more services than nova - so what you really have is an openstack inventory.
With that said - my proposal is that we put the current modules on stasis and only accept bug fixes while landing the new modules. We can maintain the old modules in a deprecated manner for several releases, but focus effort on improvements and things like a detailed guide on new modules.
Thoughts?
Hi Monty,
I left a few comments but I’m confused as to why you are re-writing modules that already exist. This will probably be confusing to have multiple modules that do the same thing. Any info you can provide would be helpful.
Thanks
Tyler
So, there’s many reasons.
Some of them are just UX related, such as providing a good end-user design rather than one that is more aimed at admins and people who have implementation knowledge of their cloud. Things like the naming of os_image and having that be able to do nova or glance commands as appropriate, or security groups targeting either nova or neutron as appropriate are good examples.
That could be worked around though. The REAL reason is that in order to get the existing modules up to being broadly usable, there’s just a ton of work. Since the current modules are 100% untested and also don’t have a detailed usage guide, and since breaking backwards compatibility in a project like ansible is pretty evil, it seems to me that a clean break with a deprecation of the old modules and produce new ones with clear and understood semantics as well as verifiable integration tests would be a good choice.
I did a compatible pass:
https://github.com/ansible/ansible/pull/8924
(which also, btw, pulled in your cinder changes) - but the amount of compat code to support something that was of dubious operational quality in the first place - but which needs to be assumed is in use somewhere - it just too high of a cost to me.
There are also things, like some of the default values, that I have zero interest in maintaining compatibility with. Defaulting auth_url to http://127.0.0.1:35357/v2.0/, for instance, means that it’s quite impossible to do what you really want to do - which is to give the user an early warning if they have produced a crazy attempt at a config - and the number of people for whom that value is valid are, hopefully, far outweighed by the number of people for whom it is not. So while it makes it easier for someone hacking on a devstack install - or a product install - it makes it harder to do parameter validation for actual end users.
Finally, with config processing improvements like os-client-config being adopted by python-openstackclient and others upstream, having custom local config reading and auth processing logic that is different from and doesn’t consume similar artifacts is nothing but confusing. That the old system was less flexible AND more verbose as well means that attempting to maintain config semantics with the nova.ini files while adding it AND maintaining the compatibility with the set of currently expected default values becomes un-intelligible.
This plan is largely dependent on the recent work to be able to mark modules as deprecated - without that, I agree that having two sets of active modules will be very confusing.