Digital Ocean v2 module

I’ve finally gotten around to implementing a working version of the digital ocean v2 module. Currently, only the droplet API works–I’m now moving on to SSH keys and domains. Since this is my first module and I’m new to Python, I was hoping someone could have a look at my implementation and let me know if everything checks out and is in the ansible spirit.

First of all, this module depends on the Skiff library: https://github.com/Shrugs/skiff

Here’s the v2 droplet module: https://github.com/rvangundy/ansible/blob/module-digital_ocean_v2_api/library/cloud/digital_ocean_droplet

Thanks!

Hi Ryan,

I was just talking about how this would be nice today.

I’d appreciate testing from those who use DO - but I now this has been a popular request, I think mainly to get “slugs” going, I don’t know what other features people may want.

It would be nice if “exact_count” type semantics like the AWS module could also be there.

Is this mostly argument-for-argument compatible in your view?

I’m not sure about Skiff yet because of the low fork count - but also it’s new - and the existing dopy did need upgrades - does DO recommend any particular bindings?

–Michael

Summarizing some issues with the DO module, would like some feedback on how best to go about the implementation, especially from some other DO users.

1.) Library use. I’m currently using the Skiff library, https://github.com/Shrugs/skiff. It’s still a young library. The author is actively working on this and has been extremely responsive in terms of fixes. There’s good opportunity for some parallel development, but it’s worth assessing whether this is the right library to use or if it’s worth waiting for the dopy library to update to v2, or potentially abandon both and work directly with the API.

2.) v2 API maturity. The current API is still in beta and is lacking in a couple areas that would be convenient. There are two issues I’ve encountered so far. You can only refer to public images by slug, so if you’ve created a private image/snapshot, the API requires you to refer to it by ID which is not convenient for the user. I’ve addressed this by querying for all images available to an account and looping through until I find the image with the given name. The same issue applies to SSH keys. You have to refer to keys either by ID or fingerprint, neither of which is convenient. I handle this in a similar way by querying for all keys and looping through. In the case of both images and SSH keys, my implementation allows for you to use ID, name, slug, and fingerprint for keys–it will return whatever it finds. Is the Ansible module a good place to implement this extra level of usability? Could the performance hit resulting from these queries be an issue? Should these features be pushed back in to the Skiff library as contributions there?

3.) v1 backwards compatibility. The current digital_ocean module has some unusual functionality. You can call it either using “command=SSH” or “command=droplet”. If you run in SSH mode, it behaves exactly like the digital_ocean_sshkey module. When setting up SSH keys for a new droplet, from what I can tell there’s no functional difference between running the digital_ocean module twice using each mode, or running the digital_ocean_sshkey module before running the digital_ocean module. Additionally, backwards compatibility won’t be entirely possible because of the use of a single token for authentication vs. the client_id and api_key in the previous API. My solution is to create a totally new digital_ocean_droplet module and leave the current digital_ocean module as-is. This is also in better alignment with the way the DO v2 API is organized. Would like to hear thoughts on how best to migrate to the v2 API, whether to implement separately as digital_ocean_droplet, and whether to leave out the “command” argument in favor of exclusively using the digital_ocean_sshkey module for working with keys.

4.) New v2 API features. The v2 API has several more features. In particular, it supports actions. There are image actions and droplet actions. These are pretty useful, such as rebooting or powering off a droplet. I’m imagining these being implemented using an “action” parameter in the module. This is low-priority, but something to consider going forward. If the v2 module implements the “command” argument, will it be confusing to also have the “action” argument?

Thanks for any feedback!

Another area to consider in the v2 DO module. Other cloud modules use arguments such as “count”, “exact_count”, etc. These could be of use in the digital_ocean module as well.

Original post from me that was direct to Ryan since he missed a reply-to-all:

"Based on the v2 API and also how some of the other cloud modules are organized, this module is now called “digital_ocean_droplet”

I’d like to avoid a module name change here, and instead replace the existing module, keeping existing inputs. We could make the image_id parameter undocumented and alias to “image” of course. The goal would be to not surprise people by removing a module name and getting a lot of questions :slight_smile:

Can you elaborate on what specifically you see mixed together with the keys bits?

Launching with a key is pretty standard for the cloud modules, while another module to upload a key is also fair.

When you say extended the basic functionality of the API, I imagine this is from extending Skiff? Could those patches go back upstream? If so, we may wish for this to hold until skiff pushes an update on those features.

(Just trying to coordinate…)

Summarizing some issues with the DO module, would like some feedback on
how best to go about the implementation, especially from some other DO
users.

1.) Library use. I'm currently using the Skiff library,
https://github.com/Shrugs/skiff. It's still a young library. The author
is actively working on this and has been extremely responsive in terms of
fixes. There's good opportunity for some parallel development, but it's
worth assessing whether this is the right library to use or if it's worth
waiting for the dopy library to update to v2, or potentially abandon both
and work directly with the API.

Directly with the API might be reasonable. I don't know the relative
complexity of what we need to do, but I'm a little scared because this is
so new and you've listed some problems with it.

2.) v2 API maturity. The current API is still in beta and is lacking in a
couple areas that would be convenient.

Ok this is interesting and makes me think we can hold off a bit maybe.

There are two issues I've encountered so far. You can only refer to public
images by slug, so if you've created a private image/snapshot, the API
requires you to refer to it by ID which is not convenient for the user.

I think the most common request right now is just public slugs, but it
might be they don't plan to do this at all - and that maybe ok. It might
be worth asking DO to weigh in on this issue.

I've addressed this by querying for all images available to an account and
looping through until I find the image with the given name. The same issue
applies to SSH keys. You have to refer to keys either by ID or fingerprint,
neither of which is convenient. I handle this in a similar way by querying
for all keys and looping through. In the case of both images and SSH keys,
my implementation allows for you to use ID, name, slug, and fingerprint for
keys--it will return whatever it finds. Is the Ansible module a good place
to implement this extra level of usability? Could the performance hit
resulting from these queries be an issue? Should these features be pushed
back in to the Skiff library as contributions there?

Seems fine for Ansible to do this and not out of the ordinary relative to
things we might do in other modules.

Not concerned with performance as we are talking about a very small number
of lookups.

3.) v1 backwards compatibility. The current digital_ocean module has some
unusual functionality. You can call it either using "command=SSH" or
"command=droplet". If you run in SSH mode, it behaves exactly like the
digital_ocean_sshkey module. When setting up SSH keys for a new droplet,
from what I can tell there's no functional difference between running the
digital_ocean module twice using each mode, or running the
digital_ocean_sshkey module before running the digital_ocean module.

Thanks for highlighting this. In this case I'm fine with dropping the
command parameter by undocumenting it and making it raise an error when
used with the value "ssh" about how to fix it. The command=droplet should
continue to work.

We don't like verbs in ansible modules, so this needs to go.

Additionally, backwards compatibility won't be entirely possible because
of the use of a single token for authentication vs. the client_id and
api_key in the previous API.

My solution is to create a totally new digital_ocean_droplet module and
leave the current digital_ocean module as-is.

Ideally, we need to go with one module.

Are you saying they changed the API auth system so the old auth system
doesn't work with the new API? That's kind of crap ... but we'll have to
deal with it. We should make it so each DO module takes both if we can -
but if we can't, seeing the level at which this has been executed I am ok
with perhaps deprecating the old module and putting a scary note in the
docs after all.

And we can delete that old module after 2-3 releases.

This is also in better alignment with the way the DO v2 API is organized.
Would like to hear thoughts on how best to migrate to the v2 API, whether
to implement separately as digital_ocean_droplet, and whether to leave out
the "command" argument in favor of exclusively using the
digital_ocean_sshkey module for working with keys.

Sounds like I'm ok with the above, but I'd like all the modules to accept
the new auth mechanisms.

4.) New v2 API features. The v2 API has several more features. In
particular, it supports actions. There are image actions and droplet
actions. These are pretty useful, such as rebooting or powering off a
droplet. I'm imagining these being implemented using an "action" parameter
in the module. This is low-priority, but something to consider going
forward. If the v2 module implements the "command" argument, will it be
confusing to also have the "action" argument?

This maybe needs to be a digital_ocean_action module to keep it from
polluting the provisioning modules too much. We should also make sure
common DO code lives in a common module snippet, like we have for ec2, to
avoid repeats.

I know there is a lot going on here, do you feel ok with taking all of this
and kind of consolidating it either into one PR, or emailing when all the
related PRs are done so we can review/execute together?

I think the worst would be if it were half-done and unclear when to use
which, or when auth was different between different module types.

Has DO said they plan to retire v1 on any particular date?

I ideally want to see all of our modules support exact_count.

There are some that are weird - like google - where you have to supply individual names. This is a pain on the user.

However in Rackspace, this was worked around, so it determines the names for you - and so while the Rackspace API doesn’t do an exact_count, it was done in the module, so you have that feature available - which is pretty cool stuff.

Yeah, I like adding a little “sugar” at the module level to make things consistent with the ease-of-use of the rest of Ansible. Just because a provider doesn’t offer a particular feature, doesn’t mean it shouldn’t be implemented. And in the future, if the provider offers that feature, update the module code to use it.

With regard to the different authorizations. Yes, they changed how they authenticate. I think that it would be really easy to make it so that depending on the authentication scheme you use, it infers which version of the API to use. Almost all the other arguments can remain the same and it will be entirely backwards compatible.

I generally like the idea of dropping the library dependencies and just writing the API calls. That way, running against different versions of the API is trivial.

This definitely expands the scope of what I planned on getting in to, but this seems like the right approach. I’ll start by formalizing the spec, probably by working through the docs. Another member of my team is well versed in Python. I might pull in his help and see if we can knock this out, it would be nice to have.

This sounds excellent – thanks very much and keep us posted – happy to help with questions as things come along!

FYI the core issue has been moved to the ansible-modules-core project: https://github.com/ansible/ansible-modules-core/issues/209

Ryan,

How are things coming along with this? Have you made much progress? Be keen to help in anyway I can (although final stages of my PhD come first over next couple of months)!

Cheers,

Steve