Request to improve RDS module to use boto rds2

Hi folks,

The goal was to provision an rds instance using ansible, which was easy enough using this module: http://docs.ansible.com/rds_module.html

Unfortunately this module doesn’t have the option to allow the RDS to pass in publicly_accessible and ec2 defaults making it publicly accessible to false. We’d like this seed our data on instance creation, proper security groups are used ofcourse.

I tried modifying the module to include publicly_accessible where needed but it looks the boto rds doesn’t support it:

<localhost> REMOTE_MODULE rds size=10 password=VALUE_HIDDEN publicly_accessible=true command=create instance_type=db.m1.small instance_name=prod-rds2 engine_version=5.6 aws_access_key=AKIAJKMT2FVVHTPMEI6Q aws_secret_key=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 region=us-west-1 vpc_security_groups=sg-905fb6f5 db_engine=MySQL subnet=prod-rds username=admin <localhost> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009 && echo $HOME/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009'] <localhost> PUT /tmp/tmpDEylEv TO /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds <localhost> EXEC ['/bin/sh', '-c', u'LC_CTYPE=C LANG=C /usr/bin/python /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds; rm -rf /home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/ >/dev/null 2>&1'] failed: [localhost] => (item={'username': 'admin', 'subnet': 'prod-public-rds', 'db_engine': 'MySQL', 'name': 'prod-rds2', 'instance_type': 'db.m1.small', 'password': 'omitted_from_mailing_list`', ‘engine_version’: 5.6, ‘size’: 10}) => {“failed”: true, “item”: {“db_engine”: “MySQL”, “engine_version”: 5.6, “instance_type”: “db.m1.small”, “name”: “prod-rds2”, “password”: “omitted_from_mailing_list”, “size”: 10, “subnet”: “prod-public-rds”, “username”: “admin”}, “parsed”: false}
invalid output was: Traceback (most recent call last):
File “/home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds”, line 2048, in
main()
File “/home/iain/.ansible/tmp/ansible-tmp-1402373676.78-265376028601009/rds”, line 487, in main
result = conn.create_dbinstance(instance_name, size, instance_type, username, password, publicly_accessible, **params)
TypeError: create_dbinstance() got an unexpected keyword argument ‘publicly_accessible’

`

Boto does have rds2 which does seem to support this: https://github.com/boto/boto/blob/develop/boto/rds2/layer1.py

Is anyone interested in creating an rds2 module in ansible for this? Is this the proper place to request such a module or are they generally user provided?

Thank you,
iain

It seems very non-Ansible-way to have another copy of the module that just uses a different protocol.

Couldn’t this just be changed up to support the new parameter?

–Michael

Hi Michael,

Thanks for the reply! That makes sense, I’m still learning the ansible/correct way :slight_smile:

Do you think we should we pursue modifying the boto rds interface to support the param or should we modify the ansible rds module to use rds2 (which seems to support option groups and a couple other things like publicly_accessible).

I don’t really understand & couldn’t find any history as to why boto added “rds2” instead of iterating on the existing rds interface:

https://github.com/boto/boto/tree/develop/boto/rds

https://github.com/boto/boto/tree/develop/boto/rds2

Cheers,
iain

This I don’t know.

Maybe so other RDS using folks would have input.

Digging up an old thread (as I came across https://github.com/ansible/ansible/issues/7564 looking at tagging RDS instances).

It looks like the rds2 module is automatically generated so that it keeps up to date with upstream AWS capabilities, and this change broke backward compatibility.
http://boto.readthedocs.org/en/latest/migrations/rds_v1_to_v2.html

Do we have a minimum supported boto version? We could rewrite the RDS module to use rds2, but that might affect users on earlier boto versions. Supporting both seems like a headache. However, I suspect we could just use rds2 for the features that just aren’t in rds (so make a boto.rds2.RDSConnection if we need to do instance tagging or publicly available).

I’m hoping I can use the ec2_tag module to work around my problem but it’s worth discussing in more general terms - we can deprecate old Ansible conventions, but when do we deprecate old boto libraries?

Will

“we can deprecate old Ansible conventions, but when do we deprecate old boto libraries?”

Generally each module is responsible for yelling if it tries to do something it cannot.

But boto is obviously not something we release, so we can’t “deprecate it”, we can only raise an error when the version is too old.

I am ok with forcing folks to update their boto from time to time, especially how AWS usually forces the issue on it’s own occasionally by not working with older versions.

–Michael

Ah-ha!
This might very well be the cause for my issues: https://github.com/ansible/ansible/issues/7958 & https://github.com/ansible/ansible/issues/7952

I’m going to say rds vs rds2 is unrelated to your two issues, unfortunately.

I’ve updated 7958 - my refactoring* of the rds module won’t currently affect this behaviour (although it could if I understood the requirements better).

7952 is part of a different module (rds_params_group) that I currently have no plans to touch.

  • I say refactoring, git diff thinks it’s a complete rewrite, even though I haven’t touched some lines at all (mostly the module declaration). I am reusing a lot of the existing logic.

Will

Thanks for checking that.
You’re totally write about the #7952; was thinking about something else there.

I’ve rewritten rds to be able to use boto.rds and boto.rds2. The latter is only six months old, and insisting on it for all RDS users might be a little disruptive.

Removing boto.rds support would greatly simplify the results, but I’m happier with the new structure of the rds module than with the old structure. I could probably remove a lot of the additional classes that I’m using to unify the rds and rds2 code were we to insist on rds2 only.

My tests have been reasonably extensive (and I’ve greatly improved rds test coverage in general) but I’d welcome further tests.

https://github.com/ansible/ansible/pull/8834

Will