Hello list,
I anticipate provisioning 10-20 VMs using Ansible, then assigning floating IPs to each, then waiting for SSH to become available for each VM. I would like to do this in parallel instead of serially. Specifically:
- Start the VMs, but don’t block
- Assign the IPs, but don’t block
- Wait on SSH until all VMs respond
I saw the nova_compute “wait: ‘no’” option, but when I use it I get a stack trace:
failed: [localhost] => (item=1) => {“failed”: true, “item”: 1, “parsed”: false}
invalid output was: Traceback (most recent call last):
File “/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute”, line 1490, in
main()
File “/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute”, line 266, in main
_create_server(module, nova)
File “/home/ubuntu/.ansible/tmp/ansible-tmp-1405028178.0-234314980043958/nova_compute”, line 194, in _create_server
private = [ x[‘addr’] for x in getattr(server, ‘addresses’).itervalues().next() if x[‘OS-EXT-IPS:type’] == ‘fixed’]
StopIteration
Perhaps I’m using it incorrectly:
- name: Launch cluster VM on Openstack
nova_compute:
name: “{{ os_username }}_cluster1”
state: present
login_username: “{{ os_username }}”
login_tenant_name: “{{ os_tenant }}”
login_password: “{{ os_password }}”
image_id: “{{ os_image_id }}”
key_name: “{{ os_username }}_controller_key”
wait: “no”
flavor_id: “{{ os_flavor_id }}”
auth_url: “{{ os_url }}”
user_data: “#cloud-config\nmanage_etc_hosts: true”
So, two questions:
- Am I using “wait” correctly?
- Should I use “wait” to get to my desired parallel VM launch, as described above, or should I use something else, e.g. “async”?
Thanks,
-Kurt
Well, “async” is totally a bust. I got a message:
`
fatal: [localhost] => lookup plugins (with_*) cannot be used with async tasks
`
I guess that error is because I put a “with_items” in there.
How does everyone else do this? I don’t understand how to loop asynchronously. See pseudo-code:
< start all 5 at once: >
< start up openstack host >
< assign it a floating ip >
< capture the floating ip >
< end when all 5 have a floating ip >
< wait for all 5 floating IPs to have an open SSH port >
So some of the provisioning modules, like AWS in particular, support spinning up “N” modules at a time by passing “count” or “exact_count”.
Rackspace I believe does this with manual looping (for now), but I could be wrong and that might have just been historical truth.
If the OpenStack API says we can launch 10 at once, it could be made to do similar things.
So should I make a custom module which loops over nova_compute asynchronously, and also assigns floating IPs?
Would such a module be useful to the wider community, or is it too specialized to contribute back?
“So should I make a custom module which loops over nova_compute asynchronously, and also assigns floating IPs?”
I’d first rather know where the openstack API allows simultaneous creation of N virtual machines of the same image type.
I expect the floating IP stuff is fast and a usual with_items loop isn’t a problem there, once those guests exist.
(Using neutron, I assume?)
There’s an undocumented min_count and max_count option in python-novaclient ServerManager.create() which looks like what you want [1].
-Kurt
[1] https://github.com/openstack/python-novaclient/blob/cc7364067f995bd952149cd172cebf083fe4dc99/novaclient/v3/servers.py#L427
To answer your other question about neutron: I use “quantum_floating_ip” to assign IPs on Openstack.
It would be very useful if these API calls could also be parallelized. I imagine a lot more time would be spent on 100 serial API calls than doing them in parallel.
-Kurt
Yep, I think we’d be open to that.
Is this a “pull requests welcome” situation? If so, I can try my best…
Opened two github issues:
Minor process note:
If you’re going to submit the code to try, don’t open the feature requests, just clutters up the queue.
If you aren’t, opening an RFE usually doesn’t mean the code fairies implement it anyway, since we’re likely going to attend to existing bug reports/PRs and feature PRs first
I dug a bit further. The API does allow min_count and max_count, much the way boto does for AWS.
When you submit a request with min_count, your instances are named -. That’s acceptable, though not ideal.
I’m taking a look at the Ansible ec2 module, and the code for boto instance-launching code looks very different than the nova_compute instance-launching code. I haven’t run it yet; I need to dig around to find my ec2 creds so I can run a test.
The Ansible ec2 module also allows one to assign public IPs while launching multiple instances. The Ansible nova_compute module does not permit this ATM.
Additions of new params to add IP spawning behavior would be reasonable.
(assign_public_ip, True/False, etc)
What might you prefer on names?
I found a different approach to this problem.
First create a cluster configuration file.
`
$ cat cluster.yml
I forgot to include a critical piece of this solution. Here’s how you invoke the task to tell Openstack to start the VMs:
`
- name: Create cluster
hosts: os_api
gather_facts: no
roles: [instantiate] # “instantiate” is a role that includes the Openstack startup task
`
Perhaps this kind of example should be included in docs for parallel execution in Ansible. Let me know if you want me to help with that.
-Kurt
So I currently have an inventory script that creates me “fake” devices which I run the nova_compute task against. This allows me to provision all the vms in parallel using the native parallelization that ansible provides which is nice cause I can set it up to only provision 10 - 20 vms at a time using the serial flag.
So this method provisions the VMs great, but since the “fake” devices do not have connection information I cannot use any other modules later in the playbook. I am trying to add these newly created VMs to a new group using the add_host module but I cannot get it to add more than one device to the running inventory.
I saw the fake device approach on another thread and liked the ability to use ansible to throttle my provisioning as well as choose the names for the vms. I was wondering though if it would be possible to update host with the connection information ( specifically the ansible_ssh_host, ansible_ssh_private_key_file, and whatever variable specifies how to connect to the host ) rather than creating new hosts.
I also am stuck trying to get add_hosts to get all of my hosts. I assumed that add_hosts would also run in parallel but after searching around all of the examples I have seen show it using a loop. If I need to use a loop is there a good way to get the registered output from nova into a list rather than per host? I could not figure out any way to use set_fact to append the per host values of the registered nova variable into a single variable.
I felt like this approach was pretty clean, but I am definitely open to other ideas. My goal is to be able to provision a scalability test environment so I am expecting to be provisioning 500 VMs in a private Openstack network.
Tony
I’m not sure why you think add_host needs to run in parallel, it completes in almost zero time since it’s just doing some internal inventory “math”.
That all being said, what we really want is OpenStack’s module having the equivalent of exact_count in AWS/Rax other (though the Rackspace implementation is a bit more low level).
Hey Kurt,
Do you happen to have a full example in a repo somewhere? I would be very interested to check it out.
Thanks in advance,
Bob