Currently, I am working with AWX to simplify a part of our deployment process, which involves two use cases:
Launching one, several, or all machines in an inventory group.
Launching a machine on AWS that automatically triggers an Ansible playbook via a provisioning callback embedded in the machine’s user data.
However, I have encountered several issues that I am struggling to resolve:
Credential Limitation: I am dealing with two regions, each requiring a different PEM file. AWX allows only one PEM per execution. Therefore, if I select deployment in both regions, it fails because AWX cannot use two PEM files simultaneously.
Workflow and Template Configuration: To address the above issue, I created a template and a workflow that uses a survey to collect the region values, allowing the selection of one or both regions. The workflow includes a verification step like this:
---
- name: Verify if useast is in the selection
hosts: localhost
gather_facts: no
tasks:
- name: Check if useast is selected
fail:
msg: "useast not selected"
when: "'useast' not in regions"
register: useast_check
- name: Debug output for useast
debug:
msg: "useast is selected"
when: "'useast' in regions"
If the verification passes, it executes the template with the correct PEM. If both regions are selected, the same template is executed with different PEMs, which works. However, this solution introduces additional problems:
The provisioning callback works for one region but fails for the other because it calls a template associated with a specific PEM.
The templates include a region variable in the extra-vars, which can be either useast or eucentral. Workflow Execution Marked as Failed: The previous verification has an issue. If I select only one region, the check for the other region fails, and although the deployment works correctly for the selected region, the overall workflow execution is marked as failed.
My goal is to execute a single template or workflow, but I am facing two main obstacles:
A. Single Region Credential: I thought about creating a custom credential that includes both PEMs, but I am unsure how to implement it. Ideally, it should select the correct PEM based on the survey variable or a similar mechanism.
B. Region Variable in Extra-vars: The region variable should be dynamic based on the survey. However, the survey returns a list that can contain two values, requiring the workflow to execute twice, once for each region, which seems unviable.
Could you suggest a solution to this problem? Your guidance would be invaluable, and I thank you sincerely for your time and expertise.
What problems do you face if you have two different templates in this workflow? Something like template A pointing to workflow A template B pointing to workflow B, for example. You can then use the limit field on the job template to select the desired region.
I have no experience with running AWX with anything AWS related, so I can’t offer any specific advice. However, I do have a more abstract suggestion to give.
Create a playbook (and potentially a role if it makes sense) and job template that is abstract enough to run against any region. By design, this playbook should fail against any region without sufficiently defined variables passed at execution (not in the playbook itself). You can pass a vars file on cli for testing purposes.
In AWX, create a job template with the appropriate prompts to successfully run the playbook against any 1 region.
Then create a workflow template that runs against all regions by adding the previously created job template once per region you need to run against. Then you can add a survey to the workflow template for any vars that might apply globally to all regions.
Thank you for your suggestions. I’d like to provide some clarifications and details regarding my current setup and the issues I’m facing.
I have implemented exactly what you suggested: using two different templates in the workflow. It works, but it increases maintenance since the templates are essentially the same, only differing by the PEM file and the region variable in the extra-vars. The goal was to simplify the process, not to add more maintenance overhead. Additionally, with the check I shared earlier, if both regions are selected in the survey, it works fine. However, if only one region is selected, one of the checks fails, and while the template executes correctly for the selected region, the workflow is marked as failed due to the failed check, which is the expected behavior but not desirable.
I am migrating a setup that was previously executed on localhost and has the region variable deeply integrated. Extracting it would require a lot of work. Currently, the survey asks for regions in a list format, and the variable expects an array, causing issues.
When I trigger the execution from an AWS instance user data, I can modify the extra-vars so the region variable is not a limitation, but the PEM file is. I can only select one PEM file per template execution, so one of the two regions will not work.
When I launch the execution from AWX, I have it controlled to deploy in one zone or the other, or in both. In this case, the problem is that if I launch in only one zone, the check will cause the workflow to be marked as failed even though everything is correct
In summary, my main issues are:
Credential Limitation: Only one PEM file can be used per template execution, which restricts the functionality when dealing with multiple regions.
Workflow Execution: If only one region is selected in the survey, the workflow is marked as failed due to the failed check for the other region, even though the execution for the selected region is successful.
I appreciate your guidance and look forward to any further suggestions or solutions you might have to address these challenges.
@alvaroc20
Hello, I apologize for not being able to respond for a while since you first reached out via DM (Thanks for moving the topic to here!).
I think I can provide some information that might be helpful.
By using Custom Credential Types, you can pass arbitrary strings (such as the contents of a PEM file) to the execution environment as files, environment variables, or extra_vars.
This means you can pass all the PEM files for each region into the execution environment as files.
In addition, the ansible_ssh_private_key_file can be dynamically set within a playbook task using the set_fact module. Example:
- name: set ssh key for us east
ansible.builtin.set_fact:
ansible_ssh_private_key_file: ssh/useast.pem
when: region == 'useast'
- name: set ssh key for us west
ansible.builtin.set_fact:
ansible_ssh_private_key_file: ssh/uswest.pem
when: region == 'uswest'
Although I haven’t tested this myself, these should allow you to dynamically change the ansible_ssh_private_key_file based on the extra_vars provided at callback execution time, ensuring that the correct key file is used for SSH connections in subsequent tasks.
If the above is feasible, you probably won’t need to use the workflow.
However, if you still want to continue using the workflow, you can place some nodes after “On Failure” of the job template to ensure success, which would prevent the entire workflow from failing. It’s a bit roundabout, though.
I’ve been trying to test the custom credential setup for a few days, but I haven’t been able to get it to work. I created a credential type with the following configuration:
I then created a credential of this type, added the PEM files in the respective fields, and updated the template to deploy using this new credential. However, I encountered the following error: Failed to connect to the host via ssh, which indicates that it is not working correctly.
Could you provide any insights into what I might be doing wrong or what I might be missing?
Regarding your second suggestion, it would be ideal, but the issue is that it would only allow execution for one region at a time, not both simultaneously. Am I mistaken? Additionally, I think that path to the PEM file is needed, but if the PEM is stored in AWX credentials, how can I reference it correctly or verify its contents?
Thank you very much for always responding to my messages.
By passing region: east or region: west as extra_vars, we can confirm that the PEM file from the custom credentials for correct regions is used to connect remote host via SSH.
PLAY [all] *********************************************************************
TASK [EAST | Update user] ******************************************************
ok: [demo]
TASK [EAST | Update SSH private key] *******************************************
ok: [demo]
TASK [EAST | Test connection] **************************************************
changed: [demo]
TASK [EAST | Dump test connection] *********************************************
ok: [demo] => {
"test_connection.stdout": "I'm on EAST region!"
}
PLAY [all] *********************************************************************
TASK [WEST | Update user] ******************************************************
ok: [demo]
TASK [WEST | Update SSH private key] *******************************************
ok: [demo]
TASK [WEST | Test connection] **************************************************
changed: [demo]
TASK [WEST | Dump test connection] *********************************************
ok: [demo] => {
"test_connection.stdout": "I'm on WEST region!"
}
Notes
While testing, I found a few issues:
In custom credential types, we can specify ssh_private_key as format, but it wasn’t very practical because it couldn’t keep line breaks in the key file, even though an error occurs if there is no header line.
The documentation introduces awx.filename, but it must be tower.filename instead.
