My team and I have been using AWX 13 installed in Docker on CentOS 8 for several years to back up our 300 network device configs. It still works great, but since it’s installed on CentOS 8, we decided to upgrade to the newest AWX on Ubuntu 20.04 LTS. Our new AWX 21.10 deployment is up and running and synced up with our project on our GitLab server. However, we’re having issues with the playbook that we were using in the old AWX deployment. We made some changes to it here and there since things work differently with AWX 21.10 and we’ve made some progress. The theme for the playbook is as follows:
- Reach out to the network device and pull the running configuration.
- Copy the running configuration to /var/lib/awx/projects/_16__network_device_configs within the minikube pod which is hosting AWX.
- Edit the file and remove portions of configuration which are irrelevant and would cause GIT to be updated on every push (timestamps for example)
- Copy ssh key from /etc/ssh/ssh_host_ed25519_key to /etc/ssh/temp_ed25519_key and create restricted permissions for the GIT push usage.
- Perform a “git pull” to sync the _16__network_device_configs directory with the master branch.
- Perform a “git add” to stage new files for committal.
- Commit the changed files.
- Push the files to GIT.
The problem that we’re having is that we get errors saying the directory doesn’t exist and our SSH key that we use to talk to Git doesn’t exist either. Obviously, the directory and the SSH key both exist otherwise we would have never been able to sync AWX with our project that exists on our Gitlab server. Based on what I’ve read, AWX is apparently running tasks from the playbook outside of the Minikube pod that hosts AWX.
Since AWX 21 has some fundamental changes to how it works, I’m looking for some guidance on how I might change our playbook to retain the same functionality as described in the theme above. Thanks for your advice.
I’ve attached the playbook file should anyone want to check it out.
(attachments)
ios_config_mgmt.yml (1.49 KB)
Yes jobs run in separate pods, and only a copy of the job’s private data dir (including the source github project folders) is passed in. So you cannot rely on referencing local files that are present on the control node.
is the playbook that is running also part of the _16__network_device_configs project? if so, the EE pod that is running your job can access the project files under /runner/project directory
if the playbook is NOT running from the _16__network_device_configs project, then you’ll have to find a way to mount an additional directory into this running pod. Look at the “Paths to expose to isolated jobs” setting in Settings > Job Settings in the app
https://docs.ansible.com/automation-controller/4.1.4/html/userguide/execution_environments.html#ee-mount-options
AWX Team
Yes, the playbook is part of the _16__network_device_configs project. Give me some time to investigate and try your suggestions out and I’ll report back. Thanks for your help on this.
The playbook is part of the _16__network_device_configs project. It appears that AWX is able to copy the project files from my gitlab server over to /runner and execute the playbook. It was able to successfully download the configuration files from the switches in question. However, it looks to me like the EE pod is unable to save the config file backups to the /runner/project directory. I’ve attached the playbook as well as the debug output from AWX. Thanks for your help.
(attachments)
job_445.txt (16.9 KB)
nxos_config_mgmt.yml (1.43 KB)
This error is repeated in the output.
“msg”: “Destination directory /var/lib/awx/projects/_16__network_device_configs does not exist”
Walter
Yes I know. I’ve confirmed the directory exists. I still get the error. I don’t know where to go from here.
/var/lib/awx/projects/_16__network_device_configs is not going to exist in the pod that is running your job. A copy of the project files (we call it private data dir) are located under /runner inside of the pod.
is your playbook hardcoding access to /var/lib/awx/projects/_16__network_device_configs? if so, that isn’t going to work
AWX Team
I understand all of that now, and yes my playbook “hard codes the directory”. I’ve actually attached the playbook if you want to look and see what I did. Again, this works on our old AWX deployment, but there have been fundamental AWX changes, which I get. I just don’t know how to make my playbook work with the new AWX changes. Anyone know of a good document that explains how to write playbooks so they’ll work with new AWX? Thanks.
I don’t see the playbook attached. Is it possible it got removed from your mail relay? Can you link to it somewhere? Maybe in a gist?
-The AWX Team
Hello,
Thank you for providing that additional information. You can refer to this PR: https://github.com/ansible/awx/pull/11659 to gain some more insight about this.
-AWX Team
From the github link that you posted, I believe that you’re suggesting that I use the “Paths to expose to isolated jobs” under ‘Job Settings’?
It says that this is a list of paths that would otherwise be hidden and will be exposed to isolated jobs.
Enter one path per line.
Volumes will be mounted from the execution node to the container.
The supported format is
HOST-DIR[:CONTAINER-DIR[:OPTIONS]].
Mine currently looks like this (default):
[
“/etc/pki/ca-trust:/etc/pki/ca-trust:O”,
“/usr/share/pki:/usr/share/pki:O”
]
I tried to expose my git repository directories on my AWX host to the container that is running the job, but I must be using the wrong syntax or something. I can’t get it to work:
“/etc/ssh/ssh_host_ed25519_key:/mnt3:rw”,
“/etc/ssh/temp_ed25519_key:/mnt3:rw”,
“/var/lib/awx/projects/_31__network_device_configs:/mnt3:rw”
I actually made a typo. It should have been:
“/var/lib/awx/projects/_16__network_device_configs:/mnt3:rw”
Is the concept to mount my AWX host local git hub repository and directories required for git auth to the container with read/write permissions so that it can run the playbook without any issue?
Anyway, it’s still not working.
After thinking about this more, we don’t think this is going to be feasible for what you are trying to do.
First, the .git folder is no longer pushed around to the execution node (so its not really even a repo anymore). Second, the required bits to run the playbook are zipped up and sent over to the execution node so there is no guarantee that /var/lib/awx/ will even be in the path any longer; it may be /tmp//<other_stuff>. That being said, you might want to attempt to modify your playbook to do a git clone, git add, git push directly from the target execution node. To do this, you will likely need to add a custom credential type to house your git credentials (one of the builtins might also work depending on your exact needs).
Let us know if this might be a good solution for you.
Thanks for your help.
I updated the playbook and did away with trying to write files to the local git folder. Now I’m just writing the files to the /tmp directory in the execution node. (see the attached new playbook). This has allowed me to get all of the way to the final step in the playbook without encountering errors.However, the problem still remains where the execution node doesn’t have access to the location where the ssh key is located and therefore cannot authenticate with gitlab in order to write the backup files to the gitlab repository. (see the attached error in JSON format).
regarding your suggestion:
“modify your playbook to do a git clone, git add, git push directly from the target execution node. To do this, you will likely need to add a custom credential type to house your git credentials”
I think I understand what you’re saying, but don’t have any idea on how to go about doing this. Do you have an example or a link to some documentation that might spell this out?
Am I going about this all wrong? Is there a better way to do this? What are other people doing? Surely there’s a best practice for getting the collected data from the execution node to a location somewhere where it can be used.
Thanks again.
(attachments)
nxos_config_mgmt.yml (1.1 KB)
Commit to gitlab error.txt (2.53 KB)
I created a custom credential type with a custom credential that goes with it. See attached screen shots. From here it appears that I need to write a new playbook file (and create an associated template in AWX that uses these custom creds) which will clone a particular repository from gitlab into the execution node. At that point, I’m assuming that the execution node will have what it needs to run the configuration backup job that I’ve been talking about in this particular conversation. Am I on the right track, or way off? Thanks.
(attachments)
yes that sounds right, let us know how it goes and if you run into any stumbling blocks
AWX Team
