Installing collection dependencies in awx-task?

Archives AWX Project
, , ,
1

I am trying to use the azure.azcollection Galaxy collection with awx 15.0.1, and it has quite a lot of azure libraries as dependencies. As far as I can tell, nothing inside awx will do anything to install those dependencies. To make matters worse, the base awx install does seem to include older versions of some of the dependencies, so some things work and some don’t.

I started building my own awx image, and updating ansible-requirements.txt.in to reflect the collection’s requirements instead of ansible 2.9’s but that doesn’t build fully.

What is the intended way to do this?
Also, what is the intended way in general to get new binaries or libraries into the stock awx container? (with k8s - so I guess some kind of init container)

If I need to create a seperate virtualenv for this, what else needs to be in it for awx to use it? (I guess parts of ansible at least) And again, how do you initialise it without needing to maintain a fork?

Thanks,

Howard

2

Having spent a day on variations of virtualenv and requirements.yaml, it seems that I couldn’t actually get a newer version of azure.azcollection to run in awx. I had a choice between missing libraries (azure.mgmt.privatedns is not in the ansible venv but required by azure.azcollection) or a strange bug where the azure_rm_keyvaultsecret* modules take 20 minutes to do anything (but same playbook takes seconds on my system).

Can you actually use different versions of collections in different projects (or different from the bundled ones) with awx? The collections folder has no version component in the path. I have to manage the collection dependencies in a virtualenv manually, which would be doable with a virtualenv per project, I guess.

Does anyone else use this stuff?

3

Hi Howard

I wouldn’t recommend trying to modify the awx-task or -web images. These are a large part of the backbone to AWX.
You’ve hit the upstream vs. stable-but-older versions, which has always been a problem with Azure in my experience.

Eventually something called Execution Environments will come along and do what you want much better. AWX already uses something called ansible-runner for a standardised way to do ansible runtime, and coupled with something called ansible-builder, they will do the collections and pre-reqs for you inside a container build.

For now, you need to use virtualenvs. I found this excellent write up which went through a lot of your pain. Checkout the update at the bottom which will make things easier: https://fixes.co.za/ansible/awx-giving-a-project-a-custom-virtual-env-in-container-based-install/#update-use-the-custom_venv_dir-variable
Rather than doing a rebuild/replace of awx-task, it’ll bind mount the config into the container.

You can then assign the venv to the project, org or JT. Each venv can have a different collection version. I’d leave the default venv alone and create your own as required.

Phil.

4

Thanks for the reply, Phil! But there are some issues with this:

The bind-mount thing looks like something docker-compose specific - but we're running on Kubernetes. My intention if I ever got this working was to do something similar with an init-container and a PVC.

As I mentioned elsewhere, you actually can't put venvs outside of /var/lib/awx/venv - the API won't let you register them. You can add new ones in there, which is what I ended up doing, but not elsewhere. I guess this is something that has changed, since the Tower manual recommends you do this too, but the 15.0.1 API doesn't allow it. Anyway, I have my venv in /var/lib/awx/venv/local, with all the dependencies for azcollection 1.2.0 in it and registered with awx.

I can verify that the project sync IS getting the right collection version and stuffing it into a project-specific cache directory. I can specify that my project runs with the appropriate venv, and confirm that in the job run metadata, but then I get errors that are basically impossible to debug because they're in temporary files.

AttributeError: module 'azure.mgmt.resource.resources._resource_management_client' has no attribute 'VERSION'
in /var/lib/awx/.ansible/tmp/ansible-tmp-1607684279.7399247-880-87049114153030/AnsiballZ_azure_rm_resourcegroup.py

The error is mentioned in a closed azcollection github ticket about the wrong versions of things being run (loaded from galaxy vs bundled in ansible).

And sure enough, in the debug logs:

Using module file /usr/lib/python3\.6/site\-packages/ansible/modules/cloud/azure/azure\_rm\_resourcegroup\.py

with a fully-qualified module name in my playbook.

Naturally, I can't recreate this on my desktop :slight_smile:

What version of awx does current Tower correspond to? I'm curious to try this "mature and stable" life...

5

The bind-mount thing is for kubernetes as well, as it’s a Linux feature, I use it with Code Ready Containers (CRC) which is effectively Openshift.
If you look in awx/installer/roles/kubernetes/defaults/main.yml , you’ll find it being set. Whether it’s being used or theres a bug is another matter.

I think you can’t put it outside of /var/lib/awx because of container runtime permissions that’s all. If you try to use /opt or /tmp it works. The awx-task and awx-web containers run as awx, which is non-root but in the root group. It doesn’t have permissions to create subdirs most likely. This doesn’t happen in a non-container install like most Tower deployment as you can hop onto the platform and mkdir.

What version of ansible are you using with that collection? Have you tried others? I’d suggest 2.9.15 as a min unless stated otherwise.

The Tower and AWX versions don’t match up necessarily. AWX is pure upstream so you can expect this fluidity and change.