AWS just doesn't pull down collections

Archives AWX Project
, , ,
1

Good day,

Yet another thread on installing collections. Running AWX 19 in K8s. I’ve tried many different things. I have a requirements.yml under the collections directory. My playbook references the collection as well. Yet AWX just doesn’t even try to install the collection, it keeps telling me it can’t find the module I’m using. Outside of AWX, the collection I need installs just fine with ansible-galaxy collection install cisco.aci. My requirements.yml looks like this:

collections:

  • name: cisco.aci

and my playbook like this:

2

Oh and I do have “enable collections download” and “enable role download” checked under Settings.

3

And kubectl logs -f for awx-ee and awx-task show nothing related to pulling down collections from requirements.yml

awx-ee:

INFO 2021/04/26 22:48:25 Client connected to control service
INFO 2021/04/26 22:48:25 Control service closed
INFO 2021/04/26 22:48:25 Client disconnected from control service
INFO 2021/04/26 22:48:27 Client connected to control service
INFO 2021/04/26 22:48:27 Control service closed
INFO 2021/04/26 22:48:27 Client disconnected from control service
INFO 2021/04/26 22:48:27 Client connected to control service
INFO 2021/04/26 22:48:32 Client connected to control service
INFO 2021/04/26 22:48:32 Control service closed
INFO 2021/04/26 22:48:32 Client disconnected from control service
ERROR 2021/04/26 22:48:32 Error updating status file /tmp/receptor/awx-b5f6cf4d4-4rrgl/8oHAqn9Q/status: open /tmp/receptor/awx-b5f6cf4d4-4rrgl/8oHAqn9Q/status.lock: no such file or directory.

DEBUG 2021/04/26 22:49:08 Sending service advertisement: &{awx-b5f6cf4d4-4rrgl control 2021-04-26 22:49:08.314775763 +0000 UTC m=+68645.603393494 1 map }
DEBUG 2021/04/26 22:50:08 Sending service advertisement: &{awx-b5f6cf4d4-4rrgl control 2021-04-26 22:50:08.314902671 +0000 UTC m=+68705.603520367 1 map }

awx-task:

2021-04-26 22:52:11,279 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(1)
2021-04-26 22:52:11,525 INFO [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.commands.run_callback_receiver Event processing is finished for Job 130, sending notifications
2021-04-26 22:52:11,525 INFO [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.commands.run_callback_receiver Event processing is finished for Job 130, sending notifications
2021-04-26 22:52:12,100 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.tasks job 130 (running) finished running, producing 10 events.
2021-04-26 22:52:12,110 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.analytics.job_lifecycle job-130 post run
2021-04-26 22:52:12,282 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.analytics.job_lifecycle job-130 finalize run
2021-04-26 22:52:12,291 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(2)
2021-04-26 22:52:12,291 DEBUG [-] awx.main.commands.run_callback_receiver JobEvent.objects.bulk_create(2)
2021-04-26 22:52:12,305 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch task 3c3c5599-3525-4351-a45c-7357a488ab2c starting awx.main.tasks.update_inventory_computed_fields([1])
2021-04-26 22:52:12,323 WARNING [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch job 130 (failed) encountered an error (rc=None), please see task stdout for details.
2021-04-26 22:52:12,326 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch task a56fbbd1-ea71-4c6d-8029-b426804a956b starting awx.main.tasks.handle_work_error([‘a56fbbd1-ea71-4c6d-8029-b426804a956b’])
2021-04-26 22:52:12,340 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.tasks Executing error task id a56fbbd1-ea71-4c6d-8029-b426804a956b, subtasks: [{‘type’: ‘job’, ‘id’: 130}]
2021-04-26 22:52:12,438 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.models.inventory Going to update inventory computed fields, pk=1
2021-04-26 22:52:12,454 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch task a56fbbd1-ea71-4c6d-8029-b426804a956b starting awx.main.tasks.handle_work_success([])
2021-04-26 22:52:12,459 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch task 743029d7-cb96-4909-824f-a3a19fc285ff starting awx.main.scheduler.tasks.run_task_manager()
2021-04-26 22:52:12,464 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.scheduler Running Tower task manager.
2021-04-26 22:52:12,491 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.models.inventory Finished updating inventory computed fields, pk=1, in 0.053 seconds
2021-04-26 22:52:12,508 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.dispatch task 50689ea0-cc32-4267-9805-4da095881ab1 starting awx.main.scheduler.tasks.run_task_manager(*)
2021-04-26 22:52:12,510 DEBUG [0bd71e067b8e4a89b56d6a2b2c24567b] awx.main.scheduler Running Tower task manager.

I’m going to try to build a custom EE with the collection installed.

4

I solved the issue by building a custom EE with the collections required. Follow the steps here, it worked like a charm for me: https://www.linkedin.com/pulse/creating-custom-ee-awx-phil-griffiths/

5

Can i ask for some details on how you did this? I am trying to do the exact same thing. I am new to this style of install with AWX. I used the old (pre 18) versions. I was able to install it using the quickstart guide. . Once I create the execution environment, push it up to a registry, do I then consume that my vanilla install of AWX and the execution environment then modifies my AWX? I am a little confused.

6

There is an execution environment section

7

Thanks for the response, I am not sure what you mean by there is an execution environment section. I guess maybe I dont understand what an Execution Environment is used for. At the end of this process I need an AWX Web Ui installed and running that has the hashicorp_vault module installed so when i run playbooks that use it they will work as expected.

The way I THINK it works

Install AWX using quickstart guide.
Build a custom EE environment
Push up to a registry
Login to AWX and add my custom EE
It somehow modifies my AWX. (I dont see how this would work, but its all I can come up from the small amount of documentation i am able to find)

8

I have learned a bit more about EE. I have learned that my playbooks will run USING these EE, not AWX like in the past.

9

AWX spins up a container with your EE when the time has come to run a playbook. Ansible really runs inside your EE container. You simply point your job templates or entire project to any EE you want. You can see it’s indeed your EE running when observing the stdout stream while a playbook is executed, if your EE contains collections or a version of Ansible which differs from the default EE.

10

I see that a workflow job template runs each job in a separate EE container, is it possible to configure either at workflow or other place, to run all the jobs in the same EE pod/container?

11

No. We rely on containers for isolation (we used to use bubble wrap for this which I think used cgroups under the hood, but I digress).

12

Thank you. In thase case we need to use ‘set_facts’ module to pass output from a job as input to the next jobs in a workflow template, is that right?
Let me know if there are any other recommended methods to do the same.

13

Typo, it should be ‘set_stats’ not ‘set_facts’.