Conflicts while building AWX-EE custom image

motorbass · November 28, 2023, 2:18pm

I’m currently building AWX EE custom image with the latest available but it seems latest versions got some conflicts with pip modules. My custom image was working great in July (awx-ee version 22.X 23.X)

I used to set collections/modules I need in requirements files as below. However, I got some conflicts with pip module I didn’t set (so I assume there are “internal” or managed by AWX-EE base image) and I didn’t know how to get rid off them.

I use ansible-builder for all of this task.

Here’s my execution-environment.yml :

---
version: 1

build_arg_defaults:
  EE_BASE_IMAGE: 'quay.io/ansible/awx-ee:latest'

dependencies:
  galaxy: requirements.yml
  python: requirements.txt

additional_build_steps:
  prepend: |
    RUN cat /etc/os-release

  append:
   RUN whoami

Here’s the content of requirements.yml

---
collections:
-  name: nutanix.ncp
-  name: community.hashi_vault
-  name: arubanetworks.aos_switch
-  name: community.general

and requirements.txt :

hvac

The error message I got is the following. I read some pip modules are in conflicts but I don’t know where those modules have been configured/asked to be installed.
To get a workaround, I set hvac module to a very lower version than 1.2.X but error message is the same.

The user requested requests>=2.4.2 and ~=2.26.0
210.3     azure-core 1.28.0 depends on requests>=2.18.4
210.3     awxkit 13.0.0 depends on requests
210.3     hvac 1.2.1 depends on requests<3.0.0 and >=2.27.1
210.3
210.3 To fix this you could try to:
210.3 1. loosen the range of package versions you've specified
210.3 2. remove package versions to allow pip attempt to solve the dependency conflict
210.3
210.3 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
------
Dockerfile:54
--------------------
  52 |     COPY _build/requirements.txt requirements.txt
  53 |     RUN $PYCMD /output/scripts/introspect.py introspect --sanitize --user-pip=requirements.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
  54 | >>> RUN /output/scripts/assemble
  55 |
  56 |     # Final build stage
--------------------
ERROR: failed to solve: process "/bin/sh -c /output/scripts/assemble" did not complete successfully: exit code: 1

Also, I tried to remove azure-core cause I didn’t need it ,directly in execution-environment.yml but it fails cause module isn’t installed yet in the prepend part.

RUN python3 -m pip uninstall azure-core

About the solution provided by pip, I don’t know how I can loosen/remove a package version that I didn’t ask for in my requirement file.

210.3 1. loosen the range of package versions you've specified
210.3 2. remove package versions to allow pip attempt to solve the dependency conflict

I also tried to set a requirements.txt empty but still got the same kind of error.

My question is : How can I set version of pip module that seems to be embedded in AWX-EE image ?

motorbass · November 28, 2023, 3:16pm

Finally found out the root cause : nutanix.ncp collection got a requirements.txt file that is responsible for asking ‘requests’ ~=2.26.0

I’ll try to override that or found a workaround.

EDIT:
After looking at ansible-runner debug and execution-environment.yml it seems that this file is used because collections installed are exactly the same (+ those I set in my execution-environment.yml file) but I have no idea how to override it. (for instance, delete all azure_* collection which I don’t need)

thedoubl3j · November 28, 2023, 6:45pm

Apologies if I am misinterpreting the ask/comments but you could build your own EE rather than use ours/build on top of. This is the one we ship with purely because those things are included out of the box in AWX (azure for example because of the credential).

You can build one that has only the things you need an use that and awx can run things in that EE. All the base line requirements to run ansible are included in every ansible-builder image so you don’t have to worry about package clashing like you see here if its something you don’t need.

motorbass · November 29, 2023, 7:52am

Hi @thedoubl3j

Thanks for your comment. I wasn’t aware I can build a fresh new EE instead of build on top.

Correct me if i’m wrong but, it means :

I should create my own awx-ee BASE_IMAGE with a Dockerfile and launch build with docker cli
use ansible-builder with this new image with requirements I need right ?

(even if I assume ansible-builder raise a docker build command ?)

If you got an example (probably something like this ?)

motorbass · November 29, 2023, 1:07pm

Finally found this great documentation , I succeed to build an image from centos9:stream with only requirements I need

Something weird is, if I keep nutanix.ncp in requirements.yml, it still goes to conflict.
If I remove it, but install it manually (in a docker run for instance) everything works perfectly.

I’ve tried to add the following instruction in any append phasis to workaround this bug, but it doesn’t seems ot work as collection isn’t installed (but image building is working)
RUN ansible-galaxy collection install nutanix.ncp

Thanks a lot for your advices @thedoubl3j

thedoubl3j · November 29, 2023, 7:38pm

yup, those are the docs and interesting. I wonder if something else requires requests @motorbass could look at what it (ncp) depends on collection wise if it has any. really also wish that collections don’t set hard deps but soft if possible.

The other collections might have clashing dep needs as well. might need to segment off ncp into its own EE since it is the one with the hard dep.

motorbass · November 29, 2023, 8:11pm

I’ll double check tomorrow and share my files cause I made a lot of changes. awx.kit seems to ask for requests too (i assume awx collection in requirements.yml ask for awx kit in pip)

About the ncp requirements, i’ll also have a double check, but this morning i saw that the requirements.txt file is like 1 y.o and no updates since… even if the collection is regurarly updated.

The other collections might have clashing dep needs as well. might need to segment off ncp into its own EE since it is the one with the hard dep.

Yes that’s what i was thinking too. Else, keep this custom EE without ncp, but add the collection as a requirements in my Ansible projects when needed , it means it will be automatically installed while the project requests it, and it should work.

I’ll have to give some try tomorrow

motorbass · November 30, 2023, 9:55am

What do you mean when you’re talking about hard deps vs. soft deps ?
Does it means that some ansible collection may get hard coded deps or version into their own requirements ?

My requirements.txt only contains hvac without settings any version (so i assume it will get the last available)
My requirements.yml contains :

---
collections:
-  name: community.hashi_vault
-  name: awx.awx
-  name: kubernetes.core
-  name: ansible.posix
-  name: ansible.windows
-  name: arubanetworks.aos_switch
-  name: community.general
-  name: nutanix.ncp

After enabling again nutanix.ncp again into requirements.yml, here’s the conflicts :

#28 41.46 The conflict is caused by:
#28 41.46     The user requested requests~=2.26.0
#28 41.46     awxkit 13.0.0 depends on requests
#28 41.46     hvac 2.0.0 depends on requests<3.0.0 and >=2.27.1
#28 41.46     The user requested requests~=2.26.0
#28 41.46     awxkit 13.0.0 depends on requests
#28 41.46     hvac 1.2.1 depends on requests<3.0.0 and >=2.27.1

From what I saw, hvac module import ‘requests’ but without settings any specific version too
GitHub : hvac/hvac/blob/main/hvac/aws_utils.py) but also /hvac/hvac/blob/main/hvac/adapters.py

(sorry for this format but I received some message from the community that I can’t put those link into my post, don’t understand why)

So I assume as nutanix.ncp GitHub : nutanix/nutanix.ansible/blob/main/requirements.txt), that’s the faulty one.
There’s nothing about hvac but probably it’s mandatory for another python requirements ?

What’s weird is after building image without nutanix.ncp i can successfully add it manually without any conflicts. I don’t understand the conflicts while building and out-of building.

In any case, i’ll stay right there with my image and just add collection requirements in my Ansible projects, if it works, it works

gwmngilfen · November 30, 2023, 10:18am

Apologies for the system flagging you as spam @motorbass - I’ve added the Ansible ReadTheDocs site to our URL whitelist now

felixfontein · November 30, 2023, 12:19pm

The requirements.txt in nutanix.ncp is very likely not meant to be an EE dependency file (its contents suggest that it is a requirements file for folks who want to work on the collection itself), but ansible-builder still treats it as one by default. Unfortunately there is no way to turn that off right now, except for the collection itself (the collection would have to include an EE description file which doesn’t reference this requirements.txt).

Unfortunately ansible-builder’s documentation is not very helpful for collection authors; the page Collection-level Metadata — Ansible Builder Documentation mentions meta/execution-environment.yml, but provides no information how that file should look like. I don’t know how I figured it out in the past, but that file can look like this: https://github.com/ansible-collections/community.crypto/blob/main/meta/execution-environment.yml - mentioning another Python dependency file (even an empty one) than requirements.txt in the root would fix the problem for this collection.

thedoubl3j · December 5, 2023, 9:43pm

apologies for my delay, was on holiday so away from my machine.

hard and soft here mean uncapped or capped. so if something is a hard dep, what I mean is that a specific package version can only be used. ex: requests == 2.2.6 (I don’t know their versions, this is just an example). Soft being an uncapped version requirement, ex: requests >= 2.2.5, 2.3.0 <=, essentially not locking people in to a specific version if something else might need requests as well (which you are finding out the hard way when something does).

glad you got things working though, apologies for the confusing terms. will try to be more clear in future posts.

motorbass · December 6, 2023, 8:00am

Thanks for the details
Don’t feel sorry, I’m not accustomed to some syntax so here’s my fault too.

Thanks again for your help and precious sharing !