Looking for an AWX + AKS howto for dummies

Get Help
,
1

Hello,

I am looking for an AWX on AKS howto for dummies. At the moment my test lab is running on K3S and that’s running fine, but the production environment should run on AKS and I can t get things working.

I have zero knowledge of Kubernetes, and I’ve tried (lord, have I tried) to understand it but it’s all Chinese to me. I have accepted that I will never understand k8s.

So I’m looking at a howto that describes the following items in detail (so not ‘create the ingress’, but 'Use this and this command to create the ingress, changing parameter X and Y for your situation).

  • How to create the AKS cluster (preferrably as private cluster)
  • How to create the Postgresql database (preferrably as private database)
  • How to install AWX
  • How to integrate AWX with a loadbalancer

I tried copying the K3S configuration (thank you Kurokobo for your excellent guide!), and everything seems to come online, but I can’t reach the web interface so I’m doing something wrong.

Joost

2

No one?

The documentation for AWX is horrible. Take for instance the basic install guide. Line 1 already gives an error:

https://ansible.readthedocs.io/projects/awx-operator/en/latest/installation/basic-install.html

git clone git@github.com:ansible/awx-operator.git

This results in:

Cloning into 'awx-operator'...
git@github.com: Permission denied (publickey).
3

Try git clone https://github.com/ansible/awx-operator.git.

Kubernetes is complicated, the cloud is complicated, AKS is kubernetes on the cloud, it’s going to be very complicated. You need to be familiar with all sorts of complexities such as private endpoints, network segments for the cloud. For kubernetes, you need to familiarize yourself with deployments, services, ingresses, custom resource definitions. Asking for AWX + AKS for dummies is like asking for brain surgery + quantum physics for dummies. This tends to be why we get paid a lot in our field, if it were easy, they wouldn’t need us.

Sorry you’re having a tough time, but all these technologies move fast and documentation is often the last priority for most people trying to keep our heads above water. Kurokobo’s stuff is great, and while there may be mistakes in documentation, you’re going to have to troubleshoot on your own. There won’t be a guide out there when a user comes to you and asks why their job is failing.

If you’re getting started, it may be better to continuing to experiment with a local k3s instance. That’ll help you learn these concepts for AWX/kubernetes on a smaller, more digestible scale. Then uhhh do something with Azure since that’s a beast unto itself. Then combine the two and then job hop for a 40% salary increase.

4

Yes, I know how to clone a git repository. But if the documentation can’t even get a basic thing like cloning a git repo correct, how can I trust it for the more complex stuff?

In order to drive a car, I don’t need to know how a combustion engine works.

I would love to start with k3s and Kurokobo’s tutorial, but unfortunately I inherited an environment which runs on AKS and the person who installed it (and then left the company) left ZERO documentation. I’m trying to rebuild the environment in a test AKS cluster but can’t even get the most basic things, like a working ingress, working, because the documentation is written for people who understand combustion engines, and who understand that if the documentation says ‘valve’ it could also mean ‘piston’, ‘spark plug’, or something else.

5

AKS and ingresses can go so many ways it’s almost impossible to consolidate it all down to anything that might be considered easy, in my opinion. I imagine you don’t want your AKS environment publicly accessible, right? You probably also want it to be able to reach internally networked systems as well. If so, then you’ll need a private endpoint. In order to get a private endpoint, you’ll need a subnet in your subscription assigned to a VNET. You need to know how many IP addresses AKS consumes so that your subnet is large enough to accomodate it, but not too large that you’re wasting internal IP address spaces. You need to have some type of network pipe setup between your org and Azure that allows that internal connectivity, whether it be an express route or a VPN (this is where my Azure knowledge gets fuzzy). Then, your network guys will need to slice off a network segment allocated for Azure.

Once all that is done and dusted, you can set up an ingress that redirects to AKS. I tend to use CNAMEs, so you’d create a CNAME in whatever DNS provider controls DNS in your org that points to your AKS cluster’s name.

I have no idea how anyone would write a “for dummies” version of all this because so much of it depends on business rules and/or what decisions your org has made.

environment which runs on AKS and the person who installed it (and then left the company) left ZERO documentation

I don’t blame you for being frustrated, it sounds like you got screwed on this one. Ask for more money during review time, squeaky wheel gets the grease.

But if the documentation can’t even get a basic thing like cloning a git repo correct, how can I trust it for the more complex stuff?

I understand what you’re saying, however, another way of looking at it could be: if the reader is unable to solve a git clone issue, how are they going to be able to route an ingress through a VNET peered to the company from Azure?

6

The git clone operation was clearly written for someone who has a stored ssh key for cloning (documentation frequently is limited in this way - it’s written by people who did the work for themselves, and is up to the reader to adapt as necessary). If you do not have a stored key then you need to use a username and password over https. This is git 101.

If you are looking for a step by step instruction manual on AKS and Azure, good luck, and let me know when you find one.

To mcen1s point, if you aren’t able to work around this or recognize the difference for step 1 then I don’t think you are going to have much luck with AKS or AWX.

1 Like
7

Deploying AWX on Azure AKS

I deployed AWX on Azure AKS, and I found that it is not significantly more complex than deploying other pods on AKS.

Prerequisites

Before getting started, it’s essential to have a good understanding of Kubernetes.

Pro Tip

I recommend deploying a VM in your AKS subnet and installing all necessary management tools on it, such as:

  • Azure CLI
  • kubectl
  • git

From this VM, you can then run the AWX Operator

Configuration Complexity

The complexity of the configuration depends on your specific environment. It can be simple or complex, depending on your requirements and setup.

8

I managed to get everything working. Operator in my test environment is now 2.19.1 and AWX is now 24.6.1. All jobs seem to run OK.

Next problem: I see the following in the kubernetes logs for the task pod:

2025-03-11 09:23:32,445 ERROR    [42fad36abd6347e0b05c55e8a3dced39] awx.main.scheduler Failed to list pods for container group AWX - Timeout Workaround-5
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/scheduler/kubernetes.py", line 28, in list_active_jobs
    for pod in pm.kube_api.list_namespaced_pod(pm.namespace, label_selector='ansible-awx={}'.format(settings.INSTALL_UUID)).to_dict().get('items', []):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15823, in list_namespaced_pod
    return self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 15942, in list_namespaced_pod_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '884eb906-50fc-478f-8f08-93b8256308ff', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '8c87fe40-0bbd-44cf-abe8-5b13ccfacb2f', 'X-Kubernetes-Pf-Prioritylevel-Uid': '37fa4d23-016b-4bbb-86e4-3c2a5825b70b', 'Date': 'Tue, 11 Mar 2025 09:23:32 GMT', 'Content-Length': '283'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:awxtest:awxtest\" cannot list resource \"pods\" in API group \"\" in the namespace \"awx-test\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

I searched for this error and found a few blogs that explained a role/rolebinding should be created for the service account. I created the following role and rolebinding and applied it:

kind: Role
metadata:
  name: pod-reader
  namespace: awxtest
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: awxtest
subjects:
- kind: ServiceAccount
  name: awxtest
  namespace: awxtest
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

But this didn’t solve the problem.

Questions:

  • Is this really a problem, or just a cosmetic error?
  • If it’s a problem, how can this be solved?