Awx k3s offline installation

Get Help
,
1

I am attempting to install AWX using K3s in an offline environment. I have set up K3s successfully and have also configured a local container registry where I have pushed the necessary AWX, Operator, PostgreSQL, and Redis images. However, I am encountering difficulties in proceeding with the installation process due to unclear next steps.

Requested Help

I need assistance in understanding and completing the remaining steps for offline installation of AWX on K3s. Specifically, I am unsure about:

  • How to deploy AWX using these pre-pulled images in an offline k3s
  • Any additional configurations or steps required for the deployment process.

has anyone here had experience with deploying AWX on K3s using the repository at

GitHub - kurokobo/awx-on-k3s: An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.?

I would appreciate insights, tips, or documentation links that can help clarify the deployment process in an offline environment.

2

@smoohiii
Hi,

To install AWX Operator in the air-gaped cluster, to replace default images, you can specify private images under images in your kustomization.yaml in this document: Basic install - Ansible AWX Operator Documentation

To install AWX using AWX Operator, you can use some parameters to replace default images with your private images. Refer to the documentation to see available parameters: Deploying a specific version of awx - Ansible AWX Operator Documentation

1 Like
3

Hi Kurokobo,

thansk for your reply. Can you tell me where i can find the kustomization.yaml file? and also, you are refering to “some parameters”. Wich parameters do you mean?

4

Hi, please read the docs that I’ve provided first. There is an example kustomization.yaml and list of the available parameters.

2 Likes
5

Hi Kurokobo,

been away for a while, but still not working. For the operator i have placed the operator image and the kube-rbac-proxy in the kustimization.

  # Find the latest tag here: https://github.com/ansible/awx-operator/releases
  - /home/awxuser/awx-operator-2.19.0/config/manager/manager.yaml


# Set the image tags to match the git version from above
images:
  - name: gcr.io/kubebuilder/kube-rbac-proxy
    newName: localhost:5000/kube-rbac-proxy
    newTag: v0.15.0
  - name: quay.io/ansible/awx-operator:latest
    newName: localhost:5000/operator-2.19.0
    newTag: latest

# Specify a custom namespace in which to install AWX
namespace: awx
type or paste code here

is the resource correct? couldnt find the correct path in the documentation.

i have also tryed to manualy install with kubectl apply -k /config/default … the pod will be created, but then tries to pull images from quay.io and grc.

6

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/ansible/awx-operator/config/default?ref=2.19.1

images:
  - name: gcr.io/kubebuilder/kube-rbac-proxy
    newName: localhost:5000/kube-rbac-proxy
    newTag: v0.15.0

  - name: quay.io/ansible/awx-operator
    newName: localhost:5000/operator-2.19.0
    newTag: latest

namespace: awx

result:

$ kustomize build . | grep -E "\s+image:\s"
        image: localhost:5000/kube-rbac-proxy:v0.15.0
        image: localhost:5000/operator-2.19.0:latest
7

the file you provided does not work

kubectl apply -k .

error: accumulating resources: accumulation err=‘accumulating resources from ‘github.com/ansible/awx-operator/config/default?ref=2.19.0’: evalsymlink failure on ‘/home/awxuser/awx-operator-2.19.0/github.com/ansible/awx-operator/config/default?ref=2.19.0’ : lstat /home/awxuser/awx-operator-2.19.0/github.com: no such file or directory’: hit 27s timeout running ‘/usr/bin/git fetch --depth=1 GitHub - ansible/awx-operator: An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖 2.19.0’

when i put this as resource, it builds 1/1 pods.

  • …/awx-operator-2.19.0/config/crd
  • …/awx-operator-2.19.0/config/rbac
  • …/awx-operator-2.19.0/config/manager
  • …/awx-operator-2.19.0/awx-instance.yaml

I’m not sure if this is the correct approach. The guides indicate there should be 2/2 pods. If this is indeed the right way, then the next step is to install AWX, but I’m a bit stuck on that part. I found this topic and tried following it, but only the PostgreSQL pod appears. It’s showing problems with pulling the image, as it seems to be trying to pull it from the internet.

Events:
Type Reason Age From Message

Normal Scheduled 52s default-scheduler Successfully assigned awx/awx-postgres-15-0 to awx
Warning Failed 22s kubelet Failed to pull image “Quay”: failed to pull and unpack image “Quay”: failed to resolve reference “Quay”: failed to do request: Head “https://quay.io/v2/sclorg/postgresql-15-c9s/manifests/latest”: dial tcp 3.222.172.189:443: i/o timeout
Warning Failed 22s kubelet Error: ErrImagePull
Normal BackOff 22s kubelet Back-off pulling image “Quay”
Warning Failed 22s kubelet Error: ImagePullBackOff
Normal Pulling 10s (x2 over 52s) kubelet Pulling image “Quay”

Thanks so far

8

It’s definitely an issue with your environment. This occurs if you have installed kubectl or kustomize via Snap, or if the version of Git is very old.

If you want to proceed this way, specify config/default directory instead.

9 
kubectl get pods -n awx
NAME                                               READY   STATUS             RESTARTS   AGE
awx-operator-controller-manager-788b86c756-f26tc   2/2     Running            0          2m46s

The operator is operational, now i am having a hard time to get awx running. so in the main folder of awx-operator i have created awx.yaml

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  image: localhost:5000/ee
  image_version: latest
  image_pull_policy: IfNotPresent
  image_pull_secrets:
    - pull_secret_name
  redis_image: localhost:5000/redis:latest
  postgres_image: localhost:5000/postgres:latest
  ee_images:
    - name: my-custom-awx-ee
      image: myorg/my-custom-awx-ee
  control_plane_ee_image: localhost:5000/ee
  init_container_image: localhost:5000/ee
  init_container_image_version: latest
  init_projects_container_image: localhost:5000/centos:latest

So when I run kubectl apply -f awx.yaml, it first starts to create a pod named awx-postgres-15-0. However, this fails because it’s pulling from quay.io instead of the local repository. I tried modifying the StatefulSet, but that didn’t work either.

Events:
  Type     Reason                           Age                 From               Message
  ----     ------                           ----                ----               -------
  Normal   Scheduled                        118s                default-scheduler  Successfully assigned awx/awx-postgres-15-0 to awx
  Warning  Failed                           44s (x2 over 89s)   kubelet            Failed to pull image "quay.io/sclorg/postgresql-15-c9s:latest": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "quay.io/sclorg/postgresql-15-c9s:latest": failed to resolve reference "quay.io/sclorg/postgresql-15-c9s:latest": failed to do request: Head "https://quay.io/v2/sclorg/postgresql-15-c9s/manifests/latest": dial tcp 3.211.251.234:443: i/o timeout
  Warning  Failed                           44s (x2 over 89s)   kubelet            Error: ErrImagePull
  Normal   BackOff                          29s (x2 over 88s)   kubelet            Back-off pulling image "quay.io/sclorg/postgresql-15-c9s:latest"
  Warning  Failed                           29s (x2 over 88s)   kubelet            Error: ImagePullBackOff
  Warning  FailedToRetrieveImagePullSecret  17s (x5 over 119s)  kubelet            Unable to retrieve some image pull secrets (pull_secret_name); attempting to pull the image may not succeed.
  Normal   Pulling                          17s (x3 over 119s)  kubelet            Pulling image "quay.io/sclorg/postgresql-15-c9s:latest"

Am I missing something here? I added the postgres image to the kustomization.yaml file as well, but that didn’t work either

10

You have to explicitly specify all of redis_image, redis_image_version, postgres_image, and postgres_image_version.

In addition, image and image_version are for AWX image instead of EE.

11

thanks, that did the trick… so now i encounter the next problem.

Back-off pulling image "localhost:5000/postgres:latest"
  Warning  Failed   3m22s (x9 over 56m)  kubelet  Failed to pull image "localhost:5000/postgres:latest": rpc error: code = Canceled desc = failed to pull and unpack image "localhost:5000/postgres:latest": context canceled

I am able to pull localhost:5000/postgres:latest with Podman without any issues, but for some reason k3s won’t pull it. Any ideas?

12

Hi Smoohii,

I have been running AWX on K3s for a few years and I wanted to mention that there is an alternative to modifying the kustomization.yaml or AWX operator yaml. We actually followed this document Private Registry Configuration | K3s to tell K3s to redirect image pulls that are specified by AWX to a different location.

13

Hi BlueJ,

So I solved the issue mentioned above by changing from localhost:5000 to <node-ip>:5000 and also adding this as a mirror in the registries.yaml.

I’m kind of curious how you managed this without using a kustomization or awx.yaml? I still have to add the <node-ip>:5000 to my file.