# Awx k3s offline installation **URL:** \ **Category:** Get Help\ **Tags:** awx, kubernetes\ **Created:** [July 1, 2024, 2:32pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874 "2024-07-01T14:32:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [July 1, 2024, 2:32pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/1 "2024-07-01T14:32:48Z")
I am attempting to install AWX using K3s in an offline environment. I have set up K3s successfully and have also configured a local container registry where I have pushed the necessary AWX, Operator, PostgreSQL, and Redis images. However, I am encountering difficulties in proceeding with the installation process due to unclear next steps. #### Requested Help I need assistance in understanding and completing the remaining steps for offline installation of AWX on K3s. Specifically, I am unsure about: - How to deploy AWX using these pre-pulled images in an offline k3s - Any additional configurations or steps required for the deployment process. has anyone here had experience with deploying AWX on K3s using the repository at > [GitHub - kurokobo/awx-on-k3s: An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.](https://github.com/kurokobo/awx-on-k3s)? I would appreciate insights, tips, or documentation links that can help clarify the deployment process in an offline environment. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [July 10, 2024, 3:24pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/2 "2024-07-10T15:24:53Z")
@smoohiii Hi, To install AWX Operator in the air-gaped cluster, to replace default images, you can specify private images under `images` in your `kustomization.yaml` in this document: [Basic install - Ansible AWX Operator Documentation](https://ansible.readthedocs.io/projects/awx-operator/en/latest/installation/basic-install.html) To install AWX using AWX Operator, you can use some parameters to replace default images with your private images. Refer to the documentation to see available parameters: [Deploying a specific version of awx - Ansible AWX Operator Documentation](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/deploying-a-specific-version-of-awx.html) ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [July 17, 2024, 12:47pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/3 "2024-07-17T12:47:49Z")
Hi Kurokobo, thansk for your reply. Can you tell me where i can find the kustomization.yaml file? and also, you are refering to “some parameters”. Wich parameters do you mean? ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [July 17, 2024, 1:44pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/4 "2024-07-17T13:44:48Z")
Hi, please read the docs that I’ve provided first. There is an example `kustomization.yaml` and list of the available parameters. ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [October 1, 2024, 12:24pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/5 "2024-10-01T12:24:00Z")
Hi Kurokobo, been away for a while, but still not working. For the operator i have placed the operator image and the kube-rbac-proxy in the kustimization. ```auto # Find the latest tag here: https://github.com/ansible/awx-operator/releases - /home/awxuser/awx-operator-2.19.0/config/manager/manager.yaml # Set the image tags to match the git version from above images: - name: gcr.io/kubebuilder/kube-rbac-proxy newName: localhost:5000/kube-rbac-proxy newTag: v0.15.0 - name: quay.io/ansible/awx-operator:latest newName: localhost:5000/operator-2.19.0 newTag: latest # Specify a custom namespace in which to install AWX namespace: awx type or paste code here ``` is the resource correct? couldnt find the correct path in the documentation. i have also tryed to manualy install with kubectl apply -k /config/default … the pod will be created, but then tries to pull images from [quay.io](http://quay.io) and grc. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [October 1, 2024, 2:31pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/6 "2024-10-01T14:31:32Z")
`kustomization.yaml`: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - github.com/ansible/awx-operator/config/default?ref=2.19.1 images: - name: gcr.io/kubebuilder/kube-rbac-proxy newName: localhost:5000/kube-rbac-proxy newTag: v0.15.0 - name: quay.io/ansible/awx-operator newName: localhost:5000/operator-2.19.0 newTag: latest namespace: awx ``` result: ```bash $ kustomize build . | grep -E "\s+image:\s" image: localhost:5000/kube-rbac-proxy:v0.15.0 image: localhost:5000/operator-2.19.0:latest ``` ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [October 1, 2024, 2:45pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/7 "2024-10-01T14:45:24Z")
the file you provided does not work kubectl apply -k . > error: accumulating resources: accumulation err=‘accumulating resources from ‘[github.com/ansible/awx-operator/config/default?ref=2.19.0](http://github.com/ansible/awx-operator/config/default?ref=2.19.0)’: evalsymlink failure on ‘/home/awxuser/awx-operator-2.19.0/github.com/ansible/awx-operator/config/default?ref=2.19.0’ : lstat /home/awxuser/awx-operator-2.19.0/github.com: no such file or directory’: hit 27s timeout running ‘/usr/bin/git fetch --depth=1 [GitHub - ansible/awx-operator: An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖](https://github.com/ansible/awx-operator) 2.19.0’ when i put this as resource, it builds 1/1 pods. > - …/awx-operator-2.19.0/config/crd > - …/awx-operator-2.19.0/config/rbac > - …/awx-operator-2.19.0/config/manager > - …/awx-operator-2.19.0/awx-instance.yaml I’m not sure if this is the correct approach. The guides indicate there should be 2/2 pods. If this is indeed the right way, then the next step is to install AWX, but I’m a bit stuck on that part. I found this [topic](https://groups.google.com/g/awx-project/c/e_amqVXwROE) and tried following it, but only the PostgreSQL pod appears. It’s showing problems with pulling the image, as it seems to be trying to pull it from the internet. > Events: > Type Reason Age From Message > > * * * > > Normal Scheduled 52s default-scheduler Successfully assigned awx/awx-postgres-15-0 to awx > Warning Failed 22s kubelet Failed to pull image “[Quay](http://quay.io/sclorg/postgresql-15-c9s:latest)”: failed to pull and unpack image “[Quay](http://quay.io/sclorg/postgresql-15-c9s:latest)”: failed to resolve reference “[Quay](http://quay.io/sclorg/postgresql-15-c9s:latest)”: failed to do request: Head “[https://quay.io/v2/sclorg/postgresql-15-c9s/manifests/latest](https://quay.io/v2/sclorg/postgresql-15-c9s/manifests/latest)”: dial tcp 3.222.172.189:443: i/o timeout > Warning Failed 22s kubelet Error: ErrImagePull > Normal BackOff 22s kubelet Back-off pulling image “[Quay](http://quay.io/sclorg/postgresql-15-c9s:latest)” > Warning Failed 22s kubelet Error: ImagePullBackOff > Normal Pulling 10s (x2 over 52s) kubelet Pulling image “[Quay](http://quay.io/sclorg/postgresql-15-c9s:latest)” Thanks so far ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [October 1, 2024, 3:19pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/8 "2024-10-01T15:19:54Z")
> [@smoohiii](#): > > error: accumulating resources: accumulation err=‘accumulating resources from ‘[github.com/ansible/awx-operator/config/default?ref=2.19.0](http://github.com/ansible/awx-operator/config/default?ref=2.19.0)’: evalsymlink failure on ‘/home/awxuser/awx-operator-2.19.0/github.com/ansible/awx-operator/config/default?ref=2.19.0’ : lstat /home/awxuser/awx-operator-2.19.0/github.com: no such file or directory’: hit 27s timeout running ‘/usr/bin/git fetch --depth=1 [GitHub - ansible/awx-operator: An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖](https://github.com/ansible/awx-operator) 2.19.0’ It’s definitely an issue with your environment. This occurs if you have installed kubectl or kustomize via Snap, or if the version of Git is very old. > [@smoohiii](#): > > - …/awx-operator-2.19.0/config/crd > - …/awx-operator-2.19.0/config/rbac > - …/awx-operator-2.19.0/config/manager If you want to proceed this way, specify `config/default` directory instead. ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [October 3, 2024, 9:36am UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/9 "2024-10-03T09:36:22Z")
```auto kubectl get pods -n awx NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-788b86c756-f26tc 2/2 Running 0 2m46s ``` The operator is operational, now i am having a hard time to get awx running. so in the main folder of awx-operator i have created awx.yaml ```auto apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: service_type: nodeport image: localhost:5000/ee image_version: latest image_pull_policy: IfNotPresent image_pull_secrets: - pull_secret_name redis_image: localhost:5000/redis:latest postgres_image: localhost:5000/postgres:latest ee_images: - name: my-custom-awx-ee image: myorg/my-custom-awx-ee control_plane_ee_image: localhost:5000/ee init_container_image: localhost:5000/ee init_container_image_version: latest init_projects_container_image: localhost:5000/centos:latest ``` So when I run `kubectl apply -f awx.yaml`, it first starts to create a pod named `awx-postgres-15-0`. However, this fails because it’s pulling from [quay.io](http://quay.io) instead of the local repository. I tried modifying the StatefulSet, but that didn’t work either. ```auto Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 118s default-scheduler Successfully assigned awx/awx-postgres-15-0 to awx Warning Failed 44s (x2 over 89s) kubelet Failed to pull image "quay.io/sclorg/postgresql-15-c9s:latest": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "quay.io/sclorg/postgresql-15-c9s:latest": failed to resolve reference "quay.io/sclorg/postgresql-15-c9s:latest": failed to do request: Head "https://quay.io/v2/sclorg/postgresql-15-c9s/manifests/latest": dial tcp 3.211.251.234:443: i/o timeout Warning Failed 44s (x2 over 89s) kubelet Error: ErrImagePull Normal BackOff 29s (x2 over 88s) kubelet Back-off pulling image "quay.io/sclorg/postgresql-15-c9s:latest" Warning Failed 29s (x2 over 88s) kubelet Error: ImagePullBackOff Warning FailedToRetrieveImagePullSecret 17s (x5 over 119s) kubelet Unable to retrieve some image pull secrets (pull_secret_name); attempting to pull the image may not succeed. Normal Pulling 17s (x3 over 119s) kubelet Pulling image "quay.io/sclorg/postgresql-15-c9s:latest" ``` Am I missing something here? I added the postgres image to the kustomization.yaml file as well, but that didn’t work either ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [October 3, 2024, 11:22am UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/10 "2024-10-03T11:22:20Z")
You have to explicitly specify all of `redis_image`, `redis_image_version`, `postgres_image`, and `postgres_image_version`. In addition, `image` and `image_version` are for AWX image instead of EE. ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [October 3, 2024, 2:20pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/11 "2024-10-03T14:20:40Z")
> [@kurokobo](#): > > n, `image` and `image_version` are for AWX image instead of EE. thanks, that did the trick… so now i encounter the next problem. ```auto Back-off pulling image "localhost:5000/postgres:latest" Warning Failed 3m22s (x9 over 56m) kubelet Failed to pull image "localhost:5000/postgres:latest": rpc error: code = Canceled desc = failed to pull and unpack image "localhost:5000/postgres:latest": context canceled ``` I am able to pull `localhost:5000/postgres:latest` with Podman without any issues, but for some reason k3s won’t pull it. Any ideas? ---
**Author:** ![BlueJ](https://avatars.discourse-cdn.com/v4/letter/b/eb8c5e/32.png) [@BlueJ](https://forum.ansible.com/u/BlueJ)\ **Post date:** [October 3, 2024, 3:02pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/12 "2024-10-03T15:02:47Z")
Hi Smoohii, I have been running AWX on K3s for a few years and I wanted to mention that there is an alternative to modifying the kustomization.yaml or AWX operator yaml. We actually followed this document [Private Registry Configuration | K3s](https://docs.k3s.io/installation/private-registry#mirrors) to tell K3s to redirect image pulls that are specified by AWX to a different location. ---
**Author:** ![smoohiii](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/smoohiii/32/4004_2.png) [@smoohiii](https://forum.ansible.com/u/smoohiii)\ **Post date:** [October 4, 2024, 1:26pm UTC](https://forum.ansible.com/t/awx-k3s-offline-installation/6874/13 "2024-10-04T13:26:32Z")
Hi BlueJ, So I solved the issue mentioned above by changing from l`ocalhost:5000 to :5000` and also adding this as a mirror in the `registries.yaml.` I’m kind of curious how you managed this without using a` kustomization or awx.yaml?` I still have to add the `:5000` to my file.