Install issues

Archives AWX Project
1

Hello,

I’m a bit new here, I have already AWX 21.1.0 and I wanted to upgrade, unfortunately, it failed so I create a new virtual and I want to install fresh awx. I followed the git installation guide (basic) and I couldn’t get it running.

My install fails on getting up awx-web and awx-task with the error “Apply resources failed”:
NAME READY STATUS RESTARTS AGE awx-postgres-13-0 1/1 Running 0 29m awx-web-565fbfb879-qhsp5 1/3 CrashLoopBackOff 18 (4m19s ago) 26m awx-task-57fd466894-sj8cd 0/4 Init:CrashLoopBackOff 10 (29s ago) 28m

I tried awx-operator 2.4.0 and 2.5.0 with the same result, My OS is Debian 11.

If anyone can help or give me a hint I will be very grateful, thank you.
Best regards,
Gertz

2

can you print out the logs of your web container?

also kubectl describe your web pod?

AWX Team

3

Hello,

Unfornutally this is output of commands:
:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
awx-postgres-13-0 1/1 Running 1 (83s ago) 2d4h
awx-task-6f445b58f4-xhdfz 0/4 Init:CrashLoopBackOff 14 (40s ago) 2d4h
awx-operator-controller-manager-66c5b94884-qv9q8 2/2 Running 3 (49s ago) 2d4h
awx-web-67fc6f7f4b-7wnmz 1/3 CrashLoopBackOff 27 (36s ago) 2d4h

:~# kubectl -n awx logs deploy/awx -c awx-web
Error from server (NotFound): deployments.apps “awx” not found

:~# kubectl -n awx logs deploy/awx -c awx-task
Error from server (NotFound): deployments.apps “awx” not found

:~# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
awx-task 0/1 1 0 2d4h
awx-web 0/1 1 0 2d4h
awx-operator-controller-manager 1/1 1 1 2d4h

:~# kubectl describe pods/awx-web-67fc6f7f4b-7wnmz
Name: awx-web-67fc6f7f4b-7wnmz
Namespace: awx
Priority: 0
Service Account: awx
Node: awx-prod.feel4.cz/192.168.2.136
Start Time: Wed, 02 Aug 2023 15:01:06 +0200
Labels: app.kubernetes.io/component=awx
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=awx-web
app.kubernetes.io/operator-version=2.5.0
app.kubernetes.io/part-of=awx
app.kubernetes.io/version=22.6.0
pod-template-hash=67fc6f7f4b
Annotations: checksum-configmaps-config: 038a749ff3e48ce0951d7836cd2fa73d2448d2e2
checksum-secret-bundle_cacert: da39a3ee5e6b4b0d3255bfef95601890afd80709
checksum-secret-ldap_cacert: da39a3ee5e6b4b0d3255bfef95601890afd80709
checksum-secret-receptor_ca: a4b9adecff1e5a9bea5994ae24b51cf91897b450
checksum-secret-receptor_work_signing: 82f7c1af84a3736708617686cbd81764c174952c
checksum-secret-route_tls: da39a3ee5e6b4b0d3255bfef95601890afd80709
checksum-secret-secret_key: 1bcfe15d30df9e3faaca21e321630000821f8080
checksum-secrets-app_credentials: 21a7d43de9993c97e2db99bca1ac000410b13708
checksum-storage-persistent: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
kubectl.kubernetes.io/default-container: awx-web
Status: Running
IP: 10.42.0.62
IPs:
IP: 10.42.0.62
Controlled By: ReplicaSet/awx-web-67fc6f7f4b
Containers:
redis:
Container ID: containerd://2563665136a228aada7684d7476fdeace97d6d845ae5ef7d74ba356765a1a9ef
Image: docker.io/redis:7
Image ID: docker.io/library/redis@sha256:08a82d4bf8a8b4dd94e8f5408cdbad9dd184c1cf311d34176cd3e9972c43f872
Port:
Host Port:
Args:
redis-server
/etc/redis.conf
State: Running
Started: Fri, 04 Aug 2023 19:25:22 +0200
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Wed, 02 Aug 2023 15:01:06 +0200
Finished: Fri, 04 Aug 2023 19:25:19 +0200
Ready: True
Restart Count: 1
Requests:
cpu: 50m
memory: 64Mi
Environment:
Mounts:
/data from awx-redis-data (rw)
/etc/redis.conf from awx-redis-config (ro,path=“redis.conf”)
/var/run/redis from awx-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2jlwv (ro)
awx-web:
Container ID: containerd://6f50bee2df69cb52507e4cfbb683b8014ec0b4a5cbe3bc64ba6dafe1e3a7b916
Image: quay.io/ansible/awx:22.6.0
Image ID: quay.io/ansible/awx@sha256:64a95fa73f203c7b92ea97043b170e48b1a3fb1b24bc6386b76d39edace922e7
Port: 8052/TCP
Host Port: 0/TCP
Args:
/usr/bin/launch_awx_web.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 127
Started: Fri, 04 Aug 2023 19:28:32 +0200
Finished: Fri, 04 Aug 2023 19:28:32 +0200
Ready: False
Restart Count: 15
Requests:
cpu: 100m
memory: 128Mi
Environment:
AWX_COMPONENT: web
SUPERVISOR_CONFIG_PATH: /etc/supervisord_web.conf
MY_POD_NAMESPACE: awx (v1:metadata.namespace)
MY_POD_IP: (v1:status.podIP)
UWSGI_MOUNT_PATH: /
Mounts:
/etc/nginx/nginx.conf from awx-nginx-conf (ro,path=“nginx.conf”)
/etc/receptor/tls/ca/mesh-CA.crt from awx-receptor-ca (ro,path=“tls.crt”)
/etc/receptor/tls/ca/mesh-CA.key from awx-receptor-ca (ro,path=“tls.key”)
/etc/receptor/work_public_key.pem from awx-receptor-work-signing (ro,path=“work-public-key.pem”)
/etc/tower/SECRET_KEY from awx-secret-key (ro,path=“SECRET_KEY”)
/etc/tower/conf.d/credentials.py from awx-application-credentials (ro,path=“credentials.py”)
/etc/tower/conf.d/execution_environments.py from awx-application-credentials (ro,path=“execution_environments.py”)
/etc/tower/conf.d/ldap.py from awx-application-credentials (ro,path=“ldap.py”)
/etc/tower/settings.py from awx-settings (ro,path=“settings.py”)
/etc/tower/uwsgi.ini from awx-uwsgi-config (ro,path=“uwsgi.conf”)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/redis from awx-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2jlwv (ro)
awx-rsyslog:
Container ID: containerd://bac185d6bfdc6904f998c4e8c8b1749608cb81677d17a50063a3975e83d89c7c
Image: quay.io/ansible/awx:22.6.0
Image ID: quay.io/ansible/awx@sha256:64a95fa73f203c7b92ea97043b170e48b1a3fb1b24bc6386b76d39edace922e7
Port:
Host Port:
Args:
/usr/bin/launch_awx_rsyslog.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 127
Started: Fri, 04 Aug 2023 19:28:32 +0200
Finished: Fri, 04 Aug 2023 19:28:32 +0200
Ready: False
Restart Count: 15
Requests:
cpu: 100m
memory: 128Mi
Environment:
SUPERVISOR_CONFIG_PATH: /etc/supervisord_rsyslog.conf
Mounts:
/etc/tower/SECRET_KEY from awx-secret-key (ro,path=“SECRET_KEY”)
/etc/tower/conf.d/credentials.py from awx-application-credentials (ro,path=“credentials.py”)
/etc/tower/settings.py from awx-settings (ro,path=“settings.py”)
/var/run/awx-rsyslog from rsyslog-socket (rw)
/var/run/redis from awx-redis-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2jlwv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
awx-receptor-ca:
Type: Secret (a volume populated by a Secret)
SecretName: awx-receptor-ca
Optional: false
awx-receptor-work-signing:
Type: Secret (a volume populated by a Secret)
SecretName: awx-receptor-work-signing
Optional: false
awx-application-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: awx-app-credentials
Optional: false
awx-secret-key:
Type: Secret (a volume populated by a Secret)
SecretName: awx-secret-key
Optional: false
awx-settings:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-awx-configmap
Optional: false
awx-nginx-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-awx-configmap
Optional: false
awx-redis-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-awx-configmap
Optional: false
awx-uwsgi-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-awx-configmap
Optional: false
awx-redis-socket:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
awx-redis-data:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
rsyslog-socket:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
receptor-socket:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
awx-receptor-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: awx-awx-configmap
Optional: false
kube-api-access-2jlwv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 2d4h default-scheduler Successfully assigned awx/awx-web-67fc6f7f4b-7wnmz to awx-prod.feel4.cz
Normal Pulled 2d4h kubelet Container image “docker.io/redis:7” already present on machine
Normal Created 2d4h kubelet Created container redis
Normal Started 2d4h kubelet Started container redis
Normal Pulled 2d4h (x3 over 2d4h) kubelet Container image “quay.io/ansible/awx:22.6.0” already present on machine
Normal Created 2d4h (x3 over 2d4h) kubelet Created container awx-web
Normal Started 2d4h (x3 over 2d4h) kubelet Started container awx-web
Normal Pulled 2d4h (x3 over 2d4h) kubelet Container image “quay.io/ansible/awx:22.6.0” already present on machine
Normal Created 2d4h (x3 over 2d4h) kubelet Created container awx-rsyslog
Normal Started 2d4h (x3 over 2d4h) kubelet Started container awx-rsyslog
Warning BackOff 2d4h (x2 over 2d4h) kubelet Back-off restarting failed container awx-rsyslog in pod awx-web-67fc6f7f4b-7wnmz_awx(f7fa6845-764b-4643-ae6d-a9e637b5abe6)
Warning BackOff 2d3h (x143 over 2d4h) kubelet Back-off restarting failed container awx-web in pod awx-web-67fc6f7f4b-7wnmz_awx(f7fa6845-764b-4643-ae6d-a9e637b5abe6)
Normal SandboxChanged 5m25s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 5m24s kubelet Container image “docker.io/redis:7” already present on machine
Normal Created 5m24s kubelet Created container redis
Normal Started 5m24s kubelet Started container redis
Normal Pulled 5m10s (x2 over 5m24s) kubelet Container image “quay.io/ansible/awx:22.6.0” already present on machine
Normal Created 5m10s (x2 over 5m24s) kubelet Created container awx-web
Normal Started 5m10s (x2 over 5m24s) kubelet Started container awx-web
Normal Pulled 5m10s (x2 over 5m24s) kubelet Container image “quay.io/ansible/awx:22.6.0” already present on machine
Normal Created 5m10s (x2 over 5m24s) kubelet Created container awx-rsyslog
Normal Started 5m10s (x2 over 5m23s) kubelet Started container awx-rsyslog
Warning BackOff 5m8s (x4 over 5m23s) kubelet Back-off restarting failed container awx-rsyslog in pod awx-web-67fc6f7f4b-7wnmz_awx(f7fa6845-764b-4643-ae6d-a9e637b5abe6)
Warning BackOff 17s (x26 over 5m23s) kubelet Back-off restarting failed container awx-web in pod awx-web-67fc6f7f4b-7wnmz_awx(f7fa6845-764b-4643-ae6d-a9e637b5abe6)

Thank you for help,
Gertz.

Dne pátek 4. srpna 2023 v 19:21:01 UTC+2 uživatel AWX Project napsal:

4

Okey, thank you for your support I finally find out the solution to this, looks like there are increased HW requirements so is required x86-v64-v2+aes. Is there a posibility to add HW requirements to the awx install manual?

Thank you for help,
Gertz.

Dne pátek 4. srpna 2023 v 19:32:57 UTC+2 uživatel Gertz napsal:

5

There used to be some instructions for minimum sizing in the install guide but we think that should have all been removed. Sizing can be very tricky because it really depends on your automation usage levels along with API usage factors.

Can you tell us what the difference was from your old hardware to your new hardware?

-The AWX Team

6

I didn’t change physical HW because I use virtualization but AWX request X86-64-v2 with aes so default setup for virtual like proxmox is KVM which doesn’t have set x86-64-v2 neither aes. it’s supported instruction set. This information is crytical for ppl who use virtualization, in git I created issue (befora I wrote here) for this and there was also another guy whit same problem. So in basic instalation guide should be information like “when AWX-operator is installed as virtual machine make sure CPU is set for x86-64-v2+aes support”

note: x86-64-v2 means that a CPU support not only the basic x86-64 instructions set, but also other instructions like SSE4. 2, SSSE3 or POPCNT

Dne středa 9. srpna 2023 v 20:46:00 UTC+2 uživatel AWX Project napsal: