How to enable https 443 on AWX running on top of K8S

Get Help
1

HI all
I have installed latest version of AWX on top of K8S, the http 80 port is running smoothly. Now I want to enable https 443 port for AWX. But I could not find any documentation or explanation, anyone who has suggestion is welcome

2

Hope my comment on the similar topic helps you:

2 Likes
3

Hi Kurokobo

I am trying to do the same config on my running awx with http only (migrated from old release)
So can you please provide a details command on how to do it ?

Thank for your support

4

My comment above and the link there says most of the information you need.
Also I’ve answered for your issue in my repo: Trusting a Custom Certificate Authority (enable https) on running AWX instance currently without SSL support · Issue #284 · kurokobo/awx-on-k3s · GitHub

I have already provided you with enough information, so could you please check the answers first and give it a try?

5

Hi Kurokobo

I run kubectl apply -f base (with tls cert files)
But now I have below error messages:

localhost : ok=67 changed=1 unreachable=0 failed=1 skipped=68 rescued=0 ignored=1

{“level”:“error”,“ts”:“2023-12-10T20:53:27Z”,“msg”:“Reconciler error”,“controller”:“awx-controller”,“object”:{“name”:“awx”,“namespace”:“awx”},“namespace”:“awx”,“name”:“awx”,“reconcileID”:“fef220ad-d09b-4a39-808b-a25477e79eed”,“error”:“event runner on failed”,“stacktrace”:“sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235”}
{“level”:“info”,“ts”:“2023-12-10T20:53:28Z”,“logger”:“logging_event_handler”,“msg”:“[playbook task start]”,“name”:“awx”,“namespace”:“awx”,“gvk”:“awx.ansible.com/v1beta1, Kind=AWX”,“event_type”:“playbook_on_task_start”,“job”:“9073996043421960506”,“EventData.Name”:“Verify imagePullSecrets”}

I still can not copy/paste all the output logs but find below interesting part

"Deployment does not have minimum availability.", "reason": "MinimumReplicasUnavailable"

Thanks for your support

6

Let me know if you need more info

Note that I am not confirmed kubernetes user

Thanks

7

Lack of information. You have various ways to paste complete logs; posting as code block (use ```), using Gist, using Pastebin, etc.

Provide fresh logs by:

  1. Uncomment no_log: false in your base/awx.yaml and kubectl apply -f base again
  2. Restart Operator by kubectl -n awx delete pod -l control-plane=controller-manager
  3. Gather logs by kubectl -n awx logs -f deployments/awx-operator-controller-manager
1 Like
8

Still in error>
kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-postgres-13-0 1/1 Running 13 (4h8m ago) 310d
awx-task-75484964fd-p4d5n 4/4 Running 0 6m7s
awx-operator-controller-manager-66d787886f-mvjzk 2/2 Running 0 3m26s
awx-web-6574bf9676-2tzq4 2/3 CrashLoopBackOff 5 (41s ago) 6m

Still limited by number of characters …

[{\\\"key\\\":\\\"secret_key\\\",\\\"path\\\":\\\"SECRET_KEY\\\"}],\\\"secretName\\\":\\\"awx-secret-key\\\"}},{\\\"configMap\\\":{\\\"items\\\":[{
\\\"key\\\":\\\"settings\\\",\\\"path\\\":\\\"settings.py\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-settings\\\"},{\\\"configMap\\\":
{\\\"items\\\":[{\\\"key\\\":\\\"nginx_conf\\\",\\\"path\\\":\\\"nginx.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-nginx-conf\\\"}
,{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"redis_conf\\\",\\\"path\\\":\\\"redis.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\
"awx-redis-config\\\"},{\\\"configMap\\\":{\\\"items\\\":[{\\\"key\\\":\\\"uwsgi_conf\\\",\\\"path\\\":\\\"uwsgi.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap
\\\"},\\\"name\\\":\\\"awx-uwsgi-config\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-redis-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"awx-redis-data\
\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"rsyslog-socket\\\"},{\\\"emptyDir\\\":{},\\\"name\\\":\\\"receptor-socket\\\"},{\\\"configMap\\\":{\\\"items\\\":[{
\\\"key\\\":\\\"receptor_conf\\\",\\\"path\\\":\\\"receptor.conf\\\"}],\\\"name\\\":\\\"awx-awx-configmap\\\"},\\\"name\\\":\\\"awx-receptor-config\\\"},{\\\"
name\\\":\\\"awx-projects\\\",\\\"persistentVolumeClaim\\\":{\\\"claimName\\\":\\\"awx-projects-claim\\\"}}]}}}}\"}, \"creationTimestamp\": \"2023-06-11T17:44
:08Z\", \"generation\": 4, \"labels\": {\"app.kubernetes.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name
\": \"awx-web\", \"app.kubernetes.io/operator-version\": \"2.8.0\", \"app.kubernetes.io/part-of\": \"awx\", \"app.kubernetes.io/version\": \"23.5.0\"}, \"mana
gedFields\": [{\"apiVersion\": \"apps/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:kubectl.kubernete
s.io/last-applied-configuration\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kuber
netes.io/name\": {}, \"f:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}, \"f:app.kubernetes.io/version\": {}}, \"f:ownerReferen
ces\": {\".\": {}, \"k:{\\\"uid\\\":\\\"080ec523-76db-4a6a-ba32-eecb6e15bb6f\\\"}\": {}}}, \"f:spec\": {\"f:progressDeadlineSeconds\": {}, \"f:replicas\": {},
 \"f:revisionHistoryLimit\": {}, \"f:selector\": {}, \"f:strategy\": {\"f:rollingUpdate\": {\".\": {}, \"f:maxSurge\": {}, \"f:maxUnavailable\": {}}, \"f:type
\": {}}, \"f:template\": {\"f:metadata\": {\"f:annotations\": {\".\": {}, \"f:checksum-configmaps-config\": {}, \"f:checksum-secret-bundle_cacert\": {}, \"f:c
hecksum-secret-ldap_cacert\": {}, \"f:checksum-secret-receptor_ca\": {}, \"f:checksum-secret-receptor_work_signing\": {}, \"f:checksum-secret-route_tls\": {},
 \"f:checksum-secret-secret_key\": {}, \"f:checksum-secrets-app_credentials\": {}, \"f:checksum-storage-persistent\": {}, \"f:kubectl.kubernetes.io/default-co
ntainer\": {}}, \"f:labels\": {\".\": {}, \"f:app.kubernetes.io/component\": {}, \"f:app.kubernetes.io/managed-by\": {}, \"f:app.kubernetes.io/name\": {}, \"f
:app.kubernetes.io/operator-version\": {}, \"f:app.kubernetes.io/part-of\": {}, \"f:app.kubernetes.io/version\": {}}}, \"f:spec\": {\"f:containers\": {\"k:{\\
\"name\\\":\\\"awx-rsyslog\\\"}\": {\".\": {}, \"f:args\": {}, \"f:env\": {\".\": {}, \"k:{\\\"name\\\":\\\"SUPERVISOR_CONFIG_PATH\\\"}\": {\".\": {}, \"f:nam
e\": {}, \"f:value\": {}}}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}
, \"f:memory\": {}}}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/etc/to
wer/SECRET_KEY\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/conf.d/c
redentials.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/settings.
py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\"}\": {\
".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}, \"k:{\\\
"name\\\":\\\"awx-web\\\"}\": {\".\": {}, \"f:args\": {}, \"f:env\": {\".\": {}, \"k:{\\\"name\\\":\\\"AWX_COMPONENT\\\"}\": {\".\": {}, \"f:name\": {}, \"f:v
alue\": {}}, \"k:{\\\"name\\\":\\\"MY_POD_IP\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"MY_P
OD_NAMESPACE\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}, \"k:{\\\"name\\\":\\\"SUPERVISOR_CONFIG_PATH\\\"}\": {\".\
": {}, \"f:name\": {}, \"f:value\": {}}, \"k:{\\\"name\\\":\\\"UWSGI_MOUNT_PATH\\\"}\": {\".\": {}, \"f:name\": {}, \"f:value\": {}}}, \"f:image\": {}, \"f:im
agePullPolicy\": {}, \"f:name\": {}, \"f:ports\": {\".\": {}, \"k:{\\\"containerPort\\\":8052,\\\"protocol\\\":\\\"TCP\\\"}\": {\".\": {}, \"f:containerPort\"
: {}, \"f:protocol\": {}}}, \"f:resources\": {}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\
\"mountPath\\\":\\\"/etc/nginx/nginx.conf\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\
\\":\\\"/etc/receptor/tls/ca/mesh-CA.crt\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\
\":\\\"/etc/receptor/tls/ca/mesh-CA.key\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\
":\\\"/etc/receptor/work_public_key.pem\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\
":\\\"/etc/tower/SECRET_KEY\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/t
ower/conf.d/credentials.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/to
wer/conf.d/execution_environments.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\
\\"/etc/tower/conf.d/ldap.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/
tower/settings.py\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/tower/uwsgi
.ini\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\"}\":
 {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/awx-rsyslog\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}, \
"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}, \"k:{\\\"name\\\":\\\"redis\\\"}\": {\".\": {}, \"f:args\
": {}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {\".\": {}, \"f:requests\": {\".\": {}, \"f:cpu\": {}, \"f:memory\": {}}},
 \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\": {}, \"k:{\\\"mountPath\\\":\\\"/data\\\"}\": {\".\": {}, \"
f:mountPath\": {}, \"f:name\": {}}, \"k:{\\\"mountPath\\\":\\\"/etc/redis.conf\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}, \"f:readOnly\": {}, \"
f:subPath\": {}}, \"k:{\\\"mountPath\\\":\\\"/var/run/redis\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}}, \"f:dnsPolicy\": {}, \"f:initContaine
rs\": {\".\": {}, \"k:{\\\"name\\\":\\\"init\\\"}\": {\".\": {}, \"f:command\": {}, \"f:image\": {}, \"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\
": {}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}}, \"k:{\\\"name\\\":\\\"init-projects\\\"}\": {\".\": {}, \"f:command\": {}, \"f:e
nv\": {\".\": {}, \"k:{\\\"name\\\":\\\"MY_POD_NAME\\\"}\": {\".\": {}, \"f:name\": {}, \"f:valueFrom\": {\".\": {}, \"f:fieldRef\": {}}}}, \"f:image\": {}, \
"f:imagePullPolicy\": {}, \"f:name\": {}, \"f:resources\": {}, \"f:terminationMessagePath\": {}, \"f:terminationMessagePolicy\": {}, \"f:volumeMounts\": {\".\
": {}, \"k:{\\\"mountPath\\\":\\\"/var/lib/awx/projects\\\"}\": {\".\": {}, \"f:mountPath\": {}, \"f:name\": {}}}}}, \"f:restartPolicy\": {}, \"f:schedulerNam
e\": {}, \"f:securityContext\": {}, \"f:serviceAccount\": {}, \"f:serviceAccountName\": {}, \"f:terminationGracePeriodSeconds\": {}, \"f:volumes\": {\".\": {}
, \"k:{\\\"name\\\":\\\"awx-application-credentials\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:
secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-nginx-conf\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": 
{}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-projects\\\"}\": {\".\": {}, \"f:name\": {}, \"f:persistentVolumeClaim\": {\".\": {}, \"f:claimName\": {}}}, \
"k:{\\\"name\\\":\\\"awx-receptor-ca\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:secretName\": {}}}, \"k:{\\\"nam
e\\\":\\\"awx-receptor-config\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{
\\\"name\\\":\\\"awx-receptor-work-signing\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:secretName\": {}}}, \"k:{\
\\"name\\\":\\\"awx-redis-config\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {}, \"f:name\": {}}, \"f:name\": {}}, \"
k:{\\\"name\\\":\\\"awx-redis-data\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-redis-socket\\\"}\": {\".\": {}, \"f:emp
tyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-secret-key\\\"}\": {\".\": {}, \"f:name\": {}, \"f:secret\": {\".\": {}, \"f:defaultMode\": {}, \"f:it
ems\": {}, \"f:secretName\": {}}}, \"k:{\\\"name\\\":\\\"awx-settings\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\": {},
 \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"awx-uwsgi-config\\\"}\": {\".\": {}, \"f:configMap\": {\".\": {}, \"f:defaultMode\": {}, \"f:items\":
 {}, \"f:name\": {}}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"receptor-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}, \"k:{\\\"name\\\":\\\"rs
yslog-socket\\\"}\": {\".\": {}, \"f:emptyDir\": {}, \"f:name\": {}}}}}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2023-12-
11T16:37:49Z\"}, {\"apiVersion\": \"apps/v1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:metadata\": {\"f:annotations\": {\"f:deployment.kubernetes.io/
revision\": {}}}, \"f:status\": {\"f:conditions\": {\".\": {}, \"k:{\\\"type\\\":\\\"Available\\\"}\": {\".\": {}, \"f:lastTransitionTime\": {}, \"f:lastUpdat
eTime\": {}, \"f:message\": {}, \"f:reason\": {}, \"f:status\": {}, \"f:type\": {}}, \"k:{\\\"type\\\":\\\"Progressing\\\"}\": {\".\": {}, \"f:lastTransitionT
ime\": {}, \"f:lastUpdateTime\": {}, \"f:message\": {}, \"f:reason\": {}, \"f:status\": {}, \"f:type\": {}}}, \"f:observedGeneration\": {}, \"f:replicas\": {}
, \"f:unavailableReplicas\": {}, \"f:updatedReplicas\": {}}}, \"manager\": \"k3s\", \"operation\": \"Update\", \"subresource\": \"status\", \"time\": \"2023-1
2-11T16:43:08Z\"}], \"name\": \"awx-web\", \"namespace\": \"awx\", \"ownerReferences\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWX\", \"na
me\": \"awx\", \"uid\": \"080ec523-76db-4a6a-ba32-eecb6e15bb6f\"}], \"resourceVersion\": \"50492950\", \"uid\": \"57a8c23d-ec30-4ad5-831c-ff07698617aa\"}, \"s
pec\": {\"progressDeadlineSeconds\": 600, \"replicas\": 1, \"revisionHistoryLimit\": 10, \"selector\": {\"matchLabels\": {\"app.kubernetes.io/component\": \"a
wx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx-web\"}}, \"strategy\": {\"rollingUpdate\": {\"maxSurge\": \"25%\",
 \"maxUnavailable\": \"25%\"}, \"type\": \"RollingUpdate\"}, \"template\": {\"metadata\": {\"annotations\": {\"checksum-configmaps-config\": \"f1a4580cfa08a13
faa1b970b250cd3d94c25877e\", \"checksum-secret-bundle_cacert\": \"da39a3ee5e6b4b0d3255bfef95601890afd80709\", \"checksum-secret-ldap_cacert\": \"da39a3ee5e6b4
b0d3255bfef95601890afd80709\", \"checksum-secret-receptor_ca\": \"2cafb60d6872d8fd903be8e7f17cb735a5638897\", \"checksum-secret-receptor_work_signing\": \"b51
4325f1474ca610436a2f3fc3f14a46ad24977\", \"checksum-secret-route_tls\": \"da39a3ee5e6b4b0d3255bfef95601890afd80709\", \"checksum-secret-secret_key\": \"2009b7
d217c4539d2a17c25253963c3e3acafee8\", \"checksum-secrets-app_credentials\": \"23dbca0a186a11df9e40de3c3c92ebd1c707d8ad\", \"checksum-storage-persistent\": \"a
dc83b19e793491b1c6ea0fd8b46cd9f32e592fc\", \"kubectl.kubernetes.io/default-container\": \"awx-web\"}, \"creationTimestamp\": null, \"labels\": {\"app.kubernet
es.io/component\": \"awx\", \"app.kubernetes.io/managed-by\": \"awx-operator\", \"app.kubernetes.io/name\": \"awx-web\", \"app.kubernetes.io/operator-version\
": \"2.8.0\", \"app.kubernetes.io/part-of\": \"awx\", \"app.kubernetes.io/version\": \"23.5.0\"}}, \"spec\": {\"containers\": [{\"args\": [\"redis-server\", \
"/etc/redis.conf\"], \"image\": \"docker.io/redis:7\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"redis\", \"resources\": {\"requests\": {\"cpu\": \"5
0m\", \"memory\": \"64Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\":
 \"/etc/redis.conf\", \"name\": \"awx-redis-config\", \"readOnly\": true, \"subPath\": \"redis.conf\"}, {\"mountPath\": \"/var/run/redis\", \"name\": \"awx-re
dis-socket\"}, {\"mountPath\": \"/data\", \"name\": \"awx-redis-data\"}]}, {\"args\": [\"/usr/bin/launch_awx_web.sh\"], \"env\": [{\"name\": \"AWX_COMPONENT\"
, \"value\": \"web\"}, {\"name\": \"SUPERVISOR_CONFIG_PATH\", \"value\": \"/etc/supervisord_web.conf\"}, {\"name\": \"MY_POD_NAMESPACE\", \"valueFrom\": {\"fi
eldRef\": {\"apiVersion\": \"v1\", \"fieldPath\": \"metadata.namespace\"}}}, {\"name\": \"MY_POD_IP\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": \"v1\", 
\"fieldPath\": \"status.podIP\"}}}, {\"name\": \"UWSGI_MOUNT_PATH\", \"value\": \"/\"}], \"image\": \"quay.io/ansible/awx:23.5.0\", \"imagePullPolicy\": \"IfN
otPresent\", \"name\": \"awx-web\", \"ports\": [{\"containerPort\": 8052, \"protocol\": \"TCP\"}], \"resources\": {}, \"terminationMessagePath\": \"/dev/termi
nation-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/tower/uwsgi.ini\", \"name\": \"awx-uwsgi-config\", \"readOnly\
": true, \"subPath\": \"uwsgi.conf\"}, {\"mountPath\": \"/etc/tower/conf.d/execution_environments.py\", \"name\": \"awx-application-credentials\", \"readOnly\
": true, \"subPath\": \"execution_environments.py\"}, {\"mountPath\": \"/etc/tower/conf.d/credentials.py\", \"name\": \"awx-application-credentials\", \"readO
nly\": true, \"subPath\": \"credentials.py\"}, {\"mountPath\": \"/etc/tower/conf.d/ldap.py\", \"name\": \"awx-application-credentials\", \"readOnly\": true, \
"subPath\": \"ldap.py\"}, {\"mountPath\": \"/etc/tower/SECRET_KEY\", \"name\": \"awx-secret-key\", \"readOnly\": true, \"subPath\": \"SECRET_KEY\"}, {\"mountP
ath\": \"/etc/tower/settings.py\", \"name\": \"awx-settings\", \"readOnly\": true, \"subPath\": \"settings.py\"}, {\"mountPath\": \"/etc/nginx/nginx.conf\", \
"name\": \"awx-nginx-conf\", \"readOnly\": true, \"subPath\": \"nginx.conf\"}, {\"mountPath\": \"/var/run/redis\", \"name\": \"awx-redis-socket\"}, {\"mountPa
th\": \"/var/run/awx-rsyslog\", \"name\": \"rsyslog-socket\"}, {\"mountPath\": \"/var/lib/awx/projects\", \"name\": \"awx-projects\"}, {\"mountPath\": \"/etc/
receptor/tls/ca/mesh-CA.crt\", \"name\": \"awx-receptor-ca\", \"readOnly\": true, \"subPath\": \"tls.crt\"}, {\"mountPath\": \"/etc/receptor/tls/ca/mesh-CA.ke
y\", \"name\": \"awx-receptor-ca\", \"readOnly\": true, \"subPath\": \"tls.key\"}, {\"mountPath\": \"/etc/receptor/work_public_key.pem\", \"name\": \"awx-rece
ptor-work-signing\", \"readOnly\": true, \"subPath\": \"work-public-key.pem\"}]}, {\"args\": [\"/usr/bin/launch_awx_rsyslog.sh\"], \"env\": [{\"name\": \"SUPE
RVISOR_CONFIG_PATH\", \"value\": \"/etc/supervisord_rsyslog.conf\"}], \"image\": \"quay.io/ansible/awx:23.5.0\", \"imagePullPolicy\": \"IfNotPresent\", \"name
\": \"awx-rsyslog\", \"resources\": {\"requests\": {\"cpu\": \"100m\", \"memory\": \"128Mi\"}}, \"terminationMessagePath\": \"/dev/termination-log\", \"termin
ationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/etc/tower/conf.d/credentials.py\", \"name\": \"awx-application-credentials\", \"readOnly
\": true, \"subPath\": \"credentials.py\"}, {\"mountPath\": \"/etc/tower/SECRET_KEY\", \"name\": \"awx-secret-key\", \"readOnly\": true, \"subPath\": \"SECRET
_KEY\"}, {\"mountPath\": \"/etc/tower/settings.py\", \"name\": \"awx-settings\", \"readOnly\": true, \"subPath\": \"settings.py\"}, {\"mountPath\": \"/var/run
/redis\", \"name\": \"awx-redis-socket\"}, {\"mountPath\": \"/var/run/awx-rsyslog\", \"name\": \"rsyslog-socket\"}]}], \"dnsPolicy\": \"ClusterFirst\", \"init
Containers\": [{\"command\": [\"/bin/sh\", \"-c\", \"\"], \"image\": \"quay.io/ansible/awx-ee:latest\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"ini
t\", \"resources\": {}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\"}, {\"command\": [\"/bin/sh\", \"-c\", \"c
hmod 775 /var/lib/awx/projects\\nchgrp 1000 /var/lib/awx/projects\\n\"], \"env\": [{\"name\": \"MY_POD_NAME\", \"valueFrom\": {\"fieldRef\": {\"apiVersion\": 
\"v1\", \"fieldPath\": \"metadata.name\"}}}], \"image\": \"quay.io/centos/centos:stream9\", \"imagePullPolicy\": \"IfNotPresent\", \"name\": \"init-projects\"
, \"resources\": {}, \"terminationMessagePath\": \"/dev/termination-log\", \"terminationMessagePolicy\": \"File\", \"volumeMounts\": [{\"mountPath\": \"/var/l
ib/awx/projects\", \"name\": \"awx-projects\"}]}], \"restartPolicy\": \"Always\", \"schedulerName\": \"default-scheduler\", \"securityContext\": {}, \"service
Account\": \"awx\", \"serviceAccountName\": \"awx\", \"terminationGracePeriodSeconds\": 30, \"volumes\": [{\"name\": \"awx-receptor-ca\", \"secret\": {\"defau
ltMode\": 420, \"secretName\": \"awx-receptor-ca\"}}, {\"name\": \"awx-receptor-work-signing\", \"secret\": {\"defaultMode\": 420, \"secretName\": \"awx-recep
tor-work-signing\"}}, {\"name\": \"awx-application-credentials\", \"secret\": {\"defaultMode\": 420, \"items\": [{\"key\": \"credentials.py\", \"path\": \"cre
dentials.py\"}, {\"key\": \"ldap.py\", \"path\": \"ldap.py\"}, {\"key\": \"execution_environments.py\", \"path\": \"execution_environments.py\"}], \"secretNam
e\": \"awx-app-credentials\"}}, {\"name\": \"awx-secret-key\", \"secret\": {\"defaultMode\": 420, \"items\": [{\"key\": \"secret_key\", \"path\": \"SECRET_KEY
\"}], \"secretName\": \"awx-secret-key\"}}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"settings\", \"path\": \"settings.py\"}], \"name\": 
\"awx-awx-configmap\"}, \"name\": \"awx-settings\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"nginx_conf\", \"path\": \"nginx.conf\"}], 
\"name\": \"awx-awx-configmap\"}, \"name\": \"awx-nginx-conf\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"redis_conf\", \"path\": \"redi
s.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-redis-config\"}, {\"configMap\": {\"defaultMode\": 420, \"items\": [{\"key\": \"uwsgi_conf\", \"
path\": \"uwsgi.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-uwsgi-config\"}, {\"emptyDir\": {}, \"name\": \"awx-redis-socket\"}, {\"emptyDir\"
: {}, \"name\": \"awx-redis-data\"}, {\"emptyDir\": {}, \"name\": \"rsyslog-socket\"}, {\"emptyDir\": {}, \"name\": \"receptor-socket\"}, {\"configMap\": {\"d
efaultMode\": 420, \"items\": [{\"key\": \"receptor_conf\", \"path\": \"receptor.conf\"}], \"name\": \"awx-awx-configmap\"}, \"name\": \"awx-receptor-config\"
}, {\"name\": \"awx-projects\", \"persistentVolumeClaim\": {\"claimName\": \"awx-projects-claim\"}}]}}}, \"status\": {\"conditions\": [{\"lastTransitionTime\"
: \"2023-06-11T17:44:08Z\", \"lastUpdateTime\": \"2023-12-11T16:37:52Z\", \"message\": \"ReplicaSet \\\"awx-web-6574bf9676\\\" has successfully progressed.\",
 \"reason\": \"NewReplicaSetAvailable\", \"status\": \"True\", \"type\": \"Progressing\"}, {\"lastTransitionTime\": \"2023-12-11T16:43:08Z\", \"lastUpdateTime
\": \"2023-12-11T16:43:08Z\", \"message\": \"Deployment does not have minimum availability.\", \"reason\": \"MinimumReplicasUnavailable\", \"status\": \"False
\", \"type\": \"Available\"}], \"observedGeneration\": 4, \"replicas\": 1, \"unavailableReplicas\": 1, \"updatedReplicas\": 1}}}\n\r\nPLAY RECAP *************
********************************************************\r\nlocalhost                  : ok=62   changed=0    unreachable=0    failed=1    skipped=64   rescue
d=0    ignored=0   \n","job":"800397080604192267","name":"awx","namespace":"awx","error":"exit status 2","stacktrace":"github.com/operator-framework/operator-
sdk/internal/ansible/runner.(*runner).Run.func1\n\t/workspace/internal/ansible/runner/runner.go:269"}
{"level":"error","ts":"2023-12-11T16:45:22Z","msg":"Reconciler error","controller":"awx-controller","object":{"name":"awx","namespace":"awx"},"namespace":"awx
","name":"awx","reconcileID":"4bd31b07-bd7b-4b99-b687-33e28a062b41","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal
/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/contro
ller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controlle
r.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/interna
l/controller/controller.go:235"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Verify imagePullSecrets] *************************************************
task path: /opt/ansible/playbooks/awx.yml:10

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-11T16:45:23Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.c
om/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2421458714998890432","EventData.Name":"Verify imagePullSecrets"}
{"level":"info","ts":"2023-12-11T16:45:24Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/aw
x/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":""
,"Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}}
{"level":"info","ts":"2023-12-11T16:45:24Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.c
om/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2421458714998890432","EventData.Name":"Create imagePullSecret"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Create imagePullSecret] **************************************************
task path: /opt/ansible/playbooks/awx.yml:17

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ******************************** 
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-11T16:45:24Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1
beta1, Kind=AWX","event_type":"runner_on_ok","job":"2421458714998890432","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

TASK [installer : Check for presence of old awx Deployment] ********************
9

Read my comment please.

posting as code block (use ````` ), using Gist, using Pastebin, etc.

1 Like
10

Hi

all posted logs are not enough ?

Do not have more

Best regards

11

kubectl -n awx logs -f deployments/awx-operator-controller-manager → provide too many logs
How can I copy/paste all the logs ?

Thanks

12

We are also limiteed in number of characters :

An error occurred: Body is limited to 32000 characters; you entered 242461.

Best regards

13

As I commented, use Gist or Pastebin.

Omitting the logs is not a good idea. In fact, the logs provided by you do not allow us to determine which tasks failed and why.

14

Hi

Can you let me know how I can troubleshoot my error by myself (procedure, methodology) ?

Thanks

15

It is the same as finding the cause of Playbook failure. Carefully read the logs to identify the task that failed, identify its cause, and resolve it.

As you can see, the logs are complex and difficult to read, but the logs of failed tasks are definitely there and can be identified by careful reading.

1 Like