AWX Openshift pods not starting

Get Help
, , ,
1

We have upgraded the AWX operator from version 2.12.2 to 2.16.1.

Since then, I am unable to see the awx-web and awx-task pods in the AWX namespace.

image
image2360×950 177 KB

please help on fixing this.

2

and i am unable to access the console

image
image2780×900 159 KB

3

You will need to post the logs from the AWX controller to see what is failing. Looks like something in your upgrade is going wrong since you have Postgres 13 and 15 pods existing simultaneously.

4

Hi @mcen1

Here are the logs from the AWX controller

Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
W0325 06:05:23.327415       1 kube-rbac-proxy.go:155] 
==== Deprecation Warning ======================

Insecure listen address will be removed.
Using --insecure-listen-address won't be possible!

The ability to run kube-rbac-proxy without TLS certificates will be removed.
Not using --tls-cert-file and --tls-private-key-file won't be possible!

For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187

===============================================

		
I0325 06:05:23.327542       1 kube-rbac-proxy.go:284] Valid token audiences: 
I0325 06:05:23.327590       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0325 06:05:24.018910       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0325 06:05:24.019306       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443
5

I think those are the logs for your kube rbac proxy. You’ll need the logs from awx-operator-controller-manager. Typically those logs have a lot of Ansible playbook output during an upgrade.

6

Hi @mcen1

i am sharing the awx-operator-controller-manager logs as you asked

https://drive.google.com/file/d/1ScbKiebE3n5_4YXfdq6smXvGmah8DvpA/view?usp=sharing

7

Database migration from 13 to 15 is failing when data is being copied to new postgres15 database.

[installer : Stream backup from pg_dump to the new postgresql container] ***\r\n\u001b[1;30mtask path: /opt/ansible/roles/installer/tasks/upgrade_postgres.yml:99\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that ‘no_log: true’ was specified for this result", "changed": true}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=63 \u001b[0m \u001b[0;33mchanged=3 \u001b[0m unreachable=0 \u001b[0;31mfailed=1 \u001b[0m \u001b[0;36mskipped=24 \u001b[0m rescued=0 ignored=0 \n",“job”:“7937547529177739800”,“name”:“awx”,“namespace”:“tp-awx”,“error”:“exit status 2”,“stacktrace”:“github.com/operator-framework/ansible-operator-plugins/internal/ansible/runner.(*runner).Run.func1\n\tansible-operator-plugins/internal/ansible/runner/runner.go:269”}

I can think, most probably the issue is there could be some tables created in postgres13 database that are not owned awx admin user. If there any tables like that remove them and try.