Hi Kurokobo
I am still trying to use our custom certification authority certs following your recommendation, but it does not work
I request a certificate (csr) from our security team, I replaced tls.crt and tls.key with new certificate and private key and run kubectl apply -k base
I used kubectl v1.28.4+k3s2 with operator 2.8.0 and awx 23.5.0 with your awx-on-k3s repo
I used your recommendation on an existing awx running instance with a self signed certificate,
Do I need to reinstall from scratch with new tls.crt and tls.key ?
When I tryed I have : 404 page not found
Following awx doc we also need to create a CA secret (via cli):
# kubectl create secret generic <resourcename>-custom-certs \
--from-file=ldap-ca.crt=<PATH/TO/YOUR/CA/PEM/FILE> \
--from-file=bundle-ca.crt=<PATH/TO/YOUR/CA/PEM/FILE>
“changed": false}\n\r\nTASK [installer : Update new resource pod name as a variable.] *****************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:280\nok: [localhost] => {"ansible_facts": {"awx_task_pod_name": "awx-task-75fbddf9f6-9m4b4"}, "changed": false}\n\r\nTASK [installer : Verify the resource pod name is populated.] ******************\r\ntask path: /opt/ansible/roles/installer/tasks/resources_configuration.yml:286\nok: [localhost] => {\r\n "changed": false,\r\n "msg": "All assertions passed"\r\n}\n\r\nTASK [installer : Check for pending migrations] ********************************\r\ntask path: /opt/ansible/roles/installer/tasks/install.yml:97\n[DEPRECATION WARNING]: The ‘return_code’ return key is being renamed to ‘rc’. \r\nBoth keys are being returned for now to allow users to migrate their \r\nautomation. This feature will be removed from kubernetes.core in version 4.0.0.\r\n Deprecation warnings can be disabled by setting deprecation_warnings=False in \r\nansible.cfg.\r\nok: [localhost] => {"changed": false, "rc": 0, "return_code": 0, "stderr": "Traceback (most recent call last):\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect\n self.connection = self.get_new_connection(conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection\n connection = self.Database.connect(**conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect\n raise ex.with_traceback(None)\npsycopg.OperationalError: connection failed: password authentication failed for user \"awx\"\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/bin/awx-manage\", line 8, in \n sys.exit(manage())\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 159, in manage\n if (connection.pg_version // 10000) < 12:\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py\", line 15, in getattr\n return getattr(self._connections[self._alias], item)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py\", line 57, in get\n res = instance.dict[self.name] = self.func(instance)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 436, in pg_version\n with self.temporary_connection():\n File \"/usr/lib64/python3.9/contextlib.py\", line 119, in enter\n return next(self.gen)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 705, in temporary_connection\n with self.cursor() as cursor:\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 330, in cursor\n return self._cursor()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 306, in _cursor\n self.ensure_connection()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py\", line 91, in exit\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect\n self.connection = self.get_new_connection(conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection\n connection = self.Database.connect(**conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect\n raise ex.with_traceback(None)\ndjango.db.utils.OperationalError: connection failed: password authentication failed for user \"awx\"\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect", " self.connection = self.get_new_connection(conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection", " connection = self.Database.connect(**conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect", " raise ex.with_traceback(None)", "psycopg.OperationalError: connection failed: password authentication failed for user \"awx\"", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", " File \"/usr/bin/awx-manage\", line 8, in ", " sys.exit(manage())", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 159, in manage", " if (connection.pg_version // 10000) < 12:", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py\", line 15, in getattr", " return getattr(self._connections[self._alias], item)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py\", line 57, in get", " res = instance.dict[self.name] = self.func(instance)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 436, in pg_version", " with self.temporary_connection():", " File \"/usr/lib64/python3.9/contextlib.py\", line 119, in enter", " return next(self.gen)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 705, in temporary_connection", " with self.cursor() as cursor:", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 330, in cursor", " return self._cursor()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 306, in _cursor", " self.ensure_connection()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py\", line 91, in exit", " raise dj_exc_value.with_traceback(traceback) from exc_value", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect", " self.connection = self.get_new_connection(conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection", " connection = self.Database.connect(**conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect", " raise ex.with_traceback(None)", "django.db.utils.OperationalError: connection failed: password authentication failed for user \"awx\""], "stdout": "0\n", "stdout_lines": ["0"]}\n\r\nTASK [installer : Migrate the database if the K8s resources were updated] ******\r\ntask path: /opt/ansible/roles/installer/tasks/install.yml:108\nskipping: [localhost] => {"changed": false, "false_condition": "(database_check.stdout|trim) != ‘0’", "skip_reason": "Conditional result was False"}\n\r\nTASK [installer : Initialize Django] *******************************************\r\ntask path: /opt/ansible/roles/installer/tasks/install.yml:139\nincluded: /opt/ansible/roles/installer/tasks/initialize_django.yml for localhost\n\r\nTASK [installer : Check if there are any super users defined.] *****************\r\ntask path: /opt/ansible/roles/installer/tasks/initialize_django.yml:2\nfatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last):\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect\n self.connection = self.get_new_connection(conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection\n connection = self.Database.connect(**conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect\n raise ex.with_traceback(None)\npsycopg.OperationalError: connection failed: password authentication failed for user \"awx\"\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/bin/awx-manage\", line 8, in \n sys.exit(manage())\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 159, in manage\n if (connection.pg_version // 10000) < 12:\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py\", line 15, in getattr\n return getattr(self._connections[self._alias], item)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py\", line 57, in get\n res = instance.dict[self.name] = self.func(instance)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 436, in pg_version\n with self.temporary_connection():\n File \"/usr/lib64/python3.9/contextlib.py\", line 119, in enter\n return next(self.gen)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 705, in temporary_connection\n with self.cursor() as cursor:\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 330, in cursor\n return self._cursor()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 306, in _cursor\n self.ensure_connection()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py\", line 91, in exit\n raise dj_exc_value.with_traceback(traceback) from exc_value\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection\n self.connect()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect\n self.connection = self.get_new_connection(conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection\n connection = self.Database.connect(**conn_params)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect\n raise ex.with_traceback(None)\ndjango.db.utils.OperationalError: connection failed: password authentication failed for user \"awx\"\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect", " self.connection = self.get_new_connection(conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection", " connection = self.Database.connect(**conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect", " raise ex.with_traceback(None)", "psycopg.OperationalError: connection failed: password authentication failed for user \"awx\"", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", " File \"/usr/bin/awx-manage\", line 8, in ", " sys.exit(manage())", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py\", line 159, in manage", " if (connection.pg_version // 10000) < 12:", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py\", line 15, in getattr", " return getattr(self._connections[self._alias], item)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py\", line 57, in get", " res = instance.dict[self.name] = self.func(instance)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 436, in pg_version", " with self.temporary_connection():", " File \"/usr/lib64/python3.9/contextlib.py\", line 119, in enter", " return next(self.gen)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 705, in temporary_connection", " with self.cursor() as cursor:", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 330, in cursor", " return self._cursor()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 306, in _cursor", " self.ensure_connection()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py\", line 91, in exit", " raise dj_exc_value.with_traceback(traceback) from exc_value", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 289, in ensure_connection", " self.connect()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py\", line 270, in connect", " self.connection = self.get_new_connection(conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py\", line 26, in inner", " return func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py\", line 275, in get_new_connection", " connection = self.Database.connect(**conn_params)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py\", line 728, in connect", " raise ex.with_traceback(None)", "django.db.utils.OperationalError: connection failed: password authentication failed for user \"awx\""], "stdout": "", "stdout_lines": }\r\n…ignoring\n\r\nTASK [installer : Create super user via Django if it doesn’t exist.] ***********\r\ntask path: /opt/ansible/roles/installer/tasks/initialize_django.yml:16\nfatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that ‘no_log: true’ was specified for this result", "changed": true}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost : ok=71 changed=2 unreachable=0 failed=1 skipped=65 rescued=0 ignored=1 \n”,“job”:“3378832938213667160”,“name”:“awx”,“namespace”:“awx”,“error”:“exit status 2”,“stacktrace”:“github.com/operator-framework/ansible-operator-plugins/internal/ansible/runner.(*runner).Run.func1\n\tansible-operator-plugins/internal/ansible/runner/runner.go:269”}
----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) -----
PLAY RECAP *********************************************************************
localhost : ok=71 changed=2 unreachable=0 failed=1 skipped=65 rescued=0 ignored=1
{“level”:“error”,“ts”:“2023-12-21T12:04:46Z”,“msg”:“Reconciler error”,“controller”:“awx-controller”,“object”:{“name”:“awx”,“namespace”:“awx”},“namespace”:“awx”,“name”:“awx”,“reconcileID”:“b5a1c506-1f72-4848-9f04-950a425152e5”,“error”:“event runner on failed”,“stacktrace”:“sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235”}
{“level”:“info”,“ts”:“2023-12-21T12:04:47Z”,“logger”:“logging_event_handler”,“msg”:“[playbook task start]”,“name”:“awx”,“namespace”:“awx”,“gvk”:“awx.ansible.com/v1beta1, Kind=AWX”,“event_type”:“playbook_on_task_start”,“job”:“9004064123518322156”,“EventData.Name”:“Verify imagePullSecrets”}
--------------------------- Ansible Task StdOut -------------------------------
Thanks for your support