Awx-task stuck on wait-for-migrations

fabeer · November 13, 2023, 9:17am

Hi,
On a RHEL8.8, I’ve installed AWX on single node K3s using AWX Operator using kurokobo procedure (GitHub - kurokobo/awx-on-k3s: An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.)


Client Version: v1.27.6+k3s1
Kustomize Version: v5.0.1
Server Version: v1.27.6+k3s1

$ kubectl get pod -n awx
NAME                                               READY   STATUS    RESTARTS        AGE
awx-postgres-13-0                                  1/1     Running   0               2d17h
awx-web-85db9b66cb-9ncsd                           3/3     Running   0               2d17h
awx-operator-controller-manager-6c55dc66bd-kzwvk   2/2     Running   0               2d17h
awx-task-bdd85964b-njnsq                           4/4     Running   1 (2d16h ago)   2d17h

All seems Ok, but after few minute the web gui hangs and on the awx-task pod
On the awx-task pod I see the following errors:

kubectl logs -f awx-task-bdd85964b-njnsq  -n awx
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6
[wait-for-migrations] Waiting 16 seconds before next attempt
[wait-for-migrations] Attempt 7
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 8
[wait-for-migrations] Waiting 30 seconds before next attempt
...
...
...

If I run the awx-manage migrate --noinput, inside the awx-web pod, the following error is reported:

$ kubectl -n awx exec -it awx-web-85db9b66cb-9ncsd -c  awx-web -- bin/bash
bash-5.1$ awx-manage migrate --noinput
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
    if (connection.pg_version // 10000) < 12:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
    with self.temporary_connection():
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
    with self.cursor() as cursor:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known

After restarting the awx-task pod, everything starts working correctly again, only to have the same problem again after a few minutes

TheRealHaoLiu · November 13, 2023, 2:49pm

fabeer:

  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

it looks like there’s a problem connecting to the postgres service
can you post your AWX resource to show us any customization that you have added

fabeer · November 14, 2023, 10:06am

Hi,
these are the configurations I used:

awx.yaml

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # These parameters are designed for use with:
  # - AWX Operator: 2.0.0
  #   https://github.com/ansible/awx-operator/blob/2.0.0/README.md
  # - AWX: 22.0.0
  #   https://github.com/ansible/awx/blob/22.0.0/INSTALL.md

  admin_user: admin
  admin_password_secret: awx-admin-password

  ingress_type: ingress
  ingress_tls_secret: awx-secret-tls
  hostname: myhost.mydomain.com

  postgres_configuration_secret: awx-postgres-configuration

  postgres_storage_class: awx-postgres-volume
  postgres_storage_requirements:
    requests:
      storage: 8Gi

  projects_persistence: true
  projects_existing_claim: awx-projects-claim

  web_replicas: 1
  task_replicas: 1

  postgres_init_container_resource_requirements: {}
  postgres_resource_requirements: {}
  web_resource_requirements: {}
  task_resource_requirements: {}
  ee_resource_requirements: {}
  init_container_resource_requirements: {}

  # Uncomment to reveal "censored" logs
  #no_log: false

kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: awx

generatorOptions:
  disableNameSuffixHash: true

secretGenerator:
  - name: awx-secret-tls
    type: kubernetes.io/tls
    files:
      - tls.crt
      - tls.key

  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=awx-postgres-13
      - port=5432
      - database=awx
      - username=awx
      - password=xxxxxxx
      - type=managed

  - name: awx-admin-password
    type: Opaque
    literals:
      - password=xxxxxxxx

resources:
  - pv.yaml
  - pvc.yaml
  - awx.yaml

pvc.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: awx-projects-claim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi
  storageClassName: awx-projects-volume

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-postgres-13-volume
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 8Gi
  storageClassName: awx-postgres-volume
  hostPath:
    path: /data/postgres-13

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-projects-volume
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 2Gi
  storageClassName: awx-projects-volume
  hostPath:
    path: /data/projects```

TheRealHaoLiu · November 16, 2023, 6:58pm

Are you using the Postgres that the operator deployed or are you using an externally managed Postgres.

fabeer · November 17, 2023, 2:10pm

I am using Postgres distributed by the operator

TheRealHaoLiu · November 17, 2023, 6:37pm

Is the Postgres pod healthy?

fabeer · November 20, 2023, 8:11am

It would seem so. Do you need the output of some other specific command?

$ kubectl get pods -n awx

NAME                                               READY   STATUS    RESTARTS        AGE
awx-operator-controller-manager-76b545976d-f9c9z   2/2     Running   0               3d21h
awx-postgres-13-0                                  1/1     Running   0               3d21h
awx-web-78b79668fb-dhvhj                           3/3     Running   0               3d21h
awx-task-79675b847b-6nnvc                          4/4     Running   1 (2d15h ago)   2d17h

$ kubectl logs -f awx-postgres-13-0 -n awx

PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-11-16 10:18:07.635 UTC [1] LOG:  starting PostgreSQL 13.13 (Debian 13.13-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2023-11-16 10:18:07.635 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-11-16 10:18:07.635 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-11-16 10:18:07.639 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-11-16 10:18:07.687 UTC [28] LOG:  database system was interrupted; last known up at 2023-11-16 09:30:46 UTC
2023-11-16 10:18:07.728 UTC [28] LOG:  database system was not properly shut down; automatic recovery in progress
2023-11-16 10:18:07.733 UTC [28] LOG:  redo starts at 0/350ECF8
2023-11-16 10:18:07.733 UTC [28] LOG:  invalid record length at 0/35100C0: wanted 24, got 0
2023-11-16 10:18:07.733 UTC [28] LOG:  redo done at 0/3510088
2023-11-16 10:18:07.764 UTC [1] LOG:  database system is ready to accept connections
2023-11-16 11:05:59.758 UTC [50] LOG:  could not receive data from client: Connection reset by peer
2023-11-16 13:21:04.813 UTC [2781] LOG:  could not receive data from client: Connection reset by peer
2023-11-16 13:21:15.743 UTC [2780] LOG:  could not receive data from client: Connection reset by peer

$ kubectl logs -f awx-web-78b79668fb-dhvhj -n awx

...
...
...
10.42.0.8 - - [17/Nov/2023:15:01:26 +0000] "GET /api/v2/credentials/?credential_type__namespace=galaxy_api_token&order_by=name&page=1&page_size=5 HTTP/1.1" 200 1513 "https://sflnx004345.utenze.bankit.it/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "10.42.0.1"
[pid: 20|app: 0|req: 263/1465] 10.42.0.8 () {68 vars in 1645 bytes} [Fri Nov 17 15:01:26 2023] GET /api/v2/credentials/?credential_type__namespace=galaxy_api_token&order_by=name&page=1&page_size=5 => generated 1513 bytes in 138 msecs (HTTP/1.1 200) 14 headers in 585 bytes (1 switches on core 0)
10.42.0.8 - - [17/Nov/2023:15:01:44 +0000] "OPTIONS /api/v2/execution_environments/ HTTP/1.1" 200 10972 "https://sflnx004345.utenze.bankit.it/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "10.42.0.1"
[pid: 19|app: 0|req: 440/1466] 10.42.0.8 () {70 vars in 1569 bytes} [Fri Nov 17 15:01:44 2023] OPTIONS /api/v2/execution_environments/ => generated 10972 bytes in 71 msecs (HTTP/1.1 200) 14 headers in 586 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 342/1467] 10.42.0.8 () {68 vars in 1639 bytes} [Fri Nov 17 15:01:44 2023] GET /api/v2/execution_environments/?or__organization__isnull=True&order_by=name&page=1&page_size=5 => generated 1271 bytes in 80 msecs (HTTP/1.1 200) 14 headers in 585 bytes (1 switches on core 0)
10.42.0.8 - - [17/Nov/2023:15:01:44 +0000] "GET /api/v2/execution_environments/?or__organization__isnull=True&order_by=name&page=1&page_size=5 HTTP/1.1" 200 1271 "https://sflnx004345.utenze.bankit.it/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "10.42.0.1"
2023/11/17 15:18:35 [error] 18#18: *1965 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 10.42.0.8, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "sflnx004345.utenze.bankit.it"
10.42.0.8 - - [17/Nov/2023:15:18:35 +0000] "GET /websocket/ HTTP/1.1" 101 1325 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "10.42.0.1"
2023/11/17 15:19:05 [error] 18#18: *1968 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 10.42.0.26, server: _, request: "GET /websocket/relay/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/relay/", host: "10.42.0.12:8052"
10.42.0.26 - - [17/Nov/2023:15:19:05 +0000] "GET /websocket/relay/ HTTP/1.1" 101 760 "-" "Python/3.9 aiohttp/3.8.3" "-"
2023-11-17 15:19:05,002 INFO     [-] awx.main.consumers client 'specific.ae9a4c7a732147a3aab282ecfa6a6968!92c1503a41e941448f0efb457e682979' disconnected from the broadcast group.
2023-11-17 15:19:05,002 INFO     [-] awx.main.consumers client 'specific.ae9a4c7a732147a3aab282ecfa6a6968!92c1503a41e941448f0efb457e682979' disconnected from the broadcast group.
2023-11-17 15:19:05,002 INFO     client 'specific.ae9a4c7a732147a3aab282ecfa6a6968!92c1503a41e941448f0efb457e682979' disconnected from the broadcast group.

$ kubectl logs -f awx-task-79675b847b-6nnvc -n awx

[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6
[wait-for-migrations] Waiting 16 seconds before next attempt
...
...
...
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 4507
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 4508
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 4509
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 4510
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 4511
[wait-for-migrations] Waiting 30 seconds before next attempt

$ kubectl edit pods awx-postgres-13-0 -n awx

    persistentVolumeClaim:
      claimName: postgres-13-awx-postgres-13-0
  - name: kube-api-access-c68qh
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T10:17:58Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T10:18:07Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T10:18:07Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-11-16T10:17:58Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://468bd77c4cea694e7626e39b11229f838063751f051742e4a27bf7e0faa4bd05
    image: docker.io/library/postgres:13
    imageID: docker.io/library/postgres@sha256:41f295f5776399a443eecde995f08d32aedd28aa1a51eb760c7adb1f7973bd38
    lastState: {}
    name: postgres
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-11-16T10:18:07Z"
  hostIP: 10.120.231.26
  phase: Running
  podIP: 10.42.0.10
  podIPs:
  - ip: 10.42.0.10
  qosClass: BestEffort
  startTime: "2023-11-16T10:17:58Z"

kurokobo · November 20, 2023, 2:09pm

Could you provide the output from following commands?

sudo systemctl status firewalld
sudo systemctl status nm-cloud-setup

kubectl -n kube-system get deployment,pod,svc -o wide
kubectl -n awx run -it --rm --restart=Never busybox --image=busybox:1.28 -- nsl
ookup awx-postgres-13
kubectl -n awx run -it --rm --restart=Never busybox --image=busybox:1.28 -- cat
 /etc/resolv.conf

kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash -c 'PGPASSWORD=An
sible123! psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'

Also try restarting CoreDNS:

kubectl -n kube-system delete pod -l k8s-app=kube-dns

fabeer · November 20, 2023, 3:22pm

$ sudo systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

$ sudo systemctl status nm-cloud-setup
Unit nm-cloud-setup.service could not be found.

$ kubectl -n kube-system get deployment,pod,svc -o wide

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS               IMAGES                                    SELECTOR
deployment.apps/coredns                  1/1     1            1           4d5h   coredns                  rancher/mirrored-coredns-coredns:1.10.1   k8s-app=kube-dns
deployment.apps/local-path-provisioner   1/1     1            1           4d5h   local-path-provisioner   rancher/local-path-provisioner:v0.0.24    app=local-path-provisioner
deployment.apps/traefik                  1/1     1            1           4d5h   traefik                  rancher/mirrored-library-traefik:2.10.5   app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
deployment.apps/metrics-server           1/1     1            1           4d5h   metrics-server           rancher/mirrored-metrics-server:v0.6.3    k8s-app=metrics-server

NAME                                          READY   STATUS      RESTARTS   AGE    IP          NODE                           NOMINATED NODE   READINESS GATES
pod/coredns-6799fbcd5-hjzhs                   1/1     Running     0          4d5h   10.42.0.4   sflnx004345.utenze.bankit.it   <none>           <none>
pod/local-path-provisioner-84db5d44d9-jqxzv   1/1     Running     0          4d5h   10.42.0.2   sflnx004345.utenze.bankit.it   <none>           <none>
pod/helm-install-traefik-crd-bp28f            0/1     Completed   0          4d5h   10.42.0.5   sflnx004345.utenze.bankit.it   <none>           <none>
pod/helm-install-traefik-sm6jp                0/1     Completed   1          4d5h   10.42.0.3   sflnx004345.utenze.bankit.it   <none>           <none>
pod/svclb-traefik-bc4c520d-nvhln              2/2     Running     0          4d5h   10.42.0.7   sflnx004345.utenze.bankit.it   <none>           <none>
pod/traefik-f4564c4f4-jfprx                   1/1     Running     0          4d5h   10.42.0.8   sflnx004345.utenze.bankit.it   <none>           <none>
pod/metrics-server-67c658944b-z56t6           1/1     Running     0          4d5h   10.42.0.6   sflnx004345.utenze.bankit.it   <none>           <none>

NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE    SELECTOR
service/kube-dns         ClusterIP      10.43.0.10     <none>          53/UDP,53/TCP,9153/TCP       4d5h   k8s-app=kube-dns
service/metrics-server   ClusterIP      10.43.77.250   <none>          443/TCP                      4d5h   k8s-app=metrics-server
service/traefik          LoadBalancer   10.43.81.86    10.120.231.26   80:32554/TCP,443:32683/TCP   4d5h   app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik

$ kubectl -n awx run -it --rm --restart=Never busybox --image=busybox:1.28 – nslookup awx-postgres-13

Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      awx-postgres-13
Address 1: 10.42.0.10 10-42-0-10.awx-postgres-13.awx.svc.cluster.local
pod "busybox" deleted

$ kubectl -n awx run -it --rm --restart=Never busybox --image=busybox:1.28 – cat /etc/resolv.conf

search awx.svc.cluster.local svc.cluster.local cluster.local utenze.bankit.it ac.bankit.it infra.bankit.it
nameserver 10.43.0.10
options ndots:5
pod "busybox" deleted

$ kubectl -n awx exec -it deployment/awx-web -c awx-web – bash -c ‘PGPASSWORD=my-password
psql -h awx-postgres-13 -p 5432 -U awx awx -c “\l”’


                             List of databases
   Name    | Owner | Encoding |  Collate   |   Ctype    | Access privileges
-----------+-------+----------+------------+------------+-------------------
 awx       | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx
 template1 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx
(4 rows)

kurokobo · November 22, 2023, 12:15pm

Thanks, so is it possible that connect PSQL from awx-task container?

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=<your password> psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'

If awx-manage still shows Name or service not known error,

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- awx-manage showmigrations
$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- awx-manage migrate

you should review the credentials.py.

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- cat /etc/tower/conf.d/credentials.py

fabeer · November 23, 2023, 9:46am

kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=Ansible-Dais psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'`
psql: error: could not translate host name "awx-postgres-13" to address: Name or service not known`

After restart pod:

kubectl delete  pod  $(kubectl get pod  -n awx | awk '{ print $1 }' |grep task) -n awx
pod "awx-task-79675b847b-8c8wr" deleted

 kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=Ansible-Dais psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'
                             List of databases
   Name    | Owner | Encoding |  Collate   |   Ctype    | Access privileges
-----------+-------+----------+------------+------------+-------------------
 awx       | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx
 template1 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx
(4 rows)

kubectl -n awx exec -it deployment/awx-task -c awx-task -- awx-manage showmigrations
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
 [X] 0012_alter_user_first_name_max_length
conf
 [X] 0001_initial
 [X] 0002_v310_copy_tower_settings
 [X] 0003_v310_JSONField_changes
 [X] 0004_v320_reencrypt
 [X] 0005_v330_rename_two_session_settings
 [X] 0006_v331_ldap_group_type
 [X] 0007_v380_rename_more_settings
 [X] 0008_subscriptions
 [X] 0009_rename_proot_settings
 [X] 0010_change_to_JSONField
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
main
 [X] 0001_initial
 [X] 0002_squashed_v300_release (18 squashed migrations)
 [X] 0003_squashed_v300_v303_updates (9 squashed migrations)
 [X] 0004_squashed_v310_release (6 squashed migrations)
 [X] 0005_squashed_v310_v313_updates (3 squashed migrations)
 [X] 0006_v320_release
 [X] 0007_v320_data_migrations
 [X] 0008_v320_drop_v1_credential_fields
 [X] 0009_v322_add_setting_field_for_activity_stream
 [X] 0010_v322_add_ovirt4_tower_inventory
 [X] 0011_v322_encrypt_survey_passwords
 [X] 0012_v322_update_cred_types
 [X] 0013_v330_multi_credential
 [X] 0014_v330_saved_launchtime_configs
 [X] 0015_v330_blank_start_args
 [X] 0016_v330_non_blank_workflow
 [X] 0017_v330_move_deprecated_stdout
 [X] 0018_v330_add_additional_stdout_events
 [X] 0019_v330_custom_virtualenv
 [X] 0020_v330_instancegroup_policies
 [X] 0021_v330_declare_new_rbac_roles
 [X] 0022_v330_create_new_rbac_roles
 [X] 0023_v330_inventory_multicred
 [X] 0024_v330_create_user_session_membership
 [X] 0025_v330_add_oauth_activity_stream_registrar
 [X] 0026_v330_delete_authtoken
 [X] 0027_v330_emitted_events
 [X] 0028_v330_add_tower_verify
 [X] 0030_v330_modify_application
 [X] 0031_v330_encrypt_oauth2_secret
 [X] 0032_v330_polymorphic_delete
 [X] 0033_v330_oauth_help_text
 [X] 0034_v330_delete_user_role
 [X] 0035_v330_more_oauth2_help_text
 [X] 0036_v330_credtype_remove_become_methods
 [X] 0037_v330_remove_legacy_fact_cleanup
 [X] 0038_v330_add_deleted_activitystream_actor
 [X] 0039_v330_custom_venv_help_text
 [X] 0040_v330_unifiedjob_controller_node
 [X] 0041_v330_update_oauth_refreshtoken
 [X] 0042_v330_org_member_role_deparent
 [X] 0043_v330_oauth2accesstoken_modified
 [X] 0044_v330_add_inventory_update_inventory
 [X] 0045_v330_instance_managed_by_policy
 [X] 0046_v330_remove_client_credentials_grant
 [X] 0047_v330_activitystream_instance
 [X] 0048_v330_django_created_modified_by_model_name
 [X] 0049_v330_validate_instance_capacity_adjustment
 [X] 0050_v340_drop_celery_tables
 [X] 0051_v340_job_slicing
 [X] 0052_v340_remove_project_scm_delete_on_next_update
 [X] 0053_v340_workflow_inventory
 [X] 0054_v340_workflow_convergence
 [X] 0055_v340_add_grafana_notification
 [X] 0056_v350_custom_venv_history
 [X] 0057_v350_remove_become_method_type
 [X] 0058_v350_remove_limit_limit
 [X] 0059_v350_remove_adhoc_limit
 [X] 0060_v350_update_schedule_uniqueness_constraint
 [X] 0061_v350_track_native_credentialtype_source
 [X] 0062_v350_new_playbook_stats
 [X] 0063_v350_org_host_limits
 [X] 0064_v350_analytics_state
 [X] 0065_v350_index_job_status
 [X] 0066_v350_inventorysource_custom_virtualenv
 [X] 0067_v350_credential_plugins
 [X] 0068_v350_index_event_created
 [X] 0069_v350_generate_unique_install_uuid
 [X] 0070_v350_gce_instance_id
 [X] 0071_v350_remove_system_tracking
 [X] 0072_v350_deprecate_fields
 [X] 0073_v360_create_instance_group_m2m
 [X] 0074_v360_migrate_instance_group_relations
 [X] 0075_v360_remove_old_instance_group_relations
 [X] 0076_v360_add_new_instance_group_relations
 [X] 0077_v360_add_default_orderings
 [X] 0078_v360_clear_sessions_tokens_jt
 [X] 0079_v360_rm_implicit_oauth2_apps
 [X] 0080_v360_replace_job_origin
 [X] 0081_v360_notify_on_start
 [X] 0082_v360_webhook_http_method
 [X] 0083_v360_job_branch_override
 [X] 0084_v360_token_description
 [X] 0085_v360_add_notificationtemplate_messages
 [X] 0086_v360_workflow_approval
 [X] 0087_v360_update_credential_injector_help_text
 [X] 0088_v360_dashboard_optimizations
 [X] 0089_v360_new_job_event_types
 [X] 0090_v360_WFJT_prompts
 [X] 0091_v360_approval_node_notifications
 [X] 0092_v360_webhook_mixin
 [X] 0093_v360_personal_access_tokens
 [X] 0094_v360_webhook_mixin2
 [X] 0095_v360_increase_instance_version_length
 [X] 0096_v360_container_groups
 [X] 0097_v360_workflowapproval_approved_or_denied_by
 [X] 0098_v360_rename_cyberark_aim_credential_type
 [X] 0099_v361_license_cleanup
 [X] 0100_v370_projectupdate_job_tags
 [X] 0101_v370_generate_new_uuids_for_iso_nodes
 [X] 0102_v370_unifiedjob_canceled
 [X] 0103_v370_remove_computed_fields
 [X] 0104_v370_cleanup_old_scan_jts
 [X] 0105_v370_remove_jobevent_parent_and_hosts
 [X] 0106_v370_remove_inventory_groups_with_active_failures
 [X] 0107_v370_workflow_convergence_api_toggle
 [X] 0108_v370_unifiedjob_dependencies_processed
 [X] 0109_v370_job_template_organization_field
 [X] 0110_v370_instance_ip_address
 [X] 0111_v370_delete_channelgroup
 [X] 0112_v370_workflow_node_identifier
 [X] 0113_v370_event_bigint
 [X] 0114_v370_remove_deprecated_manual_inventory_sources
 [X] 0115_v370_schedule_set_null
 [X] 0116_v400_remove_hipchat_notifications
 [X] 0117_v400_remove_cloudforms_inventory
 [X] 0118_add_remote_archive_scm_type
 [X] 0119_inventory_plugins
 [X] 0120_galaxy_credentials
 [X] 0121_delete_toweranalyticsstate
 [X] 0122_really_remove_cloudforms_inventory
 [X] 0123_drop_hg_support
 [X] 0124_execution_environments
 [X] 0125_more_ee_modeling_changes
 [X] 0126_executionenvironment_container_options
 [X] 0127_reset_pod_spec_override
 [X] 0128_organiaztion_read_roles_ee_admin
 [X] 0129_unifiedjob_installed_collections
 [X] 0130_ee_polymorphic_set_null
 [X] 0131_undo_org_polymorphic_ee
 [X] 0132_instancegroup_is_container_group
 [X] 0133_centrify_vault_credtype
 [X] 0134_unifiedjob_ansible_version
 [X] 0135_schedule_sort_fallback_to_id
 [X] 0136_scm_track_submodules
 [X] 0137_custom_inventory_scripts_removal_data
 [X] 0138_custom_inventory_scripts_removal
 [X] 0139_isolated_removal
 [X] 0140_rename
 [X] 0141_remove_isolated_instances
 [X] 0142_update_ee_image_field_description
 [X] 0143_hostmetric
 [X] 0144_event_partitions
 [X] 0145_deregister_managed_ee_objs
 [X] 0146_add_insights_inventory
 [X] 0147_validate_ee_image_field
 [X] 0148_unifiedjob_receptor_unit_id
 [X] 0149_remove_inventory_insights_credential
 [X] 0150_rename_inv_sources_inv_updates
 [X] 0151_rename_managed_by_tower
 [X] 0152_instance_node_type
 [X] 0153_instance_last_seen
 [X] 0154_set_default_uuid
 [X] 0155_improved_health_check
 [X] 0156_capture_mesh_topology
 [X] 0157_inventory_labels
 [X] 0158_make_instance_cpu_decimal
 [X] 0159_deprecate_inventory_source_UoPU_field
 [X] 0160_alter_schedule_rrule
 [X] 0161_unifiedjob_host_status_counts
 [X] 0162_alter_unifiedjob_dependent_jobs
 [X] 0163_convert_job_tags_to_textfield
 [X] 0164_remove_inventorysource_update_on_project_update
 [X] 0165_task_manager_refactor
 [X] 0166_alter_jobevent_host
 [X] 0167_project_signature_validation_credential
 [X] 0168_inventoryupdate_scm_revision
 [X] 0169_jt_prompt_everything_on_launch
 [X] 0170_node_and_link_state
 [X] 0171_add_health_check_started
 [X] 0172_prevent_instance_fallback
 [X] 0173_instancegroup_max_limits
 [X] 0174_ensure_org_ee_admin_roles
 [X] 0175_workflowjob_is_bulk_job
 [X] 0176_inventorysource_scm_branch
 [X] 0177_instance_group_role_addition
 [X] 0178_instance_group_admin_migration
 [X] 0179_change_cyberark_plugin_names
 [X] 0180_add_hostmetric_fields
 [X] 0181_hostmetricsummarymonthly
 [X] 0182_constructed_inventory
 [X] 0183_pre_django_upgrade
 [X] 0184_django_indexes
 [X] 0185_move_JSONBlob_to_JSONField
 [X] 0186_drop_django_taggit
 [X] 0187_hop_nodes
oauth2_provider
 [X] 0001_initial
 [X] 0002_auto_20190406_1805
 [X] 0003_auto_20201211_1314
 [X] 0004_auto_20200902_2022
 [X] 0005_auto_20211222_2352
sessions
 [X] 0001_initial
sites
 [X] 0001_initial
 [X] 0002_alter_domain_unique
social_django
 [X] 0001_initial (2 squashed migrations)
 [X] 0002_add_related_name (2 squashed migrations)
 [X] 0003_alter_email_max_length (2 squashed migrations)
 [X] 0004_auto_20160423_0400 (2 squashed migrations)
 [X] 0005_auto_20160727_2333 (1 squashed migrations)
 [X] 0006_partial
 [X] 0007_code_timestamp
 [X] 0008_partial_timestamp
 [X] 0009_auto_20191118_0520
 [X] 0010_uid_db_index
sso
 [X] 0001_initial
 [X] 0002_expand_provider_options
 [X] 0003_convert_saml_string_to_list

kubectl -n awx exec -it deployment/awx-task -c awx-task -- awx-manage migrate
Operations to perform:
  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso
Running migrations:
  No migrations to apply.

kubectl -n awx exec -it deployment/awx-task -c awx-task -- cat /etc/tower/conf.d/credentials.py
DATABASES = {
    'default': {
        'ATOMIC_REQUESTS': True,
        'ENGINE': 'awx.main.db.profiled_pg',
        'NAME': "awx",
        'USER': "awx",
        'PASSWORD': "<my-password>",
        'HOST': 'awx-postgres-13',
        'PORT': "5432",
        'OPTIONS': { 'sslmode': 'prefer',
                     'keepalives': 1,
                     'keepalives_idle': 5,
                     'keepalives_interval': 5,
                     'keepalives_count': 5,
        },
    }

However, the problem returns after a few minutes

kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=Ansible-Dais psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'
psql: error: could not translate host name "awx-postgres-13" to address: Name or service not known

kubectl -n awx exec -it deployment/awx-task -c awx-task -- awx-manage migrate
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
    if (connection.pg_version // 10000) < 12:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
    return getattr(self._connections[self._alias], item)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
    with self.temporary_connection():
  File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
    with self.cursor() as cursor:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
    return self._cursor()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
    self.ensure_connection()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
command terminated with exit code 1
ansible@sflnx004345 ~ $ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=Ansible-Dais psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'
psql: error: could not translate host name "awx-postgres-13" to address: Name or service not known
command terminated with exit code 2

fabeer · November 24, 2023, 7:52am

I add as further useful information for analysis that the installation of k3s and awx was done on a server behind a proxy

kurokobo · November 27, 2023, 1:24pm

Hmm, wired

Is the resolv.conf of the awx-task container correct? (both right after startup and after the issue occurs)
- kubectl -n awx exec -it deployment/awx-task -c awx-task -- cat /etc/resolv.conf
Are there any oddities in the CoreDNS or K3s logs?
- sudo journalctl -u k3s --no-pager -n 100
- kubectl -n kube-system logs deployment/coredns --tail=100
Are there enough resources (CPU, memory, etc.) available?

It seems more like an internal K3s routing problem than an AWX problem. If you don’t mind losing all your data, I want to know uninstalling K3s and reinstalling latest v1.27.7+k3s2 k3s solves the issue or not.

fabeer · November 27, 2023, 2:42pm

kubectl -n awx exec -it deployment/awx-task -c awx-task – cat /etc/resolv.conf

search awx.svc.cluster.local svc.cluster.local cluster.local example.com ac.bankit.it infra.bankit.it
nameserver 10.43.0.10
options ndots:5

kubectl -n kube-system get deployment,pod,svc -o wide

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS               IMAGES                                    SELECTOR
deployment.apps/local-path-provisioner   1/1     1            1           11d   local-path-provisioner   rancher/local-path-provisioner:v0.0.24    app=local-path-provisioner
deployment.apps/traefik                  1/1     1            1           11d   traefik                  rancher/mirrored-library-traefik:2.10.5   app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
deployment.apps/metrics-server           1/1     1            1           11d   metrics-server           rancher/mirrored-metrics-server:v0.6.3    k8s-app=metrics-server
deployment.apps/coredns                  1/1     1            1           11d   coredns                  rancher/mirrored-coredns-coredns:1.10.1   k8s-app=kube-dns

NAME                                          READY   STATUS      RESTARTS   AGE    IP           NODE                           NOMINATED NODE   READINESS GATES
pod/local-path-provisioner-84db5d44d9-jqxzv   1/1     Running     0          11d    10.42.0.2    server1.example.com   <none>           <none>
pod/helm-install-traefik-crd-bp28f            0/1     Completed   0          11d    10.42.0.5    server1.example.com   <none>           <none>
pod/helm-install-traefik-sm6jp                0/1     Completed   1          11d    10.42.0.3    server1.example.com   <none>           <none>
pod/svclb-traefik-bc4c520d-nvhln              2/2     Running     0          11d    10.42.0.7    server1.example.com   <none>           <none>
pod/traefik-f4564c4f4-jfprx                   1/1     Running     0          11d    10.42.0.8    server1.example.com   <none>           <none>
pod/metrics-server-67c658944b-z56t6           1/1     Running     0          11d    10.42.0.6    server1.example.com   <none>           <none>
pod/coredns-6799fbcd5-r2mfp                   1/1     Running     0          5d6h   10.42.0.39   server1.example.com   <none>           <none>

NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE   SELECTOR
service/kube-dns         ClusterIP      10.43.0.10     <none>          53/UDP,53/TCP,9153/TCP       11d   k8s-app=kube-dns
service/metrics-server   ClusterIP      10.43.77.250   <none>          443/TCP                      11d   k8s-app=metrics-server
service/traefik          LoadBalancer   10.43.81.86    10.120.231.26   80:32554/TCP,443:32683/TCP   11d   app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik

kubectl -n awx exec -it deployment/awx-task -c awx-task – cat /etc/resolv.conf

search awx.svc.cluster.local svc.cluster.local cluster.local example.com
nameserver 10.43.0.10
options ndots:5

sudo journalctl -u k3s --no-pager -n 100

-- Logs begin at Fri 2023-11-24 15:02:09 CET, end at Mon 2023-11-27 15:12:49 CET. --
Nov 27 14:58:25 server1.example.com k3s[2399506]: I1127 14:58:25.116075 2399506 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5db078-45b8-4d58-821a-47f18a7cf79a" path="/var/lib/kubelet/pods/ee5db078-45b8-4d58-821a-47f18a7cf79a/volumes"
Nov 27 14:58:29 server1.example.com k3s[2399506]: I1127 14:58:29.431940 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 14:59:28 server1.example.com k3s[2399506]: time="2023-11-27T14:59:28+01:00" level=info msg="COMPACT compactRev=1243901 targetCompactRev=1244349 currentRev=1245349"
Nov 27 14:59:28 server1.example.com k3s[2399506]: time="2023-11-27T14:59:28+01:00" level=error msg="Compact failed: failed to compact to revision 1244349: database is locked"
Nov 27 14:59:29 server1.example.com k3s[2399506]: I1127 14:59:29.432434 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.455608 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.455860 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456195 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456349 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456471 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456548 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456627 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.456955 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.457196 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.457563 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.457633 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.458517 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.458774 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.458904 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.459060 2399506 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.459148 2399506 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.459365 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.459656 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.459815 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.460503 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.460611 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.460700 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.460945 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 14:59:31 server1.example.com k3s[2399506]: I1127 14:59:31.461063 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 14:59:50 server1.example.com k3s[2399506]: I1127 14:59:50.081204 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:00:25 server1.example.com k3s[2399506]: E1127 15:00:25.763202 2399506 upgradeaware.go:425] Error proxying data from client to backend: write tcp 127.0.0.1:51256->127.0.0.1:6443: write: broken pipe
Nov 27 15:00:29 server1.example.com k3s[2399506]: I1127 15:00:29.431803 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:01:29 server1.example.com k3s[2399506]: I1127 15:01:29.433773 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:02:29 server1.example.com k3s[2399506]: I1127 15:02:29.432459 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:03:29 server1.example.com k3s[2399506]: I1127 15:03:29.432607 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:04:28 server1.example.com k3s[2399506]: time="2023-11-27T15:04:28+01:00" level=info msg="COMPACT compactRev=1243901 targetCompactRev=1244735 currentRev=1245735"
Nov 27 15:04:28 server1.example.com k3s[2399506]: time="2023-11-27T15:04:28+01:00" level=info msg="COMPACT deleted 810 rows from 834 revisions in 18.422051ms - compacted to 1244735/1245735"
Nov 27 15:04:29 server1.example.com k3s[2399506]: I1127 15:04:29.432588 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.455576 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.455904 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456046 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456269 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456400 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456492 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456576 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456720 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.456980 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.457403 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.457515 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.457731 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.457841 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458001 2399506 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458076 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458296 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458743 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458857 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.458949 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.459256 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.459320 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.460519 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.460628 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:04:31 server1.example.com k3s[2399506]: I1127 15:04:31.460767 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:04:50 server1.example.com k3s[2399506]: I1127 15:04:50.088094 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:05:29 server1.example.com k3s[2399506]: I1127 15:05:29.431282 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:06:29 server1.example.com k3s[2399506]: I1127 15:06:29.433344 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:07:29 server1.example.com k3s[2399506]: I1127 15:07:29.432833 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:08:29 server1.example.com k3s[2399506]: I1127 15:08:29.432561 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:09:28 server1.example.com k3s[2399506]: time="2023-11-27T15:09:28+01:00" level=info msg="COMPACT compactRev=1244735 targetCompactRev=1245122 currentRev=1246122"
Nov 27 15:09:28 server1.example.com k3s[2399506]: time="2023-11-27T15:09:28+01:00" level=info msg="COMPACT deleted 362 rows from 387 revisions in 13.628869ms - compacted to 1245122/1246122"
Nov 27 15:09:29 server1.example.com k3s[2399506]: I1127 15:09:29.434529 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.456482 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.456645 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457058 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457184 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457230 2399506 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457467 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457747 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457858 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.457995 2399506 handler.go:232] Adding GroupVersion k3s.cattle.io v1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.458064 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.458180 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.459161 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.459587 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.459719 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.459888 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.460125 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.460308 2399506 handler.go:232] Adding GroupVersion helm.cattle.io v1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.460609 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.460751 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.460922 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461135 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461223 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461600 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461728 2399506 handler.go:232] Adding GroupVersion traefik.io v1alpha1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461837 2399506 handler.go:232] Adding GroupVersion awx.ansible.com v1beta1 to ResourceManager
Nov 27 15:09:31 server1.example.com k3s[2399506]: I1127 15:09:31.461958 2399506 handler.go:232] Adding GroupVersion traefik.containo.us v1alpha1 to ResourceManager
Nov 27 15:09:50 server1.example.com k3s[2399506]: I1127 15:09:50.089647 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:10:29 server1.example.com k3s[2399506]: I1127 15:10:29.433261 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:11:29 server1.example.com k3s[2399506]: I1127 15:11:29.433073 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
Nov 27 15:12:29 server1.example.com k3s[2399506]: I1127 15:12:29.432996 2399506 handler.go:232] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager

kubectl -n kube-system logs deployment/coredns --tail=100

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server

A few days ago I installed version v1.28.3+k3s2, but the problem remained.
Do you still recommend testing with the version of k3s you indicated?

kurokobo · November 29, 2023, 12:49pm

Ah okay so it’s not required to try 1.27.7.

The fact that the same problem occurs in multiple K3s versions could be a VXLAN issue with Flannel or a resolv.conf issue with the K3s host.

Do you have any domains with wild card in your resolv.conf?
Can you reproduce the issue with CoreDNS logging enabled and see if the PSQL hostname is queried?

Add log under .:53 { by kubectl -n kube-system edit cm coredns
Restart CoreDNS: kubectl -n kube-system delete pod -l k8s-app=kube-dns
Watch the logs: kubectl -n kube-system logs -f deployment/coredns --tail=100
Try connectiong PSQL from awx-task container, and see how the hostname is queried when it success and fails: kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=<your password> psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'

fabeer · December 11, 2023, 1:14pm

the resolv.conf is like this:

search domain1.mydomain.it domain2.mydomain.it domain3.mydomain.it
options rotate timeout:2 attempts:2
nameserver ip-dns1
nameserver ip-dns2

After enabled CoreDNS logging

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash -c 'PGPASSWORD=<my-password> psql -h awx-postgres-13 -p 5432 -U awx awx -c "\l"'
                             List of databases
   Name    | Owner | Encoding |  Collate   |   Ctype    | Access privileges
-----------+-------+----------+------------+------------+-------------------
 awx       | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | awx   | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx
 template1 | awx   | UTF8     | en_US.utf8 | en_US.utf8 | =c/awx           +
           |       |          |            |            | awx=CTc/awx

$  kubectl -n kube-system logs -f deployment/coredns --tail=100

...
...
...
[INFO] 10.42.0.87:34787 - 12400 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000063762s
[INFO] 10.42.0.87:34787 - 14195 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000024755s
[INFO] 10.42.0.87:60403 - 50469 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000065253s
[INFO] 10.42.0.87:60403 - 52792 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000098836s
[INFO] 10.42.0.87:48754 - 44432 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000075694s
[INFO] 10.42.0.87:48754 - 24466 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000118922s
[INFO] 10.42.0.87:50078 - 41936 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.0001782s
[INFO] 10.42.0.87:50078 - 10966 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000087953s
[INFO] 10.42.0.59:48933 - 7887 "AAAA IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000286996s
[INFO] 10.42.0.59:48933 - 6848 "A IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.00032691s
[INFO] 10.42.0.59:41250 - 61725 "AAAA IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000122946s
[INFO] 10.42.0.59:41250 - 41313 "A IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000146757s
[INFO] 10.42.0.59:35624 - 20541 "AAAA IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000201697s
[INFO] 10.42.0.59:35624 - 48690 "A IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000143936s
[INFO] 10.42.0.59:44484 - 15169 "AAAA IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.001519043s
[INFO] 10.42.0.59:44484 - 14526 "A IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.002342991s
[INFO] 10.42.0.59:35306 - 41511 "AAAA IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.001266478s
[INFO] 10.42.0.59:35306 - 22051 "A IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.001812394s
[INFO] 10.42.0.59:57896 - 21640 "AAAA IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.00093409s
[INFO] 10.42.0.59:57896 - 17292 "A IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.001277992s
[INFO] 10.42.0.59:53630 - 13632 "AAAA IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 200 0.001029476s
[INFO] 10.42.0.59:53630 - 9542 "A IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 138 0.00109339s
[INFO] 10.42.0.59:59865 - 43685 "A IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000104598s
[INFO] 10.42.0.59:59865 - 46242 "AAAA IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000153881s
[INFO] 10.42.0.59:52290 - 45103 "AAAA IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000058164s
[INFO] 10.42.0.59:52290 - 17453 "A IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000103554s
[INFO] 10.42.0.59:59459 - 31799 "AAAA IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000072603s
[INFO] 10.42.0.59:59459 - 53284 "A IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000095919s
[INFO] 10.42.0.59:32858 - 45190 "AAAA IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000050698s
[INFO] 10.42.0.59:32858 - 33413 "A IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000073552s
[INFO] 10.42.0.59:41387 - 4961 "AAAA IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.000089055s
[INFO] 10.42.0.59:41387 - 17516 "A IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.00010161s
[INFO] 10.42.0.59:53181 - 52041 "AAAA IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000050707s
[INFO] 10.42.0.59:53181 - 17484 "A IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000034105s
[INFO] 10.42.0.59:44092 - 27705 "AAAA IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 200 0.000049071s
[INFO] 10.42.0.59:44092 - 33596 "A IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 138 0.000028699s
[INFO] 10.42.0.59:45679 - 1782 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000133719s
[INFO] 10.42.0.59:45002 - 45588 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.00008209s
[INFO] 10.42.0.59:40287 - 35462 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000095162s
[INFO] 10.42.0.59:40287 - 42943 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000165876s
[INFO] 10.42.0.59:60074 - 27 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000062425s
[INFO] 10.42.0.59:45679 - 63230 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000180307s
[INFO] 10.42.0.59:45002 - 531 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000075696s
[INFO] 10.42.0.59:60074 - 48387 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000223719s
[INFO] 10.42.0.59:39629 - 10749 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000217455s
[INFO] 10.42.0.59:39629 - 40693 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000089069s
[INFO] 10.42.0.59:45867 - 63293 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000096633s
...
...
...
[INFO] 10.42.0.59:49770 - 63800 "A IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000025373s
[INFO] 10.42.0.59:44524 - 42119 "AAAA IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.00002737s
[INFO] 10.42.0.59:44524 - 39289 "A IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000024527s
[INFO] 10.42.0.59:38558 - 54993 "AAAA IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000024462s
[INFO] 10.42.0.59:38558 - 41683 "A IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.00001972s
[INFO] 10.42.0.59:58069 - 62600 "AAAA IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000024336s
[INFO] 10.42.0.59:58069 - 23433 "A IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000018842s
[INFO] 10.42.0.59:39124 - 55641 "AAAA IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.00002694s
[INFO] 10.42.0.59:39124 - 22107 "A IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.000018596s
[INFO] 10.42.0.59:55848 - 18101 "AAAA IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000022835s
[INFO] 10.42.0.59:55848 - 52147 "A IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000018915s
[INFO] 10.42.0.59:34504 - 56979 "AAAA IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 200 0.000067282s
[INFO] 10.42.0.59:34504 - 53650 "A IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 138 0.000025791s
[INFO] 10.42.0.59:50614 - 64738 "A IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000118839s
[INFO] 10.42.0.59:50614 - 28632 "AAAA IN git.domain1.mydomain.it.awx.svc.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000179268s
[INFO] 10.42.0.59:37935 - 13620 "A IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000099555s
[INFO] 10.42.0.59:37935 - 60467 "AAAA IN git.domain1.mydomain.it.svc.cluster.local. udp 56 false 512" NXDOMAIN qr,aa,rd 149 0.000157938s
[INFO] 10.42.0.59:51287 - 43849 "AAAA IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000052514s
[INFO] 10.42.0.59:51287 - 54869 "A IN git.domain1.mydomain.it.cluster.local. udp 52 false 512" NXDOMAIN qr,aa,rd 145 0.000125463s
[INFO] 10.42.0.59:40856 - 4211 "AAAA IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000059757s
[INFO] 10.42.0.59:40856 - 32880 "A IN git.domain1.mydomain.it.domain1.mydomain.it. udp 55 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000078748s
[INFO] 10.42.0.59:59773 - 14323 "AAAA IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.00014418s
[INFO] 10.42.0.59:59773 - 21495 "A IN git.domain1.mydomain.it.domain2.mydomain.it. udp 51 false 512" NXDOMAIN qr,aa,rd,ra 150 0.000092658s
[INFO] 10.42.0.59:46827 - 33259 "AAAA IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000064926s
[INFO] 10.42.0.59:46827 - 37367 "A IN git.domain1.mydomain.it.domain3.mydomain.it. udp 54 false 512" NXDOMAIN qr,aa,rd,ra 156 0.000033937s
[INFO] 10.42.0.59:47802 - 46766 "AAAA IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 200 0.000059984s
[INFO] 10.42.0.59:47802 - 40877 "A IN git.domain1.mydomain.it. udp 38 false 512" NOERROR qr,aa,rd,ra 138 0.000103136s
[INFO] 10.42.0.59:52754 - 32502 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000077878s
[INFO] 10.42.0.59:52754 - 35570 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000386029s
[INFO] 10.42.0.59:58414 - 21225 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000052519s
[INFO] 10.42.0.59:58414 - 24555 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000027373s
[INFO] 10.42.0.59:46332 - 55941 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000056295s
[INFO] 10.42.0.59:58321 - 47715 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000030309s
[INFO] 10.42.0.59:58321 - 53861 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000025403s
[INFO] 10.42.0.59:46332 - 28038 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000022848s
[INFO] 10.42.0.59:49197 - 24264 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000137179s
[INFO] 10.42.0.59:49197 - 18625 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000079025s
[INFO] 10.42.0.59:58705 - 3282 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000187918s
[INFO] 10.42.0.59:58705 - 54743 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000095567s
[INFO] 10.42.0.59:50428 - 6427 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000131829s
[INFO] 10.42.0.59:40818 - 7728 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000090077s
[INFO] 10.42.0.59:40818 - 45621 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000109335s
[INFO] 10.42.0.59:50428 - 18948 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000058867s
[INFO] 10.42.0.59:50094 - 11805 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000098789s
[INFO] 10.42.0.59:50094 - 36891 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000057181s
[INFO] 10.42.0.59:55721 - 22240 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" 
[INFO] 10.42.0.59:52091 - 32993 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000099509s
[INFO] 10.42.0.59:52091 - 31210 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000053967s
[INFO] 10.42.0.59:59380 - 33877 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000090434s
[INFO] 10.42.0.59:59380 - 10584 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000134079s
[INFO] 10.42.0.59:45368 - 17192 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000297696s
[INFO] 10.42.0.59:45368 - 2095 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000205249s
[INFO] 10.42.0.59:53707 - 58899 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000176266s
[INFO] 10.42.0.59:53707 - 11109 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000138792s
[INFO] 10.42.0.59:51005 - 5478 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000076623s
[INFO] 10.42.0.59:51005 - 44129 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000028424s
[INFO] 10.42.0.59:53311 - 1770 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000179498s
[INFO] 10.42.0.59:53311 - 40173 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000694986s
[INFO] 10.42.0.59:57657 - 43244 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000083664s
[INFO] 10.42.0.59:57657 - 25833 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000153485s
[INFO] 10.42.0.59:40892 - 64170 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000138791s
[INFO] 10.42.0.59:40892 - 36260 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000076626s
[INFO] 10.42.0.59:56155 - 39339 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000078808s
[INFO] 10.42.0.59:56155 - 40617 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000066045s
[INFO] 10.42.0.59:46347 - 4382 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000065682s
[INFO] 10.42.0.59:46347 - 12581 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000054672s
[INFO] 10.42.0.59:49092 - 26459 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000157439s
[INFO] 10.42.0.59:49092 - 43349 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000109994s
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] 10.42.0.59:54068 - 20054 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000366215s
[INFO] 10.42.0.59:54068 - 2143 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000439896s
[INFO] 10.42.0.61:34127 - 12550 "AAAA IN sflnx006100.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000203531s
[INFO] 10.42.0.61:50599 - 13598 "AAAA IN sflnx006099.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000099947s
[INFO] 10.42.0.61:50599 - 51228 "A IN sflnx006099.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000052595s
[INFO] 10.42.0.61:34127 - 59396 "A IN sflnx006100.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000045332s
[INFO] 10.42.0.61:33021 - 33618 "AAAA IN sflnx006100.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.00007347s
[INFO] 10.42.0.61:33021 - 46676 "A IN sflnx006100.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000049262s
[INFO] 10.42.0.61:59842 - 62269 "AAAA IN sflnx006100.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000064626s
[INFO] 10.42.0.61:59842 - 60991 "A IN sflnx006100.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000055656s
[INFO] 10.42.0.61:39391 - 29348 "A IN sflnx006099.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000087076s
[INFO] 10.42.0.61:39391 - 56742 "AAAA IN sflnx006099.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000050607s
[INFO] 10.42.0.61:43796 - 7185 "AAAA IN sflnx006099.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.00007871s
[INFO] 10.42.0.61:43796 - 13331 "A IN sflnx006099.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000051298s
[INFO] 10.42.0.61:42690 - 10283 "AAAA IN sflnx006100.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.001228021s
[INFO] 10.42.0.61:46156 - 27500 "A IN sflnx006099.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.000939439s
[INFO] 10.42.0.61:46156 - 21098 "AAAA IN sflnx006099.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.001248479s
[INFO] 10.42.0.61:43947 - 23857 "A IN sflnx006099.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000920805s
[INFO] 10.42.0.61:43947 - 9783 "AAAA IN sflnx006099.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.0070675s
[INFO] 10.42.0.61:42690 - 11305 "A IN sflnx006100.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.012375074s
[INFO] 10.42.0.61:54538 - 6476 "A IN sflnx006099.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.000813689s
[INFO] 10.42.0.61:54538 - 47950 "AAAA IN sflnx006099.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.001380277s
[INFO] 10.42.0.61:48328 - 64984 "AAAA IN sflnx006099.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 149 0.000841437s
[INFO] 10.42.0.61:48328 - 1499 "A IN sflnx006099.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 90 0.001409646s
[INFO] 10.42.0.61:52953 - 30751 "AAAA IN sflnx006100.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000771426s
[INFO] 10.42.0.61:52953 - 2845 "A IN sflnx006100.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000768848s
[INFO] 10.42.0.61:52740 - 25777 "AAAA IN sflnx006101.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000081421s
[INFO] 10.42.0.61:52740 - 58802 "A IN sflnx006101.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000057629s
[INFO] 10.42.0.61:49594 - 49337 "AAAA IN sflnx006101.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000076729s
[INFO] 10.42.0.61:49594 - 50356 "A IN sflnx006101.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000051295s
[INFO] 10.42.0.61:45111 - 25228 "AAAA IN sflnx006103.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000080521s
[INFO] 10.42.0.61:55057 - 41661 "A IN sflnx006101.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000057923s
[INFO] 10.42.0.61:55057 - 51903 "AAAA IN sflnx006101.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000053675s
[INFO] 10.42.0.61:45111 - 40578 "A IN sflnx006103.domain1.mydomain.it.awx.svc.cluster.local. udp 68 false 512" NXDOMAIN qr,aa,rd 161 0.000052488s
[INFO] 10.42.0.61:40665 - 26410 "AAAA IN sflnx006103.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000071696s
[INFO] 10.42.0.61:40665 - 8232 "A IN sflnx006103.domain1.mydomain.it.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000059545s
[INFO] 10.42.0.61:49361 - 1238 "AAAA IN sflnx006101.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.001125334s
[INFO] 10.42.0.61:38540 - 29674 "A IN sflnx006103.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.00006264s
[INFO] 10.42.0.61:38540 - 43031 "AAAA IN sflnx006103.domain1.mydomain.it.cluster.local. udp 60 false 512" NXDOMAIN qr,aa,rd 153 0.000050313s
[INFO] 10.42.0.61:49361 - 51668 "A IN sflnx006101.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.001625984s
[INFO] 10.42.0.61:45408 - 2229 "AAAA IN sflnx006100.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.001039256s
[INFO] 10.42.0.61:34434 - 25962 "AAAA IN sflnx006101.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.000805911s
[INFO] 10.42.0.61:45408 - 16050 "A IN sflnx006100.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.001744867s
[INFO] 10.42.0.61:34434 - 21623 "A IN sflnx006101.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.001574421s
[INFO] 10.42.0.61:35737 - 38999 "A IN sflnx006100.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 90 0.001429554s
[INFO] 10.42.0.61:59354 - 54456 "AAAA IN sflnx006101.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.001417532s
[INFO] 10.42.0.59:53749 - 25022 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000043652s
[INFO] 10.42.0.59:44098 - 25721 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000034118s
[INFO] 10.42.0.59:44098 - 9087 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000027673s
[INFO] 10.42.0.59:53749 - 60835 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.00002s
[INFO] 10.42.0.59:56431 - 15646 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000023327s
[INFO] 10.42.0.59:56431 - 39424 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000019403s
[INFO] 10.42.0.59:44984 - 51492 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000055712s
[INFO] 10.42.0.59:44984 - 61478 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.00002519s
[INFO] 10.42.0.61:59354 - 35517 "A IN sflnx006101.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.00761046s
[INFO] 10.42.0.61:34717 - 31673 "AAAA IN sflnx006103.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.010296709s
[INFO] 10.42.0.61:34717 - 47802 "A IN sflnx006103.domain1.mydomain.it.domain1.mydomain.it. udp 63 false 512" NXDOMAIN qr,aa,rd,ra 166 0.010231251s
[INFO] 10.42.0.61:35737 - 3928 "AAAA IN sflnx006100.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 149 0.008571783s
[INFO] 10.42.0.61:35384 - 59042 "A IN sflnx006103.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.004082468s
[INFO] 10.42.0.61:35384 - 26528 "AAAA IN sflnx006103.domain1.mydomain.it.domain2.mydomain.it. udp 59 false 512" NXDOMAIN qr,aa,rd,ra 158 0.00421049s
[INFO] 10.42.0.61:55865 - 25183 "A IN sflnx006101.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 90 0.004508872s
[INFO] 10.42.0.61:55865 - 48985 "AAAA IN sflnx006101.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 149 0.004569815s
[INFO] 10.42.0.61:36623 - 39778 "AAAA IN sflnx006103.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.00109028s
[INFO] 10.42.0.61:36623 - 64608 "A IN sflnx006103.domain1.mydomain.it.domain3.mydomain.it. udp 62 false 512" NXDOMAIN qr,aa,rd,ra 164 0.001087475s
[INFO] 10.42.0.61:60915 - 19453 "AAAA IN sflnx006103.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 149 0.000806288s
[INFO] 10.42.0.61:60915 - 9215 "A IN sflnx006103.domain1.mydomain.it. udp 46 false 512" NOERROR qr,aa,rd,ra 90 0.00096847s
[INFO] 10.42.0.59:53149 - 42634 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000096559s
[INFO] 10.42.0.59:53149 - 32910 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000161091s
[INFO] 10.42.0.59:53871 - 59816 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000308442s
[INFO] 10.42.0.59:53871 - 57747 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000380226s
[INFO] 10.42.0.59:40330 - 53383 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000105693s
[INFO] 10.42.0.59:40330 - 30593 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000111035s
[INFO] 10.42.0.59:57291 - 21958 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.00014784s
[INFO] 10.42.0.59:57291 - 19906 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000074615s
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] 10.42.0.59:57226 - 19725 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000337935s
[INFO] 10.42.0.59:57226 - 42501 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000528722s
[INFO] 10.42.0.59:52813 - 6485 "A IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 108 0.000096519s
[INFO] 10.42.0.59:52813 - 8012 "AAAA IN awx-postgres-13.awx.svc.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.00014539s
...
...
...

As soon as the problem recurs the same log contains only these entries:

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override

kurokobo · December 11, 2023, 2:54pm

Thanks for the update. So, when the issue occurs, no queries are sent to CoreDNS? I believe this is preventing all name resolution within k3s, not just AWX.

Can you try wiping all rules for iptables and restarting k3s? Check to see if the issue resolves after restarting k3s and if the issue reappears a short time after it is resolved.

# Accept all traffic and wipe all rules
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -t nat -F; sudo iptables -t mangle -F; sudo iptables -F; sudo iptables -X

# Restarting K3s re-creates all rules for K3s
sudo systemctl restart k3s

fabeer · December 12, 2023, 3:13pm

no changes, but…
… it seems to be a k3s related problem.

I create a simple Pod to use as a test environment for dns:

$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

$ kubectl get pods -o wide
NAME       READY   STATUS    RESTARTS   AGE     IP           NODE                           NOMINATED NODE   READINESS GATES
dnsutils   1/1     Running   0          3m40s   10.42.0.81   xxxx.yyyy.zzzz.   <none>           <none>

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.43.0.1

$ kubectl exec -ti dnsutils -- nslookup google.com
Server:         10.43.0.10
Address:        10.43.0.10#53

Non-authoritative answer:
Name:   google.com
Address: 142.250.180.142

But after about ten minutes the requests to the DNS began to give problems:

$ kubectl exec -i -t dnsutils -- nslookup kubernetes.default
;; connection timed out; no servers could be reached

command terminated with exit code 1

However, the pods were reachable via ip address

$ kubectl get pods -o wide -n kube-system

NAME                                      READY   STATUS      RESTARTS   AGE   IP          
local-path-provisioner-84db5d44d9-jqxzv   1/1     Running     0          26d   10.42.0.2   
svclb-traefik-bc4c520d-nvhln              2/2     Running     0          26d   10.42.0.7   
helm-install-traefik-crd-slxhl            0/1     Completed   0          13d   <none>      
helm-install-traefik-r464x                0/1     Completed   0          13d   <none>      
metrics-server-67c658944b-z56t6           1/1     Running     0          26d   10.42.0.6   
traefik-f4564c4f4-jfprx                   1/1     Running     0          26d   10.42.0.8   
coredns-6799fbcd5-4t5p7                   1/1     Running     0          18m   10.42.0.83


$ kubectl exec -i -t dnsutils -- ping 10.42.0.6
PING 10.42.0.6 (10.42.0.6) 56(84) bytes of data.
64 bytes from 10.42.0.6: icmp_seq=1 ttl=64 time=0.072 ms
64 bytes from 10.42.0.6: icmp_seq=2 ttl=64 time=0.095 ms
^C
--- 10.42.0.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1040ms
rtt min/avg/max/mdev = 0.072/0.083/0.095/0.014 ms

$ kubectl exec -i -t dnsutils -- ping 10.42.0.7
PING 10.42.0.7 (10.42.0.7) 56(84) bytes of data.
64 bytes from 10.42.0.7: icmp_seq=1 ttl=64 time=0.144 ms
64 bytes from 10.42.0.7: icmp_seq=2 ttl=64 time=0.111 ms
^C
--- 10.42.0.7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1064ms
rtt min/avg/max/mdev = 0.111/0.127/0.144/0.020 ms

$ kubectl exec -i -t dnsutils -- ping 10.42.0.83
PING 10.42.0.83 (10.42.0.83) 56(84) bytes of data.
64 bytes from 10.42.0.83: icmp_seq=1 ttl=64 time=0.096 ms
64 bytes from 10.42.0.83: icmp_seq=2 ttl=64 time=0.082 ms
^C
--- 10.42.0.83 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.082/0.089/0.096/0.007 ms

Problem disappeared for a while after restarting kube-dns…

So same behavior as awx.
I focus on the k3s side analysis

Topic		Replies	Views
Getting wait-for-migrations log inside awx-task and awx-web pods AWX Project awx , kubernetes	4	2	August 29, 2023
AWX migration from docker based env to k8s AWX Project awx , kubernetes	3	1	May 17, 2023
upgrade 1.0.0 to 1.1.4 stuck repeating [waiting for migrations] AWX Project awx	1	0	February 2, 2023
Upgrade with awx-operator 1.1.2 to 1.1.3 (or newer) fails AWX Project awx	10	2	March 7, 2023
AWX upgrade stuck Get Help awx , awx-operator , kubernetes , ansible	9	413	August 9, 2024

Awx-task stuck on wait-for-migrations

Related topics