AWX-Operator - cannot access NodePort

roelvandepaar · September 23, 2024, 11:58am

I tried to install AWX with the recommended installation method with the AWX-Operator.
(awx-operator version/tag 2.19.1)
I used this install guide: awx-operator/docs/installation/basic-install.md at devel · ansible/awx-operator · GitHub

This is what kubectl shows me:

$ kubectl get deploy,pods,svc -n awx
NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx-demo-task                     1/1     1            1           4h5m
deployment.apps/awx-demo-web                      1/1     1            1           4h5m
deployment.apps/awx-operator-controller-manager   1/1     1            1           4h28m

NAME                                                   READY   STATUS      RESTARTS         AGE
pod/awx-demo-migration-24.6.1-74wsx                    0/1     Completed   0                4h3m
pod/awx-demo-postgres-15-0                             1/1     Running     0                4h6m
pod/awx-demo-task-8588564675-9m5mc                     4/4     Running     0                4h5m
pod/awx-demo-web-76fc57c699-k7wk5                      3/3     Running     0                4h5m
pod/awx-operator-controller-manager-666ddcf9c5-92gcj   2/2     Running     33 (5m37s ago)   4h28m

NAME                                                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/awx-demo-postgres-15                              ClusterIP   None           <none>        5432/TCP       4h6m
service/awx-demo-service                                  NodePort    10.43.16.232   <none>        80:32241/TCP   4h5m
service/awx-operator-controller-manager-metrics-service   ClusterIP   10.43.64.233   <none>        8443/TCP       4h28m

Unfortunatelly port 32241 of the node is not exposed.
It is not visible with netstat and I cant access it through browser or curl.

There is a python django Exception in the logs of the awx-demo-web pod.
I cannot attach it here as it seems though.

Does someone have an idea or hints for troubleshooting this issue?

kurokobo · September 23, 2024, 3:27pm

Hi, which kubernetes distributions are you using?

The logs are important to understand your situation. Please just paste here or upload to the third-party service e.g. pastebin.

roelvandepaar · September 24, 2024, 7:05am

Hello kurokobo, thanks for the answer.
I am using k3s to deploy the cluster.

This is the log of the awx-demo-web pod:

2024-09-23 07:40:53,264 INFO RPC interface 'supervisor' initialized
2024-09-23 07:40:53,264 INFO RPC interface 'supervisor' initialized
2024-09-23 07:40:53,267 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-09-23 07:40:53,267 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-09-23 07:40:53,267 INFO supervisord started with pid 7
2024-09-23 07:40:53,267 INFO supervisord started with pid 7
2024-09-23 07:40:54,280 INFO spawned: 'superwatcher' with pid 13
2024-09-23 07:40:54,280 INFO spawned: 'superwatcher' with pid 13
2024-09-23 07:40:54,297 INFO spawned: 'nginx' with pid 14
2024-09-23 07:40:54,297 INFO spawned: 'nginx' with pid 14
2024-09-23 07:40:54,327 INFO spawned: 'uwsgi' with pid 15
2024-09-23 07:40:54,327 INFO spawned: 'uwsgi' with pid 15
2024-09-23 07:40:54,352 INFO spawned: 'daphne' with pid 16
2024-09-23 07:40:54,352 INFO spawned: 'daphne' with pid 16
2024-09-23 07:40:54,421 INFO spawned: 'awx-cache-clear' with pid 17
2024-09-23 07:40:54,421 INFO spawned: 'awx-cache-clear' with pid 17
2024-09-23 07:40:54,456 INFO spawned: 'ws-heartbeat' with pid 18
2024-09-23 07:40:54,456 INFO spawned: 'ws-heartbeat' with pid 18
READY
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.24 (64bit) on [Mon Sep 23 07:40:54 2024] ***
compiled with version: 11.4.1 20231218 (Red Hat 11.4.1-3) on 02 July 2024 20:15:47
os: Linux-5.14.0-427.33.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 28 17:34:59 UTC 2024
nodename: awx-demo-web-76fc57c699-k7wk5
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
Python version: 3.11.9 (main, Jun 11 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x7fe4b656ccf8
your server socket listen backlog is limited to 128 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 15)
spawned uWSGI worker 1 (pid: 20, cores: 1)
spawned uWSGI worker 2 (pid: 21, cores: 1)
spawned uWSGI worker 3 (pid: 22, cores: 1)
spawned uWSGI worker 4 (pid: 23, cores: 1)
spawned uWSGI worker 5 (pid: 24, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2024-09-23 07:40:56,224 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-23 07:40:56,224 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-23 07:41:24,317 INFO success: nginx entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:24,317 INFO success: nginx entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:24,323 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:24,323 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,331 INFO success: daphne entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,331 INFO success: daphne entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,332 INFO success: awx-cache-clear entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,332 INFO success: awx-cache-clear entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,332 INFO success: ws-heartbeat entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-09-23 07:41:25,332 INFO success: ws-heartbeat entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
WSGI app 0 (mountpoint='/') ready in 93 seconds on interpreter 0x7fe4b656ccf8 pid: 23 (default app)
WSGI app 0 (mountpoint='/') ready in 93 seconds on interpreter 0x7fe4b656ccf8 pid: 22 (default app)
WSGI app 0 (mountpoint='/') ready in 93 seconds on interpreter 0x7fe4b656ccf8 pid: 20 (default app)
WSGI app 0 (mountpoint='/') ready in 93 seconds on interpreter 0x7fe4b656ccf8 pid: 24 (default app)
WSGI app 0 (mountpoint='/') ready in 93 seconds on interpreter 0x7fe4b656ccf8 pid: 21 (default app)
2024-09-23 07:42:29,178 ERROR    [204a556eaab4444e80a8846ee7ca5575] django.request Internal Server Error: /
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "django_migrations" does not exist
LINE 1: SELECT 1 AS "a" FROM "django_migrations" WHERE ("django_migr...
                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 133, in __call__
    response = self.process_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/middleware.py", line 198, in process_request
    if is_migrating() and getattr(resolve(request.path), 'url_name', '') != 'migrations_notran':
       ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/common.py", line 196, in _memoizer
    value = f(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/middleware.py", line 193, in is_migrating
    return not MigrationRecorder(connection).migration_qs.filter(app='main', name=latest_name).exists()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 1241, in exists
    return self.query.has_results(using=self.db)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 598, in has_results
    return compiler.has_results()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/compiler.py", line 1530, in has_results
    return bool(self.execute_sql(SINGLE))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: relation "django_migrations" does not exist
LINE 1: SELECT 1 AS "a" FROM "django_migrations" WHERE ("django_migr...
                             ^
2024-09-23 07:42:29,178 ERROR    [204a556eaab4444e80a8846ee7ca5575] django.request Internal Server Error: /
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedTable: relation "django_migrations" does not exist
LINE 1: SELECT 1 AS "a" FROM "django_migrations" WHERE ("django_migr...
                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 133, in __call__
    response = self.process_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/middleware.py", line 198, in process_request
    if is_migrating() and getattr(resolve(request.path), 'url_name', '') != 'migrations_notran':
       ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/common.py", line 196, in _memoizer
    value = f(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/middleware.py", line 193, in is_migrating
    return not MigrationRecorder(connection).migration_qs.filter(app='main', name=latest_name).exists()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 1241, in exists
    return self.query.has_results(using=self.db)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 598, in has_results
    return compiler.has_results()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/compiler.py", line 1530, in has_results
    return bool(self.execute_sql(SINGLE))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: relation "django_migrations" does not exist
LINE 1: SELECT 1 AS "a" FROM "django_migrations" WHERE ("django_migr...
                             ^
10.42.0.1 - - [23/Sep/2024:07:42:29 +0000] "GET / HTTP/1.1" 500 3832 "-" "curl/7.76.1" "-"
[pid: 23|app: 0|req: 1/1] 10.42.0.1 () {34 vars in 371 bytes} [Mon Sep 23 07:42:28 2024] GET / => generated 3820 bytes in 649 msecs (HTTP/1.1 500) 4 headers in 205 bytes (1 switches on core 0)
2024-09-23 07:42:32,429 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
2024-09-23 07:42:32,429 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
2024-09-23 07:42:32,449 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2024-09-23 07:42:32,449 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2024-09-23 07:42:32,454 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
2024-09-23 07:42:32,454 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
2024-09-23 07:42:32,458 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
2024-09-23 07:42:32,458 INFO     Listening on TCP address 127.0.0.1:8051
10.42.0.1 - - [23/Sep/2024:07:44:01 +0000] "GET / HTTP/1.1" 302 5 "-" "curl/7.76.1" "-"
[pid: 21|app: 0|req: 1/2] 10.42.0.1 () {34 vars in 368 bytes} [Mon Sep 23 07:44:01 2024] GET / => generated 0 bytes in 575 msecs (HTTP/1.1 302) 5 headers in 220 bytes (1 switches on core 0)
10.42.0.1 - - [23/Sep/2024:07:44:50 +0000] "HEAD / HTTP/1.1" 302 0 "-" "curl/7.76.1" "-"
[pid: 21|app: 0|req: 2/3] 10.42.0.1 () {34 vars in 368 bytes} [Mon Sep 23 07:44:50 2024] HEAD / => generated 0 bytes in 81 msecs (HTTP/1.1 302) 5 headers in 220 bytes (1 switches on core 0)
2024-09-23 07:51:41,384 INFO     [-] awx.main.consumers client 'specific.f054d412121f413c99643be6b01c96a2!337ca6be43ed44869224cf687a492030' joined the broadcast group.
2024-09-23 07:51:41,384 INFO     [-] awx.main.consumers client 'specific.f054d412121f413c99643be6b01c96a2!337ca6be43ed44869224cf687a492030' joined the broadcast group.
2024-09-23 07:51:41,384 INFO     client 'specific.f054d412121f413c99643be6b01c96a2!337ca6be43ed44869224cf687a492030' joined the broadcast group.

kurokobo · September 24, 2024, 2:09pm

@roelvandepaar
Thanks for updating, it seems that the database initialization is not done correctly.

I’m concerned that the operator is repeatedly restarting due to instability. Can you investigate the cause of this?

pod/awx-operator-controller-manager-666ddcf9c5-92gcj 2/2 Running 33 (5m37s ago) 4h28m

kubectl -n awx describe pod awx-operator-controller-manager-666ddcf9c5-92gcj
kubectl -n awx logs awx-operator-controller-manager-666ddcf9c5-92gcj

roelvandepaar · September 25, 2024, 6:58am

@kurokobo
This morning I was able to get to the AWX login screen in the browser for the first time.
I did not change anything.
I could not find anything suspicious in the logs.

Maybe the VM is just too low specced with one CPU core and 4 GB of RAM for k3s and AWX.

kurokobo · September 25, 2024, 2:20pm

Indeed, this could be the reason for the failure.

I have experience deploying AWX on a VM with 2 vCPUs and 4 GB of RAM successfully, but I usually recommend running it with 4 vCPUs and 8 GB of RAM for stability.

Anyway, I’m glad to know that you’ve successfully installed everything using my guide Have fun!

roelvandepaar · September 25, 2024, 2:49pm

Thank you for the support!