AWX install gets stucked in Redis saving and UI access fails

aj-kyriba · February 27, 2024, 5:53pm

SUMMARY

While executing “make docker-compose”, the process reports a loop in redis task trying to save changes.

ENVIRONMENT

AWX version: 23.8.0 /23.8.1
AWX install method: docker
AWX deployment target: docker
Operating System: Mac

STEPS TO REPRODUCE

on AWX installer, while executing
make docker-compose

EXPECTED RESULTS

Finish AWX configuration

tools_awx_1       | 2024-02-27 17:26:13,514 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of pool_cleanup is -0.007032871246337891 seconds in the past
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.077 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.078 * Background saving started by pid 32
tools_redis_1     | 32:C 27 Feb 2024 17:27:21.084 * DB saved on disk
tools_redis_1     | 32:C 27 Feb 2024 17:27:21.084 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.179 * Background saving terminated with success
tools_awx_1       | [pid: 1087|app: 0|req: 8/14] 192.168.65.1 () {60 vars in 1145 bytes} [Tue Feb 27 17:29:14 2024] GET / => generated 2148 bytes in 105 msecs (HTTP/1.1 200) 9 headers in 438 bytes (1 switches on core 0)
tools_awx_1       | [pid: 1087|app: 0|req: 9/15] 192.168.65.1 () {58 vars in 1114 bytes} [Tue Feb 27 17:29:24 2024] GET / => generated 2148 bytes in 95 msecs (HTTP/1.1 200) 9 headers in 438 bytes (1 switches on core 0)
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.018 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.018 * Background saving started by pid 33
tools_redis_1     | 33:C 27 Feb 2024 17:32:22.025 * DB saved on disk
tools_redis_1     | 33:C 27 Feb 2024 17:32:22.025 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.119 * Background saving terminated with success
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.001 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.002 * Background saving started by pid 34
tools_redis_1     | 34:C 27 Feb 2024 17:37:23.007 * DB saved on disk
tools_redis_1     | 34:C 27 Feb 2024 17:37:23.008 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.102 * Background saving terminated with success
tools_awx_1       | 2024-02-27 17:41:13,512 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of pool_cleanup is -0.008401632308959961 seconds in the past

After running

docker exec tools_awx_1 make clean-ui ui-devel

in web UI https://localhost:8043/#/home

while https://localhost:8043/api/v2

Version 2
GET /api/v2/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx_1
X-API-Product-Name: AWX
X-API-Product-Version: 23.8.0
X-API-Query-Count: 2
X-API-Query-Time: 0.002s
X-API-Time: 0.026s

TheRealHaoLiu · February 27, 2024, 7:29pm

try running awx-manage collectstatic inside the container using docker exec -it tools_awx_1 sh

aj-kyriba · February 28, 2024, 8:09am

So I interrupted the running install

tools_awx_1       | 2024-02-28 07:37:47,512 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of send_subsystem_metrics is -0.008138895034790039 seconds in the past
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.029 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.030 * Background saving started by pid 202
tools_redis_1     | 202:C 28 Feb 2024 07:40:11.036 * DB saved on disk
tools_redis_1     | 202:C 28 Feb 2024 07:40:11.036 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.131 * Background saving terminated with success
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.026 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.027 * Background saving started by pid 203
tools_redis_1     | 203:C 28 Feb 2024 07:45:12.032 * DB saved on disk
tools_redis_1     | 203:C 28 Feb 2024 07:45:12.033 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.127 * Background saving terminated with success
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.085 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.086 * Background saving started by pid 204
tools_redis_1     | 204:C 28 Feb 2024 07:50:13.092 * DB saved on disk
tools_redis_1     | 204:C 28 Feb 2024 07:50:13.093 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.187 * Background saving terminated with success
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 3/3
 ✔ Container tools_awx_1       Stopped                                                                                                                     4.8s 
 ✔ Container tools_postgres_1  Stopped                                                                                                                     0.3s 
 ✔ Container tools_redis_1     Stopped                                                                                                                     0.2s 
canceled
make: *** [docker-compose] Error 130

Then started the containers and ran the command awx-manage collectstatic

bash-5.1$ cd /var/lib/awx/public/static/
bash-5.1$ ls
api  awx-spud-reading.svg  custom_404.html  custom_504.html   debug_toolbar	 drf-yasg  media	RedHatDisplay-Medium.ttf   rest_framework
awx  css		   custom_502.html  custom_error.css  django_extensions  js	   polymorphic	RedHatDisplay-Regular.ttf
bash-5.1$ ls -l
total 188
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 api
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 awx
-rw-r--r-- 1 awx root 27482 Feb 28 08:04 awx-spud-reading.svg
drwxr-xr-x 3 awx root  4096 Feb 28 08:04 css
-rw-r--r-- 1 awx root   150 Feb 28 08:04 custom_404.html
-rw-r--r-- 1 awx root   650 Feb 28 08:04 custom_502.html
-rw-r--r-- 1 awx root   650 Feb 28 08:04 custom_504.html
-rw-r--r-- 1 awx root  1390 Feb 28 08:04 custom_error.css
drwxr-xr-x 4 awx root  4096 Feb 27 14:12 debug_toolbar
drwxr-xr-x 5 awx root  4096 Feb 27 14:12 django_extensions
drwxr-xr-x 5 awx root  4096 Feb 28 08:04 drf-yasg
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 js
drwxr-xr-x 2 awx root 12288 Feb 28 08:04 media
drwxr-xr-x 4 awx root  4096 Feb 27 14:12 polymorphic
-rw-r--r-- 1 awx root 47388 Feb 28 08:04 RedHatDisplay-Medium.ttf
-rw-r--r-- 1 awx root 47356 Feb 28 08:04 RedHatDisplay-Regular.ttf
drwxr-xr-x 7 awx root  4096 Feb 27 14:12 rest_framework
bash-5.1$ cat api/api.
api.css  api.js   
bash-5.1$ cat awx
awx/                  awx-spud-reading.svg  
bash-5.1$ cat awx/index_awx.html 
<html>
  <head><title>UI Next Missing</title></head>
  <body style="background-color: black; color: white;">
    <div style="display: flex; justify-content: center; align-items: center; text-align: center; min-height: 100vh;">
       <h1>Oops... Looks like the UI Next wasn't properly built</h1>
    </div>
  </body>
</html>

fosterseth · February 28, 2024, 7:59pm

did this step say the UI was built successfully?

docker exec tools_awx_1 make clean-ui ui-devel

Seth

aj-kyriba · February 29, 2024, 10:05am

Finally, I managed to get the awx 23.8.0 version running.

As the installation was running in a loop with the above error message, I exited the installation which led to all the containers to stop.
Then I restarted the containers and when I ran the command

docker exec tools_awx_1 make clean-ui ui-devel

Once it was completed, I was able to access the web UI will success.

system · March 30, 2024, 10:05am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.