AWX install gets stucked in Redis saving and UI access fails

aj-kyriba · February 27, 2024, 5:53pm

SUMMARY

While executing “make docker-compose”, the process reports a loop in redis task trying to save changes.

ENVIRONMENT

AWX version: 23.8.0 /23.8.1
AWX install method: docker
AWX deployment target: docker
Operating System: Mac

STEPS TO REPRODUCE

on AWX installer, while executing
make docker-compose

EXPECTED RESULTS

Finish AWX configuration

tools_awx_1       | 2024-02-27 17:26:13,514 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of pool_cleanup is -0.007032871246337891 seconds in the past
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.077 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.078 * Background saving started by pid 32
tools_redis_1     | 32:C 27 Feb 2024 17:27:21.084 * DB saved on disk
tools_redis_1     | 32:C 27 Feb 2024 17:27:21.084 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:27:21.179 * Background saving terminated with success
tools_awx_1       | [pid: 1087|app: 0|req: 8/14] 192.168.65.1 () {60 vars in 1145 bytes} [Tue Feb 27 17:29:14 2024] GET / => generated 2148 bytes in 105 msecs (HTTP/1.1 200) 9 headers in 438 bytes (1 switches on core 0)
tools_awx_1       | [pid: 1087|app: 0|req: 9/15] 192.168.65.1 () {58 vars in 1114 bytes} [Tue Feb 27 17:29:24 2024] GET / => generated 2148 bytes in 95 msecs (HTTP/1.1 200) 9 headers in 438 bytes (1 switches on core 0)
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.018 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.018 * Background saving started by pid 33
tools_redis_1     | 33:C 27 Feb 2024 17:32:22.025 * DB saved on disk
tools_redis_1     | 33:C 27 Feb 2024 17:32:22.025 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:32:22.119 * Background saving terminated with success
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.001 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.002 * Background saving started by pid 34
tools_redis_1     | 34:C 27 Feb 2024 17:37:23.007 * DB saved on disk
tools_redis_1     | 34:C 27 Feb 2024 17:37:23.008 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 27 Feb 2024 17:37:23.102 * Background saving terminated with success
tools_awx_1       | 2024-02-27 17:41:13,512 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of pool_cleanup is -0.008401632308959961 seconds in the past

After running

docker exec tools_awx_1 make clean-ui ui-devel

in web UI https://localhost:8043/#/home

while https://localhost:8043/api/v2

Version 2
GET /api/v2/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx_1
X-API-Product-Name: AWX
X-API-Product-Version: 23.8.0
X-API-Query-Count: 2
X-API-Query-Time: 0.002s
X-API-Time: 0.026s

TheRealHaoLiu · February 27, 2024, 7:29pm

try running awx-manage collectstatic inside the container using docker exec -it tools_awx_1 sh

aj-kyriba · February 28, 2024, 8:09am

So I interrupted the running install

tools_awx_1       | 2024-02-28 07:37:47,512 WARNING  [-] awx.main.dispatch.periodic Scheduler next run of send_subsystem_metrics is -0.008138895034790039 seconds in the past
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.029 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.030 * Background saving started by pid 202
tools_redis_1     | 202:C 28 Feb 2024 07:40:11.036 * DB saved on disk
tools_redis_1     | 202:C 28 Feb 2024 07:40:11.036 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:40:11.131 * Background saving terminated with success
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.026 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.027 * Background saving started by pid 203
tools_redis_1     | 203:C 28 Feb 2024 07:45:12.032 * DB saved on disk
tools_redis_1     | 203:C 28 Feb 2024 07:45:12.033 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:45:12.127 * Background saving terminated with success
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.085 * 100 changes in 300 seconds. Saving...
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.086 * Background saving started by pid 204
tools_redis_1     | 204:C 28 Feb 2024 07:50:13.092 * DB saved on disk
tools_redis_1     | 204:C 28 Feb 2024 07:50:13.093 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
tools_redis_1     | 1:M 28 Feb 2024 07:50:13.187 * Background saving terminated with success
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 3/3
 ✔ Container tools_awx_1       Stopped                                                                                                                     4.8s 
 ✔ Container tools_postgres_1  Stopped                                                                                                                     0.3s 
 ✔ Container tools_redis_1     Stopped                                                                                                                     0.2s 
canceled
make: *** [docker-compose] Error 130

Then started the containers and ran the command awx-manage collectstatic

bash-5.1$ cd /var/lib/awx/public/static/
bash-5.1$ ls
api  awx-spud-reading.svg  custom_404.html  custom_504.html   debug_toolbar	 drf-yasg  media	RedHatDisplay-Medium.ttf   rest_framework
awx  css		   custom_502.html  custom_error.css  django_extensions  js	   polymorphic	RedHatDisplay-Regular.ttf
bash-5.1$ ls -l
total 188
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 api
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 awx
-rw-r--r-- 1 awx root 27482 Feb 28 08:04 awx-spud-reading.svg
drwxr-xr-x 3 awx root  4096 Feb 28 08:04 css
-rw-r--r-- 1 awx root   150 Feb 28 08:04 custom_404.html
-rw-r--r-- 1 awx root   650 Feb 28 08:04 custom_502.html
-rw-r--r-- 1 awx root   650 Feb 28 08:04 custom_504.html
-rw-r--r-- 1 awx root  1390 Feb 28 08:04 custom_error.css
drwxr-xr-x 4 awx root  4096 Feb 27 14:12 debug_toolbar
drwxr-xr-x 5 awx root  4096 Feb 27 14:12 django_extensions
drwxr-xr-x 5 awx root  4096 Feb 28 08:04 drf-yasg
drwxr-xr-x 2 awx root  4096 Feb 28 08:04 js
drwxr-xr-x 2 awx root 12288 Feb 28 08:04 media
drwxr-xr-x 4 awx root  4096 Feb 27 14:12 polymorphic
-rw-r--r-- 1 awx root 47388 Feb 28 08:04 RedHatDisplay-Medium.ttf
-rw-r--r-- 1 awx root 47356 Feb 28 08:04 RedHatDisplay-Regular.ttf
drwxr-xr-x 7 awx root  4096 Feb 27 14:12 rest_framework
bash-5.1$ cat api/api.
api.css  api.js   
bash-5.1$ cat awx
awx/                  awx-spud-reading.svg  
bash-5.1$ cat awx/index_awx.html 
<html>
  <head><title>UI Next Missing</title></head>
  <body style="background-color: black; color: white;">
    <div style="display: flex; justify-content: center; align-items: center; text-align: center; min-height: 100vh;">
       <h1>Oops... Looks like the UI Next wasn't properly built</h1>
    </div>
  </body>
</html>

fosterseth · February 28, 2024, 7:59pm

did this step say the UI was built successfully?

docker exec tools_awx_1 make clean-ui ui-devel

Seth

aj-kyriba · February 29, 2024, 10:05am

Finally, I managed to get the awx 23.8.0 version running.

As the installation was running in a loop with the above error message, I exited the installation which led to all the containers to stop.
Then I restarted the containers and when I ran the command

docker exec tools_awx_1 make clean-ui ui-devel

Once it was completed, I was able to access the web UI will success.

system · March 30, 2024, 10:05am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Guidance for Offline AWX Installation Redis Container doesn’t start Get Help awx , playbook , install , ansible	1	239	June 5, 2024
AWX-EE Exceeded retries for reading stdout Get Help awx , ee	0	41	October 13, 2024
Unable to find image 'awx_redis:latest' locally Get Help awx	3	161	May 29, 2024
AWX 23.6 install fails when using docker-compose Get Help awx , collections , kubernetes	0	273	February 26, 2024
Updating AWX image on Docker Get Help awx , upgrade , docker	13	1808	November 8, 2023

AWX install gets stucked in Redis saving and UI access fails

ENVIRONMENT

Related topics