AWX Login - Something went wrong

I have been configuring and setting up a cluster on AWX 23.9.0. Occasionally, I am unable to login to the UI for long stretches of time which eventually just seem to go away.

Web Logs

[pid: 25|app: 0|req: 22/49] 10.233.114.54 () {78 vars in 1463 bytes} [Mon Apr  8 14:31:48 2024] POST /api/login/ => generated 0 bytes in 540 msecs (HTTP/1.1 302) 15 headers in 737 bytes (1 switches on core 0)
2024-04-08 14:31:49,267 WARNING  [28f8f5b1c46d42ae9eb4279e74fe09f7] django.request Not Found: /api/login/api/v2/config/
2024-04-08 14:31:49,267 WARNING  [28f8f5b1c46d42ae9eb4279e74fe09f7] django.request Not Found: /api/login/api/v2/config/
10.233.114.54 - - [08/Apr/2024:14:31:49 +0000] "GET /api/login/api/v2/config/ HTTP/1.1" 404 55 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 23/50] 10.233.114.54 () {72 vars in 1373 bytes} [Mon Apr  8 14:31:49 2024] GET /api/login/api/v2/config/ => generated 55 bytes in 35 msecs (HTTP/1.1 404) 9 headers in 443 bytes (1 switches on core 0)
2024-04-08 14:31:49,378 INFO     [7302d245a6f1481d82c9586375d08ded] awx.api.generics status 401 received by user AnonymousUser attempting to access /api/v2/me/ from 10.233.114.54
2024-04-08 14:31:49,381 WARNING  [7302d245a6f1481d82c9586375d08ded] django.request Unauthorized: /api/v2/me/
2024-04-08 14:31:49,381 WARNING  [7302d245a6f1481d82c9586375d08ded] django.request Unauthorized: /api/v2/me/
10.233.114.54 - - [08/Apr/2024:14:31:49 +0000] "GET /api/v2/me/ HTTP/1.1" 401 107 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 24/51] 10.233.114.54 () {72 vars in 1345 bytes} [Mon Apr  8 14:31:49 2024] GET /api/v2/me/ => generated 107 bytes in 33 msecs (HTTP/1.1 401) 13 headers in 568 bytes (1 switches on core 0)
10.233.114.54 - - [08/Apr/2024:14:37:09 +0000] "GET /api/logout/ HTTP/1.1" 302 0 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 22|app: 0|req: 7/52] 10.233.114.54 () {72 vars in 1299 bytes} [Mon Apr  8 14:37:09 2024] GET /api/logout/ => generated 0 bytes in 88 msecs (HTTP/1.1 302) 11 headers in 439 bytes (1 switches on core 0)
10.233.114.54 - - [08/Apr/2024:14:37:09 +0000] "GET /api/ HTTP/1.1" 200 186 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 25/53] 10.233.114.54 () {72 vars in 1286 bytes} [Mon Apr  8 14:37:09 2024] GET /api/ => generated 186 bytes in 83 msecs (HTTP/1.1 200) 12 headers in 462 bytes (1 switches on core 0)

I see in those logs the 404 attempt to hit /api/v2/config is getting prefixed with /api/login.

I can still utilize the API while this happens, and it affects all users.

I’ve tried bouncing the web pod and my DB with no change in behavior.

Some more details I’ve found:

  • My last_login timestamp is incrementing each time I login
  • The number of sessions in the DB is increasing on each login