AWX Login - Something went wrong

I have been configuring and setting up a cluster on AWX 23.9.0. Occasionally, I am unable to login to the UI for long stretches of time which eventually just seem to go away.

Web Logs

[pid: 25|app: 0|req: 22/49] 10.233.114.54 () {78 vars in 1463 bytes} [Mon Apr  8 14:31:48 2024] POST /api/login/ => generated 0 bytes in 540 msecs (HTTP/1.1 302) 15 headers in 737 bytes (1 switches on core 0)
2024-04-08 14:31:49,267 WARNING  [28f8f5b1c46d42ae9eb4279e74fe09f7] django.request Not Found: /api/login/api/v2/config/
2024-04-08 14:31:49,267 WARNING  [28f8f5b1c46d42ae9eb4279e74fe09f7] django.request Not Found: /api/login/api/v2/config/
10.233.114.54 - - [08/Apr/2024:14:31:49 +0000] "GET /api/login/api/v2/config/ HTTP/1.1" 404 55 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 23/50] 10.233.114.54 () {72 vars in 1373 bytes} [Mon Apr  8 14:31:49 2024] GET /api/login/api/v2/config/ => generated 55 bytes in 35 msecs (HTTP/1.1 404) 9 headers in 443 bytes (1 switches on core 0)
2024-04-08 14:31:49,378 INFO     [7302d245a6f1481d82c9586375d08ded] awx.api.generics status 401 received by user AnonymousUser attempting to access /api/v2/me/ from 10.233.114.54
2024-04-08 14:31:49,381 WARNING  [7302d245a6f1481d82c9586375d08ded] django.request Unauthorized: /api/v2/me/
2024-04-08 14:31:49,381 WARNING  [7302d245a6f1481d82c9586375d08ded] django.request Unauthorized: /api/v2/me/
10.233.114.54 - - [08/Apr/2024:14:31:49 +0000] "GET /api/v2/me/ HTTP/1.1" 401 107 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 24/51] 10.233.114.54 () {72 vars in 1345 bytes} [Mon Apr  8 14:31:49 2024] GET /api/v2/me/ => generated 107 bytes in 33 msecs (HTTP/1.1 401) 13 headers in 568 bytes (1 switches on core 0)
10.233.114.54 - - [08/Apr/2024:14:37:09 +0000] "GET /api/logout/ HTTP/1.1" 302 0 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 22|app: 0|req: 7/52] 10.233.114.54 () {72 vars in 1299 bytes} [Mon Apr  8 14:37:09 2024] GET /api/logout/ => generated 0 bytes in 88 msecs (HTTP/1.1 302) 11 headers in 439 bytes (1 switches on core 0)
10.233.114.54 - - [08/Apr/2024:14:37:09 +0000] "GET /api/ HTTP/1.1" 200 186 "https://redacted-hostname/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" "10.233.98.0"
[pid: 25|app: 0|req: 25/53] 10.233.114.54 () {72 vars in 1286 bytes} [Mon Apr  8 14:37:09 2024] GET /api/ => generated 186 bytes in 83 msecs (HTTP/1.1 200) 12 headers in 462 bytes (1 switches on core 0)

I see in those logs the 404 attempt to hit /api/v2/config is getting prefixed with /api/login.

I can still utilize the API while this happens, and it affects all users.
Screenshot 2024-04-08 at 10.44.05 AM

I’ve tried bouncing the web pod and my DB with no change in behavior.

Some more details I’ve found:

  • My last_login timestamp is incrementing each time I login
  • The number of sessions in the DB is increasing on each login