AWX is barely functional, clicking anything takes 3-4 minutes to load. Checked logs/monitoring (except for postgres which i do not have access to) and no red flags. As a test, I spun up a local postgres docker container, connected AWX to it and the issue went away…
Other applications connected to our postgres production server don’t suffer from this, so wondering if there’s something I should be looking at on the AWX side? It’s strange that there are no errors in the logs…
Interesting, I’ve never noticed anything like this on installs with an external postgres deployment. Does postgres complain about slow queries? Are they geographically close to each other?
The slowness seems to be intermittent and the awx_task log is filled with:
[2017-12-13 22:08:12,090: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1095, started daemon)>
[2017-12-13 22:08:12,098: ERROR/MainProcess] Process ‘ForkPoolWorker-1095’ pid:1238 exited with ‘signal 9 (SIGKILL)’
[2017-12-13 22:08:12,429: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1096, started daemon)>
[2017-12-13 22:08:12,437: ERROR/MainProcess] Process ‘ForkPoolWorker-1096’ pid:1239 exited with ‘signal 9 (SIGKILL)’
[2017-12-13 22:08:12,873: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1097, started daemon)>
[2017-12-13 22:08:12,881: ERROR/MainProcess] Process ‘ForkPoolWorker-1097’ pid:1240 exited with ‘signal 9 (SIGKILL)’
[2017-12-13 22:08:15,477: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1098, started daemon)>
[2017-12-13 22:08:15,486: ERROR/MainProcess] Process ‘ForkPoolWorker-1098’ pid:1241 exited with ‘signal 9 (SIGKILL)’
[2017-12-13 22:08:16,252: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1099, started daemon)>
[2017-12-13 22:08:16,262: ERROR/MainProcess] Process ‘ForkPoolWorker-1099’ pid:1242 exited with ‘signal 9 (SIGKILL)’
[2017-12-13 22:08:16,595: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-1100, started daemon)>
[2017-12-13 22:08:16,605: ERROR/MainProcess] Process ‘ForkPoolWorker-1100’ pid:1243 exited with 'signal
This also happens when switching back to a local postgres docker container, so the issue is not external db.
This looks like this: https://github.com/celery/celery/issues/1715
Can you open an issue in the AWX issue tracker for us to track this?
the issue was my own mistake, we had a missing env variable for rabbitmq vhost, added it back and issue went away.