# Can't start AWX web, connection rabbitmq, postgres timeout **URL:** \ **Category:** AWX Project\ **Tags:** awx\ **Created:** [February 7, 2020, 3:28am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169 "2020-02-07T03:28:24Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Tat\_Dat\_Pham](https://avatars.discourse-cdn.com/v4/letter/t/34f0e0/32.png) [@Tat\_Dat\_Pham](https://forum.ansible.com/u/Tat_Dat_Pham)\ **Post date:** [February 7, 2020, 3:28am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/1 "2020-02-07T03:28:24Z")
##### ISSUE TYPE- Bug Report ##### SUMMARY##### ENVIRONMENT- AWX version: 9.1.1 - AWX install method: docker on centos8 - Ansible version: 2.9.3 - Operating System: Centos 8 - Web Browser: Chrome latest ##### STEPS TO REPRODUCE Install awx follow command ```auto ansible-playbook -i inventory install.yml ``` All container is up ```auto CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 756ed3d8a7e2 ansible/awx_task:9.1.1 "/tini -- /bin/sh -c…" About an hour ago Up About an hour 8052/tcp awx_task ffb7cfb55ba5 ansible/awx_web:9.1.1 "/tini -- /bin/sh -c…" About an hour ago Up About an hour 0.0.0.0:80->8052/tcp awx_web a3a1f2724640 postgres:10 "docker-entrypoint.s…" About an hour ago Up About an hour 5432/tcp awx_postgres ff2a5734c174 memcached:alpine "docker-entrypoint.s…" About an hour ago Up About an hour 11211/tcp awx_memcached 54685e6e1512 ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" About an hour ago Up About an hour 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp awx_rabbitmq ``` ```auto ``` ---
**Author:** ![Dan\_W1](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dan\_W1](https://forum.ansible.com/u/Dan_W1)\ **Post date:** [February 14, 2020, 10:20pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/2 "2020-02-14T22:20:04Z")
Encountered very similar issue. awx\_task is not able to reach Postgres container using standard deployment. ##### SUMMARY##### ENVIRONMENT- AWX version: 9.2.0 (latest) - AWX install method: Docker Compose on centos8 - Ansible version: 2.9.4 - Operating System: Centos 8 - Web Browser: Chrome latest - Python 3.6.8 ---
**Author:** ![Tat\_Dat\_Pham](https://avatars.discourse-cdn.com/v4/letter/t/34f0e0/32.png) [@Tat\_Dat\_Pham](https://forum.ansible.com/u/Tat_Dat_Pham)\ **Post date:** [February 18, 2020, 9:54am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/3 "2020-02-18T09:54:17Z")
Yep, that is my issue. But still not resolved yet. AWX team don’t support 🙂 Vào 05:20:05 UTC+7 Thứ Bảy, ngày 15 tháng 2 năm 2020, Dan W đã viết: ---
**Author:** ![Christopher\_Meyers](https://avatars.discourse-cdn.com/v4/letter/c/3e96dc/32.png) [@Christopher\_Meyers](https://forum.ansible.com/u/Christopher_Meyers)\ **Post date:** [February 18, 2020, 2:24pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/4 "2020-02-18T14:24:26Z")
This is a problem with the new postgres containers. They now require a superuser password to be defined on boot. We are looking in to how best to provide this password. ---
**Author:** ![Christopher\_Meyers](https://avatars.discourse-cdn.com/v4/letter/c/3e96dc/32.png) [@Christopher\_Meyers](https://forum.ansible.com/u/Christopher_Meyers)\ **Post date:** [February 18, 2020, 2:27pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/5 "2020-02-18T14:27:41Z")
You can verify that you are experiencing the same issue by issuing `docker logs -f awx_postgres`. If you are experiencing this issue you will see the below output. ```auto Error: Database is uninitialized and superuser password is not specified. You must specify POSTGRES_PASSWORD for the superuser. Use "-e POSTGRES_PASSWORD=password" to set it in "docker run". You may also use POSTGRES_HOST_AUTH_METHOD=trust to allow all connections without a password. This is *not* recommended. See PostgreSQL documentation about "trust": [https://www.postgresql.org/docs/current/auth-trust.html](https://www.postgresql.org/docs/current/auth-trust.html) ``` ---
**Author:** ![Christopher\_Meyers](https://avatars.discourse-cdn.com/v4/letter/c/3e96dc/32.png) [@Christopher\_Meyers](https://forum.ansible.com/u/Christopher_Meyers)\ **Post date:** [February 18, 2020, 3:40pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/6 "2020-02-18T15:40:28Z")
After digging deeper, it looks like you are **not** experiencing the postgres issue I mentioned earlier. Tat, from the traceback the problem is awx\_task → awx\_rabbitmq communication Tat, you basically included all of the debugging information I would have asked for and nothing looks apparently wrong. I did notice, however, that you don’t have a network entry called **awxcompose\_default** which means you have changed the default **docker\_compose\_dir** setting. That is fine, but maybe some other modification to inventory has led to this. Can you post your inventory file? Dan, Can you run the debug steps that Tat ran and post the results? ---
**Author:** ![Tat\_Dat\_Pham](https://avatars.discourse-cdn.com/v4/letter/t/34f0e0/32.png) [@Tat\_Dat\_Pham](https://forum.ansible.com/u/Tat_Dat_Pham)\ **Post date:** [February 19, 2020, 3:36am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/7 "2020-02-19T03:36:32Z")
I dont get this error on POSTGRES. Vào 21:28:06 UTC+7 Thứ Ba, ngày 18 tháng 2 năm 2020, Christopher Meyers đã viết: ---
**Author:** ![Tat\_Dat\_Pham](https://avatars.discourse-cdn.com/v4/letter/t/34f0e0/32.png) [@Tat\_Dat\_Pham](https://forum.ansible.com/u/Tat_Dat_Pham)\ **Post date:** [February 19, 2020, 3:40am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/8 "2020-02-19T03:40:05Z")
Yesterday, I try to reinstall AWX on Centos 8. - Stop all container, prune all docker data - Turn off firewall - Reinstall. And AWX started without any problem. I dont know how and what problem that i got, although i try to reinstall awx many times before. But now, AWX is running. Vào 22:40:53 UTC+7 Thứ Ba, ngày 18 tháng 2 năm 2020, Christopher Meyers đã viết: ---
**Author:** ![Dan\_W1](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dan\_W1](https://forum.ansible.com/u/Dan_W1)\ **Post date:** [February 19, 2020, 4:17pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/9 "2020-02-19T16:17:23Z")
Chris - Please see the debug details below. I’ve tried stopping/removing all the containers and re-running the Installer but have gotten the same result. Since this, I’ve moved to CentOS 7 and have installed using the same inventory file and using latest AWX. Containers are up: ` [root@awx01 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bc7c32d4f60c ansible/awx\_task:latest “/tini – /bin/sh -c…” 2 days ago Up 2 days 8052/tcp awx\_task 49df2a131c81 ansible/awx\_web:latest “/tini – /bin/sh -c…” 2 days ago Up 2 days 0.0.0.0:80-\>8052/tcp, 0.0.0.0:443-\>8053/tcp awx\_web 7b15c4dbc69d memcached:alpine “docker-entrypoint.s…” 2 days ago Up 2 days 11211/tcp awx\_memcached 059e758a6581 postgres:10 “docker-entrypoint.s…” 2 days ago Up 2 days 5432/tcp awx\_postgres 5b31f01e9ab9 ansible/awx\_rabbitmq:3.7.4 “docker-entrypoint.s…” 2 days ago Up 2 days 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp awx\_rabbitmq ` Networks are Default: ` [root@awx01 ~]# docker network ls NETWORK ID NAME DRIVER SCOPE 4eb2bdaca233 awxcompose\_default bridge local 49ed9426b273 bridge bridge local 354d4f4ad44e host host local 57310c7a7426 none null local ` IPs are configured and I’m able to ping the other containers from each: ` [root@awx01 ~]# docker inspect -f ‘{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}’ awx\_postgres awx\_task awx\_memcached awx\_web awx\_rabbitmq 172.18.0.3 172.18.0.6 172.18.0.4 172.18.0.5 172.18.0.2 ` AWX\_Postgres is up and I can access the DB using psql from within the container: ` [root@awx01 ~]# docker logs awx\_postgres PostgreSQL Database directory appears to contain a database; Skipping initialization 2020-02-16 19:03:31.638 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432 2020-02-16 19:03:31.638 UTC [1] LOG: listening on IPv6 address “::”, port 5432 2020-02-16 19:03:31.642 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432” 2020-02-16 19:03:31.664 UTC [24] LOG: database system was shut down at 2020-02-16 18:57:53 UTC 2020-02-16 19:03:31.673 UTC [1] LOG: database system is ready to accept connections ` AWX\_RABBITMQ is up: ` [root@awx01 ~]# docker logs awx\_rabbitmq 2020-02-16 19:03:39.997 [info] \<0.33.0\> Application lager started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.459 [info] \<0.33.0\> Application crypto started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.586 [info] \<0.33.0\> Application mnesia started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.586 [info] \<0.33.0\> Application xmerl started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application os\_mon started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application recon started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application cowlib started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application jsx started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application asn1 started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.593 [info] \<0.33.0\> Application public\_key started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.666 [info] \<0.33.0\> Application ssl started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.672 [info] \<0.33.0\> Application ranch started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.675 [info] \<0.33.0\> Application cowboy started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.675 [info] \<0.33.0\> Application ranch\_proxy\_protocol started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.675 [info] \<0.33.0\> Application rabbit\_common started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.684 [info] \<0.33.0\> Application amqp\_client started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.766 [info] \<0.33.0\> Application inets started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:40.779 [info] \<0.193.0\> Starting RabbitMQ 3.7.4 on Erlang 20.1.7 Copyright (C) 2007-2018 Pivotal Software, Inc. Licensed under the MPL. See [http://www.rabbitmq.com/](http://www.rabbitmq.com/) ## ## ## RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc. ########## Licensed under the MPL. See [http://www.rabbitmq.com/](http://www.rabbitmq.com/) ###### ########## Logs: Starting broker… 2020-02-16 19:03:40.800 [info] \<0.193.0\> node : rabbit@5b31f01e9ab9 home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.conf cookie hash : at619UOZzsenF44tSK3ulA== log(s) : database dir : /var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9 2020-02-16 19:03:43.259 [info] \<0.201.0\> Memory high watermark set to 6350 MiB (6659248947 bytes) of 15876 MiB (16648122368 bytes) total 2020-02-16 19:03:43.263 [info] \<0.203.0\> Enabling free disk space monitoring 2020-02-16 19:03:43.263 [info] \<0.203.0\> Disk free limit set to 50MB 2020-02-16 19:03:43.266 [info] \<0.205.0\> Limiting to approx 1048476 file handles (943626 sockets) 2020-02-16 19:03:43.266 [info] \<0.206.0\> FHC read buffering: OFF 2020-02-16 19:03:43.266 [info] \<0.206.0\> FHC write buffering: ON 2020-02-16 19:03:43.267 [info] \<0.193.0\> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9 is empty. Assuming we need to join an existing cluster or initialise from scratch… 2020-02-16 19:03:43.267 [info] \<0.193.0\> Configured peer discovery backend: rabbit\_peer\_discovery\_classic\_config 2020-02-16 19:03:43.267 [info] \<0.193.0\> Will try to lock with peer discovery backend rabbit\_peer\_discovery\_classic\_config 2020-02-16 19:03:43.267 [info] \<0.193.0\> Peer discovery backend does not support locking, falling back to randomized delay 2020-02-16 19:03:43.267 [info] \<0.193.0\> Peer discovery backend rabbit\_peer\_discovery\_classic\_config does not support registration, skipping randomized startup delay. 2020-02-16 19:03:43.268 [info] \<0.193.0\> All discovered existing cluster peers: 2020-02-16 19:03:43.268 [info] \<0.193.0\> Discovered no peer nodes to cluster with 2020-02-16 19:03:43.269 [info] \<0.33.0\> Application mnesia exited with reason: stopped 2020-02-16 19:03:43.290 [info] \<0.33.0\> Application mnesia started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:43.371 [info] \<0.193.0\> Waiting for Mnesia tables for 30000 ms, 9 retries left 2020-02-16 19:03:43.405 [info] \<0.193.0\> Waiting for Mnesia tables for 30000 ms, 9 retries left 2020-02-16 19:03:43.439 [info] \<0.193.0\> Waiting for Mnesia tables for 30000 ms, 9 retries left 2020-02-16 19:03:43.439 [info] \<0.193.0\> Peer discovery backend rabbit\_peer\_discovery\_classic\_config does not support registration, skipping registration. 2020-02-16 19:03:43.440 [info] \<0.193.0\> Priority queues enabled, real BQ is rabbit\_variable\_queue 2020-02-16 19:03:43.445 [info] \<0.374.0\> Starting rabbit\_node\_monitor 2020-02-16 19:03:43.475 [info] \<0.193.0\> message\_store upgrades: 1 to apply 2020-02-16 19:03:43.475 [info] \<0.193.0\> message\_store upgrades: Applying rabbit\_variable\_queue:move\_messages\_to\_vhost\_store 2020-02-16 19:03:43.475 [info] \<0.193.0\> message\_store upgrades: No durable queues found. Skipping message store migration 2020-02-16 19:03:43.475 [info] \<0.193.0\> message\_store upgrades: Removing the old message store data 2020-02-16 19:03:43.476 [info] \<0.193.0\> message\_store upgrades: All upgrades applied successfully 2020-02-16 19:03:43.508 [info] \<0.193.0\> Management plugin: using rates mode ‘basic’ 2020-02-16 19:03:43.509 [info] \<0.193.0\> Adding vhost ‘awx’ 2020-02-16 19:03:43.523 [info] \<0.408.0\> Making sure data directory ‘/var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9/msg\_stores/vhosts/5A7PFSQL4N28PQOL0INDFDDLU’ for vhost ‘awx’ exists 2020-02-16 19:03:43.527 [info] \<0.408.0\> Starting message stores for vhost ‘awx’ 2020-02-16 19:03:43.527 [info] \<0.412.0\> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg\_store\_transient”: using rabbit\_msg\_store\_ets\_index to provide index 2020-02-16 19:03:43.529 [info] \<0.408.0\> Started message store of type transient for vhost ‘awx’ 2020-02-16 19:03:43.529 [info] \<0.415.0\> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg\_store\_persistent”: using rabbit\_msg\_store\_ets\_index to provide index 2020-02-16 19:03:43.530 [warning] \<0.415.0\> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg\_store\_persistent”: rebuilding indices from scratch 2020-02-16 19:03:43.531 [info] \<0.408.0\> Started message store of type persistent for vhost ‘awx’ 2020-02-16 19:03:43.532 [info] \<0.193.0\> Creating user ‘guest’ 2020-02-16 19:03:43.534 [info] \<0.193.0\> Setting user tags for user ‘guest’ to [administrator] 2020-02-16 19:03:43.536 [info] \<0.193.0\> Setting permissions for ‘guest’ in ‘awx’ to ‘._', '._’, ‘.\*’ 2020-02-16 19:03:43.540 [info] \<0.453.0\> started TCP Listener on [::]:5672 2020-02-16 19:03:43.544 [info] \<0.193.0\> Setting up a table for connection tracking on this node: tracked\_connection\_on\_node\_rabbit@5b31f01e9ab9 2020-02-16 19:03:43.547 [info] \<0.193.0\> Setting up a table for per-vhost connection counting on this node: tracked\_connection\_per\_vhost\_on\_node\_rabbit@5b31f01e9ab9 2020-02-16 19:03:43.547 [info] \<0.33.0\> Application rabbit started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:43.550 [info] \<0.33.0\> Application rabbitmq\_management\_agent started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:43.550 [info] \<0.33.0\> Application rabbitmq\_web\_dispatch started on node rabbit@5b31f01e9ab9 2020-02-16 19:03:43.584 [info] \<0.510.0\> Management plugin started. Port: 15672 2020-02-16 19:03:43.584 [info] \<0.616.0\> Statistics database started. 2020-02-16 19:03:43.585 [info] \<0.33.0\> Application rabbitmq\_management started on node rabbit@5b31f01e9ab9 completed with 3 plugins. 2020-02-16 19:03:43.814 [info] \<0.5.0\> Server startup complete; 3 plugins started. - rabbitmq\_management - rabbitmq\_web\_dispatch - rabbitmq\_management\_agent ` I’ll post the awx\_task logs in a follow up. ---
**Author:** ![Dan\_W1](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dan\_W1](https://forum.ansible.com/u/Dan_W1)\ **Post date:** [February 19, 2020, 4:53pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/10 "2020-02-19T16:53:05Z")
AWX\_TASK Logs: ` docker logs awx\_task Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | FAILED! =\> { “ansible\_facts”: { “discovered\_interpreter\_python”: “/usr/libexec/platform-python” }, “changed”: false, “elapsed”: 301, “msg”: “Timeout when waiting for postgres:5432” } Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | FAILED! =\> { “ansible\_facts”: { “discovered\_interpreter\_python”: “/usr/libexec/platform-python” }, “changed”: false, “elapsed”: 301, “msg”: “Timeout when waiting for memcached:11211” } Using /etc/ansible/ansible.cfg as config file 127.0.0.1 | FAILED! =\> { “ansible\_facts”: { “discovered\_interpreter\_python”: “/usr/libexec/platform-python” }, “changed”: false, “elapsed”: 301, “msg”: “Timeout when waiting for rabbitmq:5672” } Using /etc/ansible/ansible.cfg as config file An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TCP/IP connections on port 5432? 127.0.0.1 | FAILED! =\> { “ansible\_facts”: { “discovered\_interpreter\_python”: “/usr/libexec/platform-python” }, “changed”: false, “msg”: “unable to connect to database: could not connect to server: No route to host\n\tIs the server running on host "postgres" (172.18.0.3) and accepting\n\tTCP/IP connections on port 5432?\n” } Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) psycopg2.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/usr/bin/awx-manage”, line 8, in sys.exit(manage()) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/ **init**.py”, line 152, in manage execute\_from\_command\_line(sys.argv) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/ **init**.py”, line 381, in execute\_from\_command\_line utility.execute() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/ **init**.py”, line 375, in execute self.fetch\_command(subcommand).run\_from\_argv(self.argv) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run\_from\_argv self.execute(\*args, \*\*cmd\_options) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute output = self.handle(\*args, \*\*options) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 83, in wrapped res = handle\_func(\*args, \*\*kwargs) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/commands/migrate.py”, line 87, in handle executor = MigrationExecutor(connection, self.migration\_progress\_callback) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/executor.py”, line 18, in **init** self.loader = MigrationLoader(self.connection) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/loader.py”, line 49, in **init** self.build\_graph() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/loader.py”, line 212, in build\_graph self.applied\_migrations = recorder.applied\_migrations() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/recorder.py”, line 73, in applied\_migrations if self.has\_table(): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/recorder.py”, line 56, in has\_table return self.Migration.\_meta.db\_table in self.connection.introspection.table\_names(self.connection.cursor()) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor return self.\_cursor() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in \_cursor self.ensure\_connection() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in **exit** raise dj\_exc\_value.with\_traceback(traceback) from exc\_value File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) django.db.utils.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) psycopg2.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/usr/bin/awx-manage”, line 8, in sys.exit(manage()) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/ **init**.py”, line 152, in manage execute\_from\_command\_line(sys.argv) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/ **init**.py”, line 381, in execute\_from\_command\_line utility.execute() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/ **init**.py”, line 375, in execute self.fetch\_command(subcommand).run\_from\_argv(self.argv) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run\_from\_argv self.execute(\*args, \*\*cmd\_options) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute output = self.handle(\*args, \*\*options) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/commands/shell.py”, line 92, in handle exec(sys.stdin.read()) File “”, line 1, in File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/models.py”, line 162, in create\_superuser return self.\_create\_user(username, email, password, \*\*extra\_fields) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/models.py”, line 145, in \_create\_user user.save(using=self.\_db) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/base\_user.py”, line 66, in save super().save(\*args, \*\*kwargs) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 741, in save force\_update=force\_update, update\_fields=update\_fields) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 779, in save\_base force\_update, using, update\_fields, File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 870, in \_save\_table result = self.\_do\_insert(cls.\_base\_manager, using, fields, update\_pk, raw) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 908, in \_do\_insert using=using, raw=raw) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/manager.py”, line 82, in manager\_method return getattr(self.get\_queryset(), name)(\*args, \*\*kwargs) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1186, in \_insert return query.get\_compiler(using=using).execute\_sql(return\_id) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1366, in execute\_sql with self.connection.cursor() as cursor: File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor return self.\_cursor() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in \_cursor self.ensure\_connection() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in **exit** raise dj\_exc\_value.with\_traceback(traceback) from exc\_value File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) django.db.utils.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? 2020-02-16 19:19:07,114 ERROR awx.conf.settings Database settings are not available, using defaults. Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) psycopg2.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 87, in \_ctit\_db\_wrapper yield File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 415, in **getattr** value = self.\_get\_local(name) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 331, in \_get\_local self.\_preload\_cache() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 293, in \_preload\_cache for setting in Setting.objects.filter(key\_\_in=settings\_to\_cache.keys(), user\_\_isnull=True).order\_by(‘pk’): File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in **iter** self.\_fetch\_all() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in \_fetch\_all self.\_result\_cache = list(self.\_iterable\_class(self)) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in **iter** results = compiler.execute\_sql(chunked\_fetch=self.chunked\_fetch, chunk\_size=self.chunk\_size) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1131, in execute\_sql cursor = self.connection.cursor() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor return self.\_cursor() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in \_cursor self.ensure\_connection() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in **exit** raise dj\_exc\_value.with\_traceback(traceback) from exc\_value File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure\_connection self.connect() File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect self.connection = self.get\_new\_connection(conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get\_new\_connection connection = Database.connect(\*\*conn\_params) File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/ **init**.py”, line 126, in connect conn = \_connect(dsn, connection\_factory=connection\_factory, \*\*kwasync) django.db.utils.OperationalError: could not connect to server: No route to host Is the server running on host “postgres” (172.18.0.3) and accepting TCP/IP connections on port 5432? ` ---
**Author:** ![Christopher\_Meyers](https://avatars.discourse-cdn.com/v4/letter/c/3e96dc/32.png) [@Christopher\_Meyers](https://forum.ansible.com/u/Christopher_Meyers)\ **Post date:** [February 19, 2020, 6:39pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/11 "2020-02-19T18:39:15Z")
Dan, Seems like you are experiencing a different, but similar, problem as Tat. This looks like a failure in the docker networking somehow. I would advise trying what Tat tried: - Stop all container, prune all docker data - Turn off firewall - Reinstall. ---
**Author:** ![Dan\_W1](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dan\_W1](https://forum.ansible.com/u/Dan_W1)\ **Post date:** [February 19, 2020, 10:23pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/12 "2020-02-19T22:23:55Z")
What does “prune all docker data” involve? And which Firewall is being turned off? CentOS fw or a docker-specific fw? ---
**Author:** ![Tat\_Dat\_Pham](https://avatars.discourse-cdn.com/v4/letter/t/34f0e0/32.png) [@Tat\_Dat\_Pham](https://forum.ansible.com/u/Tat_Dat_Pham)\ **Post date:** [February 20, 2020, 4:38pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/13 "2020-02-20T16:38:39Z")
@Dan, “prune all docker data” for reinstalling awx and turn-off Cenots FW and disable fw startup. Maybe you need reboot server and try to reinstall AWX. You must stop all container before prune docker data. Hope this help! Vào 05:23:55 UTC+7 Thứ Năm, ngày 20 tháng 2 năm 2020, Dan W đã viết: ---
**Author:** ![kdelee](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kdelee/32/193_2.png) [@kdelee](https://forum.ansible.com/u/kdelee)\ **Post date:** [February 21, 2020, 1:19pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/14 "2020-02-21T13:19:58Z")
@Dan I think they mean: [https://docs.docker.com/engine/reference/commandline/system\_prune/](https://docs.docker.com/engine/reference/commandline/system_prune/) ```auto docker system prune --all ``` ---
**Author:** ![Stuart\_Luscombe](https://avatars.discourse-cdn.com/v4/letter/s/a6a055/32.png) [@Stuart\_Luscombe](https://forum.ansible.com/u/Stuart_Luscombe)\ **Post date:** [February 28, 2020, 11:15am UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/15 "2020-02-28T11:15:25Z")
I was experiencing the same issue on CentOS 8 with awx\_web not connecting to the postgres container and awx\_task also giving some errors. The only thing I found that fixed it was to disable the system firewall and rebooting. I’m currently running AWX in our dev environment, but if I want to move it to prod then I’ll need to tighten up the firewall. The server I’m using is a ‘plain’ CentOS 8 install using the ‘Server’ package group with docker-ce and epel-8 being the only additions. ---
**Author:** ![Dan\_W1](https://avatars.discourse-cdn.com/v4/letter/d/e9c0ed/32.png) [@Dan\_W1](https://forum.ansible.com/u/Dan_W1)\ **Post date:** [March 3, 2020, 5:06pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/16 "2020-03-03T17:06:10Z")
Thanks Everyone for the replies, apologies for the delay in my response. Yes - can absolutely CONFIRM that this is due to the Firewall. I didn’t need to prune the docker data, just simply changed the FW Zone which applied to the awxcompose\_default network from public to trusted. Initially I had moved awxcompose\_default to the “Internal” Zone and added the postgresql service, but still wasn’t able to connect via psql on awx\_task. Is there any documentation of which ports & protocols are leveraged so that correct firewall configuration can be implemented? Thanks, Dan ---
**Author:** ![Stuart\_Luscombe](https://avatars.discourse-cdn.com/v4/letter/s/a6a055/32.png) [@Stuart\_Luscombe](https://forum.ansible.com/u/Stuart_Luscombe)\ **Post date:** [March 4, 2020, 3:12pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/17 "2020-03-04T15:12:37Z")
I tried this out myself. ` sudo firewall-cmd --permanent --zone=trusted --change-interface=br-$(sudo docker network ls -f name=awxcompose\_default -q) sudo firewall-cmd –reload ` The containers now communicate with each other correctly. Thanks Dan! ---
**Author:** ![Bharat\_Maddirala](https://avatars.discourse-cdn.com/v4/letter/b/a5b964/32.png) [@Bharat\_Maddirala](https://forum.ansible.com/u/Bharat_Maddirala)\ **Post date:** [March 6, 2020, 6:10pm UTC](https://forum.ansible.com/t/cant-start-awx-web-connection-rabbitmq-postgres-timeout/32169/18 "2020-03-06T18:10:51Z")
I ran the firewall command. Then reboot the VM, stopped the container and pruned the data. Now , the logs for awx\_tasks have changed and look normal to me but I am still getting “connection refused” when trying to connect to the front end here are the logs for awx\_web 2020-03-06 17:56:22,218 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for \> than 1 seconds (startsecs) 2020-03-06 17:56:22,220 INFO spawned: ‘nginx’ with pid 145 2020-03-06 17:56:22,220 INFO success: uwsgi entered RUNNING state, process has stayed up for \> than 1 seconds (startsecs) 2020-03-06 17:56:22,220 INFO success: daphne entered RUNNING state, process has stayed up for \> than 1 seconds (startsecs) 2020/03/06 17:56:22 [emerg] 145#0: PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) nginx: [emerg] PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) 2020-03-06 17:56:22,254 INFO exited: nginx (exit status 1; not expected) WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 128 (default app) 2020-03-06 17:56:24,266 INFO spawned: ‘nginx’ with pid 152 2020/03/06 17:56:24 [emerg] 152#0: PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) nginx: [emerg] PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) 2020-03-06 17:56:24,290 INFO exited: nginx (exit status 1; not expected) WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 127 (default app) WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 125 (default app) WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 124 (default app) WSGI app 0 (mountpoint=‘’) ready in 4 seconds on interpreter 0x18e4550 pid: 126 (default app) 2020-03-06 17:56:25,165 INFO Starting server at tcp:port=8051:interface=127.0.0.1, channel layer awx.asgi:channel\_layer. 2020-03-06 17:56:25,165 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras) 2020-03-06 17:56:25,165 INFO Using busy-loop synchronous mode on channel layer 2020-03-06 17:56:25,165 INFO Listening on endpoint tcp:port=8051:interface=127.0.0.1 2020-03-06 17:56:28,170 INFO spawned: ‘nginx’ with pid 161 2020/03/06 17:56:28 [emerg] 161#0: PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) nginx: [emerg] PEM\_read\_bio\_X509\_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get\_name:no start line:Expecting: TRUSTED CERTIFICATE) 2020-03-06 17:56:28,179 INFO exited: nginx (exit status 1; not expected) 2020-03-06 17:56:29,181 INFO gave up: nginx entered FATAL state, too many start retries too quickly RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2 OKREADY RESULT 2