Can't start AWX web, connection rabbitmq, postgres timeout

Archives AWX Project
1
ISSUE TYPE- Bug Report
SUMMARY##### ENVIRONMENT- AWX version: 9.1.1
  • AWX install method: docker on centos8
  • Ansible version: 2.9.3
  • Operating System: Centos 8
  • Web Browser: Chrome latest
STEPS TO REPRODUCE

Install awx follow command


ansible-playbook -i inventory install.yml

All container is up

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
756ed3d8a7e2        ansible/awx_task:9.1.1       "/tini -- /bin/sh -c…"   About an hour ago   Up About an hour    8052/tcp                                              awx_task
ffb7cfb55ba5        ansible/awx_web:9.1.1        "/tini -- /bin/sh -c…"   About an hour ago   Up About an hour    0.0.0.0:80->8052/tcp                                  awx_web
a3a1f2724640        postgres:10                  "docker-entrypoint.s…"   About an hour ago   Up About an hour    5432/tcp                                              awx_postgres
ff2a5734c174        memcached:alpine             "docker-entrypoint.s…"   About an hour ago   Up About an hour    11211/tcp                                             awx_memcached
54685e6e1512        ansible/awx_rabbitmq:3.7.4   "docker-entrypoint.s…"   About an hour ago   Up About an hour    4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   awx_rabbitmq
2

Encountered very similar issue. awx_task is not able to reach Postgres container using standard deployment.

SUMMARY##### ENVIRONMENT- AWX version: 9.2.0 (latest)
  • AWX install method: Docker Compose on centos8
  • Ansible version: 2.9.4
  • Operating System: Centos 8
  • Web Browser: Chrome latest
  • Python 3.6.8
3

Yep, that is my issue. But still not resolved yet. AWX team don’t support :slight_smile:

Vào 05:20:05 UTC+7 Thứ Bảy, ngày 15 tháng 2 năm 2020, Dan W đã viết:

4

This is a problem with the new postgres containers. They now require a superuser password to be defined on boot. We are looking in to how best to provide this password.

5

You can verify that you are experiencing the same issue by issuing docker logs -f awx_postgres. If you are experiencing this issue you will see the below output.

Error: Database is uninitialized and superuser password is not specified.
       You must specify POSTGRES_PASSWORD for the superuser. Use
       "-e POSTGRES_PASSWORD=password" to set it in "docker run".
       You may also use POSTGRES_HOST_AUTH_METHOD=trust to allow all connections
       without a password. This is *not* recommended. See PostgreSQL
       documentation about "trust":
       [https://www.postgresql.org/docs/current/auth-trust.html](https://www.postgresql.org/docs/current/auth-trust.html)
6

After digging deeper, it looks like you are not experiencing the postgres issue I mentioned earlier. Tat, from the traceback the problem is awx_task → awx_rabbitmq communication Tat, you basically included all of the debugging information I would have asked for and nothing looks apparently wrong. I did notice, however, that you don’t have a network entry called awxcompose_default which means you have changed the default docker_compose_dir setting. That is fine, but maybe some other modification to inventory has led to this. Can you post your inventory file?

Dan,

Can you run the debug steps that Tat ran and post the results?

7

I dont get this error on POSTGRES.

Vào 21:28:06 UTC+7 Thứ Ba, ngày 18 tháng 2 năm 2020, Christopher Meyers đã viết:

8

Yesterday, I try to reinstall AWX on Centos 8.

  • Stop all container, prune all docker data
  • Turn off firewall
  • Reinstall.
    And AWX started without any problem. I dont know how and what problem that i got, although i try to reinstall awx many times before. But now, AWX is running.

Vào 22:40:53 UTC+7 Thứ Ba, ngày 18 tháng 2 năm 2020, Christopher Meyers đã viết:

9

Chris - Please see the debug details below. I’ve tried stopping/removing all the containers and re-running the Installer but have gotten the same result. Since this, I’ve moved to CentOS 7 and have installed using the same inventory file and using latest AWX.

Containers are up:

`

[root@awx01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bc7c32d4f60c ansible/awx_task:latest “/tini – /bin/sh -c…” 2 days ago Up 2 days 8052/tcp awx_task
49df2a131c81 ansible/awx_web:latest “/tini – /bin/sh -c…” 2 days ago Up 2 days 0.0.0.0:80->8052/tcp, 0.0.0.0:443->8053/tcp awx_web
7b15c4dbc69d memcached:alpine “docker-entrypoint.s…” 2 days ago Up 2 days 11211/tcp awx_memcached
059e758a6581 postgres:10 “docker-entrypoint.s…” 2 days ago Up 2 days 5432/tcp awx_postgres
5b31f01e9ab9 ansible/awx_rabbitmq:3.7.4 “docker-entrypoint.s…” 2 days ago Up 2 days 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp awx_rabbitmq

`

Networks are Default:

`

[root@awx01 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
4eb2bdaca233 awxcompose_default bridge local
49ed9426b273 bridge bridge local
354d4f4ad44e host host local
57310c7a7426 none null local

`

IPs are configured and I’m able to ping the other containers from each:

`

[root@awx01 ~]# docker inspect -f ‘{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}’ awx_postgres awx_task awx_memcached awx_web awx_rabbitmq
172.18.0.3
172.18.0.6
172.18.0.4
172.18.0.5
172.18.0.2

`

AWX_Postgres is up and I can access the DB using psql from within the container:

`

[root@awx01 ~]# docker logs awx_postgres

PostgreSQL Database directory appears to contain a database; Skipping initialization

2020-02-16 19:03:31.638 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2020-02-16 19:03:31.638 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2020-02-16 19:03:31.642 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2020-02-16 19:03:31.664 UTC [24] LOG: database system was shut down at 2020-02-16 18:57:53 UTC
2020-02-16 19:03:31.673 UTC [1] LOG: database system is ready to accept connections

`

AWX_RABBITMQ is up:

`

[root@awx01 ~]# docker logs awx_rabbitmq
2020-02-16 19:03:39.997 [info] <0.33.0> Application lager started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.459 [info] <0.33.0> Application crypto started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.586 [info] <0.33.0> Application mnesia started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.586 [info] <0.33.0> Application xmerl started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application os_mon started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application recon started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application cowlib started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application jsx started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application asn1 started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.593 [info] <0.33.0> Application public_key started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.666 [info] <0.33.0> Application ssl started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.672 [info] <0.33.0> Application ranch started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.675 [info] <0.33.0> Application cowboy started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.675 [info] <0.33.0> Application ranch_proxy_protocol started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.675 [info] <0.33.0> Application rabbit_common started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.684 [info] <0.33.0> Application amqp_client started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.766 [info] <0.33.0> Application inets started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:40.779 [info] <0.193.0>
Starting RabbitMQ 3.7.4 on Erlang 20.1.7
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

## RabbitMQ 3.7.4. Copyright (C) 2007-2018 Pivotal Software, Inc.

########## Licensed under the MPL. See http://www.rabbitmq.com/

########## Logs:

Starting broker…
2020-02-16 19:03:40.800 [info] <0.193.0>
node : rabbit@5b31f01e9ab9
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.conf
cookie hash : at619UOZzsenF44tSK3ulA==
log(s) :
database dir : /var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9
2020-02-16 19:03:43.259 [info] <0.201.0> Memory high watermark set to 6350 MiB (6659248947 bytes) of 15876 MiB (16648122368 bytes) total
2020-02-16 19:03:43.263 [info] <0.203.0> Enabling free disk space monitoring
2020-02-16 19:03:43.263 [info] <0.203.0> Disk free limit set to 50MB
2020-02-16 19:03:43.266 [info] <0.205.0> Limiting to approx 1048476 file handles (943626 sockets)
2020-02-16 19:03:43.266 [info] <0.206.0> FHC read buffering: OFF
2020-02-16 19:03:43.266 [info] <0.206.0> FHC write buffering: ON
2020-02-16 19:03:43.267 [info] <0.193.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9 is empty. Assuming we need to join an existing cluster or initialise from scratch…
2020-02-16 19:03:43.267 [info] <0.193.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2020-02-16 19:03:43.267 [info] <0.193.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2020-02-16 19:03:43.267 [info] <0.193.0> Peer discovery backend does not support locking, falling back to randomized delay
2020-02-16 19:03:43.267 [info] <0.193.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2020-02-16 19:03:43.268 [info] <0.193.0> All discovered existing cluster peers:
2020-02-16 19:03:43.268 [info] <0.193.0> Discovered no peer nodes to cluster with
2020-02-16 19:03:43.269 [info] <0.33.0> Application mnesia exited with reason: stopped
2020-02-16 19:03:43.290 [info] <0.33.0> Application mnesia started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:43.371 [info] <0.193.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-02-16 19:03:43.405 [info] <0.193.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-02-16 19:03:43.439 [info] <0.193.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2020-02-16 19:03:43.439 [info] <0.193.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2020-02-16 19:03:43.440 [info] <0.193.0> Priority queues enabled, real BQ is rabbit_variable_queue
2020-02-16 19:03:43.445 [info] <0.374.0> Starting rabbit_node_monitor
2020-02-16 19:03:43.475 [info] <0.193.0> message_store upgrades: 1 to apply
2020-02-16 19:03:43.475 [info] <0.193.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2020-02-16 19:03:43.475 [info] <0.193.0> message_store upgrades: No durable queues found. Skipping message store migration
2020-02-16 19:03:43.475 [info] <0.193.0> message_store upgrades: Removing the old message store data
2020-02-16 19:03:43.476 [info] <0.193.0> message_store upgrades: All upgrades applied successfully
2020-02-16 19:03:43.508 [info] <0.193.0> Management plugin: using rates mode ‘basic’
2020-02-16 19:03:43.509 [info] <0.193.0> Adding vhost ‘awx’
2020-02-16 19:03:43.523 [info] <0.408.0> Making sure data directory ‘/var/lib/rabbitmq/mnesia/rabbit@5b31f01e9ab9/msg_stores/vhosts/5A7PFSQL4N28PQOL0INDFDDLU’ for vhost ‘awx’ exists
2020-02-16 19:03:43.527 [info] <0.408.0> Starting message stores for vhost ‘awx’
2020-02-16 19:03:43.527 [info] <0.412.0> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg_store_transient”: using rabbit_msg_store_ets_index to provide index
2020-02-16 19:03:43.529 [info] <0.408.0> Started message store of type transient for vhost ‘awx’
2020-02-16 19:03:43.529 [info] <0.415.0> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg_store_persistent”: using rabbit_msg_store_ets_index to provide index
2020-02-16 19:03:43.530 [warning] <0.415.0> Message store “5A7PFSQL4N28PQOL0INDFDDLU/msg_store_persistent”: rebuilding indices from scratch
2020-02-16 19:03:43.531 [info] <0.408.0> Started message store of type persistent for vhost ‘awx’
2020-02-16 19:03:43.532 [info] <0.193.0> Creating user ‘guest’
2020-02-16 19:03:43.534 [info] <0.193.0> Setting user tags for user ‘guest’ to [administrator]
2020-02-16 19:03:43.536 [info] <0.193.0> Setting permissions for ‘guest’ in ‘awx’ to ‘.', '.’, ‘.*’
2020-02-16 19:03:43.540 [info] <0.453.0> started TCP Listener on [::]:5672
2020-02-16 19:03:43.544 [info] <0.193.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@5b31f01e9ab9
2020-02-16 19:03:43.547 [info] <0.193.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@5b31f01e9ab9
2020-02-16 19:03:43.547 [info] <0.33.0> Application rabbit started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:43.550 [info] <0.33.0> Application rabbitmq_management_agent started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:43.550 [info] <0.33.0> Application rabbitmq_web_dispatch started on node rabbit@5b31f01e9ab9
2020-02-16 19:03:43.584 [info] <0.510.0> Management plugin started. Port: 15672
2020-02-16 19:03:43.584 [info] <0.616.0> Statistics database started.
2020-02-16 19:03:43.585 [info] <0.33.0> Application rabbitmq_management started on node rabbit@5b31f01e9ab9
completed with 3 plugins.
2020-02-16 19:03:43.814 [info] <0.5.0> Server startup complete; 3 plugins started.

  • rabbitmq_management
  • rabbitmq_web_dispatch
  • rabbitmq_management_agent

`

I’ll post the awx_task logs in a follow up.

10

AWX_TASK Logs:

`

docker logs awx_task
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
“ansible_facts”: {
“discovered_interpreter_python”: “/usr/libexec/platform-python”
},
“changed”: false,
“elapsed”: 301,
“msg”: “Timeout when waiting for postgres:5432”
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
“ansible_facts”: {
“discovered_interpreter_python”: “/usr/libexec/platform-python”
},
“changed”: false,
“elapsed”: 301,
“msg”: “Timeout when waiting for memcached:11211”
}
Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | FAILED! => {
“ansible_facts”: {
“discovered_interpreter_python”: “/usr/libexec/platform-python”
},
“changed”: false,
“elapsed”: 301,
“msg”: “Timeout when waiting for rabbitmq:5672”
}
Using /etc/ansible/ansible.cfg as config file
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TCP/IP connections on port 5432?
127.0.0.1 | FAILED! => {
“ansible_facts”: {
“discovered_interpreter_python”: “/usr/libexec/platform-python”
},
“changed”: false,
“msg”: “unable to connect to database: could not connect to server: No route to host\n\tIs the server running on host "postgres" (172.18.0.3) and accepting\n\tTCP/IP connections on port 5432?\n”
}
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py”, line 152, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 83, in wrapped
res = handle_func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/commands/migrate.py”, line 87, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/executor.py”, line 18, in init
self.loader = MigrationLoader(self.connection)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/loader.py”, line 49, in init
self.build_graph()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/loader.py”, line 212, in build_graph
self.applied_migrations = recorder.applied_migrations()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/recorder.py”, line 73, in applied_migrations
if self.has_table():
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/migrations/recorder.py”, line 56, in has_table
return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor
return self._cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor
self.ensure_connection()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py”, line 152, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/commands/shell.py”, line 92, in handle
exec(sys.stdin.read())
File “”, line 1, in
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/models.py”, line 162, in create_superuser
return self._create_user(username, email, password, **extra_fields)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/models.py”, line 145, in _create_user
user.save(using=self._db)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/contrib/auth/base_user.py”, line 66, in save
super().save(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 741, in save
force_update=force_update, update_fields=update_fields)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 779, in save_base
force_update, using, update_fields,
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 870, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/base.py”, line 908, in _do_insert
using=using, raw=raw)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/manager.py”, line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1366, in execute_sql
with self.connection.cursor() as cursor:
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor
return self._cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor
self.ensure_connection()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

2020-02-16 19:19:07,114 ERROR awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 87, in _ctit_db_wrapper
yield
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 415, in getattr
value = self._get_local(name)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 331, in _get_local
self._preload_cache()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 293, in _preload_cache
for setting in Setting.objects.filter(key__in=settings_to_cache.keys(), user__isnull=True).order_by(‘pk’):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1131, in execute_sql
cursor = self.connection.cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor
return self._cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor
self.ensure_connection()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No route to host
Is the server running on host “postgres” (172.18.0.3) and accepting
TCP/IP connections on port 5432?

`

11

Dan,

Seems like you are experiencing a different, but similar, problem as Tat. This looks like a failure in the docker networking somehow. I would advise trying what Tat tried:

  • Stop all container, prune all docker data
  • Turn off firewall
  • Reinstall.
12

What does “prune all docker data” involve? And which Firewall is being turned off? CentOS fw or a docker-specific fw?

13

@Dan,

“prune all docker data” for reinstalling awx and turn-off Cenots FW and disable fw startup. Maybe you need reboot server and try to reinstall AWX.
You must stop all container before prune docker data.
Hope this help!

Vào 05:23:55 UTC+7 Thứ Năm, ngày 20 tháng 2 năm 2020, Dan W đã viết:

14

@Dan I think they mean: https://docs.docker.com/engine/reference/commandline/system_prune/

docker system prune --all
15

I was experiencing the same issue on CentOS 8 with awx_web not connecting to the postgres container and awx_task also giving some errors.

The only thing I found that fixed it was to disable the system firewall and rebooting. I’m currently running AWX in our dev environment, but if I want to move it to prod then I’ll need to tighten up the firewall.

The server I’m using is a ‘plain’ CentOS 8 install using the ‘Server’ package group with docker-ce and epel-8 being the only additions.

16

Thanks Everyone for the replies, apologies for the delay in my response.

Yes - can absolutely CONFIRM that this is due to the Firewall. I didn’t need to prune the docker data, just simply changed the FW Zone which applied to the awxcompose_default network from public to trusted. Initially I had moved awxcompose_default to the “Internal” Zone and added the postgresql service, but still wasn’t able to connect via psql on awx_task. Is there any documentation of which ports & protocols are leveraged so that correct firewall configuration can be implemented?

Thanks,
Dan

17

I tried this out myself.

`

sudo firewall-cmd --permanent --zone=trusted --change-interface=br-$(sudo docker network ls -f name=awxcompose_default -q)

sudo firewall-cmd –reload

`

The containers now communicate with each other correctly.

Thanks Dan!

18

I ran the firewall command. Then reboot the VM, stopped the container and pruned the data. Now , the logs for awx_tasks have changed and look normal to me but I am still getting “connection refused” when trying to connect to the front end

here are the logs for awx_web

2020-03-06 17:56:22,218 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-06 17:56:22,220 INFO spawned: ‘nginx’ with pid 145
2020-03-06 17:56:22,220 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-06 17:56:22,220 INFO success: daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020/03/06 17:56:22 [emerg] 145#0: PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: [emerg] PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
2020-03-06 17:56:22,254 INFO exited: nginx (exit status 1; not expected)
WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 128 (default app)
2020-03-06 17:56:24,266 INFO spawned: ‘nginx’ with pid 152
2020/03/06 17:56:24 [emerg] 152#0: PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: [emerg] PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
2020-03-06 17:56:24,290 INFO exited: nginx (exit status 1; not expected)
WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 127 (default app)
WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 125 (default app)
WSGI app 0 (mountpoint=‘’) ready in 3 seconds on interpreter 0x18e4550 pid: 124 (default app)
WSGI app 0 (mountpoint=‘’) ready in 4 seconds on interpreter 0x18e4550 pid: 126 (default app)
2020-03-06 17:56:25,165 INFO Starting server at tcp:port=8051:interface=127.0.0.1, channel layer awx.asgi:channel_layer.
2020-03-06 17:56:25,165 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2020-03-06 17:56:25,165 INFO Using busy-loop synchronous mode on channel layer
2020-03-06 17:56:25,165 INFO Listening on endpoint tcp:port=8051:interface=127.0.0.1
2020-03-06 17:56:28,170 INFO spawned: ‘nginx’ with pid 161
2020/03/06 17:56:28 [emerg] 161#0: PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: [emerg] PEM_read_bio_X509_AUX(“/etc/nginx/awxweb.pem”) failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
2020-03-06 17:56:28,179 INFO exited: nginx (exit status 1; not expected)
2020-03-06 17:56:29,181 INFO gave up: nginx entered FATAL state, too many start retries too quickly
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2
OKREADY
RESULT 2