AWX via Fargate Fails to start - postgresql errors

,

Hey All,

I’ve been trying to deploy awx via ECS Fargate recently and am running into deployment issues. What I’ve done is build images based on Dockerfiles for:

  • awx_web

  • awx_task

  • memcache

  • awx_rabbitmq

that pull the ‘original’ awx images from dockerhub and inject kms encryption environment variables to the containers, which are decrypted with the aws-cli (installed via the dockerfile). These environment variables are given to the underlying container in the launch_awx.sh and launch_awx_task.sh files for their respective containers, and supervisord has access to them. The SECRET_KEY file is also created and populated with the SECRET_KEY environment variable from the injected kms secrets, then environment.sh is populated. None of this is done with installer or install.yml playbooks, all containers are deployed individually and the rabbitmq/awx_web/memcache/awx_task containers are fed environment variables that point to the ports / db info that they need.

Currently im stuck with this error:

`
2020-06-10 17:47:58,878 ERROR awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/run/postgresql/.s.PGSQL.5432”?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 76, in _ctit_db_wrapper
yield
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 404, in getattr
value = self._get_local(name)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 347, in _get_local
setting = Setting.objects.filter(key=name, user__isnull=True).order_by(‘pk’).first()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 653, in first
for obj in (self if self.ordered else self.order_by(‘pk’))[:1]:
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1138, in execute_sql
cursor = self.connection.cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 256, in cursor
return self._cursor()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 233, in _cursor
self.ensure_connection()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/base/base.py”, line 195, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
connection = Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/run/postgresql/.s.PGSQL.5432”?
`

which has to deal with awx_web and awx_task not connecting to the postgresql db, however all of my environment variables are correct and I can see the db running both in rds and when i test locally. Anyone have an idea of what could be going wrong? Not sure why awx.conf.settings says the database settings arent available.

Environment variables that I inject are:

AWX_ADMIN_PASSWORD

AWX_ADMIN_USER

DATABASE_HOST

DATABASE_NAME

DATABASE_PASSWORD

DATABASE_PORT

DATABASE_USER

MEMCACHED_HOST

MEMCACHED_PORT

RABBITMQ_HOST

RABBITMQ_PASSWORD

RABBITMQ_PORT

RABBITMQ_USER

RABBITMQ_VHOST

SECRET_KEY

One more piece of info, the awx_web and awx_task containers have the entrypoint launch_awx.sh and launch_awx_task.sh, which launch supervisord with supervisord.conf files. I assumed those two entrypoints would configure awx to use the environment variables I passed to the containers, but it doesnt seem to be doing so.