DB Migrations Failing on new install

I’m getting errors running AWX on Openshift.

Openshift version : 3.7
AWX 1.0.8.0

Setup runs through correctly but there are errors in the migration. Postgres is running 9.6 on the openshift environment also with a clean install i.e. no existing database.

awx_web logs:

*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 137)
spawned uWSGI worker 1 (pid: 139, cores: 1)
spawned uWSGI worker 2 (pid: 140, cores: 1)
spawned uWSGI worker 3 (pid: 141, cores: 1)
spawned uWSGI worker 4 (pid: 142, cores: 1)
spawned uWSGI worker 5 (pid: 143, cores: 1)
READY
2018-09-20 18:20:03,007 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-09-20 18:20:03,007 INFO success: daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-09-20 18:20:03,007 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-09-20 18:20:03,007 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-09-20 18:20:06,963 WARNING awx.conf.settings Database settings are not available, using defaults, error:
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/awx/wsgi.py”, line 80, in
application = get_wsgi_application()
File “/usr/lib/python2.7/site-packages/awx/wsgi.py”, line 77, in get_wsgi_application
return AWXWSGIHandler()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/wsgi.py”, line 151, in init
self.load_middleware()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/handlers/base.py”, line 56, in load_middleware
mw_class = import_string(middleware_path)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/module_loading.py”, line 20, in import_string
module = import_module(module_path)
File “/usr/lib64/python2.7/importlib/init.py”, line 37, in import_module
import(name)
File “/usr/lib/python2.7/site-packages/awx/sso/middleware.py”, line 18, in
from social_django import utils
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/social_django/utils.py”, line 13, in
BACKENDS = settings.AUTHENTICATION_BACKENDS
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 531, in getattr_without_cache
return getattr(self._wrapped, name)
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 444, in getattr
value = self._get_local(name)
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 361, in _get_local
self._preload_cache()
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 323, in _preload_cache
for setting in Setting.objects.filter(key__in=settings_to_cache.keys(), user__isnull=True).order_by(‘pk’):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
ProgrammingError: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…

Same error on awx_celery

Using /etc/ansible/ansible.cfg as config file
127.0.0.1 | SUCCESS => {
“changed”: false,
“db”: “awxtest”
}
Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 9, in
load_entry_point(‘awx==1.0.8.0’, ‘console_scripts’, ‘awx-manage’)()
File “/usr/lib/python2.7/site-packages/awx/init.py”, line 116, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/init.py”, line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 283, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 330, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/commands/shell.py”, line 101, in handle
exec(sys.stdin.read())
File “”, line 1, in
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py”, line 170, in create_superuser
return self._create_user(username, email, password, **extra_fields)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/models.py”, line 153, in _create_user
user.save(using=self._db)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/contrib/auth/base_user.py”, line 80, in save
super(AbstractBaseUser, self).save(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py”, line 808, in save
force_update=force_update, update_fields=update_fields)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py”, line 838, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py”, line 924, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/base.py”, line 963, in _do_insert
using=using, raw=raw)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/manager.py”, line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 1076, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 1112, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py”, line 94, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column “last_login” violates not-null constraint
DETAIL: Failing row contains (1, pbkdf2_sha256$36000$OD7sY8fSCxn3$NuGzqPNkK7cTL+ZLhlUZ3UiPtp6zpd+…, null, t, admin, , , root@localhost, t, t, 2018-09-20 18:19:58.843266+00).
2018-09-20 18:20:01,399 WARNING awx.conf.settings Database settings are not available, using defaults, error:
Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 9, in
load_entry_point(‘awx==1.0.8.0’, ‘console_scripts’, ‘awx-manage’)()
File “/usr/lib/python2.7/site-packages/awx/init.py”, line 116, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/init.py”, line 364, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/init.py”, line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 283, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 327, in execute
self.check()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 359, in check
include_deployment_checks=include_deployment_checks,
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/management/base.py”, line 346, in _run_checks
return checks.run_checks(**kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/checks/registry.py”, line 81, in run_checks
new_errors = check(app_configs=app_configs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/checks/urls.py”, line 16, in check_url_config
return check_resolver(resolver)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/core/checks/urls.py”, line 26, in check_resolver
return check_method()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/urls/resolvers.py”, line 254, in check
for pattern in self.url_patterns:
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/functional.py”, line 35, in get
res = instance.dict[self.name] = self.func(instance)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/urls/resolvers.py”, line 405, in url_patterns
patterns = getattr(self.urlconf_module, “urlpatterns”, self.urlconf_module)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/utils/functional.py”, line 35, in get
res = instance.dict[self.name] = self.func(instance)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/urls/resolvers.py”, line 398, in urlconf_module
return import_module(self.urlconf_name)
File “/usr/lib64/python2.7/importlib/init.py”, line 37, in import_module
import(name)
File “/usr/lib/python2.7/site-packages/awx/urls.py”, line 17, in
url(r’^sso/', include(‘social_django.urls’, namespace=‘social’)),
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/conf/urls/init.py”, line 50, in include
urlconf_module = import_module(urlconf_module)
File “/usr/lib64/python2.7/importlib/init.py”, line 37, in import_module
import(name)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/social_django/urls.py”, line 6, in
from . import views
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/social_django/views.py”, line 10, in
from .utils import psa
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/social_django/utils.py”, line 13, in
BACKENDS = settings.AUTHENTICATION_BACKENDS
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 531, in getattr_without_cache
return getattr(self._wrapped, name)
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 444, in getattr
value = self._get_local(name)
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 361, in _get_local
self._preload_cache()
File “/usr/lib/python2.7/site-packages/awx/conf/settings.py”, line 323, in _preload_cache
for setting in Setting.objects.filter(key__in=settings_to_cache.keys(), user__isnull=True).order_by(‘pk’):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 250, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 1118, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 53, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 899, in execute_sql
raise original_exception
ProgrammingError: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^

All of this results in jobs not running in the pending state.

Any ideas?

So I’ve figured out what is going and also fixed the issue. The kubernetes role mounts the management pod and the awx pod at the same time. The management pod is still performing a database migration when the awx pod starts up. As the scripts have not completed on the management pod, so the awx pod starts but the logs are full of python errors. I’ve modified the tasks/main.yml to delay the awx deployment until after the management pod has been deleted and all works successfully.

I’ll post details of the change to the file shortly and raise a pull request if agreed it’s the right way to go.

I’ve opened an issue https://github.com/ansible/awx/issues/2345 and pull request for this issue https://github.com/ansible/awx/pull/2346