Problems updating to 22.4.0 from 22.3.0

Archives AWX Project
1

Hi,
We have awx version 22.3.0 (operator 2.2.0) running in AKS using an external Postgres DB. I’ve recently attempted to update it to version 22.4.0 by deploying operator 2.3.0 but without success. The containers are created but the web page gives a gateway error and the awx_web logs show:

.
.
[wait-for-migrations] Attempt 28 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 29 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 30 of 30
[wait-for-migrations] ERROR: Database migrations not applied

I tried running awx-manage migrate manually but that also errors giving the following exceptions:

psycopg.errors.DuplicateTable: relation “django_migrations” already exists

django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (relation “django_migrations” already exists)

Anyone have any idea what might be going wrong?
I managed to roll it back to 22.3.0 by redeploying operator but I can try it again if there’s any logs etc that would be useful…

Thanks in advance!

2

Hello,
We are sorry to hear you’re having trouble with migrations. We’d like to gather a bit more information from you. Would you mind running awx-manage showmigrations in your task container and provide us with that output? That could be helpful as it should show where the migrations are getting hung up.

You can also log into your database and run \dt? That should show the existing tables within your database. From there you can also run select * from django_migrations; . Please provide us with that information as well.

  • AWX Team
3

Hi,
Thanks for reply and sorry for the delay in getting back to you. I’ve redeployed 22.4.0 so that I can run the commands you asked and it has broken again in the same way.
awx-manage showmigrations errors with the following output:

bash-5.1$ awx-manage showmigrations
Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py”, line 200, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py”, line 442, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py”, line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 412, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 453, in execute
self.check()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 485, in check
all_issues = checks.run_checks(
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/checks/registry.py”, line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/checks/urls.py”, line 14, in check_url_config
return check_resolver(resolver)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/checks/urls.py”, line 24, in check_resolver
return check_method()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/urls/resolvers.py”, line 494, in check
for pattern in self.url_patterns:
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py”, line 57, in get
res = instance.dict[self.name] = self.func(instance)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/urls/resolvers.py”, line 715, in url_patterns
patterns = getattr(self.urlconf_module, “urlpatterns”, self.urlconf_module)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py”, line 57, in get
res = instance.dict[self.name] = self.func(instance)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/urls/resolvers.py”, line 708, in urlconf_module
return import_module(self.urlconf_name)
File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1030, in _gcd_import
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “”, line 850, in exec_module
File “”, line 228, in _call_with_frames_removed
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/urls.py”, line 13, in
re_path(r’^api/‘, include(‘awx.api.urls’, namespace=‘api’)),
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/urls/conf.py”, line 38, in include
urlconf_module = import_module(urlconf_module)
File “/usr/lib64/python3.9/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1030, in _gcd_import
File “”, line 1007, in _find_and_load
File “”, line 986, in _find_and_load_unlocked
File “”, line 680, in _load_unlocked
File “”, line 850, in exec_module
File “”, line 228, in _call_with_frames_removed
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/urls/init.py”, line 5, in
from .urls import urlpatterns
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/urls/urls.py”, line 83, in
from .oauth2_root import urls as oauth2_root_urls
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/urls/oauth2_root.py”, line 10, in
from oauth2_provider import views
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/oauth2_provider/views/init.py”, line 18, in
from .oidc import ConnectDiscoveryInfoView, JwksInfoView, UserInfoView
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/oauth2_provider/views/oidc.py”, line 16, in
Application = get_application_model()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/oauth2_provider/models.py”, line 571, in get_application_model
return apps.get_model(oauth2_settings.APPLICATION_MODEL)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py”, line 119, in oauth2_getattribute
val = settings.OAUTH2_PROVIDER.get(attr)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/conf/settings.py”, line 527, in getattr_without_cache
return getattr(self._wrapped, name)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/conf/settings.py”, line 425, in getattr
value = self._get_local_with_cache(name)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cachetools/init.py”, line 724, in wrapper
v = method(self, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/conf/settings.py”, line 420, in _get_local_with_cache
return self._get_local(name)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/conf/settings.py”, line 339, in _get_local
self._preload_cache()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/conf/settings.py”, line 309, in _preload_cache
if settings_to_cache[setting.key] != SETTING_CACHE_NOTSET:
KeyError: b’AUTH_LDAP_GROUP_TYPE_PARAMS’

I’ve attached the output of the \dt and select commands as separate files as they’re a bit large to paste directly into the message body.

Thanks again for your help,

Neil

(attachments)

tables.txt (371 KB)
django_migrations.txt (31.3 KB)

4

can you share the awx spec file that you are using to deploy this?

also you provide output of

SELECT * FROM conf_setting where key like ‘%LDAP%’;

and remove sensitive info please

AWX Team

5

Hi,
Thanks for your message. I’ve attached the results of the query along with the awx spec (the variables are substituted depending on whether it’s the prod or test deployment)

Thanks again,

Neil

(attachments)

query_ldap.txt (15.6 KB)
awx.yaml (828 Bytes)

6

Hi,
I’ve just tried deploying operator 2.4.0 to deploy awx 22.5.0 and it fails with the same ‘waiting for migrations’ message in the awx-web container logs.

Interestingly, this time I was able to run the awx-manage command and have attached the result.

awx-manage check fails with the error:

“KeyError: b’AUTH_LDAP_GROUP_TYPE_PARAMS’”.

(I’ve attached the full output of that command too if it helps.)

We dont use ladp authentication so the settings are all at their defaults.

Thanks again,

Neil

(attachments)

awx-manage check.txt (5.92 KB)
showmigrations.txt (8.62 KB)

7

Per your last email are you starting up a new instance altogether? It looks like this might be the case since there are no migrations applied at all.
If so, are you attempting to set up LDAP via your spec file? If so, can you please send that with any sensitive information redacted with something like <redacted>.

-The AWX Team

8

Hi,
We just deploy the new operator to kubernetes and it deploys the relevant awx version. We use an external postgres DB on azure so all the settings are retained. We don’t use LDAP at all.

However, since my last message I deployed operator v2.5.0 and this time awx updated to 22.6.0 successfully. The migrations completed and everything seems to work as it should so, for now at least, all is well :slight_smile:

Thanks again for your help,

Neil