Postgres Migration error

Anyone else get this with an external Postgres server? CentOS 7 for both Postgres and AWX.

“ProgrammingError: relation “celery_taskmeta” does not exist”

I’ve done it multiple times with the same results every time. Migration starts fine and then after a while a bunch of errors relating to this.

Here’s the full output:

exc, exc_info.traceback)))
[2017-09-11 00:32:58,332: CRITICAL/MainProcess] Task awx.main.scheduler.tasks.run_task_manager[0dd39143-c648-4b8c-915c-482a75f3acf1] INTERNAL ERROR: ProgrammingError(‘relation “celery_taskmeta” does not exist\nLINE 1: …taskmeta".“hidden”, “celery_taskmeta”.“meta” FROM "celery_ta…\n ^\n’,)
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 235, in trace_task
request=task_request,
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 256, in store_result
request=request, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/djcelery/backends/database.py”, line 29, in _store_result
traceback=traceback, children=self.current_task_children(request),
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/djcelery/managers.py”, line 42, in _inner
return fun(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/djcelery/managers.py”, line 181, in store_result
‘meta’: {‘children’: children}})
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/djcelery/managers.py”, line 87, in update_or_create
return get_queryset(self).update_or_create(**kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/djcelery/managers.py”, line 70, in update_or_create
obj, created = self.get_or_create(**kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 405, in get_or_create
return self.get(**lookup), False
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 328, in get
num = len(clone)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 144, in len
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 965, in _fetch_all
self._result_cache = list(self.iterator())
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/query.py”, line 238, in iterator
results = compiler.execute_sql()
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/models/sql/compiler.py”, line 840, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/utils.py”, line 98, in exit
six.reraise(dj_exc_type, dj_exc_value, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/django/db/backends/utils.py”, line 64, in execute
return self.cursor.execute(sql, params)
ProgrammingError: relation “celery_taskmeta” does not exist
LINE 1: …taskmeta".“hidden”, “celery_taskmeta”.“meta” FROM "celery_ta…

Interesting… this is on a production environment? Were there any errors from the initial migrations?

Not production, two brand new VMs. I’ll give it a shot on Fedora and see if that makes a difference.

So Just redid both on Fedora 26 and it worked fine. Not sure what happened with CentOS.

we’ve got the same problem, but on RHEL7.3 based VMs for AWX with an external Postresql (9.2, also on RHEL 7.3)
but switching to Fedora is not a solution for us, we’re a plain RHEL shop.

damn…

You could try just re-running it again. It looks like some folks interrupt the installation the first time and the migration doesn’t fully account for that, see: https://github.com/ansible/awx/issues/138

Matthew Jones (majones@redhat.com) said:

You could try just re-running it again. It looks like some folks interrupt
the installation the first time and the migration doesn't fully account for
that, see: https://github.com/ansible/awx/issues/138

> we've got the same problem, but on RHEL7.3 based VMs for AWX with an
> external Postresql (9.2, also on RHEL 7.3)
> but switching to Fedora is not a solution for us, we're a plain RHEL shop.

In general, PG 9.2 may not work. We're designed for 9.6 or later. (But I
defer to Matt.)

Bill

re-running doesn’t help, I’ve enabled Postgresql query logging so I can see it’s doing the same thing over and over again…
ERROR: relation “conf_setting” does not exist at character 158

shall I add the sql statements to the issue 138 ?

It sounds like your migrations didn’t run. I don’t need to see the sql queries, I know what those are but I do need to see the migration run from the top of your task container logs to see what happened there.