[DAB RBAC] Migration problem from 24.2.1 to 24.3.1 (or above)

Get Help
, ,
1

Hi,

We are currently trying to upgrade from 24.2.1 to 24.6.1 but we encoutered a blocking problem during the database migration.
(Our database is an external PostgreSQL 12).

The migration job is looping and the following stacktrace can be observed :

Operations to perform:
  Apply all migrations: auth, conf, contenttypes, dab_rbac, dab_resource_registry, main, oauth2_provider, sessions, sites, social_django, sso
Running migrations:
2024-12-05 07:26:14,338 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition NotificationTemplate Admin pk=RoleDefinition object (361) with 3 permissions
2024-12-05 07:26:14,362 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization NotificationTemplate Admin pk=RoleDefinition object (362) with 5 permissions
2024-12-05 07:26:14,381 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Inventory Admin pk=RoleDefinition object (363) with 6 permissions
2024-12-05 07:26:14,405 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Inventory Admin pk=RoleDefinition object (364) with 8 permissions
2024-12-05 07:26:14,424 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Inventory Use pk=RoleDefinition object (365) with 2 permissions
2024-12-05 07:26:14,437 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Inventory Adhoc pk=RoleDefinition object (366) with 2 permissions
2024-12-05 07:26:14,446 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Inventory Update pk=RoleDefinition object (367) with 2 permissions
2024-12-05 07:26:14,457 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition JobTemplate Admin pk=RoleDefinition object (368) with 4 permissions
2024-12-05 07:26:14,479 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization JobTemplate Admin pk=RoleDefinition object (369) with 5 permissions
2024-12-05 07:26:14,503 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition JobTemplate Execute pk=RoleDefinition object (370) with 2 permissions
2024-12-05 07:26:14,523 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition InstanceGroup Admin pk=RoleDefinition object (371) with 4 permissions
2024-12-05 07:26:14,552 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition InstanceGroup Use pk=RoleDefinition object (372) with 2 permissions
2024-12-05 07:26:14,580 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Project Admin pk=RoleDefinition object (373) with 5 permissions
2024-12-05 07:26:14,608 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Project Admin pk=RoleDefinition object (374) with 7 permissions
2024-12-05 07:26:14,632 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Project Update pk=RoleDefinition object (375) with 2 permissions
2024-12-05 07:26:14,659 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Project Use pk=RoleDefinition object (376) with 2 permissions
2024-12-05 07:26:14,684 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Team Admin pk=RoleDefinition object (377) with 4 permissions
2024-12-05 07:26:14,707 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Team Member pk=RoleDefinition object (378) with 2 permissions
2024-12-05 07:26:14,731 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Credential Admin pk=RoleDefinition object (379) with 4 permissions
2024-12-05 07:26:14,752 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Credential Admin pk=RoleDefinition object (380) with 6 permissions
2024-12-05 07:26:14,773 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Credential Use pk=RoleDefinition object (381) with 2 permissions
2024-12-05 07:26:14,780 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition ExecutionEnvironment Admin pk=RoleDefinition object (382) with 3 permissions
2024-12-05 07:26:14,787 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization ExecutionEnvironment Admin pk=RoleDefinition object (383) with 5 permissions
2024-12-05 07:26:14,794 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition WorkflowJobTemplate Admin pk=RoleDefinition object (384) with 5 permissions
2024-12-05 07:26:14,801 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization WorkflowJobTemplate Admin pk=RoleDefinition object (385) with 7 permissions
2024-12-05 07:26:14,808 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition WorkflowJobTemplate Execute pk=RoleDefinition object (386) with 2 permissions
2024-12-05 07:26:14,817 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition WorkflowJobTemplate Approve pk=RoleDefinition object (387) with 2 permissions
2024-12-05 07:26:14,827 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Member pk=RoleDefinition object (388) with 2 permissions
2024-12-05 07:26:14,834 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Admin pk=RoleDefinition object (389) with 46 permissions
2024-12-05 07:26:14,844 INFO     [-] awx.main.migrations._dab_rbac Created RoleDefinition Organization Audit pk=RoleDefinition object (390) with 10 permissions
Traceback (most recent call last):
[ ... ]
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/migrations/_dab_rbac.py", line 196, in migrate_to_new_rbac
    object = object_cls.objects.get(pk=role.object_id)  # WORKAROUND, role.content_object does not work in migrations
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
__fake__.JobTemplate.DoesNotExist: JobTemplate matching query does not exist.

An old thread seemed related (with the same error message “JobTemplate matching query does not exist”) but it was something else.
Source: Need Help with awx 9.3.0 migration to 13.0.0

A migration to a closer version, 24.4.0 and then 23.4.1 has been tested but unfortunately with the same outcome.

The changelog is pretty clear on the 24.3.0 : the DAB RBAC was implemented but had a major problem which was fixed in 24.3.1.
And regarding the error message, it is clearly linked :

  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/migrations/_dab_rbac.py", line 196, in migrate_to_new_rbac
    object = object_cls.objects.get(pk=role.object_id)  # WORKAROUND, role.content_object does not work in migrations

I read all the posts/comments by @AlanCoding on the subject, but still, I cannot make it work. I browsed the Operator AWX CRD to see if there was an option to bypass that problem, with no success. I also searched throughout all the documentation I could find, but nothing on that topic.

As a solution, the only scenario that emerges right now for us is to dump the data, reinstall AWX on a empty database and then import the dump. But it sounds a bit risky and does not seem to be the legitimate choice.

I doubt it, but could it be related to the version of PostgreSQL ?

If I am missing something obvious, please enlighten me.

Thanks in advance.

2

Hi,

do you evaluate the migration to postgres 14/15?

In my lab I need to upgrade postgres then upgrade instance.

1 Like
3

I would also recommend upgrading your external PostgreSQL database as part of your AWX upgrade. As of AWX v24.0.0, the default postgres image uses PG15, and was upgraded from PG13. PG12 hasn’t been the default since… AWX v21.3.0.

Considering you are already at AWX v24.2.1, I suggest shutting down AWX, and performing a PG12 → PG13 → PG15 upgrade (I don’t think PostgreSQL supports directly upgrading from 12 to 15). Once the PG upgrade is complete, start AWX again, verify it’s working with the new database before continuing with the AWX upgrade.

1 Like
4

Ok, thanks for the replies @Denney-tech & @tanganellilore !

We were indeed planning a migration to PG 15, but we were not aware that it could block the path for the AWX migration.
We will do so and I will keep you informed.

Thanks again.

1 Like
5

Hi again,

We have completed the migration to PG15.
For the record, we have many clients using our product, so we could not start ex nihilo : we had to migrate our data.

Here is our method :

  1. Install PG15 on the same machine as PG12
  2. Change PG12 port [R]
  3. Change PG15 pg_hba access to prevent data changes from outside [R]
  4. Change PG15 port to get the main port (the one initially used by PG12) [R]
  5. Make a PG12 dump without the main_event and jobs tables
  6. Import the dump into PG15 DB
  7. Change pg_hba to reopen incoming traffic [R]
  8. Migrate AWX to 24.6.1

[R] : with restart

NB : the dump without the bulky tables is described here

But unfortunately, I have the very same error message popping in the soway-migration job …

awx_24.6.1_dm_error

Stacktrace :

  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/migrations/_dab_rbac.py", line 196, in migrate_to_new_rbac
    object = object_cls.objects.get(pk=role.object_id)  # WORKAROUND, role.content_object does not work in migrations
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
__fake__.JobTemplate.DoesNotExist: JobTemplate matching query does not exist.

I checked in the web container and the target PG is indeed in version 15 :

psql (13.14, server 15.10)
WARNING: psql major version 13, server major version 15.
         Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

awx=>

Has anyone encountered the same problem ? Any advice on this one ?

Thanks

Edit : I will try to start with an empty database, let AWX create the tables and then diff the table structures, to see whether something is missing or if a data type changed.