Upgrade from 2.1.1 to latest 2.11.0 AWX-operator

mimianddaniel · February 13, 2024, 6:41pm

Have external postgres setup using 2.1.1 AWX-operator, went through the changes in Awx-operator and awx change logs but didn’t see any mention of schema changes etc.

Noticed that there are some pending issues related to migrations where the migration is hanging:

github.com/ansible/awx-operator

Error when upgrading awx-operator from 2.1.0 to 2.2.0

opened 12:57PM - 21 Dec 23 UTC

racheldaguenethahn

needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code… of conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx-operator/issues) for duplicates. - [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response. ### Bug Summary Hi, I'm unable to upgrade AWX instance due to an issue on the migration of the managed postgresql DB. Something is wrong into the main_projectupdateevent table. Does someone know how to solve this? Table entries to delete? Logs from the awx-manager container of awx-operator 2.2.0 pod: ``` TASK [Migrate the database if the K8s resources were updated.] ******************************** fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last): File \"/usr/bin/awx-manage\", line 8, in <module> sys.exit(manage()) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage execute_from_command_line(sys.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line utility.execute() File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv self.execute(*args, **cmd_options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute output = self.handle(*args, **options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper res = handle_func(*args, **kwargs) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle post_migrate_state = executor.migrate( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate state = self._migrate_all_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards state = self.apply_migration( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration state = migration.apply(state, schema_editor) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply operation.database_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/awx-manage\", line 8, in <module>", " sys.exit(manage())", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage", " execute_from_command_line(sys.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", " utility.execute()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute", " self.fetch_command(subcommand).run_from_argv(self.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv", " self.execute(*args, **cmd_options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", " output = self.handle(*args, **options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", " post_migrate_state = executor.migrate(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", " state = self._migrate_all_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", " operation.database_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards", " raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform: Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations: Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", " Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", " Applying main.0184_django_indexes..."]} ``` Thank in advance Rachel ### AWX Operator version 2.1.0 ### AWX version 22.2.0 ### Kubernetes platform openshift ### Kubernetes/Platform version 4.12.37 ### Modifications no ### Steps to reproduce helm pull awx-operator/awx-operator --version 2.2.0 tar -zxf awx-operator-2.2.0.tgz kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxbackups.awx.ansible.com.yaml kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxrestores.awx.ansible.com.yaml kubectl apply -f awx-operator ./awx-operator/crds/customresourcedefinition-awxs.awx.ansible.com.yaml helm upgrade --install awx-operator ./awx-operator ### Expected results awx-operator is upgraded successfully AWX instance is upgraded successfully ### Actual results awx-operator is upgraded AWX instance is waiting for database migration The following errors appears into awx-operator/awx-manager container ` TASK [Migrate the database if the K8s resources were updated.] ******************************** fatal: [localhost]: FAILED! => {"changed": true, "rc": 1, "return_code": 1, "stderr": "Traceback (most recent call last): File \"/usr/bin/awx-manage\", line 8, in <module> sys.exit(manage()) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage execute_from_command_line(sys.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line utility.execute() File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv self.execute(*args, **cmd_options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute output = self.handle(*args, **options) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper res = handle_func(*args, **kwargs) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle post_migrate_state = executor.migrate( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate state = self._migrate_all_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards state = self.apply_migration( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration state = migration.apply(state, schema_editor) File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply operation.database_forwards( File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards raise ValueError(\nValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid).\n", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/awx-manage\", line 8, in <module>", " sys.exit(manage())", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 200, in manage", " execute_from_command_line(sys.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", " utility.execute()", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py\", line 436, in execute", " self.fetch_command(subcommand).run_from_argv(self.argv)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 412, in run_from_argv", " self.execute(*args, **cmd_options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 458, in execute", " output = self.handle(*args, **options)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py\", line 106, in wrapper", " res = handle_func(*args, **kwargs)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py\", line 356, in handle", " post_migrate_state = executor.migrate(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 135, in migrate", " state = self._migrate_all_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 167, in _migrate_all_forwards", " state = self.apply_migration(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py\", line 252, in apply_migration", " state = migration.apply(state, schema_editor)", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py\", line 132, in apply", " operation.database_forwards(", " File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py\", line 977, in database_forwards", " raise ValueError(", "ValueError: Found wrong number (0) of indexes for main_projectupdateevent(project_update_id, job_created, uuid)."], "stdout": "Operations to perform: Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit\nRunning migrations: Applying main.0184_django_indexes...", "stdout_lines": ["Operations to perform:", " Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit", "Running migrations:", " Applying main.0184_django_indexes..."]} ` ### Additional information _No response_ ### Operator Logs _No response_

I will be following the migration steps:
awx-operator/docs/migration/migration.md at devel · ansible/awx-operator · GitHub

Any insight into or experience upgrading/migration would be very much appreciated.
Obviously, I will be performing the work with the DB backup + K8 objects just in case.

Thanks