AWX migrations fail during install with an empty database using receptor v1.6.0 --> v1.6.6

Installing the 24.6.1 AWX containers using docker-compose fails when using a receptor version v1.6.0 up to v1.6.6 with some similar but not identical error messages.

All installs are done from scratch, with no existing database (which is not defined to be external).

There is no such issue when using the receptor version v1.5.5.

In the flow below, after trying with a specific receptor version and before trying with another one, all docker things are deleted: containers, volumes, networks, images and build-cache.

AWX Installation Workflow:

  1. cloning awx
  2. checking out 24.6.1 with a branch named 24.6.1-v1.6.0
  3. updating some git files to change/remove some pinnings
  4. pulling the awx_devel image tagged with 24.6.1-v1.6.0 from my private hub repository. This image has been built beforehand using receptor v1.6.0 .
  5. setting the git docker-compose inventory with some passwords
  6. running the containers with:
DEV_DOCKER_OWNER=example COMPOSE_TAG=24.6.1-v1.6.0  DEV_DOCKER_TAG_BASE=hub-kvm.example.com RECEPTOR_IMAGE=quay.io/ansible/receptor:v1.6.0 PG_TLS=true make docker-compose COMPOSE_UP_OPTS=--detach

leads to:

 [ -d "/awx_devel/awx.egg-info" ] || python3.11 /awx_devel/tools/scripts/egg_info_dev
 WARNING setuptools_scm._integration.setuptools pyproject.toml does not contain a tool.setuptools_scm section
 running egg_info_dev
 creating awx.egg-info
 writing awx.egg-info/PKG-INFO
 writing dependency_links to awx.egg-info/dependency_links.txt
 writing entry points to awx.egg-info/entry_points.txt
 writing top-level names to awx.egg-info/top_level.txt
 mkdir -p /var/lib/awx/; \
 if [ "/var/lib/awx/venv" ]; then \
 	. /var/lib/awx/venv/awx/bin/activate; \
 fi; \
 python3.11 -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
 
 if [ "/var/lib/awx/venv" ]; then \
 	. /var/lib/awx/venv/awx/bin/activate; \
 fi; \
 awx-manage migrate --noinput
 System check identified some issues:
 
 WARNINGS:
 ?: (staticfiles.W004) The directory '/awx_devel/awx/ui/build/static' in the STATICFILES_DIRS setting does not exist.
 ?: (staticfiles.W004) The directory '/awx_devel/awx/ui_next/build' in the STATICFILES_DIRS setting does not exist.
 Operations to perform:
   Apply all migrations: auth, conf, contenttypes, dab_rbac, dab_resource_registry, main, oauth2_provider, sessions, sites, social_django, sso
 Running migrations:
   Applying contenttypes.0001_initial... OK
...
   Applying main.0192_custom_roles...Traceback (most recent call last):
   File "/usr/local/bin/awx-manage", line 18, in <module>
     load_entry_point('awx', 'console_scripts', 'awx-manage')()
   File "/awx_devel/awx/__init__.py", line 177, in manage
     execute_from_command_line(sys.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
     utility.execute()
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 458, in execute
     output = self.handle(*args, **options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
     res = handle_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
     post_migrate_state = executor.migrate(
                          ^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
     state = self._migrate_all_forwards(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
     state = self.apply_migration(
             ^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
     state = migration.apply(state, schema_editor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
     operation.database_forwards(
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
     self.code(from_state.apps, schema_editor)
   File "/awx_devel/awx/main/migrations/_dab_rbac.py", line 296, in setup_managed_role_definitions
     object_perms = set(Permission.objects.filter(content_type=ct))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 1438, in filter
     return self._filter_or_exclude(False, args, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 1456, in _filter_or_exclude
     clone._filter_or_exclude_inplace(negate, args, kwargs)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 1466, in _filter_or_exclude_inplace
     self._query.add_q(Q(*args, **kwargs))
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 1546, in add_q
     clause, _ = self._add_q(q_object, self.used_aliases)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 1577, in _add_q
     child_clause, needed_inner = self.build_filter(
                                  ^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 1465, in build_filter
     self.check_related_objects(join_info.final_field, value, join_info.opts)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 1276, in check_related_objects
     self.check_query_object_type(value, opts, field)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/sql/query.py", line 1253, in check_query_object_type
     raise ValueError(
 ValueError: Cannot query "ContentType object (12)": Must be "DABContentType" instance.
 make: *** [Makefile:223: migrate] Error 1
 Admin password: dwdcL3XEA7c6D2PxsN4iLCdjir96eDYvm7hVMlfdK1Mv5k3oE2Y7jJAPVTdaRoLj
 Traceback (most recent call last):
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 918, in get_or_create
     return self.get(**kwargs), False
            ^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 639, in get
     raise self.model.DoesNotExist(
 awx.main.models.organization.Organization.DoesNotExist: Organization matching query does not exist.
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/signals/handlers.py", line 31, in update_resource
     resource = Resource.get_resource_for_object(instance)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/models/resource.py", line 130, in get_resource_for_object
     return cls.objects.get(object_id=obj.pk, content_type=ContentType.objects.get_for_model(obj).pk)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 639, in get
     raise self.model.DoesNotExist(
 ansible_base.resource_registry.models.resource.Resource.DoesNotExist: Resource matching query does not exist.
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/usr/local/bin/awx-manage", line 18, in <module>
     load_entry_point('awx', 'console_scripts', 'awx-manage')()
   File "/awx_devel/awx/__init__.py", line 177, in manage
     execute_from_command_line(sys.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
     utility.execute()
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 458, in execute
     output = self.handle(*args, **options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/awx_devel/awx/main/management/commands/create_preload_data.py", line 26, in handle
     o, _ = Organization.objects.get_or_create(name='Default')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 925, in get_or_create
     return self.create(**params), True
            ^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 660, in create
     obj.save(force_insert=True, using=self.db)
   File "/awx_devel/awx/main/models/base.py", line 344, in save
     super(PrimordialModel, self).save(*args, **kwargs)
   File "/awx_devel/awx/main/models/base.py", line 192, in save
     super(CreatedModifiedModel, self).save(*args, **kwargs)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 814, in save
     self.save_base(
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 892, in save_base
     post_save.send(
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
     return [
            ^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
     (receiver, receiver(signal=self, sender=sender, **named))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/signals/handlers.py", line 34, in update_resource
     resource = init_resource_from_object(instance)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/resource_registry/models/resource.py", line 217, in init_resource_from_object
     resource_type = c_type.resource_type
                     ^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 492, in __get__
     raise self.RelatedObjectDoesNotExist(
 django.contrib.contenttypes.models.ContentType.resource_type.RelatedObjectDoesNotExist: ContentType has no resource_type.
 Traceback (most recent call last):
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 918, in get_or_create
     return self.get(**kwargs), False
            ^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 639, in get
     raise self.model.DoesNotExist(
 awx.main.models.execution_environments.ExecutionEnvironment.DoesNotExist: ExecutionEnvironment matching query does not exist.
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/rbac/models/content_type.py", line 77, in get_for_model
     ct = self.get(service=query_service, app_label=opts.app_label, model=opts.model_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 639, in get
     raise self.model.DoesNotExist(
 ansible_base.rbac.models.content_type.DABContentType.DoesNotExist: DABContentType matching query does not exist.
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/usr/local/bin/awx-manage", line 18, in <module>
     load_entry_point('awx', 'console_scripts', 'awx-manage')()
   File "/awx_devel/awx/__init__.py", line 177, in manage
     execute_from_command_line(sys.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
     utility.execute()
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/management/base.py", line 458, in execute
     output = self.handle(*args, **options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/awx_devel/awx/main/management/commands/register_default_execution_environments.py", line 99, in handle
     _this_ee, ee_created = ExecutionEnvironment.objects.get_or_create(name=ee["name"], defaults={'image': ee["image"], 'credential': registry_cred})
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
     return getattr(self.get_queryset(), name)(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 925, in get_or_create
     return self.create(**params), True
            ^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 660, in create
     obj.save(force_insert=True, using=self.db)
   File "/awx_devel/awx/main/models/base.py", line 344, in save
     super(PrimordialModel, self).save(*args, **kwargs)
   File "/awx_devel/awx/main/models/base.py", line 192, in save
     super(CreatedModifiedModel, self).save(*args, **kwargs)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 814, in save
     self.save_base(
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 892, in save_base
     post_save.send(
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
     return [
            ^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
     (receiver, receiver(signal=self, sender=sender, **named))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/rbac/triggers.py", line 223, in rbac_post_save_update_evaluations
     post_save_update_obj_permissions(instance)
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/rbac/triggers.py", line 166, in post_save_update_obj_permissions
     parent_ct = permission_registry.content_type_model.objects.get_for_model(parent_cls)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/rbac/models/content_type.py", line 79, in get_for_model
     raise RuntimeError(
 RuntimeError: Could not find content type for ('shared', 'main', 'organization'), and creating new objects via get_for_model is not allowed for DAB RBAC
 Successfully registered instance awx-1
 (changed: True)
 Successfully added receptor address awx-1:2222
 (changed: True)
 Creating instance group controlplane
 (changed: True)
 Creating instance group default
 (changed: True)
 /usr/local/lib/python3.11/site-packages/supervisor/options.py:469: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
   self.warnings.warn(
 2026-07-15 15:42:33,722 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
 2026-07-15 15:42:33,722 INFO Increased RLIMIT_NOFILE limit to 4096
 2026-07-15 15:42:33,727 INFO RPC interface 'supervisor' initialized
 2026-07-15 15:42:33,728 CRIT Server 'unix_http_server' running without any HTTP authentication checking
 2026-07-15 15:42:33,728 INFO supervisord started with pid 268
 2026-07-15 15:42:34,731 INFO spawned: 'stdout' with pid 269
 2026-07-15 15:42:34,733 INFO spawned: 'rsyslog-4xx-recovery' with pid 270
 2026-07-15 15:42:34,735 INFO spawned: 'awx-dispatcher' with pid 271
 2026-07-15 15:42:34,743 INFO spawned: 'awx-receiver' with pid 272
 2026-07-15 15:42:34,744 INFO spawned: 'awx-uwsgi' with pid 273
 2026-07-15 15:42:34,749 INFO spawned: 'awx-daphne' with pid 274
 2026-07-15 15:42:34,751 INFO spawned: 'awx-nginx' with pid 275
 2026-07-15 15:42:34,758 INFO spawned: 'awx-wsrelay' with pid 277
 2026-07-15 15:42:34,769 INFO spawned: 'awx-rsyslogd' with pid 279
 2026-07-15 15:42:34,782 INFO spawned: 'awx-ws-heartbeat' with pid 287
 2026-07-15 15:42:34,788 INFO spawned: 'awx-rsyslog-configurer' with pid 290
 2026-07-15 15:42:34,796 INFO spawned: 'awx-cache-clear' with pid 293
 2026-07-15 15:42:34,807 INFO spawned: 'awx-autoreload' with pid 294
 2026-07-15 15:42:34,819 INFO spawned: 'awx-receptor' with pid 298
 make[1]: Entering directory '/awx_devel'
 make[1]: Entering directory '/awx_devel'
 make[1]: Entering directory '/awx_devel'
 2026-07-15 15:42:34,828 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
 make[1]: Entering directory '/awx_devel'
 INFO 2026/07/15 15:42:35 Running control service control {"node_id":"awx-1"}
 INFO 2026/07/15 15:42:35 Initialization complete
 2026-07-15 15:42:36,316 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: rsyslog-4xx-recovery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-wsrelay entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-ws-heartbeat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-rsyslog-configurer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-cache-clear entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-autoreload entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:36,316 INFO success: awx-receptor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 nginx -g "daemon off;"
 2026-07-15 15:42:39,411 INFO     [-] awx.main.dispatch Running worker dispatcher listening to queues ['tower_broadcast_all', 'tower_settings_change', 'awx-1']
 2026-07-15 15:42:39,789 INFO waiting for awx-rsyslogd to stop
 2026-07-15 15:42:39,791 INFO stopped: awx-rsyslogd (exit status 0)
 2026-07-15 15:42:39,804 INFO spawned: 'awx-rsyslogd' with pid 539
 2026-07-15 15:42:39,806 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
 2026-07-15 15:42:39,926 INFO     [-] awx.main.commands.run_callback_receiver Callback receiver started with pid=272
 2026-07-15 15:42:40,008 INFO     [-] awx.main.wsrelay AWX is currently migrating, retry in 10s...
 2026-07-15 15:42:40,053 WARNING  [-] awx.main.tasks.system Rejoining the cluster as instance awx-1. Prior last_seen None
 2026-07-15 15:42:40,074 INFO     [-] awx.main.dispatch Dispatcher listener connection established
 2026-07-15 15:42:40,074 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.336769819259644
 2026-07-15 15:42:40,074 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.336272954940796
 2026-07-15 15:42:40,075 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.335871934890747
 2026-07-15 15:42:40,078 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.332593202590942
 [uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
 *** Starting uWSGI 2.0.24 (64bit) on [Wed Jul 15 15:42:40 2026] ***
 compiled with version: 11.5.0 20240719 (Red Hat 11.5.0-14) on 29 January 2026 14:51:27
 os: Linux-7.0.0-27-generic #27-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 18 19:13:49 UTC 2026
 nodename: awx-1
 machine: x86_64
 clock source: unix
 detected number of CPU cores: 4
 current working directory: /awx_devel
 detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
 !!! no internal routing support, rebuild with pcre support !!!
 uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 your memory page size is 4096 bytes
 detected max file descriptor number: 4096
 lock engine: pthread robust mutexes
 thunder lock: disabled (you can enable it with --thunder-lock)
 uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
 uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 Python version: 3.11.13 (main, Jan 16 2026, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-14)]
 2026-07-15 15:42:40,080 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.330516815185547
 2026-07-15 15:42:40,104 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.30647897720337
 2026-07-15 15:42:40,135 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.27546262741089
 *** Python threads support is disabled. You can enable it with --enable-threads ***
 Python main interpreter initialized at 0x7aeaa57897d8
 uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 your server socket listen backlog is limited to 100 connections
 your mercy for graceful operations on workers is 60 seconds
 mapped 609552 bytes (595 KB) for 5 cores
 *** Operational MODE: preforking ***
 uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 *** uWSGI is running in multiple interpreter mode ***
 spawned uWSGI master process (pid: 548)
 spawned uWSGI worker 1 (pid: 558, cores: 1)
 spawned uWSGI worker 2 (pid: 559, cores: 1)
 spawned uWSGI worker 3 (pid: 560, cores: 1)
 spawned uWSGI worker 4 (pid: 561, cores: 1)
 spawned uWSGI worker 5 (pid: 562, cores: 1)
 mounting awx.wsgi:application on /
 mounting awx.wsgi:application on /
 mounting awx.wsgi:application on /
 mounting awx.wsgi:application on /
 mounting awx.wsgi:application on /
 2026-07-15 15:42:40,192 WARNING  [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.218693256378174
 2026-07-15 15:42:40,567 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
 2026-07-15 15:42:40,567 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
 2026-07-15 15:42:40,573 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
 2026-07-15 15:42:40,573 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
 2026-07-15 15:42:40,574 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
 2026-07-15 15:42:40,574 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
 2026-07-15 15:42:40,578 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
 2026-07-15 15:42:40,578 INFO     Listening on TCP address 127.0.0.1:8051
 WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x7aeaa57897d8 pid: 562 (default app)
 WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x7aeaa57897d8 pid: 560 (default app)
 WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x7aeaa57897d8 pid: 559 (default app)
 WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7aeaa57897d8 pid: 558 (default app)
 WSGI app 0 (mountpoint='/') ready in 3 seconds on interpreter 0x7aeaa57897d8 pid: 561 (default app)
 2026-07-15 15:42:50,776 INFO exited: awx-wsrelay (exit status 0; expected)
 2026-07-15 15:42:51,779 INFO spawned: 'awx-wsrelay' with pid 597
 2026-07-15 15:42:52,781 INFO success: awx-wsrelay entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
 2026-07-15 15:42:53,140 INFO     [-] awx.main.wsrelay AWX is currently migrating, retry in 10s...

If I replace the receptor version v1.6.0 in the workflow above by v1.6.1 up to v1.6.6, there are always some fatal errors during the migrations.

What is going on and how to work around/correct these errors?

Additional logs:
awx failed build with receptor v1.6.6

awx successful build with receptor v1.5.5

Hazarding a guess, there may be a version mismatch with DAB. Although I would take this with a grain of salt.

Why are you rebuilding it? the container already exists. If you’re doing it to upgrade components inside, then create a dockerfile with 24.6.1 as the base, make your changes and copy to a FROM scratch image

I don’t understand what you mean by that: “a version mismatch with DAB” between what and what?
AFAIK, the migrations are run from within tools_awx_1 container, right?
Also, they start all with an empty database,.
Finally, I don’t know what DAB is and what package could be updated and where.
So I’m clueless here.

What do you mean by rebuilding the container? With each new receptor version:

  • the database is empty
  • there are no containers, networks, cache…
  • after cloning awx, there is only one awx_devel image which is pulled from the hub repo matching the receptor version
  • tools_awx_1 container is run each time from a different awx_devel image using the command outlined in the first post
  • I’m just following the official AWX install flow, but instead of using the latest ghcr.io/ansible/awx_devel:devel image, I build my own images using each time a different receptor. These images are built beforehand.

Maybe I misused the expression “build the containers” when I meant run the containers from their respective images. I’ll edit my first post accordingly.

you stated in OP that you changed some “pinnings” in some file prior to building. I assumed that you ment that you updated dependencies. If this is incorrect then please inform.

As from where the migrations run, it’s not relevant, as they crash due to an issue somewhere between AWX and its dependent packages. To address this code diving (or more formally, debugging) is the only solution to address.

DAB is djangle-ansible-base it’s one of the deps. The version mismatch is between it and awx.

Rebuilding… awx 24.6.1 is already available on Quay. if your not using this, then you are rebuilding. Also in OP you stated you pull the image from local registry that was built earlier. Additionally you’re using the make docker-compose which rebuilds the image from source. FYI this is for development, not deployment. That’s what awx-operator is for.

Dont know about misused expression. It’s quite possible that we have skills sets that differ. My responses were in the context that you have python/django software dev skills. If you don’t changing dependencies and expecting them to work is troublesome at best. Either way not the end of the world.

It will simplify providing advice, if you could provide the “why” you are trying to run the docker-compose or the “what” your trying to achieve by running the docker-compose.