Hi,
I recently did a clean installation of AWX 10.0.0 and getting my UI screen stuck at migration screen. I can see the following warnings and errors from redis, task and web containers.
Redis
1:C 02 Apr 2020 16:26:53.487 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 02 Apr 2020 16:26:53.487 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 02 Apr 2020 16:26:53.487 # Configuration loaded 1:M 02 Apr 2020 16:26:53.488 * Running mode=standalone, port=0. 1:M 02 Apr 2020 16:26:53.489 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 1:M 02 Apr 2020 16:26:53.489 # Server initialized 1:M 02 Apr 2020 16:26:53.489 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 1:M 02 Apr 2020 16:26:53.489 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 1:M 02 Apr 2020 16:26:53.489 * The server is now ready to accept connections at /var/run/redis/redis.sock 1:signal-handler (1585844814) Received SIGTERM scheduling shutdown... 1:M 02 Apr 2020 16:26:54.894 # User requested shutdown... 1:M 02 Apr 2020 16:26:54.894 * Removing the unix socket file. 1:M 02 Apr 2020 16:26:54.895 # Redis is now ready to exit, bye bye... 1:C 02 Apr 2020 16:26:55.976 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 02 Apr 2020 16:26:55.976 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 02 Apr 2020 16:26:55.976 # Configuration loaded 1:M 02 Apr 2020 16:26:55.977 * Running mode=standalone, port=0. 1:M 02 Apr 2020 16:26:55.977 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 1:M 02 Apr 2020 16:26:55.977 # Server initialized 1:M 02 Apr 2020 16:26:55.977 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 1:M 02 Apr 2020 16:26:55.977 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 1:M 02 Apr 2020 16:26:55.978 * The server is now ready to accept connections at /var/run/redis/redis.sock
AWX_Task
`
2020-04-03 08:24:57,685 ERROR awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 76, in _ctit_db_wrapper
yield
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 404, in getattr
value = self._get_local(name)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 347, in _get_local
setting = Setting.objects.filter(key=name, user__isnull=True).order_by(‘pk’).first()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 653, in first
for obj in (self if self.ordered else self.order_by(‘pk’))[:1]:
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1140, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
2020-04-03 08:24:57,687 ERROR awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 76, in _ctit_db_wrapper
yield
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 404, in getattr
value = self._get_local(name)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 347, in _get_local
setting = Setting.objects.filter(key=name, user__isnull=True).order_by(‘pk’).first()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 653, in first
for obj in (self if self.ordered else self.order_by(‘pk’))[:1]:
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1140, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
2020-04-03 08:24:58,039 ERROR awx.conf.settings Database settings are not available, using defaults.
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 76, in _ctit_db_wrapper
yield
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 404, in getattr
value = self._get_local(name)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py”, line 347, in _get_local
setting = Setting.objects.filter(key=name, user__isnull=True).order_by(‘pk’).first()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 653, in first
for obj in (self if self.ordered else self.order_by(‘pk’))[:1]:
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 274, in iter
self._fetch_all()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 1242, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 55, in iter
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1140, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation “conf_setting” does not exist
LINE 1: …f_setting".“value”, “conf_setting”.“user_id” FROM "conf_sett…
^
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation “main_instance” does not exist
LINE 1: SELECT (1) AS “a” FROM “main_instance” WHERE “main_instance”…
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py”, line 152, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py”, line 56, in handle
reaper.reap()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/reaper.py”, line 38, in reap
(changed, me) = Instance.objects.get_or_register()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py”, line 144, in get_or_register
return (False, self.me())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/managers.py”, line 114, in me
if node.exists():
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/query.py”, line 766, in exists
return self.query.has_results(using=self.db)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/query.py”, line 522, in has_results
return compiler.has_results()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1110, in has_results
return bool(self.execute_sql(SINGLE))
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/models/sql/compiler.py”, line 1140, in execute_sql
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/utils.py”, line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation “main_instance” does not exist
LINE 1: SELECT (1) AS “a” FROM “main_instance” WHERE “main_instance”…
^
2020-04-03 08:24:58,499 INFO exited: dispatcher (exit status 1; not expected)
2020-04-03 08:24:59,504 INFO spawned: ‘dispatcher’ with pid 23566
2020-04-03 08:25:00,506 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
`
can anyone please suggest a fix?
Thanks.