Hello guys.
I’ve setup a kubernetes cluster, and I’m trying to deploy the awx project pods.
Everything seems to work, but…then, accessing the external endpoint I set for the service, the server displays the AWX migrating page and hangs there, forever…
I checked the awx_web container errors and this is what displays:
“http://10.10.150.10/migrations_notran/” “Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0” “-”
2018-02-03 16:41:35,870 ERROR Error trying to receive messages: ‘thread._local’ object has no attribute ‘consumer’
2018-02-03 16:41:40,871 ERROR Error trying to receive messages: ‘thread._local’ object has no attribute ‘consumer’
2018/02/03 16:41:43 [error] 41#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.27.74, server: _, request: “GET / HTTP/1.1”, upstream: “uwsgi://[::1]:8050”, host: “10.10.150.10”, referrer:
The awx_celery container:
[2018-02-03 16:47:10,032: DEBUG/Beat] beat: Waking up in 10.00 seconds.
[2018-02-03 16:47:18,664: DEBUG/MainProcess] heartbeat_tick : for connection 9ded4952d28545499d38a32dfa0af6ef
[2018-02-03 16:47:18,664: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 36/1048, now - 36/1078, monotonic - 578537.802219, last_heartbeat_sent - 578497.798695, heartbeat int. - 60 for connection 9ded4952d28545499d38a32dfa0af6ef
[2018-02-03 16:47:20,039: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-03 16:47:20,041: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->07587517-4f18-4a02-8199-a8e0d20235b4
[2018-02-03 16:47:20,041: DEBUG/Beat] beat: Waking up in 19.99 seconds.
[2018-02-03 16:47:38,665: DEBUG/MainProcess] heartbeat_tick : for connection 9ded4952d28545499d38a32dfa0af6ef
[2018-02-03 16:47:38,665: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 36/1078, now - 36/1108, monotonic - 578557.802595, last_heartbeat_sent - 578497.798695, heartbeat int. - 60 for connection 9ded4952d28545499d38a32dfa0af6ef
[2018-02-03 16:47:38,665: DEBUG/MainProcess] heartbeat_tick: sending heartbeat for connection 9ded4952d28545499d38a32dfa0af6ef
[2018-02-03 16:47:40,049: INFO/Beat] Scheduler: Sending due task task_manager (awx.main.scheduler.tasks.run_task_manager)
[2018-02-03 16:47:40,051: DEBUG/Beat] awx.main.scheduler.tasks.run_task_manager sent. id->7b2682f3-e77d-4479-b3cb-568e23ce51a9
[2018-02-03 16:47:40,051: INFO/Beat] Scheduler: Sending due task tower_scheduler (awx.main.tasks.awx_periodic_scheduler)
[2018-02-03 16:47:40,053: DEBUG/Beat] awx.main.tasks.awx_periodic_scheduler sent. id->c019998e-1041-4b0e-a6ea-c265b1553617
[2018-02-03 16:47:40,053: INFO/Beat] Scheduler: Sending due task cluster_heartbeat (awx.main.tasks.cluster_node_heartbeat)
[2018-02-03 16:47:40,055: DEBUG/Beat] awx.main.tasks.cluster_node_heartbeat sent. id->3403105d-2428-424b-bf2c-16cada6bff2a
[2018-02-03 16:47:40,055: DEBUG/Beat] beat: Waking up in 19.98 seconds.
awx_rabbit:
=INFO REPORT==== 3-Feb-2018::16:35:59 ===
connection <0.823.0> (127.0.0.1:44678 -> 127.0.0.1:5672): user ‘awx’ authenticated and granted access to vhost ‘awx’
This kinda happens a lot, the migration being stuck, I’d like to know how to resolve that issue.
It usually happens to us, in our current docker installed setup, whenever we want to upgrade to certain releases, it’s the same.
Thanks for your help.
-Cesar