Hello,
when i start the celery worker i get the error message
[2018-06-06 05:44:48,076: ERROR/MainProcess] pidbox command error: PreconditionFailed(406, u"PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'tower_broadcast_all' in vhost 'awx': received 'fanout' but current is 'direct'", (40, 10), 'Exchange.declare') Traceback (most recent call last): File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/pidbox.py", line 105, in dispatch reply = handle(method, kwdict(arguments)) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/pidbox.py", line 123, in handle_call return self.handle(method, arguments) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/pidbox.py", line 120, in handle return self.handlers[method](self.state, **arguments) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/control.py", line 350, in add_consumer routing_key, **options) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 401, in add_task_queue cset.add_queue(q) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/messaging.py", line 419, in add_queue queue.declare() File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/entity.py", line 521, in declare self.exchange.declare(nowait) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/entity.py", line 174, in declare nowait=nowait, passive=passive, File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/channel.py", line 622, in exchange_declare (40, 11), # Channel.exchange_declare_ok File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait return self.dispatch_method(method_sig, args, content) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method return amqp_method(self, args) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/channel.py", line 243, in _close reply_code, reply_text, (class_id, method_id), ChannelError, PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'tower_broadcast_all' in vhost 'awx': received 'fanout' but current is 'direct'
and the following error
[2018-06-06 05:44:48,688: ERROR/MainProcess] Unrecoverable error: PreconditionFailed(406, u'PRECONDITION_FAILED - unknown delivery tag 10', (60, 80), 'Basic.ack') Traceback (most recent call last): File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start self.blueprint.start(self) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start step.start(parent) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start return self.obj.start() File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 280, in start blueprint.start(self) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start step.start(parent) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 884, in start c.loop(*c.loop_args()) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop next(loop) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/async/hub.py", line 340, in create_loop cb(*cbargs) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/transport/base.py", line 164, in on_readable reader(loop) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/kombu/transport/base.py", line 146, in _read drain_events(timeout=0) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/connection.py", line 324, in drain_events return amqp_method(channel, args) File "/var/lib/awx/venv/awx/local/lib/python2.7/site-packages/amqp/channel.py", line 243, in _close reply_code, reply_text, (class_id, method_id), ChannelError, PreconditionFailed: Basic.ack: (406) PRECONDITION_FAILED - unknown delivery tag 10
AWX Version is 1.0.6.x - error message in version 1.0.6.5 and 1.0.6.14
the instnaces in the instance group on AWX switch from available to unabailable every minute. then no more jobs can be started
We have an rabbitmq cluster with 4 nodes.
can anybody help or any idea?
thanks in advance
Simon