Hi,
I’m not sure what is going wrong. AWX was working after a clean install on a centos 8 VM, all the containers were happy and then I restarted the containers and now tools_awx_1 is strugglingggg to stay up.
Below is a snippet of the container log before it crashes:
make[1]: Leaving directory ‘/awx_devel’
2022-04-27 05:13:26,719 INFO exited: awx-nginx (exit status 2; not expected)
2022-04-27 05:13:26,719 INFO gave up: awx-nginx entered FATAL state, too many start retries too quickly
2022-04-27 05:13:27,721 INFO spawned: ‘awx-autoreload’ with pid 666
2022-04-27 05:13:27,722 WARN received SIGQUIT indicating exit request
2022-04-27 05:13:27,722 INFO waiting for stdout, superwatcher, awx-dispatcher, awx-receiver, awx-uwsgi, awx-daphne, awx-wsbroadcast, awx-rsyslogd, awx-autoreload, awx-receptor to die
Processing Event: ver:3.0 server:supervisor serial:12 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:66
2022-04-27 05:13:27,723 INFO stopped: awx-receptor (terminated by SIGTERM)
make[1]: Entering directory ‘/awx_devel’
awx-autoreload stdout | make[1]: Entering directory ‘/awx_devel’
awx-autoreload stdout |
Failed to watch /awx_devel; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via /proc/sys/fs/inotify/max_user_watches'. awx-autoreload stderr | Failed to watch /awx_devel; upper limit on inotify watches reached! awx-autoreload stderr | Please increase the amount of inotify watches allowed per user via /proc/sys/fs/inotify/max_user_watches’.
awx-autoreload stderr |
make[1]: Leaving directory ‘/awx_devel’
2022-04-27 05:13:28,658 INFO stopped: awx-autoreload (exit status 0)
awx-autoreload stdout | make[1]: Leaving directory ‘/awx_devel’
awx-autoreload stdout |
2022-04-27 05:13:28,658 ERROR Exception in callback AsyncioSelectorReactor.callLater..run() at /var/lib/awx/venv/awx/lib64/python3.9/site-packages/twisted/internet/asyncioreactor.py:287
handle: <TimerHandle when=49934.896448025 AsyncioSelectorReactor.callLater..run() at /var/lib/awx/venv/awx/lib64/python3.9/site-packages/twisted/internet/asyncioreactor.py:287>
Traceback (most recent call last):
File “/usr/lib64/python3.9/asyncio/events.py”, line 80, in _run
self._context.run(self._callback, *self._args)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/twisted/internet/asyncioreactor.py”, line 290, in run
f(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/twisted/internet/asyncioreactor.py”, line 273, in stop
super().stop()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/twisted/internet/base.py”, line 646, in stop
raise error.ReactorNotRunning(
twisted.internet.error.ReactorNotRunning: Can’t stop reactor that isn’t running.
2022-04-27 05:13:28,659 INFO [-] daphne.server Killed 0 pending application instances
2022-04-27 05:13:28,659 INFO Killed 0 pending application instances
2022-04-27 05:13:28,675 INFO stopped: awx-uwsgi (terminated by SIGKILL)
2022-04-27 05:13:28,675 INFO stopped: awx-rsyslogd (exit status 0)
2022-04-27 05:13:28,674 WARNING [-] awx.main.dispatch received SIGTERM, stopping
2022-04-27 05:13:28,674 WARNING [-] awx.main.commands.run_callback_receiver received SIGTERM, stopping
make[1]: *** [Makefile:210: wsbroadcast] Terminated
2022-04-27 05:13:28,675 INFO stopped: awx-wsbroadcast (terminated by SIGTERM)
2022-04-27 05:13:28,694 WARNING [-] awx.main.tasks.system Normal shutdown signal for instance awx_1, removed self from capacity pool.
2022-04-27 05:13:28,905 INFO stopped: awx-daphne (terminated by SIGTERM)
2022-04-27 05:13:29,034 INFO stopped: awx-receiver (terminated by SIGTERM)
2022-04-27 05:13:29,073 INFO stopped: awx-dispatcher (terminated by SIGTERM)
2022-04-27 05:13:29,073 INFO stopped: superwatcher (terminated by SIGTERM)
2022-04-27 05:13:29,074 INFO stopped: stdout (terminated by SIGTERM)
The rest of the containers are running just fine (receptors, hop, redis, and posgres). I’m not sure what I am missing here. Any help is certainly appreciated. Without tools_awx_1 up, I can’t bring up the UI so I’m kind of dead in the water here.
Thanks in advance!!