Upgrade AWX from 7.0.0 to 11.2.0

Archives AWX Project
1

Hi I am trying to upgrade current awx(web & task) 7.0.0 to 11.2.0 . However after the upgrade awx web UI is not accessible. I learned that, starting from v9.0.3 , rabbitmq has been replaced with redis… I tried to replace rabbitmq with redis in the docker stack. However awx task is throwing following exceptions. “File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/connection.py”, line 557, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or director” can you please suggest what is the cause of the error. can you please suggest me setting that i need to include in the stack/compose file to solve the error. we are using docker swam and portainer. Let me know if you need any other details from my end. Regards Prabhu

2

Hi Prabhu,

Looks like the socket file is not created. What happens if you manually create /var/run/redis/redis.sock and restart the service?

Regards,
Ompragash

3

Hi Ompragsh

Thanks for the reply.

redis server is running as container an accepting the connections on defined unix socket , details below.

nf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
1:M 27 May 2020 10:25:51.414 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 27 May 2020 10:25:51.415 * Ready to accept connections
1:M 27 May 2020 10:25:51.415 * The server is now ready to accept connections at /var/run/redis/redis.sock

However awx task container still throwing the below exceptions.

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/worker/base.py”, line 122, in run
res = queue.blpop(self.queues)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/client.py”, line 1865, in blpop
return self.execute_command(‘BLPOP’, *keys)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/client.py”, line 875, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/connection.py”, line 1185, in get_connection
connection.connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/connection.py”, line 557, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 2 connecting to unix socket: /var/run/redis/redis.sock. No such file or directory.
2020-05-27 10:26:41,201 ERROR awx.main.commands.run_callback_receiver encountered an error communicating with redis
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/connection.py”, line 552, in connect
sock = self._connect()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/redis/connection.py”, line 905, in _connect
sock.connect(self.path)
FileNotFoundError: [Errno 2] No such file or directory

Let me know if need any other details.

Any help will be greatly appreciated.

Regards
Prabhu

4

I don’t know how’s your setup, but you need to mount the redis socket as a volume bind mount on the web and task containers. If you have a clustered solution you’ll need to have a redis container deployed on each node.

5

Hi Cesar Sanchez

Thank your reply…

can you please advise how to set up the redis socket as voulme bind mount on awx web and task containers? any example would be really helpful.

we are using docker swam with docker stack, I have attached snippet of the stack file .

Please let me know if you need any other details from my end.

Regards
Prabhu

(attachments)

stackfile.txt (2.88 KB)

6

Hey,

You should take a look at the docker-compose.yml file (installer/roles/local_docker/templates/docker-compose.yml.j2) and see how the volumes are mounted, and do the same for your setup:

`

volumes:

  • supervisor-socket:/var/run/supervisor
  • rsyslog-socket:/var/run/awx-rsyslog/
  • rsyslog-config:/var/lib/awx/rsyslog/
  • “{{ docker_compose_dir }}/SECRET_KEY:/etc/tower/SECRET_KEY”
  • “{{ docker_compose_dir }}/environment.sh:/etc/tower/conf.d/environment.sh”
  • “{{ docker_compose_dir }}/credentials.py:/etc/tower/conf.d/credentials.py”
  • “{{ docker_compose_dir }}/nginx.conf:/etc/nginx/nginx.conf:ro”
  • “{{ docker_compose_dir }}/redis_socket:/var/run/redis/:rw”
  • “{{ docker_compose_dir }}/memcached_socket:/var/run/memcached/:rw”

`

7

Hi Cesar Sanchez

Greetings…

AWX version : 11.0.0

As mounting the volumes as suggested , I was able to start up the awx task and web containers.

However AWX web UI is not reachable. Its says bad gateway.

/favicon.ico:1 Failed to load resource: the server responded with a status of 502 (Bad Gateway)

Failed to load resource: the server responded with a status of 502 (Bad Gateway)

I am attaching the logs from awx web and task startup.

Any pointers will be really helpful

Regards
Prabhu

(attachments)

awx_web.txt (75.9 KB)
awx_task.txt (45.7 KB)