Updating AWX image on Docker

Bor · October 20, 2023, 7:11pm

Hey all!

I deployed my AWX in a home lab in Docker, following the github instructions. This all worked fine, I picked up the docker-compose and edited it to run it via Portainer and behind nginx reverse proxy for automated management of lets Encrypt certificates.

Today I took a stab at updating AWX, from 23.0.1 to 23.3.0. Been reading the instructions and went ahead as follows.

ran git pull
ran git checkout 23.3.0
Pulled the secrets from tools/docker-compose/_sources/secrets and placed them in the tools/docker-compose/inventory
ran make docker-compose-build

Checked docker images and renamed the new devel image:
docker image tag ghcr.io/ansible/awx_devel:HEAD ghcr.io/ansible/awx_devel:23.3.0

Edited the docker-compose to load the new image:
image: "ghcr.io/ansible/awx_devel:23.3.0"

AWX starts and seems to work fine, however when I go to top right and choose about, it shows the neat cow with 23.0.1:
Ansible AWX 23.0.1.dev0+gc9190ebd8f.d20230829

The new image is really new, new hashes and different size. Perhaps I did not correctly check out git or something?

I’m not sure how to check that and correct it if thats the issue?

Any help please?

Thanks In Advance!

ps. forgot to mention, I tried to (re)make the UI:
docker exec tools_awx make clean-ui ui-devel
Didn’t help either. I think its just building the same old image …

TheRealHaoLiu · October 22, 2023, 4:06pm

When u run “make docker-compose” did u supply a COMPOSE_TAG?

AFAIR compose file is regenerated every time u run it so I think your manual edit of the docker-compose file may have been overridden

Try to set COMPOSE_TAG to your desire image tag

Bor · October 23, 2023, 5:48am

Thank you. I will try your suggestion.

Just to clarify I didn’t edit the compose file in the git branch but copy pasted it to Portainer where I edited it.

Bor · October 25, 2023, 3:12pm

No it doesn’t help. I think this basically does what I did manually, just set the tag. Without it, docker images will show tag HEAD. This just sets the tag to 23.3.0.

TheRealHaoLiu · October 27, 2023, 11:30am

I just realize ur talking about the development environment.

The development environment container is just a container with the python dependencies installed the source code is mounted in.

If I remember correctly the version come from setup_scm_tool and essentially I think it either pull from a VERSION file or the git tag

Can u run ‘make VERSION’ to check the version.

I see that u did a pull and checkout so yah this seems strange

Bor · October 28, 2023, 10:52am

# make VERSION
awx: 23.0.1.dev0+gc9190ebd8f.d20231028

So how I think updating is supposed to work in my case is to just make a new image and build the interface. Then I edit my docker-compose to point to the new image and it should start.

To reiterate the setup:
Followed the steps to clone, build and start awx, all from the terminal in the cloned dir.

I stopped it, copied the docker-compose file contents to Portainer and then edited it. Here it is redacted:

# https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md
---
version: '2.1'
services:
  # Primary AWX Development Container
  awx:
    user: "0"
    image: "ghcr.io/ansible/awx_devel:23.0.0"
    container_name: tools_awx
    hostname: awx
    command: launch_awx.sh
    restart: always
    environment:
      OS: " Operating System: Arch Linux"
      SDB_HOST: 0.0.0.0
      SDB_PORT: 7899
      AWX_GROUP_QUEUES: tower
      MAIN_NODE_TYPE: "${MAIN_NODE_TYPE:-hybrid}"
      RECEPTORCTL_SOCKET: /var/run/awx-receptor/receptor.sock
      CONTROL_PLANE_NODE_COUNT: 1
      EXECUTION_NODE_COUNT: 0
      AWX_LOGGING_MODE: stdout
      DJANGO_SUPERUSER_PASSWORD: whatever
      UWSGI_MOUNT_PATH: /
      RUN_MIGRATIONS: 1
      NETWORK_ACCESS: internal
      VIRTUAL_HOST: awx.internal.domain.bla
      VIRTUAL_PORT: 8013
      VIRTUAL_PROTO: http
      LETSENCRYPT_HOST: awx.internal.domain.bla
      LETSENCRYPT_EMAIL: nginx-proxy-acme@domain.bla
    links:
      - postgres
      - redis
    working_dir: "/awx_devel"
    volumes:
      - "/data/docker-data/awx:/awx_devel"
      - "/data/docker-data/awx/tools/docker-compose/supervisor.conf:/etc/supervisord.conf"
      - "/data/docker-data/awx/tools/docker-compose/_sources/database.py:/etc/tower/conf.d/database.py"
      - "/data/docker-data/awx/tools/docker-compose/_sources/websocket_secret.py:/etc/tower/conf.d/websocket_secret.py"
      - "/data/docker-data/awx/tools/docker-compose/_sources/local_settings.py:/etc/tower/conf.d/local_settings.py"
      - "/data/docker-data/awx/tools/docker-compose/_sources/nginx.conf:/etc/nginx/nginx.conf"
      - "/data/docker-data/awx/tools/docker-compose/_sources/nginx.locations.conf:/etc/nginx/conf.d/nginx.locations.conf"
      - "/data/docker-data/awx/tools/docker-compose/_sources/SECRET_KEY:/etc/tower/SECRET_KEY"
      - "/data/docker-data/awx/tools/docker-compose/_sources/receptor/receptor-awx-1.conf:/etc/receptor/receptor.conf"
      - "/data/docker-data/awx/tools/docker-compose/_sources/receptor/receptor-awx-1.conf.lock:/etc/receptor/receptor.conf.lock"
      # - "/data/docker-data/awx/tools/docker-compose/_sources/certs:/etc/receptor/certs"  # TODO: optionally generate certs
      - "/sys/fs/cgroup:/sys/fs/cgroup"
      - "~/.kube/config:/var/lib/awx/.kube/config"
      - "redis_socket_1:/var/run/redis/:rw"
    privileged: true
    tty: true
    #ports:
      #- "7899-7999:7899-7999"  # sdb-listen
      #- "6899:6899"
      #- "8080:8080"  # unused but mapped for debugging
      #- "8888:8888"  # jupyter notebook
      #- "8013:8013"  # http
      #- "8043:8043"  # https
      #- "2222:2222"  # receptor foo node
      #- "3000:3001"  # used by the UI dev env
    expose: 
      - "8013"
  redis:
    image: redis:latest
    container_name: tools_redis
    restart: always
    volumes:
      - "/data/docker-data/awx/tools/redis/redis.conf:/usr/local/etc/redis/redis.conf:Z"
      - "redis_socket_1:/var/run/redis/:rw"
    entrypoint: ["redis-server"]
    command: ["/usr/local/etc/redis/redis.conf"]
  # A useful container that simply passes through log messages to the console
  # helpful for testing awx/tower logging
  # logstash:
  #   build:
  #     context: ./docker-compose
  #     dockerfile: Dockerfile-logstash
  postgres:
    image: postgres:12
    container_name: tools_postgres
    # additional logging settings for postgres can be found https://www.postgresql.org/docs/current/runtime-config-logging.html
    command: postgres -c log_destination=stderr -c log_min_messages=info -c log_min_duration_statement=1000 -c max_connections=1024
    restart: always
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
      POSTGRES_USER: awx
      POSTGRES_DB: awx
      POSTGRES_PASSWORD: funny
    volumes:
      - "/data/docker-data/awx/tools/awx_db:/var/lib/postgresql/data"

volumes:
  redis_socket_1:
    name: tools_redis_socket_1
    external: true

networks:
  default:
    name: nginx-proxy_default
    external: true

This has been running for a while.
Last week I did the steps in my OP and ran into that issue.

Today, I tried something else. I made a new git clone for 23.3.1, adjusted the inventory file with my current passwords, built that image, started it and built the UI. Stopped it and pointed my portainer docker-compose to the new image. A make VERSION shows the 23.3.1 version.

This is what happens:

So, yeh somewhere Im doing something wrong but I don’t know where

TheRealHaoLiu · October 30, 2023, 3:54pm

can u do a git describe --tags on the one that’s showing the wrong version?

version is essentially based on tag and I wonder if your original clone pulls down the tag?

TheRealHaoLiu · October 30, 2023, 3:55pm

ModuleNotFoundError: No module named 'thycotic'

from your awx logs it seems like the awx_devel image doesn’t have all the needed python modules

Bor · October 30, 2023, 4:26pm

Appreciate your help!

Not at the moment, I am migrating my home server and making some changes. Will take a couple of days.

Yeh, I saw that too, of course. But I don’t think that should be happening right? Is it a problem with 23.3.1 that its not building correctly?

djyasin · November 1, 2023, 6:41pm

Hello @Bor! We just wanted to check in and see if you still need assistance. We are continuing to look into this. Please provide us with any updates you may have along the way!

fosterseth · November 1, 2023, 6:51pm

this was recently fixed here I believe Fix DevOps Secrets Vault credential plugin by andrii-zakurenyi · Pull Request #14340 · ansible/awx · GitHub

should be in next release

Bor · November 1, 2023, 6:55pm

Heya!

Yes, this is still an issue, but as I mentioned earlier I am in the process of migrating my server. Containers are down at the moment.

I’ve run into an issue with the migration (I think a bug in ESXi ) which prevents me from finishing up.

Sounds awesome! It had me pulling my hair haha!

As soon as things are back up I’ll give it another shot!

Thanks for chiming in, both of you.

Bor · November 8, 2023, 4:47pm

I just want to say thanks to the kind folk who tried to help.

I got frustrated and deleted everything. When I finally got a new image to build it wouldn’t load.

I did a new git clone, checked out 23.4.0 and it built fine.

I have my ansible code in gitlab so all I need to do is recreate the inventory, add credentials and secrets. I wasted so much time on this its crazy.

I am now recreating the docker-compose for Portainer. Hopefully I will have less trouble next time I want to update.

djyasin · November 8, 2023, 7:57pm

Hello @Bor,
We are so sorry to hear that you had so much trouble updating! We are happy to hear that you were able to get everything working, but that does sound like a frustrating experience.

Please reach out here if you have any trouble in the future and we will absolutely do our best to try to assist you!