minikube-based fresh deployment stuck on "AWX is currently upgrading"

Archives AWX Project
, ,
1

Hello everyone.

I’ve recently started my efforts of learning Ansible and AWX for a large project, and in the process, I am trying to deploy a development infrastructure.

I came across the two variations of the installation from https://github.com/ansible/awx/blob/devel/INSTALL.md : (legacy, for v 17, and Operator-based).

The one I am discussing here is the Operator-based, and my doubts about it.

First, about my environment.

I have a physical server running Ubuntu server (headless), version 20.04.
In this server I run LXD for containers, and KVM for more traditional VMs.

For my AWX deployments I chose to use a “traditional” KVM virtual machine, with Ubuntu 20.04 Desktop edition (more about that later).
The VM is configured with 9 GB of ram, 4 processors and one disk.
Minimal distribution
SSHd installed
Static IP configured
non-root user properly configured (default on Desktop installs)

It will become clear, from my comments below, that my level of knowledge of docker and kubernetes is “novice”. Still, I am very comfortable with regular Linux.

With that out of the way, I followed the AWX install instructions to the letter, or, at least, as far as I understand “to the letter is”. Here is a summary:

Install Docker (and verify that it works)
install minikube and go as far as “minikube start”. From there, go back to the AWX instructions and execute

minikube start --cpus=4 --memory=8g --addons=ingress

instead.
Deploy AWX operator (and verify it is alive)
Deploy AWX (and wait for a few minutes)
Generate the admin password:

minikube kubectl -- get secret awx-admin-password -o jsonpath='{.data.password}' | base64 --decode

This is where things start to go south:

Access AWX

  • There is no public/external IP assigned to the container(s); This is why I used the Ubuntu Desktop: I can launch a browser on the VM and see what is going on, using the local IP.

AWX’s GUI reports that AWX is currently upgrading, and stays there forever.

And that’s the "end’ of it. I’ve tried this deployment several times, in the hope that I made some mistake, or that some updated image was deployed, that would fix it, but the result was always the same.

I found several references to this issue online - including one in Japanese - but limited solutions or details about it.

One potential solution I found was this reference (summarized):

Connect to the awx-web container (which one is the web container ?) with

kubectl exec -it [pod] (-c [container]) (-n [namespace]) bin/bash

Well, minikube does not like that syntax, complaining of the “-it” not being a valid option.

Run these commands in the “web” container ( ref: https://github.com/ansible/awx/issues/6539):

awx-manage migrate --noinput
awx-manage createsuperuser --username admin

(As a side note, I do realize that these commands - potential workarounds- are from one year ago, and may not work, or may need a different syntax)

With that said, here are the objective questions:

  • Is the “upgrading” issue something known, and could you point me to a solution ?

Other questions that would be a nice-to-have ( answers for), since they are relevant to having a working, persistent development environment, along with the reasons why they are relevant:

  • How can I have an external IP for my AWX deployment ?

  • Access to the GUI from other machines in the network

  • Potentially SSH access, to populate playbooks in the projects folders

  • I suspect we need to use some variation of the LoadBalancer, like in

  • kubectl expose deployment hello-node — type=“LoadBalancer”

  • kubectl get service

  • How do I connect to the awx-web container (console) ?

  • Populate the projects folder

  • Potential troubleshooting of/ solution to the “upgrading” issue

  • The Operator/Minikube deployment creates a pod with several containers inside; We need to find out which is the awx pod with:

  • minikube kubectl get pods (get your awx-* pod. i.e.: awx-b5f6cf4d4-nfcjr)

  • Find out which is the awx-web container with:

  • minikube kubectl describe pods awx-b5f6cf4d4-nfcjr (look for awx-web)- Connect to the container with command ??? (need help here)

I admit that, upon some more digging, the method above may be a bit outdated; There are references to /var/lib/awx/projects on awx-projects, so, here are some possible replacement questions:

  • How to populate the projects folder ?

  • What is the architecture of the pods/containers ?

  • How do I connect to a specific container ?
    Not related to the “upgrading” issue, but relevant to having a stable and predictable development environment:

  • How to make sure all pods/containers will be running after a (VM) reboot ?

  • How to restart all PODs/Containers ?

  • How to assign a public IP to the awx web server ?

  • Should minikube be started with the --driver=docker option ? (minikube start --cpus=4 --memory=8g --addons=ingress --driver=docker)

I realize that there are several questions here, and appreciate the time it may take to answer them. As I understand, capturing those answers is fundamental, especially to people taking their first steps in the AWX world.

What I would like to propose is, if you find this all relevant: I could gather the answers and compile them, contributing to updating the current documentation.

Thank you,

Carlos