Seeking Guidance for Offline AWX Installation and Execution Environment Setup

Get Help
, , , ,
1

Dear Ansible Community,

I hope this message finds you well. I am reaching out to request assistance or guidance on setting up the AWX application and its dependencies in an environment without internet access. My goal is to establish a fully functional execution environment for AWX, ensuring that all its components can operate seamlessly offline.

The challenges I face primarily revolve around the initial setup, configuration, and the management of dependencies required by AWX to run effectively without internet connectivity. I understand that this might involve pre-downloading packages, container images, and dealing with potential updates or dependencies management manually.

I would greatly appreciate any form of help, be it documentation, personal experiences, advice, or even video tutorials that specifically address the following aspects:

  1. Detailed Steps for Offline Installation: A step-by-step guide on how to install AWX and its dependencies without an internet connection.
  2. Managing Dependencies: Strategies for identifying, downloading, and managing the necessary dependencies and container images required for AWX.
  3. Setting Up the Execution Environment: Advice on configuring the execution environment to ensure AWX runs smoothly, including any required settings or modifications to the default setup.
  4. Updates and Maintenance: Tips on handling updates and maintenance tasks for AWX and its dependencies in an offline setting.

I am committed to contributing back to the community by sharing my experience and solutions found throughout this process. Any documentation, personal insights, or resources you could share would be immensely valuable to me and potentially others facing similar challenges.

Thank you very much for your time and support. I look forward to any assistance you can provide and am eager to engage further with the community on this topic.

2 Likes
2

Hi, I have not really fully tested this, but here is a design that would probably be technically feasible

  1. Detailed Steps for Offline Installation: A step-by-step guide on how to install AWX and its dependencies without an internet connection.

The first step of offline installation is to determine the Kubernetes distribution to be used. One of the my favorite distribution, K3s, provides the guide Air-Gap Install, so this is a good choice for you: Air-Gap Install | K3s

As documented in air-gap installation guide for K3s, a container registry should be hosted in the Air Gapped environment. It is required not only for K3s, but also for AWX.

To achieve this, for the minimal solution, running registry on Docker/Podman is one of the simplest way. Technically it’s possible that running it on K3s, but there is a chance the images to be removed from containerd due to garbage collection on kubelet, so at least for container registry, I recommend you to running it outside of K3s.

Also Git repository should be hosted. This is useful option for AWX, to store your playbooks. Running Gitea on K3s/Docker/Podman is good choice for the minimal solution.

  1. Managing Dependencies: Strategies for identifying, downloading, and managing the necessary dependencies and container images required for AWX.

The list of required images can be obtained from the AWX Operator source code, or by creating AWX on K3s in a connected environment and obtaining the downloaded images.

Either way, the required images should be pulled and exported, carried into air-gap environment, imported as different tags, and pushed to the private container registry.

Using AWX Opeartor and Kustomize, you should be able to specify all the images you use to pull from the private registry.

  1. Setting Up the Execution Environment: Advice on configuring the execution environment to ensure AWX runs smoothly, including any required settings or modifications to the default setup.

It is recommended that building EE images by Ansible Builder in a connected environment. Various commands that run during the build, such as dnf, pip, and ansible-galaxy, require internet access, and completing them offline is a extremely rough road.

The images you built should be exported, imported, and pushed to a private registry.

  1. Updates and Maintenance: Tips on handling updates and maintenance tasks for AWX and its dependencies in an offline setting.

Basically, upgrading AWX is completed by renewing images; push new images to private registry and update AWX to use newer images.

Hope this helps.

2 Likes
3

Thank you very much for your insightful response regarding the setup of the execution environment for AWX in an offline setting. Your advice on using Ansible Builder to create Execution Environment (EE) images in a connected environment and then transferring those to a private registry is incredibly helpful.

I have a follow-up question regarding the consolidation of dependencies. Is there a method or approach to encapsulate all necessary dependencies within a single image? Specifically, I’m looking for a way to ensure that all required packages, libraries, and other dependencies are included in one comprehensive image that can be easily transferred and utilized within an offline environment.

Furthermore, I would appreciate any guidance on:

  • Identifying and gathering all dependencies needed for AWX to run smoothly in a local execution environment without internet access. This includes:
    • A checklist or method for ensuring all required dependencies are accounted for before moving to the offline setup.
    • Best practices for managing and updating these dependencies in an isolated environment.
    • Any tools or scripts that could assist in automating the process of collecting and packaging these dependencies.

Your expertise and suggestions on how to navigate this aspect would be invaluable, making the transition to an offline environment as seamless as possible.

Thank you once again for your time and assistance. I look forward to your advice.

4

Hi,

You mean something like a single big tar.gz file that includes everything, K3s, AWX Operator, AWX, etc.?
If so, no, there is no helpful resources at this moment.

I don’t think such checklist nor tools exist. Technically, I suppose it would be possible to create a script to help with that, but I think it would be very expensive to maintain it over the long term to keep up with all the dependency changes that the tool would be subject to.

1 Like
5

Hi,

Thank you for your prompt and insightful response. Understanding that there isn’t a pre-packaged solution or an extensive set of resources for consolidating AWX and its dependencies into a single deliverable, it seems a more tailored approach is necessary.

Considering the complexity and the need for a solution that encompasses K3s, AWX Operator, AWX, and potentially other components, I’m contemplating the feasibility of manually creating a comprehensive package. This would involve:

  1. Manually Identifying Dependencies: Starting with a connected setup to list all packages, libraries, and dependencies used by AWX and its components.
  2. Scripting the Collection Process: Developing a script to download and package these components into a tar.gz file. While I acknowledge the maintenance challenges mentioned, this might be an initial step to facilitate the offline migration.
  3. Migration Strategy: Transferring the tar.gz file to the offline environment, extracting it, and manually configuring the components to ensure they operate cohesively.

I understand the maintenance of such a script could become cumbersome. However, I wonder if you have insights or advice on:

  • The process of manually identifying all necessary components and dependencies.
  • Recommendations for scripting the collection process in a way that could minimize maintenance efforts or make the script more adaptable to changes.
  • Best practices for deploying and configuring these components in an offline environment after migration.

Any guidance, resources, or examples of similar endeavors would be highly appreciated. Additionally, if you foresee any specific challenges or have suggestions on alternative approaches, I’m all ears.

Thank you once again for your valuable input. Looking forward to your thoughts.

6

In the first place, I think it is not a good approach to start designing the “most ideal” deliverable without any hands-on experience.

Before you start designing about the details of your architecture, why don’t you actually build an AWX that works in an offline environment with the following goal in mind, even if you have to do everything manually?

  • Working kubernetes cluster, git repository, and container registry
  • Working AWX Operator and AWX
  • Customized EE image on container registry
  • A project that contains a playbook on Git repository
  • Job templates that complete successfully using custom EE on container registry and project on git repository
  • All data must be persistent (not volatile)

We are occasionally consulted about offline installations, so even if the procedure is designed as doing everything manually, it may be helpful for someone else.

By actually achieving offline installation, you will understand how to find out the required components and their dependencies, the room for automation/scripting, and the difficulty of what you are trying to do.

Or, if you have already worked on it, you may be able to discuss more feasible solutions by providing specifics of what you have actually tried and the issues involved.

2 Likes
7

I wanted to share an update on my progress with setting up AWX in an offline environment, following the practical approach you recommended. I’m pleased to report that I have successfully installed AWX using Docker and have it running without an internet connection. Additionally, I’ve configured playbooks and templates, aligning with the steps outlined for a foundational offline AWX deployment.

However, I’ve encountered a challenge at a crucial step in the process. The issue arises when attempting to run any job, as it requires an image of the execution environment. I’ve managed to configure a Docker image of the execution environment locally and attempted to push it to a local Docker registry. This step seems to require internet access at least once to bring that image into the local environment, which contradicts the goal of a completely offline setup.

I’m currently stuck on how to proceed with transferring the execution environment image to the local Docker registry without necessitating an initial internet connection. Here are the specific details of my challenge:

  • I have the Docker image of the execution environment prepared locally.
  • Pushing this image to the local Docker registry seems to require internet access, which isn’t feasible in the strict offline environment I’m working within.

Given your expertise and the advice you’ve provided thus far, do you have any suggestions or workarounds for this particular challenge? Any insights on how to successfully transfer the execution environment image to the local Docker registry in a truly offline setup would be greatly appreciated.

Thank you once again for your guidance and support as I navigate through this setup process. I look forward to any advice you can offer on overcoming this hurdle.

8

Is this mean your AWX is running without any kubernetes distribution (e.g. minikube, k3d,
etc.)?

Can you be more specific? I do not understand the situation where Internet access is required to push a container image to a private registry. Could it be that you simply did not include the name of the private registry in the tag for that image? What specific command are you trying?

9

My AWX instance is currently running without any Kubernetes distribution, such as Minikube or K3d.

To achieve this, I have followed the documentation provided in the official AWX GitHub repository, specifically the instructions outlined for Docker Compose deployment. Here is the direct link to the documentation I referred to: AWX Docker Compose Setup.

This approach has allowed me to run AWX using Docker directly, bypassing the need for a Kubernetes environment. I believed this method would simplify the initial setup process and suit the constraints of my offline environment.

I hope this clarifies the methodology behind my current AWX deployment. If there are any further questions or if you have advice on how to address the challenge with the execution environment image in this setup, I would greatly appreciate your insights.

10

I apologize for any confusion caused by my previous message. Let me clarify the situation regarding the execution environment image for AWX.

After setting up the AWX application using the Docker-compose method from the provided documentation, I proceeded to the next steps necessary for running job schedules. This requires the use of an execution environment, which I have successfully pushed to a local Docker registry on the same virtual machine (VM) where AWX is running.

The issue arises when attempting to execute a job within AWX. Despite the execution environment image being available locally in the private registry on the same VM, AWX appears to be attempting to access the image over the internet. Given that this VM is configured without internet access, the job fails to execute because it cannot retrieve the execution environment image.

To summarize, the specific challenge I’m facing is:

  • The execution environment image is stored locally in a private Docker registry on the same VM as AWX.
  • When a job is run, AWX seems to seek the execution environment image over the internet instead of using the local version, despite being configured to use the local Docker registry.
  • This VM does not have internet access, leading to job execution failure.

I hope this clarifies the issue I’m encountering. If you have any insights or suggestions on how to resolve this, ensuring that AWX uses the locally stored execution environment image for job execution, I would greatly appreciate your advice.

11

Note, that method is for only recommended for development or testing, not for production use.

AWX can also alternatively be installed and run in Docker, but this install path is only recommended for development/test-oriented deployments, and has no official published release.
awx/INSTALL.md at devel · ansible/awx · GitHub

How did you configure your Execution Environment and Job Templates to achieve “being configured to use the local Docker registry”?

12

Here’s how I managed the configuration:

  1. Pulling the Image: Initially, I pulled the desired Execution Environment image from quay.io using the standard Docker pull command.

  2. Pushing to Local Docker Registry: After pulling the image from quay.io, I tagged it appropriately and pushed it into my local Docker registry. This process involved ensuring the image was tagged with the local registry’s address to be recognized as a local resource.

  3. Configuring in AWX UI: With the Execution Environment image successfully pushed to the local Docker registry, I proceeded to configure this within the AWX UI. Under the Execution Environments section, I added a new environment and specified the path to the image in the local Docker registry.

This method was intended to ensure that when a job is executed, AWX would pull the Execution Environment image from the local Docker registry instead of reaching out to the internet. Despite these configurations, the challenge arises during job execution, where AWX seems to overlook the local registry configuration and attempts to access the image over the internet.

I hope this clarifies the steps I took to configure the Execution Environment to use the local Docker registry. If there are any additional settings or adjustments required to rectify this issue, I would be keen to learn about them.

13

You have to specify Execution Environment to which you want to be used at the one of the Organization, Project, Workflow, and Job Template:

image
image1906Ă—1073 113 KB

14

@DaneialSingh , about EE:

1 Like
15

Despite these configurations, I am encountering an issue during the job template execution phase. Although the Execution Environment is set up to reference the local Docker registry, and my VM does not have internet access, it appears that AWX still attempts to access the registry over the internet. This results in a failure to find and utilize the Execution Environment image for job execution since the system cannot reach out to the internet.

To summarize, the steps to configure the Execution Environment to use the local Docker registry have been followed meticulously, and the environment is correctly specified within AWX at all necessary levels. However, the challenge remains that AWX seems to overlook this local setup and instead looks for the registry online during job executions.

Is there a specific configuration or setting within AWX that I might be missing, which ensures that AWX recognizes and prioritizes the local Docker registry for Execution Environment images, especially in an offline setup?

I appreciate your continued support and any further insights you can provide on resolving this issue.

DefaultOptionLocalhost
DefaultOptionLocalhost1366Ă—647 44.6 KB

@Andersson007

16

@DaneialSingh looking at the screenshot, it doesn’t show attempts to connect to registries on the Internet. Can you list the available tags of the images on your AWX instance from command line using the image URL to test the setup as explained in the EE local registry guide?

17

@DaneialSingh
I wanted to chime in with some additional thoughts from my own experience. Much like you, I wanted to have AWX available in an offline environment. While I had quite a bit of Docker experience I had NO Kubernetes experience. Similar to you I tried getting AWX to run in docker. I soon realized that I would be further ahead if I tried to setup an online, internet connected environment that would mirror my desired offline environment. I found out I had a lot to learn about just getting Kubernetes to work + the complexities added by getting AWX to run. I have spent about the last three months just getting comfortable with Kubernetes as time has allowed. I got a lot of assistance from two co-workers who really mentored me with Kubernetes in general. I would strongly suggest trying this type of approach. I first had to learn how to make things run in my internet connected setup. Then I could transfer that knowledge to an offline install. By running a connected setup I could see which images were getting pulled in. I also spent a lot of time troubleshooting issues with storage, which will vary widely depending on your chosen platform. Storage will be crucial for data persistence.

Even now I am trying to add features to my setup such as certificates and ldap authentication. I know I will be reaching back to the community for clarification and guidance though I am very familiar and comfortable with the principles for those areas and have setup other products with certs and ldap. The questions in my mind are about how AWX does things.

I wish you much success in your quest. Sometimes you must slow down to speed up. While I was anxious to just have a working environment, I know I am much further ahead for having learned all the little things about Kubernetes along the way. I am now MUCH more comfortable using it and I still consider myself a newb in that area. Even if I have spent the last three months trying to learn it. Good luck!