AWX modernization: Ansible Jewel

[thedoubl3j (Jake)]

Hey folks, Jake here again,

We’ve got another update on the AWX modernization that we’d like to share with the Ansible community. As we’ve mentioned from the first announcement here in the forum, our goal has been to refactor AWX into a pluggable, service-based architecture.

Today I’m pleased to announce that, with the availability of the ansible/jewel source code, we’ve reached another significant milestone with our refactoring effort.

The Jewel project and AWX

The ansible/jewel repository provides a proxy layer that connects AWX, for job scheduling and execution, and other pluggable Ansible services.

In the quickstart below we’re going to spin up the Jewel API and an AWX development environment. This table shows you the services that we’ll create and their network locations:

Service	URL
Jewel API	https://localhost:8000
Jewel Proxy (Envoy)	https://localhost:443
Controller (AWX) via proxy	https://localhost:443/api/controller/
AWX direct	https://localhost:8043

To stand this up locally and try it for yourself, complete the following steps:

1. Fork or clone both the ansible/jewel and ansible/awx repositories.

2. Start an AWX development environment.
   2a. Open a terminal window in the root of the ansible/awx repository.
   2b. Build the development image with make docker-compose-build.
   2c. Start the environment with make docker-compose.
   2d. Note the superuser password. You can also find this with cat tools/docker-compose/_sources/secrets/admin_password.yml.
   2e. Check that AWX is listening on ports 8043 (HTTPS) and 8013 (HTTP). For example, try: curl -sk https://localhost:8043/api/v2/ping/.

3. Start a Jewel development environment.
   3a. Open another terminal window in the root of the ansible/jewel repository.
   3b. Build a headless image with make docker-compose-build HEADLESS=1.
   3c. Start Jewel with make docker-compose-basic HEADLESS=1.
   3d. Check that Jewel is running with curl -sk https://localhost:8000/api/gateway/v1/ -u admin:'<PASSWORD>'.
   3e. Register the AWX service with make register-services.
   3f. Ping AWX via the proxy with curl -sk https://localhost:443/api/controller/v2/ping/.

Your next step could be trying docs/service_token_authentication.md to set up JWT-based trust between Jewel and AWX so they can authenticate API calls. That requires running gw-manage generate_service_secret controller inside the Jewel container and configuring AWX’s RESOURCE_SERVER settings.

Contributions to the Jewel and AWX documentation are welcome. If you’re looking for more ways to get involved and contribute, see John’s Ansible Jewel: Connecting Ansible services post that announces the Jewel project.

Next steps

There are other goals we’ve set out here in the forum, such as simplifying the code base and making it easier to contribute. We want the community to help shape the project and steer technical direction of the AWX project. You can expect to hear more about this in the forum as follow on posts.

Useful links

• Forum: AWX topics
• Forum: Jewel topics
• Forum: Newsletter Category
• AWX project documentation
• AWX code repository

[jon-nfc (Jon)]

Great to hear that this has been released as OSS. Especially as someone whom already runs AWX, EDA server and soon HUB.

Alas this has just been released as OSS and clearly the repo was/has been designed for the downstream product (AAP). Is there going to be added as part of Jewels DEV lifecycle, the following:

• Official Builds

• UI Builds (Ansible UI project does contain the UI (platform sub-dir) and build however quay.io has access blocked {quay.io/ansible/platform})

• Operator Created

  • If no operator at least kustomize manifests so we can at least deploy it

• Documentation that at least covers features, deployment and setup.

[jon-nfc (Jon)]

The docker compose doesn’t work…

below are some patches to fix and allow building with the UI included, they address:

• There’s a bug in the version “getter” so it crashes and wont start.

• modification to make and dockerfile so that it clones the current ansible-ui repo and as part of build you can now omit HEADLESS=1 switch.

Apply the patches (i was on commit 63115747b1c4dcd675827869d287393ea9235488). run make docker-compose-build then make docker-compose-basic to start the containers. you’ll have to wait a bit for migrations to finish. I’d advise looking at the logs docker logs -f aap_gw_1 as the admin password will be in it and you’ll see when it’s ready. presto https://localhost:8000/ is available with the ui.

diff --git a/Makefile b/Makefile
index 42d7f7d4..a386117b 100644
--- a/Makefile
+++ b/Makefile
@@ -219,11 +219,18 @@ tools/generated/.django_ansible_base_head: update_django_ansible_base_hash
 
 ## Check to pull the latest platform-ui if needed
 tools/generated/.has_built_ui:
-	$(CONTAINER_ENGINE) pull quay.io/ansible/platform-ui:latest > tools/generated/last_ui_pull
-	if [ ! -f $@ ] || [ `cat tools/generated/last_ui_pull | grep "Image is up to date" | wc -l` == "0" ] ; then \
-	    echo "Updating UI"; \
-	    touch $@ ; \
-	fi
+	if [ ! -d tools/generated/ui ]; then \
+		git clone --depth=1 https://github.com/ansible/ansible-ui tools/generated/ui; \
+	fi; \
+	cd tools/generated/ui; \
+	if [ ! -d node_modules ]; then \
+		npm ci; \
+	fi; \
+	cd platform;\
+	if [ ! -f dist/index.html ]; then \
+		npm run build; \
+	fi; \
+	cd ../../../../;
 
 ## Build the cert file
 tools/generated/gateway.crt:
diff --git a/aap_gateway_api/version.py b/aap_gateway_api/version.py
index f0c4e0df..412ae90f 100644
--- a/aap_gateway_api/version.py
+++ b/aap_gateway_api/version.py
@@ -15,6 +15,10 @@ def generate_version():
             from setuptools_scm import get_version
 
             return get_version()
+
+        except LookupError:
+            return "Unknown"
+
         except ModuleNotFoundError:
             return "Unknown"

diff --git a/tools/ansible/roles/sources/templates/Dockerfile.j2 b/tools/ansible/roles/sources/templates/Dockerfile.j2
index ad1fa279..fbdde44a 100644
--- a/tools/ansible/roles/sources/templates/Dockerfile.j2
+++ b/tools/ansible/roles/sources/templates/Dockerfile.j2
@@ -103,7 +103,7 @@ ENV PYTHONPATH=/opt/aap_gateway/src
 {% if not headless | bool %}
 # Pull in the platform-ui, this can change frequently so it should be one of the last things we try and do
 # placeholder until public image available
-COPY --from=quay.io/ansible/platform-ui:latest /usr/share/nginx/html /opt/aap_gateway/platform_ui
+COPY tools/generated/ui/platform/dist /opt/aap_gateway/platform_ui
 {% endif %}
 
 USER 1000

[tomas_z]

Hi Jon,

Thanks for taking the time to dig into this and share those patches, that’s exactly the kind of contribution we’re hoping to see from the community.

To set expectations around scope: the goal of open-sourcing Jewel is to provide the source code and a working development environment. We won’t be providing the means to re-create official builds or the operator. Similarly, the container images on quay.io/ansible/platform are a part of Red Hat’s product build pipeline and not needed for upstream development.

That said, the issues you ran into with docker compose and the version getter are things we’d love to get fixed upstream. Would you be able to open issues and/or submit PRs with your patches over at GitHub - ansible/jewel · GitHub? That way we can get them reviewed and merged properly, and others hitting the same problems will benefit too.

Really appreciate the interest and the hands-on testing. Looking forward to seeing you on the repo.

[jon-nfc (Jon)]

@tomas_z

To set expectations around scope: the goal of open-sourcing Jewel is to provide the source code and a working development environment. We won’t be providing the means to re-create official builds or the operator.

Just to confirm, as i don’t want to miss-interpret this statement. Red Had will not be creating builds, artifacts or releases of or for Jewel and the accompanying UI?

[tomas_z]

Hi Jon,

that is correct, at this time there is no plan for creating releases or any release artifacts for Jewel repository as this is an upstream development repository.

[jon-nfc (Jon)]

@tomas_z

Hi Jon,

that is correct, at this time there is no plan for creating releases or any release artifacts for Jewel repository as this is an upstream development repository.

Really???

Red hat since it’s inception has been an open source company. This has given them the bragging rights of claiming as such. Now however you have products, in this case jewel (there are many others in red hat’s catalog) that you have slapped an Open Source licence on where you don’t even meet the first fundamental principle of Open Source software…

Free To Use

This imposition is not that the software is free, as in cost no money; it’s free as in you can use it.

The majority of the Open Source software community are users – Users whom are only technical enough to follow documentation to get the software working so they can use it. If you have an “Open Source” piece of software that the majority of the community is unable to use, You don’t have “Open Source” software you have code has an Open Source licence. As such, you have now lost your “bragging rights”.

Additionally Red Hat is quite happy to use members of the Open Source Community, in this case developers to their advantage. All of this whilst neglecting the user base, the largest membership group in the community.

Where ever you fall within the Open Source community, in essence it’s a social contract between all who choose to partake, do your part: be it owner, developer, contributor (other) and last and not least end user.

Shame on Red Hat

Now I’ll address your response

@tomas_z

that is correct, at this time there is no plan for creating releases or any release artifacts for Jewel repository

ok, non committal. Although the remainder of your post adds a “but” which cancels this statement out.

@tomas_z

as this is an upstream development repository.

So in essence you want all of the benefits of owning “Open Source” software, where you use and abuse the developers from the community. All whilst not providing back to the same community “you claim” to be part of. Oh and yes you have claimed it, by using an Open Source licence.

Until such time that red hat (yes lower case, cause I’m “that” petty) chooses to fulfil their part of the Open Source social contract. As a developer within said Open Source community, I will not be assisting one bit. I work for the Open Source community only and without the end user, what’s the point.

PS. All repositories are for development. No one cares if it is up or down stream.

/cc @geerlingguy, fyi and for anyone else who is sick of large companies doing what this post describes!!

[oranod (Don Naro)]

Hey @jon-nfc

Thanks for taking the time to write all this up. To be honest with you, though, I feel like there’s a bit of hyperbole that undermines your overall position.

where you use and abuse the developers from the community
All whilst not providing back to the same community “you claim” to be part of

Red Hat does, in fact, provide back to the Ansible community in a variety of ways. Sponsoring conferences, facilitating meetups, funding for infrastructure are just some of the examples off the top of my head. I get that you might not like one response but it’s a bit of a stretch to think that represents the whole.

I feel like you make a good point about the focus on users though. Users are definitely an important persona when it comes to AWX. Jake touches on that in this section of the post about AWX and Ansible UI: AWX modernization: Ansible UI

When it comes to users, one of the things that I’ve heard again and again is that the requirement for kubernetes to run AWX was a real pain for users.

Yeah, kubernetes isn’t so hard but it is certainly overkill in loads of cases. That’s especially true for users who are “only technical enough to follow documentation to get the software working so they can use it”.

This is just my 2c but it seems kind of backwards to give “official” builds to community users. If you want “free AAP” you can always grab a developer subscription. Not being snarky about that and I hope it doesn’t come across that way. I genuinely think it would be a bit of a disservice to community users to get builds of a thing that is tailored for a different audience. Know what I mean?

[jon-nfc (Jon)]

@oranod

Thanks for taking the time to write all this up. To be honest with you, though, I feel like there’s a bit of hyperbole that undermines your overall position.

where you use and abuse the developers from the community

All whilst not providing back to the same community “you claim” to be part of

Red Hat does, in fact, provide back to the Ansible community in a variety of ways. Sponsoring conferences, facilitating meetups, funding for infrastructure are just some of the examples off the top of my head. I get that you might not like one response but it’s a bit of a stretch to think that represents the whole.

“hyperbole”… You have missed the whole point of my post and the pain points the “Open Source Community,” the only community in my contentious posts. The fact you have used the “Ansible Community” in the counter to my post is a demonstration of said missed point. All you have done is stood on the highest building and “screamed to the world” “look at the good I’m doing.” All of this whilst failing to meet the social contract of being a member of “Open Source” community.

@oranod

This is just my 2c but it seems kind of backwards to give “official” builds to community users. If you want “free AAP” you can always grab a developer subscription. Not being snarky about that and I hope it doesn’t come across that way. I genuinely think it would be a bit of a disservice to community users to get builds of a thing that is tailored for a different audience. Know what I mean?

Backwards? You are clearly thinking about this wrong, why?..

• Jewel is the software in question

• Jewel has an open source licence

• Jewel is currently only code…

• red had is not providing documentation, build artifacts or releases for Jewel

• red hat wants “Open Source” Community developers to help develop Jewel

• Jewel can not be used by a majority of the “Open Source” Community

• red hat wants the benefit “Open Source developers assisting with the software” whithout the resposibilites “Not providing a means for a majority of the Open Source Community to use said software”

As such red hat is not inline with the core principles of Open Source software. This “breech” of the social contract is common amongst red hats repositories.

If you want “free AAP” you can always grab a developer subscription.

Really! Antagonistic much! How so I hear you ask? red hat “Open Sourced” the products that make up AAP and now you winge that I can get it for free. (I don’t know if you know it or not, that’s how Open Source software works.) Welcome to Open Source Software.

Not being snarky about that and I hope it doesn’t come across that way.

Snarky or not, red hat can’t have its cake and eat it too. Be an Open Source software company or be a closed source software company. Right now this is clearly a case of red hat “protecting the bottom line.” All whilst claiming to be an “Open Source” company. A company I might add, that publicly likes to flout it’s “Open Source” nature without actually fulfilling their responsibilities.

[CWollinger]

Has anyone successfully connected Jewel with AWX?

I’m following the instructions mentioned above as well as docs/service_token_authentication.md.

However, when I run:

curl -k https://localhost:8000/api/gateway/v1/settings/analytics/ \
  --header "X-ANSIBLE-SERVICE-AUTH: $TOKEN"

The response:

{"detail":"Service authentication is locked until migrate_service_data is complete."}

The service migration itself???

aap-gateway-manage migrate_service_data --api-slug controller --username admin --merge-organizations true

Output:

Found 1 services to migrate: controller

Warning: Failed to load types and permissions from controller:
401 Client Error: Unauthorized for url:
https://proxy1:9080/api/controller/v2/service-index/role-types/?page_size=200

Response content:
{"detail":"Authentication credentials were not provided. To establish a login session, visit /api/login/."}

Role definitions referencing this service's types will not be available until the next successful migration.

AWX is also running in developer mode, and RESOURCE_SERVER is configured.