AWX Applications - Internal server error

Archives AWX Project
1

Hello,
I’ve deployed in my aks cluster AWX with awx-operator 0.28.0

It’s not a clean installation, I’m using a db from a previous AWX version (quite old version).

When I click on Application menù I receive an internal server error:

err.PNG
err.PNG766×398 13 KB

From container “awx-web” I see this log error:
2022-09-21 10:08:28,819 ERROR [2f9c4cc6f70d45b4af66e8983b5f0624] django.request Internal Server Error: /api/v2/applications/
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py”, line 120, in _verify_signature
h.verify(data[-32:])
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/primitives/hmac.py”, line 69, in verify
ctx.verify(signature)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/hmac.py”, line 76, in verify
raise InvalidSignature(“Signature did not match digest.”)
cryptography.exceptions.InvalidSignature: Signature did not match digest.

Is it possible that some tables as old data with a different signature? Can I clean this tables and re-insert data with new AWX version?

Thank you,
Elia

2

Hi Elia

Have you set the awx-secret-key as it was on your old environment?

It must be the same as far as I have discovered.
When I import data from an old environment in a new environment I have to edit that secret key.

Regards Hans

3

Hi Hans,
no I haven’t set the “awx-secret-key” as it was on my old environment.

I’ve been working with the new AWX environment and the new secret key for months.
I configured the new AWX with the new secret-key I think that now I can no longer replace it with the one recovered from the old environment.

I hope it will be possible to delete all the data related to the applications (I don’t know in which tables) and re-enter everything with the new secret-key.
It’s possible?

Regards,
Elia

4

Well it is confusing for me as well.
But when I import data from my current production environment into a newly setup one.

I do the following.

  1. Drop de awx database in postgres.
  2. Import the data into postgres
  3. Set awx-secret-key.

This worked until 0.21 of the awx operator.

After that I have had no luck so far.

5

I’ve checked in my old environment (AWX 3.0.1.0) and “awx-secret-key” secret it’s differently formatted.

I can’t find “secret_key” as in my new enviroment… It’s like this:

apiVersion: v1
data:
admin_password: xxx
confd_contents: xxx
pg_password: xxx
rabbitmq_erlang_cookie: xxx
rabbitmq_password: xxx
kind: Secret

…

Which is the “secret_key” data?

Elia

6

HI Elia,

What do you get when you do : kubectl get secret -n
Is there no awx-secret-key?

Regards Hans

7

Hi Hans,
no there isn’t. In this older AWX version there is only a secret “secret_key” and it’s not the same secret.

AWX.PNG

Regards,
Elia

8

Anyone from the staff who can suggest me how to clean up the db to fix “Application” error?

Thank you very much,
Elia

9

Hello Elia

Although I am not staff, perhaps you need to elaborate on what exactly you are doing.
You have deployed 0.28 but it is not a new installation.
What does that mean?

Regards Hans-Peter

10

Is it an external database?

11

Hi Hans,
I mean that I restored an external database from a previous AWX version.
Maybe some secret in the database should have a different signature but only for “Application” I have the issue.

12

But when you installed 0.28, did you check the awx-operator-controller-manager logging?
I discovered that some tables have additional columns in new versions.

Regards Hans-Peter

13

Hi,

But have you watched the logging of awx-controller-manager after you did the : make deploy command?

What kind of kubernetes do you use?

Regards Hans

14

Yes, I saw the awx-controller-manager log but I didn’t notice anything strange.
K8s version: 1.22.6

regards Elia

15

I think that you need to:

  1. install the old version of awx-operator and have that used the database
  2. upgrade it to 0.28

IMHO you need to have awx-secret-key and set it just as in the old environment.

16

As I show you in previous messages in my old enviroment there is no secret key. Or it is in a different secret… there is no “secret_key” field.

Elia

17

What is the old awx-operator version?

18

I’ve checked in my old environment (AWX 3.0.1.0) and “awx-secret-key” secret it’s differently formatted.

I can’t find “secret_key” as in my new enviroment… It’s like this:

apiVersion: v1
data:
admin_password: xxx
confd_contents: xxx
pg_password: xxx
rabbitmq_erlang_cookie: xxx
rabbitmq_password: xxx
kind: Secret

…

Which is the “secret_key” data?

Elia

19

I’ve found a solution clearing “Applications” tables on awx database.

Main table is “main_oauth2application” and first related tables “main_oauth2accesstoken”, “oauth2_provider_refreshtoken” only for the records affected.

Now “Applications” works fine, Internal server error disappeared.

Regards,
Elia

20

Hi
Super.
Did you find it somewhere on the web?
If so can you put it here for the reference?

Regards Hans