AWX jobs failure after migrating from AWX 17.1.0 to AWX 22.2.0

Get Help
, , , , ,
1

Hi Team,

I have 2 AWX instances:
1.) AWX 17.1.0 running as a docker container
2.) AWX 22.2.0 running on kubernetes

We are trying to perform the migration from AWX 17 to AWX 22.The issue is that the migration has completed successfully i,e able to load inventores,projects,jobs_templates etc,but when I run any AWX job from the new AWX it is giving me this error


Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py", line 133, in _verify_signature
    h.verify(data[-32:])
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/primitives/hmac.py", line 72, in verify
    ctx.verify(signature)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 85, in verify
    raise InvalidSignature("Signature did not match digest.")
cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 516, in run
    passwords = self.build_passwords(self.instance, kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py", line 1163, in build_passwords
    passwords['scm_password'] = project_update.credential.get_input('password', default='')
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/models/credential/__init__.py", line 278, in get_input
    return decrypt_field(self, field_name)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field
    return smart_str(decrypt_value(key, value))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
    value = f.decrypt(encrypted)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py", line 90, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py", line 151, in _decrypt_data
    self._verify_signature(data)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py", line 135, in _verify_signature
    raise InvalidToken
cryptography.fernet.InvalidTokentype or paste code here

On investigating I came to know that there can be a SECRET key mismatch between the old awx and the new one.

So the old AWX secret key value inside /etc/tower/SECRET_KEY is awxsecret and on encoding this in base64 it is giving me YXd4c2VjcmV0.

Whereas the new AWX contains this value ZjczbU5CbmtzdVNRcXd1MFpDNnQ5S25VdVBmbDRtU0Y=
On decoding this value I am getting f73mNBnksuSQqwu0ZC6t9KnUuPfl4mSF which is currently set in /etc/tower/SECRET in awx-container inside the awx_task pod awx-task-6777c747b8-lwb5t.

So in order to make the new AWX work should I set the value to awxsecret or YXd4c2VjcmV0 inside /etc/tower/SECRET inside the awx-task container inside the pod ?

Or is there something else that I need to follow ?

Please help as AWX on k8s is something new for me

2

Hi,

Correct, to do that, update your <resourcename>-secret-key secret resource with correct (old) value, and then restart AWX pods.
See the docs for migration for further information: Migrating data from an old AWX instance - Ansible AWX Operator Documentation

1 Like
3

Hi @kurokobo ,

Thanks for the quick help.Highly appreciate it.

I added the old secret key in the awx-secret running on k8s and it worked.

But one question:
I only updated the secret key in the data key

kubectl get secret awx-secret-key -n awx -o yaml

**apiVersion: v1**
**data:**
**  **secret_key: YXd4c2VjcmV0****
**kind: Secret**
**metadata:**
**  annotations:**
**    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Secret","metadata":{"labels":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/operator-version":"2.1.0","app.kubernetes.io/part-of":"awx"},"name":"awx-secret-key","namespace":"awx"},"stringData":{"secret_key":"f73mNBnksuSQqwu0ZC6t9KnUuPfl4mSF"}}'**
**  creationTimestamp: "2024-05-14T10:50:27Z"**
**  labels:**
**    app.kubernetes.io/component: awx**
**    app.kubernetes.io/managed-by: awx-operator**
**    app.kubernetes.io/operator-version: 2.1.0**
**    app.kubernetes.io/part-of: awx**
**  name: awx-secret-key**
**  namespace: awx**
**  resourceVersion: "2512023"**
**  uid: 705d3886-eee9-46f6-b12d-dbf60b018e9a**
**type: Opaque**

**But I did'nt added it inside metadata.annotations.kubectl.kubernetes.io/last-applied-configuration.stringData.secret_key**

**metadata:**
**  annotations:**
****    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Secret","metadata":{"labels":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/operator-version":"2.1.0","app.kubernetes.io/part-of":"awx"},"name":"awx-secret-key","namespace":"awx"},"stringData":**{"secret_key":"f73mNBnksuSQqwu0ZC6t9KnUuPfl4mSF"}**}**'**

**Is this fine ? Or we need to update it there also**
**And do we need to restart all the pods among this or only awx_task***

**awx-operator-controller-manager-68f8b75b5c-x4rh9**
**awx-postgres-13-0**
**awx-task-7d74b559d6-xhwdc**
**awx-web-58898ffc9-hd4ts**

Thanks,
Mohammed Husnain

4

Hi,

Modifying only data.secret_key is fine. Any values under kubectl.kubernetes.io/last-applied-configuration should NOT be modified by hand.

You should restart awx-task and awx-web.

5

Hi @kurokobo ,

Ok.Will only update the data key and restart only those pods which you mentioned.

Thanks Alot for the help :smiley:

6

Glad I could help you :smiley:
If you don’t have any further questions, could you please mark my post as a solution? Thanks!

7

Hi @kurokobo ,

Yes I have marked it as a solution. :smiley:

Thanks Alot :innocent:

1 Like
8

Hi @kurokobo ,

Can you please have a look on this issue: Ansible task failure stating shared connection closed module failure

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.