Recovering from catastrophic failures

We are running 19.3.0 with awx-operator 0.14.0 on k8s. We suffered catastrophic failures and lost the cluster. The only thing we have is the external pair of postgresql. The data is fine. How do we recover from this?

Look like the issue is that we’ve lost the secret_key, which is preventing us from recreating the cluster with our data.

The error we are getting is:

ERROR [-] awx.main.utils.encryption Failed to decrypt Setting(pk=97).value; if you’ve recently restored from a database backup or are running in a clustered environment, check that your SECRET_KEY value is correct
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/fernet.py”, line 104, in _verify_signature
h.verify(data[-32:])
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/primitives/hmac.py”, line 66, in verify
ctx.verify(signature)
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/hmac.py”, line 74, in verify
raise InvalidSignature(“Signature did not match digest.”)
cryptography.exceptions.InvalidSignature: Signature did not match digest.

I thought I post a followup to this. I was able to recover most of the data. It turns out most of the database content in postgresql is not encrypted. So I was able to dump the tables and massage the data into importable json, then feed it to the new AWX.

FYI for anyone who run into the same issue.

:slight_smile:

Tin

Hi, we ran into similar problem, no we are at recovering the data from the DB and feed it to the AWX, can you please share the steps. thanks in advance