# Recovering from catastrophic failures

**URL:** <https://forum.ansible.com/t/recovering-from-catastrophic-failures/36649>\
**Category:** AWX Project\
**Tags:** kubernetes, awx\
**Created:** [May 21, 2022, 2:55pm UTC](https://forum.ansible.com/t/recovering-from-catastrophic-failures/36649 "2022-05-21T14:55:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tin](https://avatars.discourse-cdn.com/v4/letter/t/5f9b8f/32.png) [@tin](https://forum.ansible.com/u/tin)\
**Post date:** [May 21, 2022, 2:55pm UTC](https://forum.ansible.com/t/recovering-from-catastrophic-failures/36649/1 "2022-05-21T14:55:29Z")

</div>

We are running 19.3.0 with awx-operator 0.14.0 on k8s. We suffered catastrophic failures and lost the cluster. The only thing we have is the external pair of postgresql. The data is fine. How do we recover from this?

Look like the issue is that we’ve lost the secret\_key, which is preventing us from recreating the cluster with our data.

The error we are getting is:

ERROR [-] awx.main.utils.encryption Failed to decrypt `Setting(pk=97).value`; if you’ve recently restored from a database backup or are running in a clustered environment, check that your `SECRET_KEY` value is correct  
Traceback (most recent call last):  
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/fernet.py”, line 104, in \_verify\_signature  
h.verify(data[-32:])  
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/primitives/hmac.py”, line 66, in verify  
ctx.verify(signature)  
File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/cryptography/hazmat/backends/openssl/hmac.py”, line 74, in verify  
raise InvalidSignature(“Signature did not match digest.”)  
cryptography.exceptions.InvalidSignature: Signature did not match digest.

---

<div class="post-metadata">

**Author:** ![tin](https://avatars.discourse-cdn.com/v4/letter/t/5f9b8f/32.png) [@tin](https://forum.ansible.com/u/tin)\
**Post date:** [June 16, 2022, 3:28pm UTC](https://forum.ansible.com/t/recovering-from-catastrophic-failures/36649/2 "2022-06-16T15:28:19Z")

</div>

I thought I post a followup to this. I was able to recover most of the data. It turns out most of the database content in postgresql is not encrypted. So I was able to dump the tables and massage the data into importable json, then feed it to the new AWX.

FYI for anyone who run into the same issue.

🙂

Tin

---

<div class="post-metadata">

**Author:** ![ponnurangam\_h](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@ponnurangam\_h](https://forum.ansible.com/u/ponnurangam_h)\
**Post date:** [June 4, 2023, 4:59pm UTC](https://forum.ansible.com/t/recovering-from-catastrophic-failures/36649/3 "2023-06-04T16:59:42Z")

</div>

Hi, we ran into similar problem, no we are at recovering the data from the DB and feed it to the AWX, can you please share the steps. thanks in advance