How to manage backups?

Hello guys!

I have a problem with my backups: jobs that were in running state when i performed my backup process remain running when I restore it. It’s quite annoying because these jobs are automatically respawned. Which is not the expected behaviour.

How would you manage that? The question (and the answer) may be obvious but is not so simple as it appears.

Many thanks.

2 Likes

UP!

I can’t believe that nobody do awx database backup :smiley:

I have the same issue as you do, we backup and restore to a different AWX instance nightly, and we were receiving alerts of job failures for jobs that happened to be running during the backup upon restore to the secondary.

I was interested to see if anyone who had more knowledge would’ve replied to you as this’d help me out too. The closest I could come to finding a solution on my own was Add ability to exclude job event and job data from backup · Issue #958 · ansible/awx-operator · GitHub but excluding the tables just made the restore fail. So my crappy solution was to have my automation disable notifications for the one frequently-running job, back up AWX, then re-enable notifications on it.

Hopefully someone has the right answer out here.

Same issue. Due to me additionally having the porblem that these exact running jobs deadlocked my AWX in such a wayy that I cannot cancel these jobs, Its really annyoing.
As a workaround I am currently trying to manually restore AWX using the awx.awx.export and awx.awx.import modules, but I cannot get the import to work.

Hi
You can use awx export binary which allow you to backup your whole AWX config into a .json file. I’m doing this and it works pretty well :

awx export --conf.token your_token --conf.host https://awx.yourdomain.net -k > awx_full.json
1 Like

Sorry for the delay. Unfortunately this may works on small instances, but how do you manage a +150GB database?

What error do you get when you try to export your 150 GB Database?
Most of that will probably be job output, do you really need those? My database was 50 GB large, but only about 900 MB of that was actually data I needed. Which was the stuff I subsequently was able to export using the aforementioned tools.

1 Like

Probably use a tool like DBeaver and manually export what you need.
However as mentionned by @jahknem , do you really need all the job output ? :slight_smile:

May or may not be relevant to you and your issue, but I was having problems trying to get AWXBackup to ignore job logs. The documentation in awx-operator for tables to exclude results in an unusable database, but it seems if you specify --exclude-table-data 'main_jobevent*' --exclude-table-data '*main_systemjobevent*' --exclude-table-data '*main_activitystream_*'" it results in a usable AWXBackup of 90% reduced size for me (Update README.md for AWXBackup pg_dump_suffix by mcen1 · Pull Request #2020 · ansible/awx-operator · GitHub). I tried doing the export method and for some reason it was taking 2+ hours to export my inventories, whereas the AWXBackup only takes a few minutes to do dump everything except those data tables.

2 Likes

One thing about that. When exporting stuff via dbeaver, make sure to get all the salts and secrets that are in the settings the same. Otherwise awx will fail to read from the database. Learned that the hard way

1 Like