I have a problem with my backups: jobs that were in running state when i performed my backup process remain running when I restore it. It’s quite annoying because these jobs are automatically respawned. Which is not the expected behaviour.
How would you manage that? The question (and the answer) may be obvious but is not so simple as it appears.
I have the same issue as you do, we backup and restore to a different AWX instance nightly, and we were receiving alerts of job failures for jobs that happened to be running during the backup upon restore to the secondary.
I was interested to see if anyone who had more knowledge would’ve replied to you as this’d help me out too. The closest I could come to finding a solution on my own was Add ability to exclude job event and job data from backup · Issue #958 · ansible/awx-operator · GitHub but excluding the tables just made the restore fail. So my crappy solution was to have my automation disable notifications for the one frequently-running job, back up AWX, then re-enable notifications on it.
Same issue. Due to me additionally having the porblem that these exact running jobs deadlocked my AWX in such a wayy that I cannot cancel these jobs, Its really annyoing.
As a workaround I am currently trying to manually restore AWX using the awx.awx.export and awx.awx.import modules, but I cannot get the import to work.
What error do you get when you try to export your 150 GB Database?
Most of that will probably be job output, do you really need those? My database was 50 GB large, but only about 900 MB of that was actually data I needed. Which was the stuff I subsequently was able to export using the aforementioned tools.
May or may not be relevant to you and your issue, but I was having problems trying to get AWXBackup to ignore job logs. The documentation in awx-operator for tables to exclude results in an unusable database, but it seems if you specify --exclude-table-data 'main_jobevent*' --exclude-table-data '*main_systemjobevent*' --exclude-table-data '*main_activitystream_*'" it results in a usable AWXBackup of 90% reduced size for me (Update README.md for AWXBackup pg_dump_suffix by mcen1 · Pull Request #2020 · ansible/awx-operator · GitHub). I tried doing the export method and for some reason it was taking 2+ hours to export my inventories, whereas the AWXBackup only takes a few minutes to do dump everything except those data tables.
One thing about that. When exporting stuff via dbeaver, make sure to get all the salts and secrets that are in the settings the same. Otherwise awx will fail to read from the database. Learned that the hard way