AWX - Project persistent storage claim - ReadWriteOnce or ReadWriteMany

[disi (Marcus)]

Can the project volume claim for AWX be ReadWriteMany in Azure Kubernetes?

Today I upgraded from 22.4.0 to 23.3.0, our project is stored in a git repository and the database in an external sql instance.

The upgrade failed, because it could not release the claim and rebond the new pods with the project storage claim.

My test to make future upgrades more smooth.

I patched the storage to “Retain” from default “Delete” in Azure Kubernetes.
I deleted the deployment of awx-web and awx-task manually to release the storage claim.
Then I changed the yaml in the pvc to ReadWriteMany for this volume and let awx-operator re-reploy awx-web and awx-task.
It failed to claim the volume.

After changing the volume claim back to ReadWriteOnce, AWX was up and running in seconds and the new pods deployed.

[TheRealHaoLiu (Hao Liu)]

AFAIR (I could be wrong) default storage class for AKS does not support RWM that’s why we failed to claim the volume when using RWM

There should be a way to setup a different storage class that support RWM.

[disi (Marcus)]

Thank you for the hint. I did not think about the storage class, it is indeed “default”.

So AWX should work if I create a project claim with RWM?
I’ll look into it tomorrow

[TheRealHaoLiu (Hao Liu)]

If you get it to work would u mind providing the information back here so other people who might have the same question can benefit thanks!

[disi (Marcus)]

Hello, so today I tested RedWriteMany with azurefile storageclass.
AWX deploys, then I tried to do a project sync.
The project sync fails, because it cannot put a lock on the filesystem.
This answers my question

<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /runner/.ansible/tmp/ansible-tmp-1697202893.022734-115-235855685788987/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git clone --origin origin 'https://$encrypted$:$encrypted$@gitlab.XXX.com/ladoga/ansible/roles.git' /var/lib/awx/projects/_8__XXX_ansible_gitlab",
    "invocation": {
        "module_args": {
            "accept_hostkey": false,
            "accept_newhostkey": false,
            "archive": null,
            "archive_prefix": null,
            "bare": false,
            "clone": true,
            "depth": null,
            "dest": "/var/lib/awx/projects/_8__XXX_ansible_gitlab",
            "executable": null,
            "force": true,
            "gpg_whitelist": [],
            "key_file": null,
            "recursive": true,
            "reference": null,
            "refspec": null,
            "remote": "origin",
            "repo": "https://$encrypted$:$encrypted$@gitlab.XXX.com/ladoga/ansible/roles.git",
            "separate_git_dir": null,
            "single_branch": false,
            "ssh_opts": null,
            "track_submodules": false,
            "umask": null,
            "update": true,
            "verify_commit": false,
            "version": "main"
        }
    },
    "msg": "Cloning into '/var/lib/awx/projects/_8__XXX_ansible_gitlab'...\\nerror: chmod on /var/lib/awx/projects/_8__XXX_ansible_gitlab/.git/config.lock failed: Operation not permitted\\nfatal: could not set 'core.filemode' to 'false'",
    "rc": 128,
    "stderr": "Cloning into '/var/lib/awx/projects/_8__XXX_ansible_gitlab'...\\nerror: chmod on /var/lib/awx/projects/_8__XXX_ansible_gitlab/.git/config.lock failed: Operation not permitted\\nfatal: could not set 'core.filemode' to 'false'\\n",
    "stderr_lines": [
        "Cloning into '/var/lib/awx/projects/_8__XXX_ansible_gitlab'...",
        "error: chmod on /var/lib/awx/projects/_8__XXX_ansible_gitlab/.git/config.lock failed: Operation not permitted",
        "fatal: could not set 'core.filemode' to 'false'"
    ],
    "stdout": "",
    "stdout_lines": []
}