Awx-projects-claim stuck in pending

Get Help
,
1

I have just upgraded AWX/Operator to 24.2.0/2.15.0 and everything went OK except the awx-projects-claim. This is stuck in pending:

kubectl get pvc
NAME                                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
awx-backup-claim                        Bound     awx-backup-volume                          4Gi        RWO            awx-backup-volume     <unset>                 9d
postgres-15-ansible-awx-postgres-15-0   Bound     pvc-d7e8181b-9492-4c97-a1a1-1c74ba714f20   8Gi        RWO            local-path            <unset>                 3h36m
awx-projects-claim                      Pending                                                                        awx-projects-volume   <unset>                 52m

The pv does exist:

kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                       STORAGECLASS          VOLUMEATTRIBUTESCLASS   REASON   AGE
awx-postgres-13-volume                     8Gi        RWO            Retain           Released   awx/postgres-13-awx-postgres-13-0           awx-postgres-volume   <unset>                          16d
awx-projects-volume                        2Gi        RWO            Retain           Released   awx/awx-projects-claim                      awx-projects-volume   <unset>                          16d
awx-postgres-15-volume                     8Gi        RWO            Retain           Released   awx/postgres-13-awx-postgres-13-0           awx-postgres-volume   <unset>                          16d
awx-backup-volume                          4Gi        RWO            Retain           Bound      awx/awx-backup-claim                        awx-backup-volume     <unset>                          9d
pvc-d7e8181b-9492-4c97-a1a1-1c74ba714f20   8Gi        RWO            Delete           Bound      awx/postgres-15-ansible-awx-postgres-15-0   local-path            <unset>                          3h36m

The storage classes do too:

kubectl get storageclass
NAME                   PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path          Delete          WaitForFirstConsumer   false                  16d
awx-projects-volume    kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  68s

I have tried modifying the pvc and pv files but I cannot get it to bind. I am sure that I am missing something obvious, but I can’t see it. Could someone point me in the right direction please?

2

Hello @bjwebster, we are so sorry to hear you’re having trouble! Could you provide us with the related Kubernetes logs? This will help us better understand what may be occurring here.

Thank you for your time!

3

Looks like you may need to set projects_storage_class: local-path in your kind: AWX spec. This may require you to manually delete the PVC for the operator to apply the change.

4

Denney-tech and djyasin,
Thank you for your replies. I am on holiday at the moment with no access to my installation. I will follow-up with you when I get home early next week.

5

OK, here is the output from kubectl describe pvc awx-projects-claim:

kubectl describe pvc awx-projects-claim
Name:          awx-projects-claim
Namespace:     awx
StorageClass:  local-path
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  WaitForFirstConsumer  3m6s (x97013 over 16d)  persistentvolume-controller  waiting for first consumer to be created before binding
6

Denny-tech,
My awx spec for project storage looks like this:

  projects_persistence: true
  projects_existing_claim: awx-projects-claim

Do I just need to add the line?:

  projects_persistence: true
  projects_existing_claim: awx-projects-claim
  projects_storage_class: local-path

The pending PVC already appear to be using local-path:

kubectl get pvc
NAME                                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
awx-backup-claim                        Bound     awx-backup-volume                          4Gi        RWO            awx-backup-volume   <unset>                 26d
postgres-15-ansible-awx-postgres-15-0   Bound     pvc-d7e8181b-9492-4c97-a1a1-1c74ba714f20   8Gi        RWO            local-path          <unset>                 17d
awx-projects-claim                      Pending                                                                        local-path          <unset>                 16d
7

@bjwebster
Hi,

You should specify ReadWriteOnce as the access mode for the PVC if you want to use local-path.
The AWX Operator creates PVC for the project with the mode ReadWriteMany by default, which is not supported by local-path provisioner: Persisting projects directory - Ansible AWX Operator Documentation

Add following line to your awx spec and try again.

  projects_storage_access_mode: ReadWriteOnce
2 Likes
8

Kurokobo,
Thank you for your help.

I am not sure that I do want to use local-path, it just seems to be where I have ended up. I have been trying so many different thing over the past couple of weeks that I have got myself into a bit of a mess.

It may be better if I just delete my installation and start again. I have used your awx-on-k3s guide to install previously but I want to access the AWX on its IP address rather than its hostname. The K3S server is running on an EC3 instance in AWS and is behind and API gateway and a Loadbalancer, which terminate HTTPS, so I don’t need certs on the AWX itself.

What would be the best way to deploy AWX so that I can access it on the server’s node port?

9

I have re-installed AWX using the awx-on-k3s guide. I used the TLS option to make things simpler for myself.
Everything is back up and running now so thank you to kurokobu for providing such a comprehensive guide, and to djyasin and Denney-tech for their suggestions.

10

@bjwebster
Thank you for the update, and I’m glad to hear that you’ve made progress despite my delayed response.

I’m trying to answer your question for future reference.

I believe “the best way” depends on your requirements.
If you’re not particular about the PV provisioner, are fine with random passwords, and only need a single node, then deploying AWX with following minimal specs on K3s or Rancher should work good:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  nodeport_port: 30080
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce

Of course it’s also possible to change to use node port based on my guide.
Simply remove the awx-secret-tls block from kustomization.yaml and specify service_type and nodeport_* instead of ingress_* in awx.yaml .

Enjoy using AWX!

1 Like