Hey,
I’ve followed all the guides and it seems like the postgress pod fails scheduling:
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pod -postgres-13-0
Name: -postgres-13-0
Namespace: awx
Priority: 0
Node:
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
app.kubernetes.io/part-of=
controller-revision-hash=-postgres-13-8677ccdd5d
statefulset.kubernetes.io/pod-name=-postgres-13-0
Annotations:
Status: Pending
IP:
IPs:
Controlled By: StatefulSet/-postgres-13
Containers:
postgres:
Image: postgres:13
Port: 5432/TCP
Host Port: 0/TCP
Requests:
cpu: 10m
memory: 64Mi
Environment:
POSTGRESQL_DATABASE: <set to the key ‘database’ in secret '-postgres-configuration’> Optional: false
POSTGRESQL_USER: <set to the key ‘username’ in secret ‘-postgres-configuration’> Optional: false
POSTGRESQL_PASSWORD: <set to the key ‘password’ in secret '-postgres-configuration’> Optional: false
POSTGRES_DB: <set to the key ‘database’ in secret ‘-postgres-configuration’> Optional: false
POSTGRES_USER: <set to the key ‘username’ in secret '-postgres-configuration’> Optional: false
POSTGRES_PASSWORD: <set to the key ‘password’ in secret '-postgres-configuration’> Optional: false
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_INITDB_ARGS: --auth-host=scram-sha-256
POSTGRES_HOST_AUTH_METHOD: scram-sha-256
Mounts:
/var/lib/postgresql/data from postgres-13 (rw,path=“data”)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-glhch (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
postgres-13:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-13--postgres-13-0
ReadOnly: false
kube-api-access-glhch:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
Warning FailedScheduling 9s (x2 over 96s) default-scheduler 0/1 nodes are available: 1 node(s) didn’t find available persistent volumes to bind.
I have two storage classes created. They are the same, except the bind mode. If I set the bind mode to Immediate, for the postgres SC, the postgres pod fails to scheduled with pod has unbound immediate PersistentVolumeClaims.
The projects-storage will bind regardless of binding mode.
[sysadmin@dev-awx-01 k8awx]$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
postgres-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 2m53s
projects-storage kubernetes.io/no-provisioner Delete Immediate false 2m53s
The PVs are the same. They are just named differently and point to their own storage class.
[sysadmin@dev-awx-01 k8awx]$ cat projects-storage-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: projects-storage-pv
namespace: awx
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: projects-storage
local:
path: /data/awx
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
[sysadmin@dev-awx-01 k8awx]$ cat postgres-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: postgres-storage-pv
spec:
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
volumeMode: Filesystem
storageClassName: postgres-storage
hostPath:
path: /data/postgress
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dev-awx-0
The PVs get created properly.
We are using local storage for this:
[sysadmin@dev-awx-01 k8awx]$ df -h
/dev/mapper/ansible-awx–storage 25G 211M 25G 1% /data/awx
/dev/mapper/ansible-postgres–storage 25G 211M 25G 1% /data/postgress
[sysadmin@dev-awx-01 k8awx]$
[sysadmin@dev-awx-01 k8awx]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
postgres-storage-pv 20Gi RWX Retain Available postgres-storage 3m
projects-storage-pv 20Gi RWX Retain Bound awx/projects-storage-pvc projects-storage 3m
However, the PVC for postgres-13-*****-postgres-13-0 will not bind.
[sysadmin@dev-awx-01 k8awx]$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
postgres-13-*****-postgres-13-0 Pending postgres-storage 2m40s
projects-storage-pvc Bound projects-storage-pv 20Gi RWX projects-storage 3m3s
Describing the PVC, shows it’s waiting for the POD to be scheduled.
[sysadmin@dev-awx-01 k8awx]$ kubectl describe pvc postgres-13--postgres-13-0
Name: postgres-13--postgres-13-0
Namespace: awx
StorageClass: postgres-storage
Status: Pending
Volume:
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-****
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
Annotations:
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: ****-postgres-13-0
Events:
Type Reason Age From Message
Normal WaitForFirstConsumer 9m29s persistentvolume-controller waiting for first consumer to be created before binding
Normal WaitForPodScheduled 3m25s (x25 over 9m25s) persistentvolume-controller waiting for pod ****-postgres-13-0 to be scheduled
Below is the awx yaml and the kustomization.yaml.