We are using AWX and Kubespray to deploy Kubernetes cluster. We have multiple customers(tenants) and each customer has multiple kubernetes environments such as dev, stg, canary and prod. So each k8s cluster has its own inventory and respective configs.yml file stored in inventory location. Configs.yml has many ansible variables which has customer specific values such as docker_version, etcd_version, etc.
This is our inventory structure:
- Inventory_folder:
- hosts.ini (Inventory file)
- configs.yml (stores configurations variable)
- group_vars/
- all/all.yml
- K8s-cluster/k8s-cluster.yml
- …
In our ansible based automation we are using configs.yml file in tenant inventory to store tenant specific configurations. Some variables in this file are also defined at other places like:
-
roles/defaults/main.yml
-
roles/vars/main.yml
-
inventory/group_vars/k8s-cluster.yml
-
inventory/group_vars/all.yml
In AWX, we moved/stored configs.yml vars inside the tenant’s inventory/group_vars/all/configs.yml because they will get imported automatically when we sync inventory from SCM. Ansible has its own variable precedence rules. As per ansible variable precedence rules roles/vars/main.yml has higher precedence than inventory/group_vars, hence AWX inventory VARIABLES not overriding the default values defined in roles/vars/main.yml.
Solution is to set the configs.yml variables at template or workflow extra-vars section which takes more precedence than roles/vars but the problem is we will have to create as many number of job_templates or workflows as many tenant * number of environment we have. Lets say if we have 5 tenant and each tenant has 4 k8s-cluster there we will have to create 20 job template or workflow just because of variable precedency issue.
We don’t want to create so many templates because playbook/role are generic across all tenant and environments.
Can someone please suggest any better approach for handling this scenario?