Nested Jinja2 templates and template_module

Get Help
, , ,
1

I am using Ansible as part of my GitLab CI/CD pipeline, and I have separate playbooks for rendering configs for my app and Nginx. The folder structure looks like this

    ├── XXX/YYY/playbook.yml
    ├── templates
    │   ├── app
    │   │   └── app_template.j2
    │   │   └── common_var.inc
    │   │ 
    │   └── nginx
    │       └── sites-available
    │           └── nginx_template.j2

All works fine until I try to include some vars from common_var.inc in the nginx_template.j2. No matter what type of path I am using, relative or absolute, I have got the following error:

jinja2.exceptions.TemplateNotFound: /home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/templates/app/common_var.inc

The file exists and contains a line like this:

{% set C.domain = "domain.tld" %}

Here is a Snippet from the Playbook that I use for rendering NGINX configs:

- name: Rendering nginx configs
    template:
      src: "{{ lookup('env','CI_PROJECT_DIR') }}/templates/nginx/sites-available/{{ item }}.j2"
      dest: "/etc/nginx/sites-available/{{ item }}.stage"
      force: yes
    with_items: "{{ vhost_list.stdout_lines }}"

I added debug for the ‘ansible_search_path’ var, and it looks in the following way:

# search: ['/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/XXX/YYY/playbooks', '/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/XXX/YYY/playbooks', '/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/templates/nginx/sites-available', '/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/XXX/YYY/playbooks', '/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/templates/nginx/sites-available']

How can I add {{ lookup('env','CI_PROJECT_DIR') }}/templates/app to the ansible_search_path?

Why does the absolute path not work?

Thank you!

2

How and where did you include common_var.inc. That part is not shown.

Generally if you run Ansible with -vvvv it will show you all the paths it searched the file in.

1 Like
3

Welcome to the forum, @yokodzun.

I tried reproducing your error locally with the following directory structure and value for CI_PROJECT_DIR:

utoddl@tango:~/ansible/yokodzun$ echo $CI_PROJECT_DIR 
/home/utoddl/ansible/yokodzun
utoddl@tango:~/ansible/yokodzun$ find `pwd` -type f | sort
/home/utoddl/ansible/yokodzun/templates/app/app_template.j2
/home/utoddl/ansible/yokodzun/templates/app/common_var.inc
/home/utoddl/ansible/yokodzun/templates/nginx/sites-available/nginx_template.j2
/home/utoddl/ansible/yokodzun/XXX/YYY/playbook.yml

You didn’t show how you were trying to include app/common_var.inc from within nginx_template.j2, so I attempted it with

{% include "app/common_var.inc" %}

Regardless of how I spelled the path in that include, I was not able to find a working incantation. However, I was able to make it work in two different ways.

The first solution is to create a symbolic link to your templates directory next to your playbook(s):

ln -s ../../templates XXX/YYY/templates

The second solution is to move your playbook up two levels to the root of your project – i.e. where Ansible expects it to be.

If you do either of those things, then you can get rid of the “{{ lookup('env','CI_PROJECT_DIR') }}/templates/” part of your src: parameter and use relative paths for all your templates. (And files. And tasks. And roles.)

Ansible uses lots of different PATHs for different things (see “ansible-config list | grep PATH”), but playbooks and templates aren’t two of them. (See “ansible-config list | grep -i PATH | grep -i 'playbook\|template'”.) Ansible searches for templates relative to the file containing the current play. Yet somehow people really want to bury their playbooks in some directory other than the root of their projects. That’s their choice of course, but they should expect to bear the consequences. I don’t see any up-sides to such non-standard project structures.

3 Likes
4

How and where did you include common_var.inc. That part is not shown.

I have been trying on different ways:

{% from project + '.inc' import C %}
{% from lookup('env','CI_PROJECT_DIR') + '/templates/app/' + project + '.inc' import C %}
{% include 'ZZZ/templates/app/' + project + '.inc' %}
{% include 'templates/app/' + project + '.inc' %}
5

We have a few different apps with configs stored in Git. The GitLab runner automatically updates those configs via Ansible playbooks.

ansible-playbook -b --user deploy-user --inventory XXX/YYY/inventories/bla-bla XXX/YYY/playbooks/deploy-config-for-nginx.yml

All playbooks are common for all apps, but each app’s configs are stored in its own git repo.
To avoid copying playbooks to each config repo, we do a ‘git clone’ of the playbook repo to the config repo as the first job of the pipeline.

- git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@git.tld/team/XXX.git

So, XXX/YYY/playbook.yml is part of the playbooks repo, and templates/* is part of the other.

This is the reason why we have this non-standard structure.

6

So there’s quite a few problems to tackle with you’re approach.

Firstly, I would attempt to build your common “XXX” repo into an Ansible Collection (even if it only contains roles and playbooks). All of the re-usable code should be in roles, and the playbooks should only exist to call those roles. Any templates found in these roles should be sourced with relative pathing.

Next, in your git CI jobs, you would install this collection from git via ansible-galaxy rather than cloning it (or install from Galaxy if you upload it to the public galaxy or setup a private galaxy). This will help enable Ansible to find everything via relative paths that make sense.

This should allow you to run ansible-playbook my_namespace.my_collection.my_playbook, and that playbook is simply:

- hosts: localhost
  gather_facts: false
  roles:
    - role: my_namespace.my_collection.my_role

All of the important tasks should be in the role(s). You can also pass any variables needed as a json blob with -e '{"use_my_vars": true, "my_custom_var": "foo"}

Any custom configs or templates that you would like this Ansible role to read should go in a sensible standardized location in the git repo. .config/ansible/my_role/templates/include.j2 for e.g. and tree this however it makes sense to you.

Search paths in Ansible won’t pick up the relative paths of the current working directory, as that context can change unexpectedly during execution. So in order to use the custom configs/templates in each repo, you will need to use the absolute path.

In your common jinja template, I would try one of the following:

{% include (lookup("env", "CI_PROJECT_DIR") ~ "/.config/ansible/my_role/templates/required.j2") %}
{% include (lookup("env", "CI_PROJECT_DIR") ~ "/.config/ansible/my_role/templates/optional.j2") ignore missing %}

or

{{ lookup("template", lookup("env", "CI_PROJECT_DIR") ~ "/.config/ansible/my_role/templates/required.j2") }}
{{ lookup("template", lookup("env", "CI_PROJECT_DIR") ~ "/.config/ansible/my_role/templates/optional.j2") | default("") }}

I’m not entirely sure that the jinja2 include directive will find templates outside of its working tree, hence the second example which uses Ansible’s template lookup plugin to render the template instead.

7

What’s interesting to me is that Jinja2 is complaining about not being able to find:

/home/gitlab-runner/builds/inqzetdg/0/configs/ZZZ/templates/app/common_var.inc

Which looks like a proper absolute path. I’m wondering if common_var.inc file is present at all in GitLab Runner working dir after it clones the Git repo.

@yokodzun you were saying that you have this split into multiple GIt repos. Is it possible that GitLab Runner did not pull everything needed to run the playbook.

8

Is it possible that GitLab Runner did not pull everything needed to run the playbook.

I have checked it manually, and all the necessary files are there.

9

What comes to my mind is that depending on your GitLab Runner setup and/or CI/CD pipeline, maybe the build folder is mounted inside a container where absolute paths are not valid any more.

You can try to create a dummy CI/CD pipeline that will just run “ls -laR” or similar command inside build folder. After that you can investigate the output in runner console logs and see if anything is different than expected or missing.

1 Like
10

I applied a workaround, which seemed to solve my problem.
I have combined a file with variables and a template into one temporary file and then rendered the target config from this file. After that, I cleaned up the temporary files. It’s not an elegant solution, but at least it works.