Best practice for extra variables interpolation via ansible-inventory?

Archives AWX Project
1

Hi everyone and Happy New Year 2022!

I am turning to you for a “best practice” question about inventory variables as I am experimenting w/ AWX to run our playbooks. Let me know if this is best suited for the general Ansible mailing list.

We currently have inventory variables defined as:
79 builder-11:
80 ansible_host: W.X.Y.Z
81 externalip: “{{ ansible_host }}”
82 vmid: 118300007
83 fqdn: “{{ builders_name }}1.{{ internal_domain }}”
84 hostname: “{{ builders_name }}1”
85 internalip: A.B.C.D

The inventory references certain variables like {{ internal_domain }} which are defined in a dedicated variable file (vars.yml)

We currently run our playbooks via:
ansible-playbook -i inventory/vrubiolo-test.yml playbook.yml -e @/path/to/vrubiolo-test01/vars.yml

Due to the way AWX builds inventories via ansible-inventory, I am seeing that the associated host does not have the abovementioned variables defined/resolved when looking at its entry in the AWX UI:
{
“ansible_host”: “W.X.Y.Z”,
“externalip”: “{{ ansible_host }}”,
“fqdn”: “{{ builders_name }}1.{{ internal_domain }}”,
“hostname”: “{{ builders_name }}1”,
“internalip”: “A.B.C.D”,
“vmid”: 118300007
}

This is problematic as the variables have an incorrect value.

I have also seen this behavior w/ ansible-inventory itself, when run via
ansible-inventory -i inventory/vrubiolo-test01.yaml --list -e @/path/to/vrubiolo-test01/vars.yml

I have seen related issues with ansible-inventory like https://github.com/ansible/ansible/issues/72590 but I am not sure this applies here …

Is there a way to make it so that the variables are resolved like it happens when running ansible-playbook directly?

I guess I could try to write a custom inventory plugin which would call ansible-playbook to perform the variable evaluation, parse the output and rebuild an inventory with the variables resolved but this seems overkill for what I want to do here …

Thanks for your guidance,

Vincent

2

Let me link this:

https://github.com/ansible/ansible/pull/70970

I think what you’re doing is using the functionality that was added there. Inventory plugins could not make use of data passed vi -e / --extra-vars until that was merged.

The problem is, I don’t think that change made it into Ansible 2.9, which we still need to have working.
There’s no good way to determine in advance what Ansible version we have either.
So I would like to add some forms of support for extra variables in inventory imports, but we’re blocked on old version support.

3

Hi Alan,

First, many thanks for your assistance here, I cannot stress how useful this is to be able to get feedback and discuss issues on this list here, especially as I am trying to make our existing playbooks run via AWX (and hitting all those difficulties related to extra variables usage scheme).

I have put my comments below.

Let me link this:

https://github.com/ansible/ansible/pull/70970

Very good pointer indeed, this is exactly the functionality I am using.

I think what you’re doing is using the functionality that was added there. Inventory plugins could not make use of data passed vi -e / --extra-vars until that was merged.

Understood. This was the direction I was following to try to solve my issue actually, I am not tied to this functionality/option.

If I rephrase my original issue:

  1. Our playbooks reference host variables, e.g via {{ hostvars[host].hostname }} in a template file (we use this to configure an Opnsense instance).
  2. This variable is set within our inventory file as: hostname: “{{customer_name}}-runtime-{{ type_of_stack }} }}”
  3. {{ customer_name}} and {{type_of_stack}} are defined in an extra variable file passed to ansible-playbook via --extra-vars
  4. The Ansible variable evaluation system/lazy evaluation system correctly expands the variable during the playbook run via ansible-playbook

When run in AWX, inventory parsing occurs as a separate phase and I see the variable does not have the correct value. It remains “{{customer_name}}-runtime-{{ type_of_stack }} }}”, w/o expansion.

Matt Martz/sivel on IRC today told me the following w.r.t ansible-inventory:
> what do you mean just ansible-inventory alone? ansible-inventory does nothing with extra vars
> inventory plugins can use extra vars, so afaik that is why the option exists for ansible-inventory, but by default it doesn’t influence anything
> yeah, that is correct, --extra-vars only exists for ansible-inventory, so that inventory plugins can read it, but it doesn’t affect any vars actually listed in inventory
> lazy evaluations of variables in a play would expand it, unless AWX is marking things as unsafe, which would prevent templating

So am kind of running in a loop here as to make this work in AWX. The closest I am w/ now is to write a custom inventory plugin which would perform the expansion to make it behave like ansible-playbook does )but I don’t even know how to trigger variable expansion via Python code like ansible-playbook does).

Isn’t there another, simpler way to achieve what I want here?

The problem is, I don’t think that change made it into Ansible 2.9, which we still need to have working.
There’s no good way to determine in advance what Ansible version we have either.
So I would like to add some forms of support for extra variables in inventory imports, but we’re blocked on old version support.

Very good point on the Ansible version, I had not thought about that. Wouldn’t a custom Controller Custom Execution Environment solve this though (by including the good Ansible version?)

Thanks again for your support,

Vincent

4

Yeah, it could be that AWX is marking your variables as unsafe. That’s something that it does, because there’s a setting to control this, ALLOW_JINJA_IN_EXTRA_VARS. I don’t quite remember how inventory variables play into this, and I don’t think I understand your use well enough. I get your CLI use well enough, but I’m not sure how that is translating to objects in AWX.

After you run the inventory import and you look at the host variables, do you see the brackets there?

Wouldn’t a custom Controller Custom Execution Environment solve this though (by including the good Ansible version?)

I wish that we had metadata for Execution Environments. Because they are only pulled in the event of job runs, we don’t have any mechanism to independently pull and inspect them, or otherwise get metadata from the image registry (or image labels, I remember looking into that but couldn’t find anything usable). Some day, I expect that EE metadata will have to become a thing, but I don’t know how.

5

Hi Alan,

Yeah, it could be that AWX is marking your variables as unsafe. That’s something that it does, because there’s a setting to control this, ALLOW_JINJA_IN_EXTRA_VARS. I don’t quite remember how inventory variables play into this, and I don’t think I understand your use well enough.

Thanks for pointing out this setting, I feel we are getting closer here although it looks like this applies to job templates run, and not inventory imports.

I am running 19.4.0 and I see this setting should now be present in the UI since 19.5.0 thanks to https://github.com/ansible/awx/commit/20c4b21

I get your CLI use well enough, but I’m not sure how that is translating to objects in AWX.

From my understanding and the notes at https://github.com/ansible/awx/blob/devel/docs/inventory/scm_inventory.md#supported-file-syntax, it is ansible-inventory which is invoked to parse inventories in AWX. Looking at the AWX code, I see this as well.

After you run the inventory import and you look at the host variables, do you see the brackets there?

Yes, my point exactly. The brackets are still present after inventory import whereas the variable should have been expanded:

inventory.png
inventory.png849×573 35.5 KB

When I call ansible-inventory by hand, I also see this behavior (variables not expanded) and this matches what happens with AWX. Adding --extra-vars to ansible-inventory does not change anything w.r.t variable expansion either (matches what sivel was saying):

ansible-inventory.png
ansible-inventory.png1388×381 58 KB

It thus looks like a shortcoming with ansible-inventory and extra variables not being expanded, contrary to what ansible-playbook does when it evaluates them.

Would you happen to know how to do what I want (trigger variable expansion with ansible-inventory)?

Wouldn’t a custom Controller Custom Execution Environment solve this though (by including the good Ansible version?)

I wish that we had metadata for Execution Environments. Because they are only pulled in the event of job runs, we don’t have any mechanism to independently pull and inspect them, or otherwise get metadata from the image registry (or image labels, I remember looking into that but couldn’t find anything usable). Some day, I expect that EE metadata will have to become a thing, but I don’t know how.

Oh, I was just referring to directing users to build a custom EE w/ the right ansible version if they want to benefit from the feature. Automatic detection would be nice but I see this as somewhat an advanced feature (relying on metadata which is not there).

Thanks again for your assistance,

Vincent

6

I think I’m on the same page as you now. I cannot get ansible-inventory to template variables from ini-type inventory, and I expect YAML inventory is going to work the same. This is with devel Ansible-core, which allows the -e option for ansible-inventory.

Given that ansible-inventory has the -e option, I think that it is conceptually possible for the inventory plugin to template the variables. I think this does happen for the “auto” plugin in some scenarios (that was the point of bcoca’s patch), but doesn’t for these static inventory types. There might be a reason they didn’t want to allow this, and if you tried to submit a patch, I’d expect there might be pushback. Maybe there should be an option to the ansible-inventory CLI to tell it to template hostvars (like how playbooks do). Anyway, this path looks hard.

Is there any reason you can’t be satisfied to delay templating until the playbook run? So leave {{}} in the host variables, but provide the necessary variables to the job template somehow. With some quick testing, it seems like this should work. Here’s some scenario sketching:

https://github.com/AlanCoding/Ansible-inventory-file-examples/tree/master/static/vars_template

(attachments)

inventory.png
inventory.png849×573 35.5 KB

ansible-inventory.png
ansible-inventory.png1388×381 58 KB

7

We have the same problem. Now all the customers have a own awx instance and we make use of a var file in the github of the customer.
We are in front of moving to one central awx and manage the customers with sepate excution enviroments.
What a have now is the playbooks in a central github and every customer have its own github for inventory and host and group vars
With some common groups we define the global variables we normally had in the external var file.

The inventory is sourced from a project in AWX. You can combine multiple sources. The are are coming from our central monitoring system.
One downside is that when you use group or hosts vars you need to define that group in your inventory file that is not combined with the other source.

8

Hi Alan,

This message to thank you for your investigation + taking the time to write a smaller testcase. I am also glad you were able to reproduce my issue and see what this is about.

I have been looking into your suggestion and importing it into AWX + testing on my playbooks. I will report back with more information when I am done.

Thanks again,

Vincent

9

Hi Alan,

Sorry for the interruption, I am back full time on this issue now.

I have looked into your suggestion/repo and while I see how you can use ansible-inventory itself as an inventory script, I have a hard time understanding how this solves my issue in AWX:

  • My very first original problem was that there is no way in AWX AFAIK to specify an extra variable file in the UI of job templates.
  • In our current, non-AWX environment, each customer gets its own set of extra vars in a file (about 100 variables in the file). The contents are fed to ansible-playbook via -e @/path/to/vars_file.
  • In AWX, I cannot specify this file from the UI. Entering the values by hand is impractical (100 entries or so). I would also like them to stay in sync w/ the SCM extra variable file.
  • I have thus updated my playbooks to use an include_vars: directive to source my variables from the playbooks since I can’t get them from the extra variable argument.
  • I can get the extra variables this way in the plays but does not work for inventory variables which use the hostvars[HOSTNAME] notation, they stay unexpanded.

I have put a small testcase similar to yours at https://github.com/vrubiolo/awx-testcases. It is the variable named vincent_inv_var_jinja which has the issue/is not evaluated at inventory parsing time.

Delaying expansion of the inventory variables works indeed, I can see that if a non-evaluated inventory variable (e.g. vincent_inv_var_jinja) is provided at job template run time, it will be correctly accessible.

This does not, however, solve the problem of:

  1. Having to enter the variables by hand in the extra variables list
  2. Keeping the values in sync w/ SCM: I would have to update the job template extra variable list each time the extra variable file contents change in git.

Would you have a suggestion on how to solve/work around the problem?

Thanks again for your assistance,

Vincent