Embedding metadata in yaml files which ansible ignores

[gardar]

Hello Ansible Community,

I’m currently working on integrating Ansible with other tools and need to embed metadata directly in YAML files (such as host_vars and group_vars) that Ansible processes. The ideal solution would ensure Ansible completely ignores this metadata, treating it as invisible.

Approaches I’ve Explored:

• YAML Document Separator (---) Unfortunately, using --- to separate metadata from variables broke Ansible’s parsing of the YAML file.

• YAML Anchors: I tried defining an anchor (&my_metadata:) and skipping references to it, but Ansible still loaded the data with a “null” key.

  &my_metadata:
    one: two
    three: four
  

  turned out as:

  null:
    one: two
    three: four
  

• _ansible_ Prefix: Inspecting the Ansible source code, particularly clean.py, I found out that it strips away variables starting with _ansible_. Leveraging this, I experimented with using a variable like _ansible_my_metadata_ to store metadata. It seems Ansible ignores this variable—no errors are thrown, and it does not appear when I print all variables with {{ vars }}.

• Comments: I considered using comments for storing the metadata but parsing them is not ideal, as the external tool I’m using leverages PyYAML, which struggles with it.

• Obscure Keys: I also considered using obscure keys for my metadata, but there’s a risk that Ansible could potentially use or process the metadata, which I wish to avoid.

• Separate Files: While storing metadata separately works, I was hoping to keep everything within the Ansible YAML structure for enhanced portability.

Questions for the Community:

1. Are there any other methods for embedding metadata that Ansible will reliably ignore?
2. Are there potential drawbacks to relying on the _ansible_ prefix behavior that I should be aware of?

[Denney-tech (Caleb Denney)]

Welcome to the forum @gardar !

Can you expand a little more on what this metadata is intended for? Why would you want host_vars/group_vars to exist, but be ignored by Ansible? When and where do you want this data to be used, and by what if not Ansible?

My first thought on how you could define your metadata in an ansible way that isn’t immediately consumed, is to define them in a role. Then that metadata is only available when you call the role, not when the inventory is initialized. Otherwise you could have vars directory in the root of your project with the metadata there as a vars file.

Edit: Honestly, you could just put it anywhere outside of inventory, and tell your tool where to find it and it should work. ./vars/meta.yml ./.meta.yml ./.toolname.meta.yml or something along these lines would work. Ansible doesn’t consume vars files outside of its inventory unless it’s in a role’s defaults or if you explicitly include_vars: files.

[gardar]

Thanks for the reply, let me provide more context on the specific use case and why I am exploring these particular options.

I’ve developed a script that interfaces with the Red Hat Satellite/Foreman API to export Ansible variable overrides and convert them into the appropriate Ansible host_vars and group_vars format.

In Satellite/Foreman, Ansible variables are managed differently than in the standard Ansible CLI environment. Variables in Satellite/Foreman can have override matchers that determine their application scope. For instance:

• To apply an Ansible variable to a specific host group (akin to placing it under group_vars/mygroup.yml in Ansible), you would set an override matcher in Satellite/Foreman like hostgroup = mygroup.
• To target a variable to a specific host (host_vars/myhost.yml), you define an override matcher such as fqdn = myhost.

Moreover, it’s possible to define additional matchers based on attributes like operating system, domain, location, subnet, etc.

My script extracts these variables and their corresponding matchers from Satellite/Foreman and structures them into the format that Ansible can understand (host_vars, group_vars).

The next phase of my project involves importing these variables back into Satellite/Foreman. However, due to the differences in how variables are handled between Satellite/Foreman and Ansible, I need to include additional details about each variable within the host_vars and group_vars files. These details are crucial for correctly placing the variables back into Satellite/Foreman.

My goal is to embed this additional metadata within the Ansible variable files in a way that Ansible itself will ignore—since there’s no need for Ansible to process this metadata, and to prevent any potential issues from Ansible trying to use this extra information.

Your suggestion to use a separate directory for metadata is certainly a feasible solution, and I’m considering it as a backup plan. However, ideally, I would prefer to keep everything integrated within the existing Ansible structure for simplicity and portability reasons. Hence, I’m exploring if there’s a way to make this metadata invisible or ignored by Ansible while still being stored alongside the relevant Ansible data.

I hope this clarifies my intentions and the challenges I’m facing.

[chris (Chris Croome)]

Could you comment the metadata?

[gardar]

Thanks for the idea! Unfortunately, using comments isn’t ideal because PyYAML, which I’m using to parse the YAML, doesn’t keep the comments—it just ignores them Load comments · Issue #90 · yaml/pyyaml · GitHub. I’d need to come up with a workaround to handle the comments separately, which feels a bit clunky and hacky. I was hoping there would be a more straightforward way to do this.

[Denney-tech (Caleb Denney)]

Wow! Thank you for the in-depth explanation of your objective.

In regards to the initial issue and what you’re trying to do, even though you’re trying to structure Satellite/Foreman inventory vars in an Ansible inventory like manner, you’re wanting to preserve how it appears in Satellite/Foreman and be able to post changes back where they came from. I suggest that you try not to emulate Ansible inventory structure, because it isn’t from Ansible inventory. You should try and maintain parity with Satellite/Foreman, so that you don’t get confused while transposing between the two, and you won’t turn them into Ansible inventory vars that get consumed automatically.

Something like: foreman_vars/parameters.yml

---
hostgroups:
  - name: group1
    vars:
      - name: var_name
        value: var_value
        type: <foreman_type>
hosts:
  - name: host1
    vars:
      - name: var_name
        value: var_value
        type: <foreman_type>
orgs:
  - name: default_org
    vars:
      - name: var_name
        value: var_value
        type: <foreman_type>
...

etc.

At least, that’s how I would approach it. You might even be able to turn this into a jinja2 template if you wanted to define parameters in Satellite/Foreman as code.

That said, there’s a lot of tooling out there already for these sorts of things. There’s the redhat.satellite/theforeman.foreman and redhat.satellite_operations/theforeman.operations collections, which contain roles for building Satellite/Foreman as code.

[gardar]

It would definitely make life more easy if I just wanted to make a backup of the variables from Satellite/Foreman.
However my goal is twofold.

• I want to be able to have the variables in Satellite/Foreman backed up and stored in a git repo, as Satellite/Foreman doesn’t have any version control for ansible variables and it’s super easy to accidentally delete variable content in Satellite/Foreman.
• I want to be able to consume the variables with ansible as host/group vars, so that I can execute ansible outside of Satellite/Foreman with the same variables as in Satellite/Foreman.

There’s actually a theforeman.foreman.foreman inventory plugin available, in my script I’ve matched the host group structure that the inventory plugin has so that I can currently use my exported host_vars and group_vars files with the foreman inventory.

[Denney-tech (Caleb Denney)]

The vars can be exported by theforeman inventory plugin as part of the dynamic inventory, so your hosts will consume and inherit vars exactly as they are defined in theforeman.

[gardar]

Absolutely, exporting the variables or using them from the inventory plugin is no problem, the challenge is converting the vars from yaml host/group vars back to satellite/foreman

[Denney-tech (Caleb Denney)]

Right, so there’s no need to reinvent the wheel. Script your exported vars in a way that’s easy for you to work with when importing it back to satellite/theforeman (potentially through the existing roles), but let the inventory plugin worry about consuming it when you actually want it in your inventory. Don’t try to layer static inventory vars on top of the dynamic inventory.