Template/File path for dependencies

[Paul_Oyston]

Hi guys,

I had a bit of an idea surrounding the template and file path for dependencies, wondering what everyone thinks of this.

At the moment when you use meta/main.yml - dependencies the template path and file path is set to the role that you’re running in. For example if you were to use a example.nginx role the template path would be something like /etc/ansible/roles/example.nginx/templates.

When you’re calling dependencies it’s likely that you may want to override a template or pass a template in to the role. For example you may wish to do something like:

`

file: /etc/ansible/roles/example.app/meta/main.yml

[Michael_DeHaan1 (Michael DeHaan)]

I don’t understand what you are suggesting to implement, though this is probably a good discussion for ansible-devel instead of the main list.

We would like to see what syntax you are proposing though it sounds like you might be saying to search the various role dependency chain items to find the first file up the chain starting at the current role.

Really, role deps shouldn’t be looked at like inheritance though - they are simply NOT inheritance. They are things to do first.

It might be better to pass a parameter into the role instead (like you have) with what filename to use, I’m unclear as to why you think it needs an absolute path though.

[Ernest0x]

But we could make role dependencies allow for template overrides /
injections. Judging by discussions in several related topics, it seems
that this would be a feature that people would want to have, so I do not
understand why you are against some little level of inheritance for the
templates. In fact, I can see great potential if we make role
dependencies smart enough to allow template inheritance between
dependent roles. Jinja already has this power built-in. Why don't we
take advantage of it? Imagine an 'apache' role that provides a vhost
template with a "{% block extra %}{% endblock %}" jinja block and a
'wsgi_app' role which depends on 'apache' role and provides content for
that 'extra' block. That would allow for a really modular design of
roles, instead of having to maintain huge, monolithic roles that do
everything with a lot of conditional includes and passing "strange" role
parameters. Role parameters are exactly what you want for things such as
'http_port' or 'bind_address', but using them for practically choosing a
"more specific role for the role" is far from elegant design.

[Michael_DeHaan1 (Michael DeHaan)]

“But we could make role dependencies allow for template overrides /
injections”

I find this kind of language hard to understand.

", I can see great potential if we make role
dependencies smart enough to allow template inheritance between
dependent roles. "

I don’t even know what this means.

Ansible isn’t a programing language, I would find this easier if we did not try to use the phrase “inheritance” where it didn’t make sense.

Did you mean including a template in another template?

[Ernest0x]

The phrase “template inheritance” is what is officially used in the jinja2 docs (). I am copying from jinja2 docs: “Template inheritance allows you to build a base “skeleton” template that contains all the common elements of your site and defines blocks that child templates can override.” So, what I am suggesting is adding a new possibility on top of role dependencies, which would allow to write a template in a dependent role that extends a template of a role it depends to. In other words, a role would be allowed to have child templates that provide content for blocks defined in a ‘skeleton’ template of a role it depends to. That means when a role is being deployed because of a role dependency, its template tasks could use blocks possibly provided by any role that depends on it and is included in the active dependency chain. I have already given an example of a ‘wsgi_app’ role that depends on an ‘apache_role’. The ‘apache_role’ would provide a template with all common things as well as an “extra” block, which could be overridden by the “wsgi_app” role in order to provide wsgi-specific functionality to the apache configuration. All this allows for a really modular design of roles. An ‘apache’ role with a good skeleton template would allow to have a great variety of dependent roles such as ‘fastcgi_app’, ‘wsgi_app’, ‘django_app’, ‘ruby_on_rails_app’, etc., without having to repeat common functionality on each role and avoiding the creation of a huge ‘apache’ role that would try to do everything driven by a great amount of parameters for choosing the appropriate templates and a spaghetti of conditional includes. Ansible is not a programming language, but it already contains a powerful template language. Why not take full advantage of it for the benefit of a better role design?

[Michael_DeHaan1 (Michael DeHaan)]

“That means when a role is being deployed because of a role dependency, its template tasks could use blocks possibly provided by any role that depends on it and is included in the active dependency chain.”

The problem is 95% of our users including myself have trouble with this wording

We avoid things we can’t explain, and that would confuse everyone when they read it.

It’s worked pretty well as a guideline in making every language decision in this project so far

I would still maintain that there are some (usually mostly coming from Chef) that try to solve too many problems with role dependencies. They were added to end a certain class of repeated question, but in general, they are overused and mostly not needed.

Role dependences are not an inheritance mechanism, they are a “run these roles before these roles” mechanism.

[James_Cammarata]

Absolutely. Role dependencies are more like package dependencies - if package A depends on package B, it means you need to install package B first.

In no circumstances should package B include files that package A is going to overwrite. It would make things extremely confusing and impossible to implement, especially when you start talking about multiple levels of dependencies, and what should happen if there are multiple dependencies (package B and C require package A, but both override a file in A, who wins?).

[Ernest0x]

I am not sure if that “95%” is valid… I don’t think that template inheritance is rocket science. Thousands of users (e.g. including people in the great community of Django, which has a template language very similar to jinja2) are already using it for better design and reusability. Please, think of this workflow: - You define an empty block called ‘extra’ in a template file named “mytemplate.j2” belonging to role A. - You write a role B that depends on role A. In the dependency statement you state that: a) Role B depends on role A b) Templates of role A can be overridden by templates provided by role B. - You define a block named “extra” in a template file also named “mytemplate.j2”, but belonging to role B. Obviously, you put in that block content that is specific for role B. - When role B is applied, it first calls role A. - Role A, when it is called by role B, renders “mytemplate.j2” with the “extra” block overridden with content provided by role B’s “mytemplate.j2”. Do you really think that the above is something that normal users could not understand?

[Ernest0x]

I don’t think your concerns are valid. The last role in the dependency chain should always win. This is how it already works in systems where template inheritance can be applied (e.g. Django). Also, I propose an explicit directive to enable template inheritance in a role dependency statement (turned off by default), so people will always know what is going on.

[Michael_DeHaan1 (Michael DeHaan)]

Ok so it’s still not “inheritance” (this is an uneducated misuse of the term by Jinja, it’s encapsulation, quite a different concept), ansible roles themselves are definitely not inheritance.

(I’ve also written a lot in Django)

Ultimately the problem is we have an irresolveable communication gap about how you are describing your technical idea.

It seems to me you are saying when a template includes another template it should also look in the “template/” directories that belong to the dependent roles, so it can find things there, but this should only apply when a template includes another template.

Ultimately, I don’t think this applies well to Ansible.

Ansible looks at role dependencies just like package managers look at package dependencies. They are things that need to be applied first, but there is no concept of search paths or inheritance/encapsulation applied.

[Adam_Morris]

For the specific example given: apache role and wsgi role there may be a simple, elegant, solution. Apache can include other configuration files, I don't know if it can include directories. If so include a directory. If not use the assemble module to create your included file.

Then the wsgi role just needs to drop a fragment or a complete file.

Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?

So is there another use case that we can discuss.

I'm thinking that they wish they'd called them prerequisites not dependencies now.

Adam

[Ernest0x]

I used ‘template inheritance’ to have exact reference to a Jinja’s feature that I propose to take advantage of in role dependencies. Sure, terminology could be wrong and that’s why I am always using examples. Basically, yes. I am thinking of something that is similar to what is done with django’s “template/” directories, but only when it is explicitly asked in the dependency statement and only if a role is triggered due to a dependency statement of that kind. Please, give it some extra thought. Think of what I have already said about better modularity in the design of roles. Think of a great potential to avoid complicating roles as variety in functionality increases. Now, people often end up in writing monolithic roles that are controlled by “choose your template” parameters and piled up include statements (think of the ‘apache’ role example). This could be done better. Also, since I propose for an explicit directive in the dependency statement, current Ansible projects are not going to be affected and, without that directive being explicitly set, Ansible would still look at role dependencies the way it does now.

[Ernest0x]

For the specific example given: apache role and wsgi role there may be a simple, elegant, solution. Apache can include other configuration files, I don't know if it can include directories. If so include a directory. If not use the assemble module to create your included file.

Then the wsgi role just needs to drop a fragment or a complete file.

Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains... how far back should it go?

It is true that software that has support for include directories can
help, but it become less helpful as you add more levels in ther hierarchy

For example, consider a tree like this:

                                       'apache' role

[Ernest0x]

Oops, the line-wrapping algorithm destroyed my tree:
http://pastebin.com/SH5nTFEk

[drybjed]

Sorry, replying from phone.

For the specific example given: apache role and wsgi role there may be a simple, elegant, solution. Apache can include other configuration files, I don’t know if it can include directories. If so include a directory. If not use the assemble module to create your included file.

Then the wsgi role just needs to drop a fragment or a complete file.

Template encapsulation seems like it could get complicated fast. Particularly if you have dependency chains… how far back should it go?

It is true that software that has support for include directories can
help, but it become less helpful as you add more levels in ther hierarchy

For example, consider a tree like this:

‘apache’ role

±------------------------±------------------------+

‘wsgi_app’ role ‘php_app’ role
‘passenger_app’

±----------------------+
±-----------------------+

‘django_app’ role ‘flask_app’ role ‘redmine_app’
role ‘gitlab_app’ role

Deploying all or a subset of the above roles on the same server could be
most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same server behind nginx. I don’t see how including templates in nginx role from other roles to make that possible could help me.

So is there another use case that we can discuss.

I have a ‘common’ role that deploys /etc/hosts. In ‘cluster node’ role
luster nodes I also add lines for all nodes in the cluster. Now I am
using a really ugly loop around lineinfile for this that took me some
time before I made it place the content properly. Try to use lineinfile
with a relatively complex data structure and you will remember my words.
Instead, it would be far more easy to write an “extra_hosts” block and
do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a multiple host environment is so 1960…

I’m thinking that they wish they’d called them prerequisites not dependencies now.

Seconded.

Maciej

[Ernest0x]

I am sure that you can deploy all these and that you are happy with the result. However, I took a look at your nginx role and it seems you are doing exactly what I believe people could (and should) avoid with template encapsulation. It seems you are creating a “super-role” merging together what could be at least 3 separate roles connected through dependencies: nginx, fastcgi_app, php_app. If you prefer that monolithic way of doing things, it’s fine. You will be one of those who will not be excited by a feature like the one I propose and probably never use it. Because it would be overkill for a 2 or 3 - node cluster. But you are missing the point. dnsmasq would still have a common configuration on most hosts, as well as an extra configuration needed for cluster nodes.

[drybjed]

> It is true that software that has support for include directories can

> help, but it become less helpful as you add more levels in ther hierarchy
>
> For example, consider a tree like this:
>
> 'apache' role
> >
> +-------------------------+-------------------------+
> > >
> >
> 'wsgi_app' role 'php_app' role
> 'passenger_app'
> > >
> +-----------------------+
> +------------------------+
> > > >
> >
> 'django_app' role 'flask_app' role 'redmine_app'
> role 'gitlab_app' role
>
>
> Deploying all or a subset of the above roles on the same server could be
> most easily done if my proposal for template encapsulation was available.

Maybe most easily, but definitely not impossible. Currently I can deploy
gitlab, gitlab-ci, phpmyadmin, phpipam, etherpad, owncloud, all on the same
server behind nginx. I don't see how including templates in nginx role from
other roles to make that possible could help me.

I am sure that you can deploy all these and that you are happy with the
result. However, I took a look at your nginx role and it seems you are
doing exactly what I believe people could (and should) avoid with template
encapsulation. It seems you are creating a "super-role" merging together
what could be at least 3 separate roles connected through dependencies:
nginx, fastcgi_app, php_app. If you prefer that monolithic way of doing
things, it's fine. You will be one of those who will not be excited by a
feature like the one I propose and probably never use it.

Actually my playbook does exactly what you suggest - 'fastcgi_app' role
(say, gitlab), uses 'nginx' role as dependency and passes its configuration
via hash variable. Similarly, PHP5-based role (like ownCloud), uses 'php5'
role as a dependency to create custom php5-fpm pool for itself, and 'nginx'
role as another dependency similarly with its own custom configuration
passed via hash variable.

Inside 'nginx' role, a server template generates server configuration for
each "parent" role, puts it in its own file in /etc/nginx/sites-available/
and enables it by symlinking it to /etc/nginx/sites-available/.

The idea behind this was to abstract nginx configuration and management
away from gitlab and owncloud roles, so that you could define for example
your own SSL certificates or other server-wide configuration in one place.
I treat Ansible roles which support dependencies like these like blackboxes
(from the perspective of a role that uses them as dependency) with common
interface - how nginx configures itself shouldn't matter for gitlab. That
way I could replace nginx with apache2 (with similar interface prepared
beforehand) and gitlab role should be none the wiser.

In fact, I stumbled on a similar problem you describe just recently, while
planning WordPress integration - you can see here
http://codex.wordpress.org/Nginx that proper installation requires a lot
more configuration in nginx that should be necessary to include in base
nginx role. Because of that I'm thinking about creating a directory inside
/etc/nginx/ where other roles could place their own nginx configuration
(either via copy or template) and include it in their nginx server
configuration - but not on Ansible level, but using nginx "include" option.

As you can hopefully see, I don't want to create a super-role for nginx,
but instead I want to include just enough logic and room in that role so
that other roles could use it comfortably. It might not be optimal or
simple, as Michael would suggest, but it's a tradeoff between simplicity
and customization which I feel will be beneficial for me in the long run -
instead of creating specific nginx roles for many servers, I'm trying to
create one general and reusable one.

I have a 'common' role that deploys /etc/hosts. In 'cluster node' role
> luster nodes I also add lines for all nodes in the cluster. Now I am
> using a really ugly loop around lineinfile for this that took me some
> time before I made it place the content properly. Try to use lineinfile
> with a relatively complex data structure and you will remember my words.
> Instead, it would be far more easy to write an "extra_hosts" block and
> do things in pure jinja.

Why not deploy for example dnsmasq? I mean, using /etc/hosts for DNS in a
multiple host environment is so 1960...

Because it would be overkill for a 2 or 3 - node cluster. But you are
missing the point. dnsmasq would still have a common configuration on most
hosts, as well as an extra configuration needed for cluster nodes.

Fair enough - I don't know what are your needs. I'm using dnsmasq on a
laptop for 3-4 virtual machines in my development environment and I don't
feel it's a waste of resources, but a convenience. I can define a CNAME
or A in one place (in my inventory, in fact) and all hosts immediately know
about it.

As for dnsmasq configuration on hosts other than the server (I hope you
didn't plan to install it on all your servers), it's just one line in
/etc/resolv.conf which is managed by operating system anyway, via DHCP. So,
no extra configuration for me.

Maciej

[Ernest0x]

But it seems you do have some specialized functionality embedded in your nginx role. For example you do fastcgi stuff. By fastcgi stuff I mean things such as configuration sections, structures and directives (e.g fastcgi_index, fastcgi_pass, etc.). I do not mean configuration data (such as {{ document_root }} or {{ fastcgi_script_name }}). If I could, I would abstract it further and put fastcgi stuff in a separate role that depends on nginx role. You would ask why to do that? Shouldn’t the nginx know how to deploy fastcgi applications? Well, as you saw while planning for WordPress integration, sooner or later you will come across with a fastcgi (or php, or whatever) application A that needs different fastcgi configuration than the fastcgi configuration needed by a fastcgi application B. So you would need to abstract your fastcgi logic further. But this will also happen with your wsgi logic (which could also have its own universe with one app being deployed with modwsgi, another one with uwsgi, etc.), your php logic, your phusion passenger logic, etc. If you keep all this stuff in a single role, you will end up with a difficult to read and maintain project. So, I think splitting your deployment in multiple roles would make your life much easier. That said, I really appreciate the way you are thinking, as you do recognize the need for abstracting your basic roles, and I see that you have done what the current tools allow you to do for that purpose. It is just that Ansible does not (yet) give you a mechanism to modularize your roles in a cleaner way. Since my suggestion for allowing template encapsulation between roles is not (yet) implemented, I would encourage you to go that way and use the nginx “include” option. However, I believe that jinja’s “template inheritance” would be more flexible than the basic “include” mechanism provided by systems like nginx. For example, with jinja you could have a section with some desired content when the role would be deployed standalone, or override that section with some different (more specific) content when the role is deployed as part of a dependency. I am not sure if that can be done (easily and cleanly) with an ngix type of “include” mechanism. Also, keep in mind that there are systems that they do NOT provide such an “include” mechanism at all.

[drybjed]

I know that different applications or frameworks require different
configuration. For this purpose, in my 'nginx' role I've included a way to
select which server template is used to generate nginx configuration.
Currently there's only one (
https://github.com/ginas/ginas/blob/master/playbooks/roles/ginas.nginx/templates/etc/nginx/sites-available/default.conf.j2which
is selected by default or by setting item.type: 'default'), but that
doesn't mean there couldn't be more - I plan to write separate one for
wordpress, for example.

Yes, with time, number of these templates will grow. But hopefully I will
be able to write them abstract enough that each one will be usable for many
applications. And you can even combine different files via Jinja extension
blocks - I currently don't use it in 'nginx', but you can see example in my
'apt' role which generates Debian Preseed configuration for normal and
destructive install (
https://github.com/ginas/ginas/tree/master/playbooks/roles/ginas.apt/templates/srv/www/sites/default/public/d-i/wheezy)
- files need to be in the same directory, and AFAIK using different
directories doesn't really work, but I haven't tested that extensively. I
plan to use similar mechanism in the future to build nginx server
configurations from parts (in fact my 'type' variable was planned with this
scenario in mind).

Other way around this would be to split nginx role into base and templates
and use them somehow via dependency. Problem with that is current nginx
configuration structure - you would probably want to create separate
directory for templating role, like /etc/nginx-templates/ and keep your
generated configuration there to not interfere with base nginx config files
- otherwise you could end up with chicken and egg problem where one role
requires /etc/nginx derectory which is created by a different role. Of
course Ansible could mitigate this with file: module, careful permissions
and still keeping your generated templates in non-standard directories
inside /etc/nginx/... It's a tricky question to answer, but I still feel
that adding separate mechanism in Ansible dependency functions to support
jinja extension blocks is a stretch and doesn't look too intuitive.

Maciej

[ramon]

Hi All,

The “inheritance” discussion keeps popping up, appearantly there’s value in there somewhere… I’d like to add my 2ct, but just with regards to using galaxy roles.

First: it’s obviously not “inheritance”, I’d like to say “override” (like variables override eachother according to precedence), and limit the scope only to files/templates.

The case is, I can find a prefect role in Galaxy, with one single problem: I’d like to use my own template instead of the included template. Currently, I cannot use the role, I must duplicate the entire thing and replace the template.

It would be nice if there was a way to use my own template, in place of the role’s template. The point has been made that this is already possible with a variable for the template name, but this would require other people to change their roles [which is absolutely an option, considering they’re open source].

The way I would expect it to work is just adding the files in a roles_path that has precedence…
It was outlined here in more detail:

https://groups.google.com/d/msg/ansible-project/9xRyYIOdSro/aGF5L7F03PoJ

But kind of got buried in talk of multiple reversed depedency inheritance injection encaptulation.

Kind regards,

Ramon