Lookup-plugins in general & lsearch in particular

Archives Ansible Developer
1

Hello,

I have a pull-request for a linear-search (lsearch) in text files
lookup-plugin pending [1] and Michael requests I discuss it on the
-devel mailing list, so here I am. :wink:

In general, I feel there can't really be enough lookup plugins; they're
a marvelous feature of Ansible. The reason for lookups in plain (flat)
text files is for an environment which has neither Redis nor any other
fancy data store. (I stole the name 'lsearch' from my favorite MTA:
Exim, which has a myriad lookup types, lsearch being one of them.)

Case in point are templates which need to be pushed out to hosts. These
templates have a connection-specific string which differs on a per/host
basis. The following example looks up {{inventory_hostname}} and inserts
the resulting string into the connection:

        # {{ ansible_managed }}
        dbname: s1
        connection: tcp://{{ lookup('lsearch', inventory_hostname) }}/

The lookup-file itself looks like this:

        host1: 127.0.0.1
        host2: 192.168.1.3
        ...

(BTW, it's a bit off-topic, but IIRC Michael announced 30+ new modules
for the next version of Ansible, so what's bad about another few lookup
plugins? :wink:

Best,

        -JP

[1] https://github.com/ansible/ansible/pull/4846

2

I’ll explain the philosophy first, independent of the lsearch discussion.

This has always been the philosophy of the project, so it should not be shocking – but new folks might not have figured this out before.

So as ansible takes on everything into core, there’s more and more to skim over in documentation, and quickly there’s more to hold in your head.

While this is the natural evolution of organic software, it’s my goal to constrain and unnaturally select things so Ansible stays moving towards where we want it to go.
Ansible must be different, consistent, and minimal in terms of core language, and clean in terms of syntax.

I already see there are cases where people make Ansible content “too clever” and don’t simplify, so in giving more rope, we run the risk of greater inconsistency.

Modules are well categorized already, but my general rule is the longer something takes to explain, the more
confusing it can be, and we need to be more careful in deciding whether we need something like that or not.
Ansible legitimately wants to provide back-pressure against additions that complicate explanations or provide obscure ways to do things.

In most cases, this means providing ways to avoid {{ lookup }}, when there is a cleaner way to represent that syntax.

Syntax is very very important to me.

An individual lookup plugin is potentially bad when it’s hard to explain or would encourage confusing practices, and is too close to the way something already works.

In the case of what you have with the “lsearch”, it seems (currently) totally solved to me by:

vars_files:

  • host_ips.yml

{{ hosts[inventory_hostname] }}

And then just defining it as a YAML hash. Data structures already provide the ability to do that kind of lookup.

When I originally read the example for ‘lsearch’, it appeared to be searching for lines in a plain text file, which doesn’t seem like a data entry mechanism we want to encourage.

Maybe I’m misunderstanding the use case, and if so, I need to understand it more.

As for “more modules always being a good thing”, it’s really not always – not when a module is going to raise more questions than it solves – lineinfile is a good one for that :slight_smile:

I’m not saying this applies to lsearch but I’m still trying to understand the case for it.

I haven’t come to a decision yet, but the great thing about the modules directory and the configurable plugin paths are that – even if we decide to say no to something – it’s still user

extensible.

3

Michael,

Modules are well categorized already, but my general rule is the longer
something takes to explain, the more
confusing it can be, and we need to be more careful in deciding whether we
need something like that or not.

ACK.

In the case of lookup plugins, I strongly doubt it's difficult to
understand: a bit more description in the Ansible documentation would
suffice IMO to bring the point accross, describe the individual plugins
we have and give some examples. I don't think that's hard. :wink:

In most cases, this means providing ways to avoid {{ lookup }}, when there
is a cleaner way to represent that syntax.
Syntax is very very important to me.

I agree that {{ lookup(abc, def) }} looks ugly, and I for one miss the
cleaner old (additional?) syntax which used to exist [I believe that's
gone?]

        $PIPE()
        $REDIS()
        ...

In the case of what you have with the "lsearch", it seems (currently)
totally solved to me by:

vars_files:
    - host_ips.yml
{{ hosts[inventory_hostname] }}

May well be; I personally don't particularly care for that syntax...

lsearch was intended to be a building block for more
plugins. The project recently accepted my `etcd' lookup (thanks), and as
I said earlier: I think lookups are very useful.

When I originally read the example for 'lsearch', it appeared to be
searching for lines in a plain text file, which doesn't seem like a data
entry mechanism we want to encourage.

It does exactly that: search for lines in a plain text file which is
neatly placed next to the other roles' files.

As for "more modules always being a good thing", it's really not always --
not when a module is going to raise more questions than it solves --
lineinfile is a good one for that :slight_smile:

Yuck & LoL. :wink:

        -JP

4

In the case of lookup plugins, I strongly doubt it's difficult to
understand: a bit more description in the Ansible documentation would
suffice IMO to bring the point accross, describe the individual plugins
we have and give some examples. I don't think that's hard. :wink:

I was asking for an explanation of lsearch in this particular case as the
pull request didn't sell me on it.

I'm willing to be sold.

> In most cases, this means providing ways to avoid {{ lookup }}, when
there
> is a cleaner way to represent that syntax.
> Syntax is very very important to me.

I agree that {{ lookup(abc, def) }} looks ugly, and I for one miss the
cleaner old (additional?) syntax which used to exist [I believe that's
gone?]

        $PIPE()
        $REDIS()
        ...

This is a seperate point, but my thought is really lookup plugins should be
exposed directly as functions as well.
We've got a pull request for that somewhere.

{{ pipe('/bin/foo') }}

If not functions, then maybe even filters, though that's not quite so good
with multiple arguments, it does look cleaner:

{{ '/bin/foo' | pipe }}

Though we're crossing the streams, lookup plugins might fit as filters.

Some of them, anyway.

Beside the point though...

> In the case of what you have with the "lsearch", it seems (currently)
> totally solved to me by:
>
> vars_files:
> - host_ips.yml
> {{ hosts[inventory_hostname] }}

May well be; I personally don't particularly care for that syntax...

If it's already YAML, I don't think we need a custom function to get a
dictionary key out of YAML, extra unneccessary hoops.

etcd or redis or... etc ... are very different data sources that require
different ways to read, more or less.

lsearch was intended to be a building block for more
plugins. The project recently accepted my `etcd' lookup (thanks), and as
I said earlier: I think lookups are very useful.

I don't understand how it would be a building block, please elaborate?

> When I originally read the example for 'lsearch', it appeared to be
> searching for lines in a plain text file, which doesn't seem like a data
> entry mechanism we want to encourage.

It does exactly that: search for lines in a plain text file which is
neatly placed next to the other roles' files.

a plain text file is a bit different, in the above, I saw colons and the
search didn't show the colons.

Also a plain text file with colons in it, feels a little like making up a
data format, which I'm very unlikely to
want to include in core.

CSV, etc, is gross, but established.

key: foo
bar: baz

etc, that's something a bit off to the side and would encourage people to
not store things in YAML and then the next thing you know someone's asking
why this 'lsearch' doesn't support structured data and we've soon
reinvented a new monster :slight_smile:

5

a plain text file is a bit different, in the above, I saw colons and the
search didn't show the colons.

First off, no colons in text file (my mistake -- dead tired), no CSV, no
YAML, no *plans* for a monster. :slight_smile: Plain text, key-whitespace-value.

I'm willing to be sold.

It's the UNIX spirit. Plain text. No frills. No database. Easy to
create, grep, etc.

That's my sales pitch.

(Off-topic with regard to lsearch, but there will be others at some
point: LDAP, NIS (yup, there are still people using that), MySQL, PgSQL,
SQLite3, ... I'm not saying I'll be doing those, but I know people who
want that sort of thing in lookups; maybe not a lot, but they're there. :slight_smile:

Best,

        -JP

6

Hmmm…

While that makes sense, there are also some bad things in the Unix world, like sendmail configuration :slight_smile:

For core, I’ll likely accept lookup plugins for more or less standard (named) data formats as well as open source network data stores like we’ve already done for things like Redis.

how about CSV and TSV?

assume a file like

key,value1,value2

thus {{ lookup(“csv”,“file.foo”,"key’)[1] }}

returns value1

and without the brackets

[‘key’,‘value1’,‘value2’]

??

LDAP and NIS and others seem pretty fair game.

Another thing we sort of need to solve is the idea the way forking in Ansible works and such, many of these are executed rather often. In some cases, with a large number of hosts, we may want lookup plugins to be evaluated once and cache, or at least cache per fork (easier than just once). This may imply lookup_once existing and needing to IPC back to the main Ansible process.

7

Michael,

how about CSV and TSV?

PR submitted https://github.com/ansible/ansible/pull/4987

thus {{ lookup("csv","file.foo","key')[1] }}

I've had to change the syntax a bit, but otherwise it works that way.

Regards,

        -JP