bash_completion issues and some ideas

Hello Ansible team!

I’m having great time with Ansible and I want make my experience (and Ansible) better.
My story is related with administrating over 500 nodes with legacy and hacks, so I can’t do everything “The right way” (and this is just “my personal” inventory part :wink: ). That’s why I had to start thinking about extra options. I will describe a few thoughts and ask you for discussion. And my apologies for possible mistakes in my English. :slight_smile:

So, let’s go!
I am using my Ansible bash_completion (full of hacks) and it helps me now. So I promised to share it with community. Today, I started improvement work on it… and had first blockers. I tried to make it really cool and support all possible completions.

First one is inventory completion (groups, hosts). I have my inventory-script with extra feature – show all hosts and groups (I have got about 60+ groups and 450+ nodes). And that is really “The Feature” of CMS for me (and all my colleagues). Somewhere we have poor infrastructure, sometimes we have special tasks – so we use ansible cli for per host/group shell executions (no playbooks).
For me it is a very big reason to use second field (host_group_pattern) with bash_completion. And I failed with it today.
I can’t do in my terminal this:
ansible my_cool_group:!dead_hosts:!unused.example.com -m shell -a ‘uptime’
Bash use symbols “!”, “&” for its internal things. So we can do an “ugly hack” (this is just my opinion – no offence :slight_smile: ):
ansible “my_cool_group:!dead_hosts:!unused.example.com” -m shell -a ‘uptime’

But it is (maybe) uncompletable and totally unhandy in my background/case. So let me propose you another syntax – we can replace this characters this way:

  • “!” => “-”, # this is like numbers: positive and negative :wink:
  • “&” => “^”, # conjunction symbol
  • “:” => “,” (optional). # I really feel comma for this case – we use commas everyday for lists (appendings)
    I totally understand about changing this part in so huge project, so it could be smooth. “–future” option for this behavior for example. Think about it. My UX experience is all about this thesis.

Let’s go further, second thing: in my practice we have a very good agreement about group naming – start group name with special character (I prefer “@” and someone use “%”).
UX remark: it helps our vision to extremely fast distinguish hosts from groups – it is about, how our sight works (visual pattern at the first word/name symbol).
I’m asking you for keeping this ability (support some “special” characters in group names) and propose to think about this idea for Ansible “Best practice” (not insisting here, just sharing thoughts).

Third one!
I have my hack with inventory script for getting group and host lists, but Ansible doesn’t have it for now. I think, it is a good idea to separate inventory features from Ansible part (CMS) – I am about “–list-hosts”. If we extract this to another “tool” (for example “ansible-inventory”), we can make ansible (tool) keys more focused and at the same time make Ansible (project) more powerful about inventory integration (I will write a bit more about this later in this message).
Ansible-inventory can be improved by “–list-groups” (minimal required key) and other. And more! We could pass parameters to Inventory script – it would proxy any Inventory system functionality to ansible toolset. One tool – infinite abilities, homogeneous environment (as a UX guy, I’m double-pointing this point! :wink: )

Fourth.
Ansible have great option “–limit”, in our practice we faced with an issue: limit hosts by variable (still in command line) – can we add this option too? # Suddenly pretty short point. :slight_smile:

Next, fifth!
Last one about my bash_completion issue. I grab options for completion from “ansible --help” (and the same for other tools), but it would be much better to have some “contract” here – agreement about fetching list of keys. And the questions:

  • can I rely on current “–help” output format?
  • or you can prompt an existing way (maybe I missed it)

Great, that’s all questions based on bash_completion work. I think I have some more notes, but there are a bit outdated. I promise to examine them and to patches or reports. :wink:

Last part, a little bit more about my background – I am going to start Inventory system project and I want to have integrations with Ansible. In my environment we really need inventory system more powerful than ini-file. We work with our current solution in a bit more complex way. And I really feel to use one toolset for close tasks. And my vision is an “ansible-inventory” “glue/proxy” tool to integrate any Inventory system with this extremely powerful CMS tools – Ansible.

Thank you very much for reading, I hope we can have a very positive and fruitful discussion. I hope Michael will join this theme.

Omg! I forgot one big thing: I would be glad to do some work by myself, so it’s not just feature-request. I want to discuss, take an approvement and start working.

Hello Ansible team!

I'm having great time with Ansible and I want make my experience (and
Ansible) better.
My story is related with administrating over 500 nodes with legacy and
hacks, so I can't do everything "The right way" (and this is just "my
personal" inventory part :wink: ). That's why I had to start thinking about
extra options. I will describe a few thoughts and ask you for discussion.
And my apologies for possible mistakes in my English. :slight_smile:

So, let's go!
I am using my Ansible bash_completion (full of hacks) and it helps me now.
So I promised to share it with community. Today, I started improvement work
on it... and had first blockers. I tried to make it really cool and support
all possible completions.

Sounds like a cool project... BTW, in case you haven't got to modules yet,
check out:

https://github.com/dominis/ansible-shell

First one is inventory completion (groups, hosts). I have my
inventory-script with extra feature -- show all hosts and groups (I have
got about 60+ groups and 450+ nodes). And that is really "The Feature" of
CMS for me (and all my colleagues). Somewhere we have poor infrastructure,
sometimes we have special tasks -- so we use ansible cli for per host/group
shell executions (no playbooks).
For me it is a very big reason to use second field (host_group_pattern)
with bash_completion. And I failed with it today.
I can't do in my terminal this:
  ansible my_cool_group:!dead_hosts:!unused.example.com -m shell -a
'uptime'
Bash use symbols "!", "&" for its internal things. So we can do an "ugly
hack" (this is just my opinion -- no offence :slight_smile: ):
  ansible "my_cool_group:!dead_hosts:!unused.example.com" -m shell -a
'uptime'

But it is (maybe) uncompletable and totally unhandy in my background/case.
So let me propose you another syntax -- we can replace this characters this
way:
  - "!" => "-", # this is like numbers: positive and negative :wink:
  - "&" => "^", # conjunction symbol
  - ":" => "," (optional). # I really feel comma for this case -- we use
commas everyday for lists (appendings)

While I can see how that helps with the above, we're not in a position
where we can change things when lots of people have scripts and aliases set
to the above already, not to mention playbook conventions.

I _totally_ understand about changing this part in so huge project, so it
could be smooth. "--future" option for this behavior for example. Think
about it. My UX experience is all about this thesis.

Future seems to apply future removal, in which case we still have this
problem to a degree. I want to avoid "make it configurable" being a last
resort.

Let's go further, second thing: in my practice we have a very good
agreement about group naming -- start group name with special character (I
prefer "@" and someone use "%").
UX remark: it helps our vision to extremely fast distinguish hosts from
groups -- it is about, how our sight works (visual pattern at the first
word/name symbol).
I'm asking you for keeping this ability (support some "special" characters
in group names) and propose to think about this idea for Ansible "Best
practice" (not insisting here, just sharing thoughts).

Yeah this is probably all well and good if we were starting from scratch --
the @convention is a decent one. Alas, we are not, though I don't think
it's terribly bad either -- because 99% of the time, folks are only
matching groups (while I wouldn't write it up as such, I *would* consider
it a best practice).

Third one!
I have my hack with inventory script for getting group and host lists, but
Ansible doesn't have it for now. I think, it is a good idea to separate
inventory features from Ansible part (CMS) -- I am about "--list-hosts". If
we extract this to another "tool" (for example "ansible-inventory"), we can
make ansible (tool) keys more focused and at the same time make Ansible
(project) more powerful about inventory integration (I will write a bit
more about this later in this message).

I think a future theoretical 2.0 kinda of CLI interface would have ansible
subcommands

ansible show-inventory
ansible playbook
ansible exec
ansible pull

Alas, this really does need to wait... I don't want to be one to hold up
progress but I'll tell you know I'd want to move our P2/P3 github queue to
nearly zero before we go there and while there can and should always be a
shiny future we are looking forward to, until we can get most of all the
good things people have already done tested/merged/incorporated, and other
needs addressed, I'm not going to prioritize this -- and the sands will
likely shift underneath your pull request as we try.

So what I'd probably suggest is when we *do* decide we want to do this, we
implement that then.

Ansible-inventory can be improved by "--list-groups" (minimal required
key) and other. And more! We could pass parameters to Inventory script --
it would proxy any Inventory system functionality to ansible toolset. One
tool -- infinite abilities, homogeneous environment (as a UX guy, I'm
double-pointing this point! :wink: )

Fourth.
Ansible have great option "--limit", in our practice we faced with an
issue: limit hosts by variable (still in command line) -- can we add this
option too? # Suddenly pretty short point. :slight_smile:

It's true that Ansible doesn't have the limit by criteria features we
invented in Func (actually one of our awesome GSOC interns at the time). I
think there's two problems here.

(A) it would need some new syntax or flag as you might want to limit by
group then criteria

(B) it would require the existing /usr/bin/ansible to know to run the
Runner loop twice -- this wouldn't be catastrophic, but would take some
doing.

I've heard this request maybe 2-3 times, which is enough to make it a
thing, but it's easy to do with a playbook already:

- hosts: all
  tasks:
    - group_by: key=family-{{ ansible_os_family }}

- hosts: family-Debian
  tasks:
     - shell: echo hi

And I think I'll freely admit that while /usr/bin/ansible is super useful
(Runner), the focus of the app is largely going to be around playbooks --
and playbooks are why we invented things.

But yes, it can be done and I think it would be nice to easily use
/usr/bin/ansible to do things like ping your Debian hosts or reboot all
servers where Apache is installed, without writing a playbook. I'd
entertain patches for /usr/bin/ansible for this, provided they don't impact
runner/__init__.py.

Next, fifth!
Last one about my bash_completion issue. I grab options for completion
from "ansible --help" (and the same for other tools), but it would be much
better to have some "contract" here -- agreement about fetching list of
keys. And the questions:
  - can I rely on current "--help" output format?

I'm not really willing to say "yes" here. But I'm not anticipating change.

  - or you can prompt an existing way (maybe I missed it)

Great, that's all questions based on bash_completion work. I think I have
some more notes, but there are a bit outdated. I promise to examine them
and to patches or reports. :wink:

Last part, a little bit more about my background -- I am going to start
Inventory system project and I want to have integrations with Ansible. In
my environment we really need inventory system more powerful than ini-file.
We work with our current solution in a bit more complex way. And I really
feel to use one toolset for close tasks. And my vision is an
"ansible-inventory" "glue/proxy" tool to integrate any Inventory system
with this extremely powerful CMS tools -- Ansible.

Excellent, the whole reason external inventory scripts in Ansible exist are
to give people more options in terms of organizing how they organize [sic]
-- there are a lot for various cloud providers but it can pull from
anything you want.

I will look at ansible-shell project, added to my research list.

Let’s try to talk about Ansible 2.0. In big communities and companies changes are hard to move i production – it is the reality and I’m ok with it. So I guess, Ansible 2.0 would be released after 6 months since today (is this kind of true?) for example. First point – I’m not pushing my wants to Ansible 1.6.
Second one – as I feel, we don’t live in “Java world” with 100% backward compatibility. And Ansible migrated to Jinja templates – I fixed all my roles and playbooks (added “{{” and “}}”), you warned me about it for a few months. It was ok. I feel this syntax changes ([:!&] → [,^-]) are the same situation.
Third point – ansible and ansible-shell are good for their own cases, one tool can’t cover everything.
Fourth – this changes will give benefits (yes, I understand some discomfort about migrating – and it not killing).
Fifth, last one – ansible (command/tool) already have almost everything possible needed for non-playbook usage. Why not to improve it? Unified environment is much better for daily usage, so I always vote for founded unifications.

It seems reasonable for me to believe this changes can be done in Ansible 2.0. Are you agree with this way?
As I’m not a terrorist ( :slight_smile: ), I think I can do some simple completion for now and make one better for Ansible 2.0.

P.S. I skipped other questions for now (I’ll return to them after this one), because I am better in focused discussions – on point in a time. I hope, it’s ok.

I will look at ansible-shell project, added to my research list.

Let's try to talk about Ansible 2.0. In big communities and companies
changes are hard to move i production -- it is the reality and I'm ok with
it. So I guess, Ansible 2.0 would be released after 6 months since today
(is this kind of true?) for example. First point -- I'm not pushing my
wants to Ansible 1.6.

There's absolutely no plan for anything like this currently.

Second one -- as I feel, we don't live in "Java world" with 100% backward
compatibility.

We still fight hard for this. Nor would I agree that Java nails this well
-- vendoring of jarfiles to isolate deps runs pretty rampant.

I don't think Jinja2 can be used as an argument here.

As Ansible was young it was finding out how things wanted to be.
Supporting Jinja was one of those transformative lifestyle events.

Host specifiers are working though, and simple, and we do not need to tweak
them.

And Ansible migrated to Jinja templates -- I fixed all my roles and
playbooks (added "{{" and "}}"), you warned me about it for a few months.
It was ok. I feel this syntax changes ([:!&] -> [,^-]) are the same
situation.
Third point -- ansible and ansible-shell are good for their own cases, one
tool can't cover everything.

I think we do an excellent job of covering 95% of mainstream use cases and
our batteries included philosophy does support this -- and prove it quite
successful. Wherever possible we'll take good ideas and move them into the
project.

Here's a good time to quote Spock in Star Trek II -- "needs of the many,
outweigh the needs of the few, or the one".

It seems reasonable for me to believe this changes can be done in Ansible
2.0. Are you agree with this way?

I don't -- and we will not be changing the way we denote host specifiers
because they make a tab completion script happy.

I am interested in exploring how to do dynamic limits for /usr/bin/ansible,
but this should not be the "--limit" keyword since it already has meaning.

It should probably be "--where".

I am a bit lost – can you formulate your vision of ansible command? I think, I don’t get it in your way.
Here are questions, I’m worrying about:

  • is it (ansible command script) an important part of ansible toolset or just small script for fill some breach?
  • should people use ansible command in regular work? (it is about “how often people use is” – from almost never to every day)

I attached bash_completion file – try to use it (it is a bit outdated, but still works ok for test case).

About “–where” (“–limit”, as you like): I haven’t discovered script, so I’ll just try to offer an example of syntax structure just as one of possible ideas: (–where host:expressions), (–where group:expressions), (–where vars:path.to.var.name::expression).

(attachments)

ansible.gz (523 Bytes)

Strange, I can’t see your last message in GG (only in mail). I agree with you about --where. Great!

About syntax operators – I rewrote my points:

  • [!&] are pretty dangerous symbols for bash (I follow Raskin and others in UX-questions)
  • my guess: people use non-“:” operator rarely
  • current documentation about them generates user mistakes (no quotes in examples)
  • this could make new users unhappy for that moments (when bash writes errors after “trying correct examples from docs”)
  • people are not robots and can do mistakes (I do them sometimes :slight_smile: ) – anyone can forget quotes
  • people do more errors under stress (10 min downtime in my service costs about 10 000$, so I write commands quick, sometimes with errors) – this make: pain = pain*2
  • I and somebody else can be unhappy, when bash showing errors caused by “!”, “&” instead of doing real job
  • my group names are longer, than 15 characters – it is really hard to live without completion

Can we add “,”, “^”, “-” operators to current list and update doc part about Patterns? Everything would still work. But no one will do mistake (if he follow new syntax).
No mistakes – no unhappiness/sad. Advanced bash_competion for comfortable work.

You made my worklife better, I shared my experience with other people. Maybe I am in 5-10% users, who use Ansible for not only common cases. But we want improvements for us too, we want be happier and share our experience with more success-stories.

So adding new operators will decrease user mistakes (UX will be better), user comfort will grow. Happy user is good for project.

Does it sound better now?

Strange, I can't see your last message in GG (only in mail). I agree with
you about --where. Great!

About syntax operators -- I rewrote my points:
  * [!&] are pretty dangerous symbols for bash (I follow Raskin and others
in UX-questions)

When quoted they don't go to bash.

  * my guess: people use non-":" operator rarely

They are used a fair amount in playbooks. /usr/bin/ansible also gets a
decent amount of usage.

  * current documentation about them generates user mistakes (no quotes in
examples)

Let's fix those.

Can we _add_ ",", "^", "-" operators to current list and update doc part
about Patterns? Everything would still work. But no one will do mistake (if
he follow new syntax).

not interested, sorry.

Can I buy this feature?

Have I understood you right? My potential pull-request about [&!:] would be rejected?

We will not be changing the way “&” and “!” work, or adding alternative syntax.