Avoiding the extra shell on remote execution

Archives Ansible Project
1

Hello,

I am testing ansible, and I stumbled over a curiosity in the remote
execution. If I run

  % ansible localhost -m command -a 'ps -f'

I notice that ansible spawns a shell on the "remote":

  madduck 19221 19204 0 15:57 pts/12 00:00:00 /bin/sh -c /usr/bin/python /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/command; rm -rf /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/ >/dev/null 2>&1
  madduck 19228 19221 0 15:57 pts/12 00:00:00 /usr/bin/python /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/command
  madduck 19229 19228 0 15:57 pts/12 00:00:00 ps -f

Correct me if I am wrong, but this shell could IMHO be avoided, if
the Python script that is being executed would close stdout and
stderr itself, and finally remove the working directory.

Is this something you'd consider?

2

Definitely, though I thought this was covered already since we were
executing the shebang line directly without going through the shell.

(What version of ansible are you using? Run "ansible --version"?)

3

For some modules we could even pipe the script directly to the interpreter on the remote system, avoiding the creation of a temporary directory, the transfer of the script and the removal of the temporary directory.

But this would mean a different code path for modules that don't require a temporary directory (for other transfers) and those that do. I think we discussed this once on IRC...

4

I'm open to experimentation provided any changes made work for all
connection types
when the pull request is submitted :slight_smile:

5

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.14.0234 +1300]:

Definitely, though I thought this was covered already since we were
executing the shebang line directly without going through the shell.

I don't understand what you mean. If you use execvp()
(or subprocess.Popen), then there is not shell involved to run
a script with a shebang line. At least on Linux, the linker handles
this, not the shell. And I am sure it's the same on other Unices
too.

(What version of ansible are you using? Run "ansible --version"?)

ansible 1.1 (devel 6266571c15) last updated 2013/02/09 19:12:25 (GMT +1300)

6

Show me a patch of what you are talking about please.

7

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.0130 +1300]:

Show me a patch of what you are talking about please.

I am working on it, but first I have a question:

In ansible/runner/__init__.py in function _execute_module, you go
through a bit of trouble to extract the shebang line and execute the
interpreter yourself, rather than just adding +x to the "command"
file you transferred and asking the linker to execute it.

Can you please tell me why you are doing this?

The only reason I could imagine is because you want to ensure things
are working if noexec is set on the mountpoint (thus preventing the
linker from working). However, in that case I think the proper
solution would be to require the admin to chose an appropriate
temporary directory instead.

Are there other reasons to extract the shebang line and thereby
foreclosing the linker's work?

8

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.0130 +1300]:

Mr. DeHaan misreads as also sprach Zarathustra and anticipates the
black monolith and soundtrack :slight_smile:

I am working on it, but first I have a question:

In ansible/runner/__init__.py in function _execute_module, you go
through a bit of trouble to extract the shebang line and execute the
interpreter yourself, rather than just adding +x to the "command"
file you transferred and asking the linker to execute it.

Can you please tell me why you are doing this?

Absolutely.

As I recall, lots of people have different preferences on shells, and
many don't have a /bin/sh.

I think the problem is caused by some random operating system that
only has fish, but I can't remember :slight_smile:

Ansible is a balance of juggling lots of various interests.

It also simplified some code that had to mangle the module if the
interpreter was in a different location, and there may have been some
benefit to not spawing the actual shell.

To be honest, it's been something of an evolution.

The only reason I could imagine is because you want to ensure things
are working if noexec is set on the mountpoint (thus preventing the
linker from working). However, in that case I think the proper
solution would be to require the admin to chose an appropriate
temporary directory instead.

That may have actually been a bit a part of it, it avoids the need for
the chmod and you can just execute the file by fully pathing Python
(or bash, or Ruby).

Shouldn't have any noexec problems in that case.

Are there other reasons to extract the shebang line and thereby
foreclosing the linker's work?

Not sure exactly what you mean by this last part.

I tend to think of linkers in gcc terms.

9

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.1149 +1300]:

Mr. DeHaan misreads as also sprach Zarathustra and anticipates the
black monolith and soundtrack :slight_smile:

I was thinking more of Nietzsche when I composed my muttrc sometime
shortly after puberty.

Thank you for your explanation. You made me realise that Ansible
lets me override interpreters per-system, e.g. by setting
ansible_python_interpreter. That is very cool and reason enough to
do with the shebang line whatever you are doing.

However, and with reference to
https://groups.google.com/forum/#!msg/ansible-project/ccZ72NSJr2Q/8pa5RW0fACsJ,
allow me to make a few comments:

As I recall, lots of people have different preferences on shells, and
many don't have a /bin/sh.

The shell has nothing to do with shebang lines. Instead, the shell
passes the file to be executed to the exec()-function-family, and
that's where shebang processing happens. If the referenced file
starts with #!, then the exec()-functions will automatically run the
specified interpreter instead and feed it the file.

Since Bash and Python's subprocess, sshd, and pretty much everything
else uses exec()-functions, you could just invoke scripts directly
and let these functions handle the shebang line. '/bin/sh' is rarely
needed.

Finally, I want to note that systems without a /bin/sh are not
POSIX-compliant. I understand you are juggling a lot of various
interests, but there's got to be a baseline, and I think that ought
to be POSIX, at least. Otherwise you cannot assume presence of e.g.
mkdir, which you use to create temporary directories remotely.

I am working on a patch to remove all the extraneous shell use…

Cheers,

10

The shell has nothing to do with shebang lines. Instead, the shell
passes the file to be executed to the exec()-function-family, and
that's where shebang processing happens. If the referenced file
starts with #!, then the exec()-functions will automatically run the
specified interpreter instead and feed it the file.

Yes, but that's true if you chmod +x the file, though if something was
mounted noexec we had the shell call in there before to avoid that
problem.

Basically if bash detects a shebang line that is not bash when
executing a script explicitly it will launch the other shell, thus you
can usually do crazy
stuff like

/bin/sh foo.py

and it will work.

Unless it's late and I'm remembering wrongly :slight_smile:

Since Bash and Python's subprocess, sshd, and pretty much everything
else uses exec()-functions, you could just invoke scripts directly
and let these functions handle the shebang line. '/bin/sh' is rarely
needed.

Except for the noexec case. Right?

Finally, I want to note that systems without a /bin/sh are not
POSIX-compliant. I understand you are juggling a lot of various
interests, but there's got to be a baseline, and I think that ought
to be POSIX, at least. Otherwise you cannot assume presence of e.g.
mkdir, which you use to create temporary directories remotely.

I generally would be inclined to agree.

11

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.1636 +1300]:

Basically if bash detects a shebang line that is not bash when
executing a script explicitly it will launch the other shell, thus you
can usually do crazy
stuff like

/bin/sh foo.py

and it will work.

I would consider this a bug, not a feature.

> Since Bash and Python's subprocess, sshd, and pretty much everything
> else uses exec()-functions, you could just invoke scripts directly
> and let these functions handle the shebang line. '/bin/sh' is rarely
> needed.

Except for the noexec case. Right?

Right, but if noexec is set, then it's set for a reason and you
could work around. I often have /home mounted noexec, but I also put
stuff into /tmp/ansible, not ~/.ansible/tmp.

> Finally, I want to note that systems without a /bin/sh are not
> POSIX-compliant. I understand you are juggling a lot of various
> interests, but there's got to be a baseline, and I think that
> ought to be POSIX, at least. Otherwise you cannot assume
> presence of e.g. mkdir, which you use to create temporary
> directories remotely.

I generally would be inclined to agree.

Phew :wink:

12

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.1636 +1300]:

Basically if bash detects a shebang line that is not bash when
executing a script explicitly it will launch the other shell, thus you
can usually do crazy
stuff like

/bin/sh foo.py

and it will work.

I would consider this a bug, not a feature.

Either way, it's what we did to get around the noexec stuff. We used
to just chmod it.

> Since Bash and Python's subprocess, sshd, and pretty much everything
> else uses exec()-functions, you could just invoke scripts directly
> and let these functions handle the shebang line. '/bin/sh' is rarely
> needed.

Except for the noexec case. Right?

Right, but if noexec is set, then it's set for a reason and you
could work around. I often have /home mounted noexec, but I also put
stuff into /tmp/ansible, not ~/.ansible/tmp.

Yeah, but /tmp/ansible is a shared dir, and that's bad for systems
management apps.

Plus, lots of people have /tmp ansible mounted noexec

We do it the way we do it so that it maximally "just works".

(That being said, if your shell tweak patches keep all that behavior
working, they are good... I haven't had time to look just yet though,
but I'll give them the benefit of the doubt for now!)

13

also sprach Michael DeHaan <michael.dehaan@gmail.com> [2013.02.15.1726 +1300]:

> I would consider this a bug, not a feature.

Either way, it's what we did to get around the noexec stuff. We used
to just chmod it.

No, you are using it because you provide interpreter injection based
on per-host variables. That's a good reason. Everything else I'd
consider a cop-out :wink:

Yeah, but /tmp/ansible is a shared dir, and that's bad for systems
management apps.

Plus, lots of people have /tmp ansible mounted noexec

No Debian system will have /tmp mounted noexec, but sure…

I use /tmp/ansible/$user, not just /tmp/ansible.

(That being said, if your shell tweak patches keep all that behavior
working, they are good... I haven't had time to look just yet though,
but I'll give them the benefit of the doubt for now!)

Please don't, but scrutinise them. I worked under time constraints
and with a 10 month old around.

14

Plus, lots of people have /tmp ansible mounted noexec

No Debian system will have /tmp mounted noexec, but sure…

I use /tmp/ansible/$user, not just /tmp/ansible.

More of a somewhat-infrequent-but-common-enough 'after installation'
hardening policy thing.

(That being said, if your shell tweak patches keep all that behavior
working, they are good... I haven't had time to look just yet though,
but I'll give them the benefit of the doubt for now!)

Please don't, but scrutinise them. I worked under time constraints
and with a 10 month old around.

Np, thanks!