# Avoiding the extra shell on remote execution **URL:** \ **Category:** Ansible Project\ **Created:** [February 13, 2013, 2:59am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348 "2013-02-13T02:59:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 13, 2013, 2:59am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/1 "2013-02-13T02:59:49Z")
Hello, I am testing ansible, and I stumbled over a curiosity in the remote execution. If I run   % ansible localhost -m command -a 'ps -f' I notice that ansible spawns a shell on the "remote":   madduck 19221 19204 0 15:57 pts/12 00:00:00 /bin/sh -c /usr/bin/python /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/command; rm -rf /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/ \>/dev/null 2\>&1   madduck 19228 19221 0 15:57 pts/12 00:00:00 /usr/bin/python /home/madduck/.ansible/tmp/ansible-1360724263.23-4932750123022/command   madduck 19229 19228 0 15:57 pts/12 00:00:00 ps -f Correct me if I am wrong, but this shell could IMHO be avoided, if the Python script that is being executed would close stdout and stderr itself, and finally remove the working directory. Is this something you'd consider? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 13, 2013, 1:34pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/2 "2013-02-13T13:34:08Z")
Definitely, though I thought this was covered already since we were executing the shebang line directly without going through the shell. (What version of ansible are you using? Run "ansible --version"?) ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [February 13, 2013, 1:52pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/3 "2013-02-13T13:52:08Z")
For some modules we could even pipe the script directly to the interpreter on the remote system, avoiding the creation of a temporary directory, the transfer of the script and the removal of the temporary directory. But this would mean a different code path for modules that don't require a temporary directory (for other transfers) and those that do. I think we discussed this once on IRC... ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 13, 2013, 2:43pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/4 "2013-02-13T14:43:15Z")
I'm open to experimentation provided any changes made work for all connection types when the pull request is submitted πŸ™‚ ---
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 13, 2013, 7:12pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/5 "2013-02-13T19:12:13Z")
also sprach Michael DeHaan \ [2013.02.14.0234 +1300]: > Definitely, though I thought this was covered already since we were > executing the shebang line directly without going through the shell. I don't understand what you mean. If you use execvp() (or subprocess.Popen), then there is not shell involved to run a script with a shebang line. At least on Linux, the linker handles this, not the shell. And I am sure it's the same on other Unices too. > (What version of ansible are you using? Run "ansible --version"?) ansible 1.1 (devel 6266571c15) last updated 2013/02/09 19:12:25 (GMT +1300) ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 14, 2013, 12:30pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/6 "2013-02-14T12:30:50Z")
Show me a patch of what you are talking about please. ---
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 14, 2013, 10:05pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/7 "2013-02-14T22:05:07Z")
also sprach Michael DeHaan \ [2013.02.15.0130 +1300]: > Show me a patch of what you are talking about please. I am working on it, but first I have a question: In ansible/runner/\_\_init\_\_.py in function \_execute\_module, you go through a bit of trouble to extract the shebang line and execute the interpreter yourself, rather than just adding +x to the "command" file you transferred and asking the linker to execute it. Can you please tell me why you are doing this? The only reason I could imagine is because you want to ensure things are working if noexec is set on the mountpoint (thus preventing the linker from working). However, in that case I think the proper solution would be to require the admin to chose an appropriate temporary directory instead. Are there other reasons to extract the shebang line and thereby foreclosing the linker's work? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 14, 2013, 10:49pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/8 "2013-02-14T22:49:19Z")
> also sprach Michael DeHaan \ [2013.02.15.0130 +1300]: Mr. DeHaan misreads as also sprach Zarathustra and anticipates the black monolith and soundtrack πŸ™‚ > I am working on it, but first I have a question: > > In ansible/runner/\_\_init\_\_.py in function \_execute\_module, you go > through a bit of trouble to extract the shebang line and execute the > interpreter yourself, rather than just adding +x to the "command" > file you transferred and asking the linker to execute it. > > Can you please tell me why you are doing this? Absolutely. As I recall, lots of people have different preferences on shells, and many don't have a /bin/sh. I think the problem is caused by some random operating system that only has fish, but I can't remember πŸ™‚ Ansible is a balance of juggling lots of various interests. It also simplified some code that had to mangle the module if the interpreter was in a different location, and there may have been some benefit to not spawing the actual shell. To be honest, it's been something of an evolution. > The only reason I could imagine is because you want to ensure things > are working if noexec is set on the mountpoint (thus preventing the > linker from working). However, in that case I think the proper > solution would be to require the admin to chose an appropriate > temporary directory instead. That may have actually been a bit a part of it, it avoids the need for the chmod and you can just execute the file by fully pathing Python (or bash, or Ruby). Shouldn't have any noexec problems in that case. > Are there other reasons to extract the shebang line and thereby > foreclosing the linker's work? Not sure exactly what you mean by this last part. I tend to think of linkers in gcc terms. ---
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 15, 2013, 12:20am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/9 "2013-02-15T00:20:37Z")
also sprach Michael DeHaan \ [2013.02.15.1149 +1300]: > Mr. DeHaan misreads as also sprach Zarathustra and anticipates the > black monolith and soundtrack πŸ™‚ I was thinking more of Nietzsche when I composed my muttrc sometime shortly after puberty. Thank you for your explanation. You made me realise that Ansible lets me override interpreters per-system, e.g. by setting ansible\_python\_interpreter. That is very cool and reason enough to do with the shebang line whatever you are doing. However, and with reference to [https://groups.google.com/forum/#!msg/ansible-project/ccZ72NSJr2Q/8pa5RW0fACsJ](https://groups.google.com/forum/#!msg/ansible-project/ccZ72NSJr2Q/8pa5RW0fACsJ), allow me to make a few comments: > As I recall, lots of people have different preferences on shells, and > many don't have a /bin/sh. The shell has nothing to do with shebang lines. Instead, the shell passes the file to be executed to the exec()-function-family, and that's where shebang processing happens. If the referenced file starts with #!, then the exec()-functions will automatically run the specified interpreter instead and feed it the file. Since Bash and Python's subprocess, sshd, and pretty much everything else uses exec()-functions, you could just invoke scripts directly and let these functions handle the shebang line. '/bin/sh' is rarely needed. Finally, I want to note that systems without a /bin/sh are not POSIX-compliant. I understand you are juggling a lot of various interests, but there's got to be a baseline, and I think that ought to be POSIX, at least. Otherwise you cannot assume presence of e.g. mkdir, which you use to create temporary directories remotely. I am working on a patch to remove all the extraneous shell use… Cheers, ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 15, 2013, 3:36am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/10 "2013-02-15T03:36:45Z")
> The shell has nothing to do with shebang lines. Instead, the shell > passes the file to be executed to the exec()-function-family, and > that's where shebang processing happens. If the referenced file > starts with #!, then the exec()-functions will automatically run the > specified interpreter instead and feed it the file. Yes, but that's true if you chmod +x the file, though if something was mounted noexec we had the shell call in there before to avoid that problem. Basically if bash detects a shebang line that is not bash when executing a script explicitly it will launch the other shell, thus you can usually do crazy stuff like /bin/sh foo.py and it will work. Unless it's late and I'm remembering wrongly πŸ™‚ > Since Bash and Python's subprocess, sshd, and pretty much everything > else uses exec()-functions, you could just invoke scripts directly > and let these functions handle the shebang line. '/bin/sh' is rarely > needed. Except for the noexec case. Right? > Finally, I want to note that systems without a /bin/sh are not > POSIX-compliant. I understand you are juggling a lot of various > interests, but there's got to be a baseline, and I think that ought > to be POSIX, at least. Otherwise you cannot assume presence of e.g. > mkdir, which you use to create temporary directories remotely. I generally would be inclined to agree. ---
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 15, 2013, 3:47am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/11 "2013-02-15T03:47:38Z")
also sprach Michael DeHaan \ [2013.02.15.1636 +1300]: > Basically if bash detects a shebang line that is not bash when > executing a script explicitly it will launch the other shell, thus you > can usually do crazy > stuff like > > /bin/sh foo.py > > and it will work. I would consider this a bug, not a feature. > \> Since Bash and Python's subprocess, sshd, and pretty much everything > \> else uses exec()-functions, you could just invoke scripts directly > \> and let these functions handle the shebang line. '/bin/sh' is rarely > \> needed. > > Except for the noexec case. Right? Right, but if noexec is set, then it's set for a reason and you could work around. I often have /home mounted noexec, but I also put stuff into /tmp/ansible, not ~/.ansible/tmp. > \> Finally, I want to note that systems without a /bin/sh are not > \> POSIX-compliant. I understand you are juggling a lot of various > \> interests, but there's got to be a baseline, and I think that > \> ought to be POSIX, at least. Otherwise you cannot assume > \> presence of e.g. mkdir, which you use to create temporary > \> directories remotely. > > I generally would be inclined to agree. Phew πŸ˜‰ ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 15, 2013, 4:26am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/12 "2013-02-15T04:26:59Z")
> also sprach Michael DeHaan \ [2013.02.15.1636 +1300]: > > > Basically if bash detects a shebang line that is not bash when > > executing a script explicitly it will launch the other shell, thus you > > can usually do crazy > > stuff like > > > > /bin/sh foo.py > > > > and it will work. > > I would consider this a bug, not a feature. Either way, it's what we did to get around the noexec stuff. We used to just chmod it. > > \> Since Bash and Python's subprocess, sshd, and pretty much everything > > \> else uses exec()-functions, you could just invoke scripts directly > > \> and let these functions handle the shebang line. '/bin/sh' is rarely > > \> needed. > > > > Except for the noexec case. Right? > > Right, but if noexec is set, then it's set for a reason and you > could work around. I often have /home mounted noexec, but I also put > stuff into /tmp/ansible, not ~/.ansible/tmp. Yeah, but /tmp/ansible is a shared dir, and that's bad for systems management apps. Plus, lots of people have /tmp ansible mounted noexec We do it the way we do it so that it maximally "just works". (That being said, if your shell tweak patches keep all that behavior working, they are good... I haven't had time to look just yet though, but I'll give them the benefit of the doubt for now!) ---
**Author:** ![martin\_f\_krafft](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@martin\_f\_krafft](https://forum.ansible.com/u/martin_f_krafft)\ **Post date:** [February 15, 2013, 8:13am UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/13 "2013-02-15T08:13:19Z")
also sprach Michael DeHaan \ [2013.02.15.1726 +1300]: > \> I would consider this a bug, not a feature. > > Either way, it's what we did to get around the noexec stuff. We used > to just chmod it. No, you are using it because you provide interpreter injection based on per-host variables. That's a good reason. Everything else I'd consider a cop-out πŸ˜‰ > Yeah, but /tmp/ansible is a shared dir, and that's bad for systems > management apps. > > Plus, lots of people have /tmp ansible mounted noexec No Debian system will have /tmp mounted noexec, but sure… I use /tmp/ansible/$user, not just /tmp/ansible. > (That being said, if your shell tweak patches keep all that behavior > working, they are good... I haven't had time to look just yet though, > but I'll give them the benefit of the doubt for now!) Please don't, but scrutinise them. I worked under time constraints and with a 10 month old around. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [February 15, 2013, 1:16pm UTC](https://forum.ansible.com/t/avoiding-the-extra-shell-on-remote-execution/12348/14 "2013-02-15T13:16:30Z")
> > Plus, lots of people have /tmp ansible mounted noexec > > No Debian system will have /tmp mounted noexec, but sure… > > I use /tmp/ansible/$user, not just /tmp/ansible. More of a somewhat-infrequent-but-common-enough 'after installation' hardening policy thing. > > (That being said, if your shell tweak patches keep all that behavior > > working, they are good... I haven't had time to look just yet though, > > but I'll give them the benefit of the doubt for now!) > > Please don't, but scrutinise them. I worked under time constraints > and with a 10 month old around. Np, thanks!