Ping fail with error 127 '/bin/bash: line 1: xxx: command not found\n'

Laurent_ROGER · January 13, 2025, 3:33pm

Hi,
I’ve run same installation task with ansible successfully on a few Debian 11 and 12 servers, first step is to check if all is setup with ansible all -m ping. Everything works fine except with one Debian 12 server

ansible server (Debian 11) :

ansible all -m ping -vvv
ansible [core 2.12.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Dec  1 2024, 12:12:57) [GCC 10.2.1 20210110]
  jinja version = 2.11.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
script declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
auto declined parsing /etc/ansible/hosts as it did not pass its verify_file() method
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
Parsed /etc/ansible/hosts inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
<remote ip> ESTABLISH SSH CONNECTION FOR USER: xxxxx
<remote ip> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="xxxxx"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/3f945fea3f"' remote ip '/bin/sh -c '"'"'echo ~xxxxx && sleep 0'"'"''
<remote ip> (127, b'/home/xxxxx\n', b'/bin/sh: 1: sleep: not found\n')
<remote ip> Failed to connect to the host via ssh: /bin/sh: 1: sleep: not found
<remote ip> ESTABLISH SSH CONNECTION FOR USER: xxxxx
<remote ip> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="xxxxx"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/3f945fea3f"' remote ip '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/xxxxx/.ansible/tmp `"&& mkdir "` echo /home/xxxxx/.ansible/tmp/ansible-tmp-1736781642.3676438-935005-233802510071625 `" && echo ansible-tmp-1736781642.3676438-935005-233802510071625="` echo /home/xxxxx/.ansible/tmp/ansible-tmp-1736781642.3676438-935005-233802510071625 `" ) && sleep 0'"'"''
<remote ip> (127, b'', b'/bin/sh: 1: mkdir: not found\n')
<remote ip> Failed to connect to the host via ssh: /bin/sh: 1: mkdir: not found
remote ip | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo /home/xxxxx/.ansible/tmp `\"&& mkdir \"` echo /home/xxxxx/.ansible/tmp/ansible-tmp-1736781642.3676438-935005-233802510071625 `\" && echo ansible-tmp-1736781642.3676438-935005-233802510071625=\"` echo /home/xxxxx/.ansible/tmp/ansible-tmp-1736781642.3676438-935005-233802510071625 `\" ), exited with result 127",
    "unreachable": true
}

target server Debian 12 (bookworm)

python3 -m pip -V
pip 23.0.1 from /usr/lib/python3/dist-packages/pip (python 3.11)

if I connect to the remote server with ssh and user xxxxxx, I have access, I have a bash and can run sleep, mkdir or whatever command.

I’ve tried to add in ansible.cfg

[defaults]
executable = /bin/bash

Not better ! Any idea ?
Thank you
Best wishes for 2025 !

chris · January 13, 2025, 5:42pm

Try:

[defaults]
executable = /usr/bin/bash

It’s moved!

However I’m not sure this is the issue, isn’t the problem here?

bcoca · January 13, 2025, 10:00pm

Are the target machines POSIX/linux? Do they restrict the shell of the login user? is /bin/sh present? is PATH set for batch logins?

Laurent_ROGER · January 13, 2025, 10:22pm

The target is Debian 12 as other Debian 12 where the ansible ping works fine, so I try to figure out what are the differences
How can I check if it restricts the shell of the login user ? (I wrote I can connect via ssh, the shell is bash) → I will try @chris suggestion for the bash path.
I don’t know if /bin/sh is present (why it would not be) → will check
but you can see that

/bin/sh: 1: mkdir: not found

means that /bin/sh exists but not mkdir (very strange)

how to check if iPATH is set for batch logins ?

bcoca · January 13, 2025, 10:26pm

something like ssh user@host 'env |grep PATH'

Laurent_ROGER · January 14, 2025, 9:40am

On the target Debian,

$ which bash
/usr/bin/bash

$ ls -al /bin
lrwxrwxrwx 1 root root 7 Nov 12 14:43 /bin -> usr/bin

So anyway /bin/bash is correct
I’ve changed to /usr/bin/bash => not solved

<remote ip> ESTABLISH SSH CONNECTION FOR USER: xxxxx
<remote ip> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="xxxxx"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/3f945fea3f"' remote ip '/usr/bin/bash -c '"'"'echo ~xxxxx && sleep 0'"'"''
<remote ip> (127, b'/home/xxxxx\n', b'/usr/bin/bash: line 1: sleep: command not found\n')
<remote ip> Failed to connect to the host via ssh: /usr/bin/bash: line 1: sleep: command not found

Laurent_ROGER · January 14, 2025, 9:40am

$ env |grep PATH
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin


$ ls -al /bin/sh
lrwxrwxrwx 1 root root 4 Jan  5  2023 /bin/sh -> dash

Laurent_ROGER · January 14, 2025, 10:01am

next experiment :

$ ssh  xxxxx@remoteip "pwd"
/home/xxxxx

$ ssh  xxxxx@remoteip "cd /etc; pwd"
/etc

$ ssh  xxxxx@remoteip "cd /etc; pwd; ls"
/etc
bash: line 1: ls: command not found

$ ssh  xxxxx@remoteip "ps"
bash: line 1: ps: command not found

$ ssh  xxxxx@remoteip "/usr/bin/ps"
    PID TTY          TIME CMD
 381570 ?        00:00:00 systemd
 381571 ?        00:00:00 (sd-pam)
 381617 ?        00:00:00 sshd
 381619 ?        00:00:00 ps

I found no difference in /etc/ssh/sshd_config compared to another Debian12 where the ansible ping works.

bcoca · January 14, 2025, 7:19pm

^ i specifically asked for it over ssh as your login will create a different environment than a ‘batch’ ssh command run

From man ssh:

   If a command is specified, it will be executed on the remote host instead of a login shell.  A complete command line may be specified as command, or it may have ad‐
     ditional arguments.  If supplied, the arguments will be appended to the command, separated by spaces, before it is sent to the server to be executed.

Ansible does not do interactive logins but sends ‘ssh commands’.

Laurent_ROGER · January 15, 2025, 8:35am

ssh xxxxx@remoteip 'env | grep PATH'
bash: line 1: env: command not found
bash: line 1: grep: command not found

If I run the same command on another host where the ping command works, of course this works, I can get PATH from env with bash

ssh xxxxx@anotherremoteip 'env | grep PATH'
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
PATH=/usr/local/bin:/usr/bin:/bin:/usr/games

it looks like the bash command has lost the PATH, so the command is not found ?

Let’s try this, force PATH before the commands

ssh xxxxx@remoteip 'PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin; env | grep PATH'
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

ok, so this is PATH related issue

Laurent_ROGER · January 15, 2025, 9:11am

I modified the .bashrc to ensures that the PATH is set even when the shell is non-interactive. I’ve added at top of .bashrc

if [ -z "$PS1" ]; then
    # Non-interactive shell
    export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
fi

and now it works for the ansible ping on the 2 hosts having the issue

ansible all -m ping
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details
remoteip | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

But the mystery is still here : why it works without any modification on most hosts and need to be fixed on others ?
Anyway thank you @bcoca @chris for your help.

Laurent_ROGER · January 15, 2025, 9:21am

On one host, I had to get back ansible.cfg to /bin/bash, because /usr/bin is not link to /bin, bash is in /bin/bash

[defaults]
executable = /bin/bash

dnmvisser · January 15, 2025, 12:35pm

I would reinstall the system, with ansible, so you at least start with reproducible hosts

Topic		Replies	Views
unable to ping the remote hosts AWX Project	2	5	June 22, 2022
Ansible win_ping fails Ansible Project windows	1	13	January 27, 2020
The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error Ansible Project	1	27	July 17, 2023
ansible win_ping fails with 'transport_message': 'Bad HTTP Ansible Project windows	0	1	July 28, 2021
ESTABLISH SSH CONNECTION FOR USER: ubuntu Ansible Project ubuntu	1	0	August 5, 2021

Ping fail with error 127 '/bin/bash: line 1: xxx: command not found\n'

Related topics