Your best bet to figure out what is going on is to strace the ansible process on the remote end, also lsof might help to see if it is stuck on accessing a file.
But reading this:
Note that client1 should be pingable but not reachable via ssh
Makes me think it is getting stuck on the firewall while trying to ssh. Ansible uses ssh by default, unless you give it another connection plugin/method of accessing the machine, Ansible won’t be able to function.
Those timeouts apply to some specific parts of the connection, not the total execution (see task timeout for that), so you can get to a point where those timeouts are not relevant anymore and still get stuck.
seems it is stuck waiting for input, which could be a firewall issue or a resources one. Since you already said ssh is not allowed, I’m going to guess its a firewall issue.
no firewall it seems that host is simply not reachable via ssh but only pingable
from time to time ansible command stuck on ssh
If I reboot the server issue disappear
[ playbooks]# ansible -i vmtesttat1cdbin9944, -a “rpm -q falcon-sensor” all -v
Using /playbooks/ansible.cfg as config file
[WARNING]: Unhandled error in Python interpreter discovery for host vmtesttat1cdbin9944: Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host
vmtesttat1cdbin9944 | UNREACHABLE! => {
“changed”: false,
“msg”: “Data could not be sent to remote host "vmtesttat1cdbin9944". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host\r\n”,
“unreachable”: true
}
[ playbooks]# ansible -i vmtesttat1cdbin9944, -a “rpm -q falcon-sensor” all -v
Using /playbooks/ansible.cfg as config file
[WARNING]: Unhandled error in Python interpreter discovery for host vmtesttat1cdbin9944: Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host
vmtesttat1cdbin9944 | UNREACHABLE! => {
“changed”: false,
“msg”: “Data could not be sent to remote host "vmtesttat1cdbin9944". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host\r\n”,
“unreachable”: true
}
[ playbooks]# ansible -i vmtesttat1cdbin9944, -a “rpm -q falcon-sensor” all -v
Using /playbooks/ansible.cfg as config file
[WARNING]: Unhandled error in Python interpreter discovery for host vmtesttat1cdbin9944: Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host
vmtesttat1cdbin9944 | UNREACHABLE! => {
“changed”: false,
“msg”: “Data could not be sent to remote host "vmtesttat1cdbin9944". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host\r\n”,
“unreachable”: true
}
[ playbooks]# ansible -i vmtesttat1cdbin9944, -a “rpm -q falcon-sensor” all -v
Using /playbooks/ansible.cfg as config file
This is not an Ansible problem. You’re literally being denied access by the remote host. Either the firewall is blocking you, or PAM is. Instead of testing with ansible ad-hoc commands, test with ssh.
As I said before, this is not really an Ansible issue. Just because you can ping the host over ICMP, that doesn’t mean you can establish a tcp connection to port 22 on that host. Something is denying your ssh connection. If you can prove that you can ssh <username>@vmtesttat1cdbin9944 successfully from where you’re trying to run Ansible, then we can at least rule out networking/firewall issues.
Testing locally, setting the timeout at the play level does appear to apply to the implicit gather_facts tasks too, but it’s possible it’s a bug in the version you’re using…
(obviously, if you’re testing via adhoc there is no playbook, so you have to use the second link I shared to configure it)
(and to be clear, it won’t “make it work”, it will just prevent the hang since you said “What I am waiting is that Ansible timeout and do no stay hang”)
I think @birb is saying that they expect some hosts to fail the SSH connection from time to time, but they don’t want that failure to cause such a long delay in the execution of the rest of the playbook or workflow.
That is what the task timeout can do (big hammer), but not the connection timeout.
I know this might be counter intuitive, but connections have many phases and each with it’s own timeout, the timeout in the connection plugins normally maps to ‘how long a does each interaction/response take w/o a ping/alive’ … but it is stuck long before that applies in ‘initial connection waiting for response’.
The ssh client configuration has dozens of timeouts, one of them applies to this case, but not the ‘connection_timeout’.
You probably want to set ConnectTimeout to not match the TCP timeout, which sometimes does not trigger due to how the firewall prevents ssh access.
Well don’t mind me then…