We have 2 environments, both showing the same problem.
ENV 1. Jenkins worker is in a Docker Container, running on a linux server
ENV 2. Jenkins worker is a K8s pod.
The container the work is being ran from is the same in each env. This is the same container I am using locally to test.
We have tons of jobs that all use this same ssh through a bastion setup, but for some reason, just this one is having problem, even though other jobs, targeting the same servers are working fine.
We generate a ssh.cfg, which specifies the bastion and its configurations. The exact commands below, run just fine on everyone laptops, fail in the Jenkins pipeline.
[defaults]
stdout_callback = debug
ansible_connection = ssh
ansible_port = 22
retry_files_enabled = False
callbacks_enabled = timer, profile_tasks
gathering = smart
timeout = 60
forks = 1
serial = 1
[ssh_connection]
ssh_args = '-F ./ssh.cfg -o ControlMaster=auto -o ControlPersist=30m -o StrictHostKeyChecking=no -o ForwardAgent=yes -o ServerAliveInterval=30 -o UserKnownHostsFile=/dev/null'
pipelining = True
no_proxy=’’ ANSIBLE_CONFIG=ansible-private.cfg ansible -i inventory/private_linux_aws_ec2.yaml ‘tag_DeploymentID_${DEPLOYMENT_ID}:!tag_Role_SSH_Bastion_Host’ -m ping -e ‘ansible_python_interpreter=/usr/bin/python2’
no_proxy=’’ ANSIBLE_CONFIG=ansible-private.cfg ansible-playbook -i inventory/private_linux_aws_ec2.yaml playbooks/single_private_linux.yml -e “deployment_id=${DEPLOYMENT_ID} env=${ENV}”
no_proxy=’’ ANSIBLE_CONFIG=ansible-private.cfg ansible -i inventory/private_windows_aws_ec2.yaml ‘tag_DeploymentID_${DEPLOYMENT_ID}:!tag_Role_RDP_Bastion_Host’ -m win_ping
no_proxy=’’ ANSIBLE_CONFIG=ansible-private.cfg ansible-playbook -i inventory/private_windows_aws_ec2.yaml playbooks/single_windows.yml -e “deployment_id=${DEPLOYMENT_ID} env=${ENV}”
When running -vvv you can copy/paste the SSH command it connects just fine.
Errors look like this, someones its all servers, sometime a few.
TASK [Set Fact - Public key] ***************************************************
Tuesday 07 December 2021 19:56:53 +0000 (0:00:00.018) 0:00:04.729 ******
ok: [ip-172-16-0-10.us-gov-west-1.compute.internal]
ok: [ip-172-16-1-5.us-gov-west-1.compute.internal]
ok: [ip-172-16-1-10.us-gov-west-1.compute.internal]
ok: [ip-172-16-0-5.us-gov-west-1.compute.internal]
TASK [Remove New Authorized Keys file if exists] ******************************
Tuesday 07 December 2021 19:56:54 +0000 (0:00:00.054) 0:00:04.784 ******
changed: [ip-172-16-0-10.us-gov-west-1.compute.internal]
fatal: [ip-172-16-1-5.us-gov-west-1.compute.internal]: UNREACHABLE! => {
“changed”: false,
“unreachable”: true
}
MSG:
Data could not be sent to remote host “172.16.1.5”. Make sure this host can be reached over ssh: Connection timed out during banner exchange
ok: [ip-172-16-1-10.us-gov-west-1.compute.internal]
fatal: [ip-172-16-0-5.us-gov-west-1.compute.internal]: UNREACHABLE! => {
“changed”: false,
“unreachable”: true
}TASK [Set Fact - Public key] ***************************************************
Tuesday 07 December 2021 19:56:53 +0000 (0:00:00.018) 0:00:04.729 ******
ok: [ip-172-16-0-10.us-gov-west-1.compute.internal]
ok: [ip-172-16-1-5.us-gov-west-1.compute.internal]
ok: [ip-172-16-1-10.us-gov-west-1.compute.internal]
ok: [ip-172-16-0-5.us-gov-west-1.compute.internal]
TASK [Remove New Authorized Keys file if exists] ******************************
Tuesday 07 December 2021 19:56:54 +0000 (0:00:00.054) 0:00:04.784 ******
changed: [ip-172-16-0-10.us-gov-west-1.compute.internal]
fatal: [ip-172-16-1-5.us-gov-west-1.compute.internal]: UNREACHABLE! => {
“changed”: false,
“unreachable”: true
}
MSG:
Data could not be sent to remote host “172.16.1.5”. Make sure this host can be reached over ssh: Connection timed out during banner exchange
ok: [ip-172-16-1-10.us-gov-west-1.compute.internal]
fatal: [ip-172-16-0-5.us-gov-west-1.compute.internal]: UNREACHABLE! => {
“changed”: false,
“unreachable”: true
}