# RHEL6.6 and ControlPersist

**URL:** <https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517>\
**Category:** Ansible Project\
**Tags:** rhel\
**Created:** [October 15, 2014, 2:57pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517 "2014-10-15T14:57:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\
**Post date:** [October 15, 2014, 2:57pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/1 "2014-10-15T14:57:33Z")

</div>

Hi,

As some of you may know, Red Hat backported the ControlPersist functionality to the OpenSSH version that ships with RHEL6.

This is terrific since RHEL users can now use this technique to speed up Ansible.

However, after some testing it seems to fail for the very first connection. What happens is that the first connection, when the persistent connection has not been set up yet, fails. Any subsequent connection seems to work fine, but obviously this fails to work properly with Ansible.

I think this is a bug, has anyone tested this ?  
Or am I doing something wrong here ?

---

<div class="post-metadata">

**Author:** ![Todd\_Zullinger](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@Todd\_Zullinger](https://forum.ansible.com/u/Todd_Zullinger)\
**Post date:** [October 15, 2014, 5:08pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/2 "2014-10-15T17:08:11Z")

</div>

Dag Wieers wrote:

> As some of you may know, Red Hat backported the ControlPersist functionality to the OpenSSH version that ships with RHEL6.
> 
> This is terrific since RHEL users can now use this technique to speed up Ansible.
> 
> However, after some testing it seems to fail for the very first connection. What happens is that the first connection, when the persistent connection has not been set up yet, fails. Any subsequent connection seems to work fine, but obviously this fails to work properly with Ansible.
> 
> I think this is a bug, has anyone tested this ? Or am I doing something wrong here ?

I noticed this after updating to one of the later test packages from the bugzilla ticket where this was requested (it did not occur initially). I also thought it might be something peculiar to my environment and didn't notice it soon enough to bring it up in the ticket, unfortunately.

I haven't installed the final update yet (on CentOS), but I was hoping that it might have corrected the problem. At least I planned to test the latest package before considering it a package bug.

Knowing it seems to affect more than me, I imagine a new bug report is in order to resolve the problem.

It is fantastic to have ControlPersist on EL6 though. 😉

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [October 15, 2014, 11:07pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/3 "2014-10-15T23:07:42Z")

</div>

“I think this is a bug, has anyone tested this ?”

Sounds like this should be reported with RHEL, definitely.

Please do and post the bugzilla here if you can.

While we could add special code to say “don’t try CP on EL6 if ssh says it CAN CP” it seems shipping the feature broken would have been detected by them.

---

<div class="post-metadata">

**Author:** ![nirik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/nirik/32/686_2.png) [@nirik](https://forum.ansible.com/u/nirik)\
**Post date:** [October 15, 2014, 11:49pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/4 "2014-10-15T23:49:51Z")

</div>

I ran into this as well, but haven't had too much time to isolate it.

However, I did find that controlpersist works with ansible, but if you  
enable pipelining, then it hangs on the first connection.

So, perhaps there's a issue around pipelining?

kevin

---

<div class="post-metadata">

**Author:** ![Todd\_Zullinger](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@Todd\_Zullinger](https://forum.ansible.com/u/Todd_Zullinger)\
**Post date:** [October 16, 2014, 1:01am UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/5 "2014-10-16T01:01:55Z")

</div>

Kevin Fenzi wrote:

> > "I think this is a bug, has anyone tested this ?"
> > 
> > Sounds like this should be reported with RHEL, definitely.
> > 
> > Please do and post the bugzilla here if you can.
> > 
> > While we could add special code to say "don't try CP on EL6 if ssh says it CAN CP" it seems shipping the feature broken would have been detected by them.
> 
> I ran into this as well, but haven't had too much time to isolate it.
> 
> However, I did find that controlpersist works with ansible, but if you enable pipelining, then it hangs on the first connection.
> 
> So, perhaps there's a issue around pipelining?

With pipelining = True commented out on el6, things still fail for me on the first run. They do fail with a different error, but perhaps that's just due to slightly different code paths with pipelining on and off, I haven't looked at the ansible code at all.

In case it matters, I adjusted the control\_path variable to avoid using ~/.ansible since $HOME is on NFS in my environment.

With pipelining disabled:

$ ansible web -m ping -o  
...  
web12 | FAILED =\> failed to resolve remote temporary directory from /var/tmp/ansible-tmp-1413420275.15-6744099385111: `mkdir -p /var/tmp/ansible-tmp-1413420275.15-6744099385111 && chmod a+rx /var/tmp/ansible-tmp-1413420275.15-6744099385111 && echo /var/tmp/ansible-tmp-1413420275.15-6744099385111` returned empty string

With pipelining enabled:

$ ansible web -m ping -o  
...  
web12 | FAILED \>\> {"failed": true, "msg": "", "parsed": false}

This is still using openssh-5.3p1-100.el6.x86\_64 which was a scratch build that Petr made per RHBZ #953088. I have not checked whether there are any differences in the -104 packages included in the latest RHEL updates (and they haven't made it to my CentOS mirror yet).

This is also with ansible-1.6.10-1.el6 from EPEL. I have not yet updated to 1.7.x (I see 1.7.2-1 is in epel-testing now, so I'll prbably wait for that to hit stable).

HTH,

---

<div class="post-metadata">

**Author:** ![nirik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/nirik/32/686_2.png) [@nirik](https://forum.ansible.com/u/nirik)\
**Post date:** [October 16, 2014, 1:44am UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/6 "2014-10-16T01:44:58Z")

</div>

ansible-1.7.2-1.el6.noarch  
openssh-5.3p1-104.el6.x86\_64

with pipeline disabled:

% ansible -m ping -o junk02\*  
junk02.phx2.fedoraproject.org | success \>\> {"changed": false, "ping": "pong"}

with pipeline enabled:

% ansible -m ping -o junk02\*  
junk02.phx2.fedoraproject.org | FAILED \>\> {"failed": true, "msg": "", "parsed": false}

kevin

---

<div class="post-metadata">

**Author:** ![Jacob\_Weber1](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@Jacob\_Weber1](https://forum.ansible.com/u/Jacob_Weber1)\
**Post date:** [October 24, 2014, 10:46pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/7 "2014-10-24T22:46:53Z")

</div>

Same results here. I have ansible 1.7.2-2 and openssh-5.3p1-104 (installed from the CentOS Continuous Releases Repository), on CentOS 6.5.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [October 24, 2014, 10:51pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/8 "2014-10-24T22:51:46Z")

</div>

“with pipeline disabled:”

Hmmmmm, curious.

Worst case we could detect RHEL 6 and auto-disable pipelining on that platform, what say ye?

---

<div class="post-metadata">

**Author:** ![Jacob\_Weber1](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@Jacob\_Weber1](https://forum.ansible.com/u/Jacob_Weber1)\
**Post date:** [October 24, 2014, 10:58pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/9 "2014-10-24T22:58:41Z")

</div>

Just to clarify – I was seeing the same results as Kevin, not Todd. Removing “pipelining = True” from my config fixed the problem.

Since that’s the default, I don’t know if you really need to change Ansible – it never worked on RHEL6 before, so nothing’s changed in that regard. But I hope they’re able to fix this issue in OpenSSH…did someone report it?

---

<div class="post-metadata">

**Author:** ![Todd\_Zullinger](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@Todd\_Zullinger](https://forum.ansible.com/u/Todd_Zullinger)\
**Post date:** [October 24, 2014, 11:24pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/10 "2014-10-24T23:24:15Z")

</div>

Jacob Weber wrote:

> Just to clarify -- I was seeing the same results as Kevin, not Todd. Removing "pipelining = True" from my config fixed the problem.

I've since updated to 1.7.2 from epel-testing and I see the same results as you and Kevin. I've disabled pipelining as well for now, but it's definitely a performance hit (better to be slower and accurate though).

> But I hope they're able to fix this issue in OpenSSH....did someone report it?

I haven't seen anything myself. I don't have a RHEL contract so I was hoping someone that did might file a ticket so it's more likely to get attention.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [October 25, 2014, 1:59am UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/11 "2014-10-25T01:59:20Z")

</div>

I’ve filed a github for now to include (in 1.8) a check to auto-disable pipelining on RHEL 6.6+ (but not EL7), which should resolve most of the confusion.

We also may make it print a warning if it was on.

But yeah, bugzilla seems appropriate.

Bugzilla from someone with a nice friendly Red Hat TAM even more so 🙂

---

<div class="post-metadata">

**Author:** ![Adam\_Miller](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Miller](https://forum.ansible.com/u/Adam_Miller)\
**Post date:** [October 28, 2014, 4:35pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/12 "2014-10-28T16:35:09Z")

</div>

> I've filed a github for now to include (in 1.8) a check to auto-disable  
> pipelining on RHEL 6.6+ (but not EL7), which should resolve most of the  
> confusion.
> 
> We also may make it print a warning if it was on.
> 
> But yeah, bugzilla seems appropriate.
> 
> Bugzilla from someone with a nice friendly Red Hat TAM even more so 🙂

I've submitted a ticket to my team's TAM through Red Hat support  
channels, if/when there is a public bugzilla as a side effect I'll  
link it here.

-AdamM

---

<div class="post-metadata">

**Author:** ![mtovey](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mtovey](https://forum.ansible.com/u/mtovey)\
**Post date:** [October 29, 2014, 8:55pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/13 "2014-10-29T20:55:16Z")

</div>

When I saw this discussion thread, I was thrilled because I am using OEL6.5 with OpenSSH-5.3. Since that is equivalent to RHEL6.5, that meant that there should be an update to OpenSSH for OEL too. Sure enough, there is (openssh-5.3p1-104.el6.x86\_64). But when I installed it, I could no longer use Ansible to copy files to other servers. I did not make any other changes (ssh\_args, scp\_if\_ssh, control\_path, and pipelining are all still commented out), and my ssh\_config does not include any of the “Control\*” parameters. Output from my quick tests are below:

Without “scp\_if\_ssh”:

sinudy36-\> ansible sinudm07 -m copy -a “src=testfile dest=/var/tmp”  
sinudm07 | FAILED \>\> {  
“failed”: true,  
“md5sum”: “d41d8cd98f00b204e9800998ecf8427e”,  
“msg”: “\u001b]2;pdxmft @ :/home/pdxmft\u0007/usr/bin/python: can’t open file ‘\u001b]2’: [Errno 2] No such file or directory\r\n/bin/sh: pdxmft: command not found\r\n”,  
“parsed”: false  
}

With “scp\_if\_ssh”:

sinudy36-\> ansible sinudm07 -m copy -a “src=testfile dest=/var/tmp”  
sinudm07 | FAILED =\> failed to transfer file to /home/pdxmft/.ansible/tmp/ansible-tmp-1414615053.51-185971585352740/source:

scp: /home/pdxmft/.ansible/tmp/ansible-tmp-1414615053.51-185971585352740/source: No such file or directory

This is identical to what I observed when I tried to specify ssh instead of paramiko (-c ssh) and prior to upgrading OpenSSH. I opened another thread about that yesterday (Cannot copy a file to a server when using ssh) before I saw this thread, but at that time I had to specify “-c ssh” in the command line to get this reaction while here it just happens regardless of what I do. Meanwhile normal scp and sftp from the command line functions just fine; I get the failures only when I try to use Ansible to copy files.  
Something about the way that Ansible is calling scp or sftp appears to trigger this bug in the latest version of OpenSSH. This will be a major problem as it now means I cannot upgrade OpenSSH for any reason and still be able to use Ansible. For now I have rolled back to openssh-5.3p1-94.el6. I hope somebody finds a solution soon.  
-Mark

---

<div class="post-metadata">

**Author:** ![mtovey](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mtovey](https://forum.ansible.com/u/mtovey)\
**Post date:** [October 29, 2014, 10:51pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/14 "2014-10-29T22:51:17Z")

</div>

I deployed a new OEL6.5 server, then upgraded OpenSSH to the new ControlPersist release and installed Ansible onto it. Without making any configuration changes to anything other than adding server names to the hosts file, I tried to use Ansible to copy a file to another server, and it failed as before. I downgraded OpenSSH to the previous version and tried the copy again. It worked perfectly.  
So, there is definitely a mismatch of some sort between Ansible and the newer release of OpenSSH on Linux 6.  
-Mark

---

<div class="post-metadata">

**Author:** ![Pythagoras\_Watson](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@Pythagoras\_Watson](https://forum.ansible.com/u/Pythagoras_Watson)\
**Post date:** [November 5, 2014, 9:36pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/15 "2014-11-05T21:36:20Z")

</div>

I ran into similar issues using the new ControlPersist option as well as the ProxyCommand option. A [Red Hat bugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=1160487) was created that has the details. I think the part in [comment 1](https://bugzilla.redhat.com/show_bug.cgi?id=1160487#c1) starting at “The commands and output below show that ControlPersist=yes does not work as expected.” is what you are referring to. There is a patch for openssh-5.3p1-104.el6.src.rpm attached to the bug that is from me backporting code from the RHEL 7 openssh related to the ControlPersist option. I don’t run Ansible, so I have no way of testing to see if the patch fixes your issue. However, I would be interested to know if it does.

---

<div class="post-metadata">

**Author:** ![Azul](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@Azul](https://forum.ansible.com/u/Azul)\
**Post date:** [November 11, 2014, 11:35am UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/16 "2014-11-11T11:35:37Z")

</div>

just a heads up,

I run RH6.5, not able to upgrade at the moment to 6.6 (and it looks like it wouldn’t help either), I have worked around the ControlPersist issue by installing a openssh6 client on my control host box (/opt/openssh6),  
I then have a wrapper script that calls ansible-playbook and sets the PATH to collect ssh and friends from /opt/openssh6/bin before /usr/bin.

because it only uses the openssh client (no daemons running), there’s no conflict with the normal redhat packages.  
Its so much faster

---

<div class="post-metadata">

**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\
**Post date:** [November 13, 2014, 1:40pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/17 "2014-11-13T13:40:56Z")

</div>

Let me break you the news that Red Hat has released an openssh update that fixes the reported issue with ControlPersist.

&nbsp;&nbsp;&nbsp;&nbsp;\* Thu Nov 06 2014 Petr Lautrbach \<plautrba@redhat.com\> 5.3p1-104.1  
&nbsp;&nbsp;&nbsp;&nbsp;- Fix ControlPersist option with ProxyCommand (#1160487)

And it works well. Joy !

---

<div class="post-metadata">

**Author:** ![mtovey](https://avatars.discourse-cdn.com/v4/letter/m/f0a364/32.png) [@mtovey](https://forum.ansible.com/u/mtovey)\
**Post date:** [November 13, 2014, 7:58pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/18 "2014-11-13T19:58:13Z")

</div>

I checked the Oracle repository and found openssh-5.3p1-104.el6\_6.1. I installed that and tested. Nice!!! It looks like that patch fixed it.  
-Mark

---

<div class="post-metadata">

**Author:** ![Jacob\_Weber1](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@Jacob\_Weber1](https://forum.ansible.com/u/Jacob_Weber1)\
**Post date:** [November 13, 2014, 9:23pm UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/19 "2014-11-13T21:23:17Z")

</div>

Got it on CentOS too, and turned pipelining on. Can’t say that I’m seeing much of a performance difference, but I’m not getting the errors either. I’ll do some testing on a longer playbook later.

---

<div class="post-metadata">

**Author:** ![Jacob\_Weber1](https://avatars.discourse-cdn.com/v4/letter/j/94ad74/32.png) [@Jacob\_Weber1](https://forum.ansible.com/u/Jacob_Weber1)\
**Post date:** [November 14, 2014, 12:10am UTC](https://forum.ansible.com/t/rhel6-6-and-controlpersist/16517/20 "2014-11-14T00:10:09Z")

</div>

Yeah, my ~50 minute playbook is down to about 46 minutes now. Not sure why I’m not seeing the difference that others are. I do see the ControlPersist files being created in ~ansible/cp. It’s running about 80 plays on each of about 20 hosts. I guess the SSH part of Ansible wasn’t adding that much overhead to begin with.