-c Paramiko hangs if stdout is large?

Archives Ansible Project
1

Hello all,

I don’t know if this is a paramiko problem, or our connection implementation for it, but the SSH connection seems to handle this okay.

From running a ‘command: yum update -y’ for CentOS 5 machines on the recent upgrade, paramiko would sometimes hang indefinitely.
Checking running processes and logs on the remote host would show that the yum update completed successfully.

I believe the problem is having a large stdout returned.
(I tested on localhost dumping big randomly generated string into a file and just catting from a playbook.)

SSH spits out a buffered amount at a time, while paramiko just hangs.
(no load on either local or remote host, no disk io happening, the process is just sitting around)

Another case that may be related is an EC2 playbook for setting up cloud9.
I believe the error started showing up after I switched back to paramiko from Fireball.
The first sourcemint install (which dumps a whole lot to stdout) tends to fail.
The error message claims mismatching quotes, but the install was successful.

I thought I should bring this up in case anyone runs into similar behaviour.

I remember something similar before in the mailing list, but I couldn’t find it.
I think it was related to filesizes and paramiko??

PS: My cloud 9 playbook can be found here.
https://github.com/fdavis/ansible-ec2-env/tree/master/c9-servers/c9

2

We have a large output test for this reason, but tests are currently
running -c local (they were not always).

How large was large?

3

fdavis wrote:

Hello all,

I don't know if this is a paramiko problem, or our connection
implementation for it, but the SSH connection seems to handle this okay.

From running a 'command: yum update -y' for CentOS 5 machines on the

recent upgrade, paramiko would sometimes hang indefinitely.
Checking running processes and logs on the remote host would show that the
yum update completed successfully.

I believe the problem is having a large stdout returned.
(I tested on localhost dumping big randomly generated string into a file
and just catting from a playbook.)

SSH spits out a buffered amount at a time, while paramiko just hangs.
(no load on either local or remote host, no disk io happening, the process
is just sitting around)

Another case that may be related is an EC2 playbook for setting up cloud9.
I believe the error started showing up after I switched back to paramiko
from Fireball.
The first sourcemint install (which dumps a whole lot to stdout) tends to
fail.
The error message claims mismatching quotes, but the install was
successful.

I thought I should bring this up in case anyone runs into similar
behaviour.

I remember something similar before in the mailing list, but I couldn't
find it.
I think it was related to filesizes and paramiko??

This should be fixed on devel. Please test it again and see if it's still
an issue.

Daniel

4

Michael DeHaan wrote:

We have a large output test for this reason, but tests are currently
running -c local (they were not always).

Yeah, running this test with -c paramiko triggered the issue.

Daniel

5

My test file for the cat playbook was 100k.
I estimated yum’s output to be about that order of magnitude, but didn’t get any real metrics on that.
(stdout size: 80 char lines * 400 pkgs * 4 iterations (install, download, verify, cleanup on each package) ~= 128k)

At any rate, it is fixed. So thank you :slight_smile:

6

Seems reasonable to make the tests take the connection string from an
environment variable, like

TEST_CONNECTION=<type> or similar?

The reason we did local was so you didn't need SSH setup for one, but
also when making quick changes you could execute the test battery
faster.

7

This is great to hear. In retrospect, this probably explains hangs we have seen from time to time, but given the fact there was no output we did not relate this to the amount of output.