[Fab-user] Fabric hangs on run() command

[n***@nathanpeters.com]

I have a simple fabric script that hangs on only one of the 1100 servers I
tried to connect to when I attempt the run command. I have also tried to
debug this by manually running the command over ssh and it works fine over
ssh.

Here is the script (the [INFO] blocks were just there for me to debug
whether it reached a certain line):

SCRIPT
======

import platform
import re
from fabric.api import settings, env, run, put, sudo

import logging
logging.basicConfig(level=logging.DEBUG)

env.abort_on_prompts = True
env.skip_bad_hosts = True

env.hosts = open('openauditlist.txt', 'r').readlines()

def upload_audit_script():
print ("[INFO] we are about to check the os version")
os_version = run ('cat /etc/redhat-release', timeout=5)
print ("[INFO] we just got the os version")
if re.search('CentOS', os_version):
print ("[INFO] OS Matched CentOS ...")
put ('audit_linux.sh', '/etc/cron.daily/audit_linux.sh', use_sudo=True)
sudo ('chmod +x /etc/cron.daily/audit_linux.sh')
sudo ('/etc/cron.daily/audit_linux.sh')

PARAMIKO DEBUG
==============

***@DWDEV6-NPETERS ~/cssh/openaudit_upload
$ fab upload_audit_script --initial-password-prompt --show=debug
Using fabfile '/home/nathan.peters/cssh/openaudit_upload/fabfile.py'
Initial value for env.password:
Commands to run: upload_audit_script
Parallel tasks now using pool size of 415
[indexer1b-dc1-ma-maqa2-van] Executing task 'upload_audit_script'
[INFO] we are about to check the os version
[indexer1b-dc1-ma-maqa2-van] run: /bin/bash -l -c "cat /etc/redhat-release"
DEBUG:paramiko.transport:starting thread (client mode): 0xff3d4550L
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_6.6)
DEBUG:paramiko.transport:kex algos:[u'curve25519-***@libssh.org',
u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521',
u'diffie-hellman-group-excha nge-sha256',
u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1',
u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa'] client
encrypt:[u'aes128-ct r', u'aes192-ctr', u'aes256-ctr', u'arcfour256',
u'arcfour128', u'aes128-***@openssh.com', u'aes256-***@openssh.com',
u'chacha20-***@openssh.com', u'aes128-cbc', u'3des-cbc',
u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour',
u'rijndael-***@lysator.liu.se'] server encrypt:[u'aes128-ctr',
u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128',
u'aes128-***@openssh.com', u'aes256-***@openssh.com',
u'chacha20-***@openssh.com', u'aes128-cbc', u'3des-cbc', u'bl
owfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour',
u'rijndael-***@lysator.liu.se'] client mac:[u'hmac-md5-***@openssh.com',
u'hmac-sha1-***@openss h.com', u'umac-64-***@openssh.com',
u'umac-128-***@openssh.com', u'hmac-sha2-256-***@openssh.com',
u'hmac-sha2-512-***@openssh.com', u'hmac-ripemd160-***@openssh.com' ,
u'hmac-sha1-96-***@openssh.com', u'hmac-md5-96-***@openssh.com',
u'hmac-md5', u'hmac-sha1', u'umac-***@openssh.com',
u'umac-***@openssh.com', u'hmac-sha2-256', u'hma c-sha2-512',
u'hmac-ripemd160', u'hmac-***@openssh.com', u'hmac-sha1-96',
u'hmac-md5-96'] server mac:[u'hmac-md5-***@openssh.com',
u'hmac-sha1-***@openssh.com', u'umac-64-***@openssh.com',
u'umac-128-***@openssh.com', u'hmac-sha2-256-***@openssh.com',
u'hmac-sha2-512-***@openssh.com', u'hmac-ripemd160-***@openssh.com', u'hma
c-sha1-96-***@openssh.com', u'hmac-md5-96-***@openssh.com', u'hmac-md5',
u'hmac-sha1', u'umac-***@openssh.com', u'umac-***@openssh.com',
u'hmac-sha2-256', u'hmac-sha2- 512', u'hmac-ripemd160',
u'hmac-***@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] client
compress:[u'none', u'***@openssh.com'] server compress:[u'none', u'zl
***@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
DEBUG:paramiko.transport:Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
DEBUG:paramiko.transport:using kex diffie-hellman-group14-sha1; server key
type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local
hmac-sha1, remote hma c-sha1; compression: local none, remote none
DEBUG:paramiko.transport:Switch to new keys ...

DEBUG:paramiko.transport:EOF in transport thread

MANUAL SSH DEBUG
================

This is what happens when I run the command manually over SSH (works
fine). I will include both with and without the -t flag. As you can see in
both cases the command properly returned.

***@DWDEV6-NPETERS ~
$ ssh -t ***@indexer1b-dc1-ma-maqa2-van "/bin/bash -l -c \"cat
/etc/redhat-release\""
Welcome to GR! All activities are logged and monitored.
Password:
cat: cannot open /etc/redhat-release
Connection to indexer1b-dc1-ma-maqa2-van closed.

***@DWDEV6-NPETERS ~
$ ssh ***@indexer1b-dc1-ma-maqa2-van "/bin/bash -l -c \"cat
/etc/redhat-release\""
Welcome to GR! All activities are logged and monitored.
Password:
cat: cannot open /etc/redhat-release

OUTPUT FROM OTHER SERVERS
=========================

What's so maddening about this, is it works fine on other servers. Here
are 3 servers where it did what I expected (failed to cat the nonexistent
file and moved on to the next server) :

[zk1-dc1-ma-maqa2-van] Executing task 'upload_audit_script'
[zk1-dc1-ma-maqa2-van] run: cat /etc/redhat-release
[zk1-dc1-ma-maqa2-van] out: cat: cannot open /etc/redhat-release
[zk1-dc1-ma-maqa2-van] out:

Warning: run() received nonzero return code 2 while executing 'cat
/etc/redhat-release'!

[smtp1-dc1-ma-maqa2-van] Executing task 'upload_audit_script'
[smtp1-dc1-ma-maqa2-van] run: cat /etc/redhat-release
[smtp1-dc1-ma-maqa2-van] out: cat: cannot open /etc/redhat-release
[smtp1-dc1-ma-maqa2-van] out:

Warning: run() received nonzero return code 2 while executing 'cat
/etc/redhat-release'!

[squirrel1-dc1-ma-maqa2-van] Executing task 'upload_audit_script'
[squirrel1-dc1-ma-maqa2-van] run: cat /etc/redhat-release
[squirrel1-dc1-ma-maqa2-van] out: cat: cannot open /etc/redhat-release
[squirrel1-dc1-ma-maqa2-van] out:

Warning: run() received nonzero return code 2 while executing 'cat
/etc/redhat-release'!