[Fab-user] Can't save output to a CSV file if running task in parallel

[Felix Almeida]

Hi all,


I'm trying to run a simple task in parallel that captures the OS type (Linux, AIX, HP-UX, etc.) of a set of UNIX servers and save this information into a CSV file.


However, if I add the @parallel decorator to the task the CSV file is left empty (only the header is saved), but if I remove the @parallel decorator then everything goes well. Note that the output to the screen via puts works fine in both cases.


Please, any ideas of what I'm doing wrong?

Perhaps this is not even a fabric question but a Python one (I'm not sure, sorry).


Here is the complete code:


import csv
from fabric.api import env, task, runs_once, parallel, execute, run, puts, hide

env.abort_on_prompts = True
env.always_use_pty = False
env.command_timeout = 3
env.disable_known_hosts = True
env.eagerly_disconnect = True
env.timeout = 3
env.use_shell = False
env.warn_only = True

@parallel(pool_size=5)
def _run(csv_output):
try:
with hide('output', 'running'):
out = run('uname')
except SystemExit:
msg = 'error: a password is being requested'
else:
msg = str(out)
puts(msg)
csv_output.writerow([env.host, msg])
return str(out)

@task
@runs_once
def run_uname(hosts_file=None):
if hosts_file:
with open(hosts_file) as input_:
host_list = [line.strip() for line in input_]
env.hosts.extend(host_list)
with open('output.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerow(['HOSTNAME', 'UNAME'])
with hide('running', 'status'):
execute(_run, writer)


I run it like this: fab run_uname:test_hosts.txt

My environment is this: RHEL4, Python 2.7.10, Fabric 1.10.2, Paramiko 1.15.2


Thank you!

Felix





________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et réception de courriels se fait strictement suivant les modalités énoncées dans l'avis publié à www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________

[Brandon Whaley]

Parallel uses multiprocessing, so you're opening output.csv in write mode
once per host. You'll need to have a master task consolidate the return
values of your parallel subtask (use execute) and write the csv once
everything is done. Remember that execute returns a dictionary whose keys
are the host_strings of each host and the values are the return value of
the function you're executing.

On Thu, Sep 10, 2015 at 4:46 PM Felix Almeida <***@rci.rogers.com>
wrote:

> Hi all,
>
>
> I'm trying to run a simple task in parallel that captures the OS type
> (Linux, AIX, HP-UX, etc.) of a set of UNIX servers and save this
> information into a CSV file.
>
>
> However, if I add the *@parallel* decorator to the task the CSV file is
> left empty (only the header is saved), but if I remove the *@parallel*
> decorator then everything goes well. Note that the output to the screen via
> *puts* works fine in both cases.
>
>
> Please, any ideas of what I'm doing wrong?
>
> Perhaps this is not even a fabric question but a Python one (I'm not sure,
> sorry).
>
>
> Here is the complete code:
>
>
> import csv
> from fabric.api import env, task, runs_once, parallel, execute, run, puts,
> hide
>
> env.abort_on_prompts = True
> env.always_use_pty = False
> env.command_timeout = 3
> env.disable_known_hosts = True
> env.eagerly_disconnect = True
> env.timeout = 3
> env.use_shell = False
> env.warn_only = True
>
> *@parallel(pool_size=5)*
> def _run(csv_output):
> try:
> with hide('output', 'running'):
> out = run('uname')
> except SystemExit:
> msg = 'error: a password is being requested'
> else:
> msg = str(out)
> puts(msg)
> csv_output.writerow([env.host, msg])
> return str(out)
>
> @task
> @runs_once
> def run_uname(hosts_file=None):
> if hosts_file:
> with open(hosts_file) as input_:
> host_list = [line.strip() for line in input_]
> env.hosts.extend(host_list)
> with open('output.csv', 'wb') as output:
> writer = csv.writer(output)
> writer.writerow(['HOSTNAME', 'UNAME'])
> with hide('running', 'status'):
> execute(_run, writer)
>
>
> I run it like this: fab run_uname:test_hosts.txt
>
> My environment is this: RHEL4, Python 2.7.10, Fabric 1.10.2, Paramiko
> 1.15.2
>
> Thank you!
>
> Felix
>
>
>
>
>
>
> ------------------------------
> This communication is confidential. We only send and receive email on the
> basis of the terms set out at www.rogers.com/web/content/emailnotice
>
>
>
> Ce message est confidentiel. Notre transmission et réception de courriels
> se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel
>
> ------------------------------
> _______________________________________________
> Fab-user mailing list
> Fab-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/fab-user
>

[Felix Almeida]

Thanks Brandon!

But that's what I thought I was doing when I decorated my master task with "runs _once" and used the "parallel" decorator only on my subtask (which I call with "execute")... since I open the CSV file in the master task I assumed (incorrectly, I guess) it would work.

Please, what do you think it would be a good approach here? Create a queue in the master task and make the subtask send messages to it?



-------- Original message --------
From: Brandon Whaley <***@gmail.com>
Date: 2015-09-10 5:10 PM (GMT-05:00)
To: Felix Almeida <***@rci.rogers.com>, fab-***@nongnu.org
Subject: Re: [Fab-user] Can't save output to a CSV file if running task in parallel

Parallel uses multiprocessing, so you're opening output.csv in write mode once per host. You'll need to have a master task consolidate the return values of your parallel subtask (use execute) and write the csv once everything is done. Remember that execute returns a dictionary whose keys are the host_strings of each host and the values are the return value of the function you're executing.

On Thu, Sep 10, 2015 at 4:46 PM Felix Almeida <***@rci.rogers.com<mailto:***@rci.rogers.com>> wrote:

Hi all,


I'm trying to run a simple task in parallel that captures the OS type (Linux, AIX, HP-UX, etc.) of a set of UNIX servers and save this information into a CSV file.


However, if I add the @parallel decorator to the task the CSV file is left empty (only the header is saved), but if I remove the @parallel decorator then everything goes well. Note that the output to the screen via puts works fine in both cases.


Please, any ideas of what I'm doing wrong?

Perhaps this is not even a fabric question but a Python one (I'm not sure, sorry).


Here is the complete code:


import csv
from fabric.api import env, task, runs_once, parallel, execute, run, puts, hide

env.abort_on_prompts = True
env.always_use_pty = False
env.command_timeout = 3
env.disable_known_hosts = True
env.eagerly_disconnect = True
env.timeout = 3
env.use_shell = False
env.warn_only = True

@parallel(pool_size=5)
def _run(csv_output):
try:
with hide('output', 'running'):
out = run('uname')
except SystemExit:
msg = 'error: a password is being requested'
else:
msg = str(out)
puts(msg)
csv_output.writerow([env.host, msg])
return str(out)

@task
@runs_once
def run_uname(hosts_file=None):
if hosts_file:
with open(hosts_file) as input_:
host_list = [line.strip() for line in input_]
env.hosts.extend(host_list)
with open('output.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerow(['HOSTNAME', 'UNAME'])
with hide('running', 'status'):
execute(_run, writer)


I run it like this: fab run_uname:test_hosts.txt

My environment is this: RHEL4, Python 2.7.10, Fabric 1.10.2, Paramiko 1.15.2


Thank you!

Felix





________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et r?ception de courriels se fait strictement suivant les modalit?s ?nonc?es dans l'avis publi? ? www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________
_______________________________________________
Fab-user mailing list
Fab-***@nongnu.org<mailto:Fab-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/fab-user




________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et r?ception de courriels se fait strictement suivant les modalit?s ?nonc?es dans l'avis publi? ? www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________

[Brandon Whaley]

I'm fairly sure this is what you want:

import csv
from fabric.api import env, task, runs_once, parallel, execute, run, puts,
hide

env.abort_on_prompts = True
env.always_use_pty = False
env.command_timeout = 3
env.disable_known_hosts = True
env.eagerly_disconnect = True
env.timeout = 3
env.use_shell = False
env.warn_only = True

@parallel(pool_size=5)
def _run():
try:
with hide('output', 'running'):
out = run('uname')
except SystemExit:
msg = 'error: a password is being requested'
else:
msg = str(out)
puts(msg)
return msg

@task
@runs_once
def run_uname(hosts_file=None):
if hosts_file:
with open(hosts_file) as input_:
host_list = [line.strip() for line in input_]
env.hosts.extend(host_list)
with open('output.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerow(['HOSTNAME', 'UNAME'])
with hide('running', 'status'):
outputs = execute(run)
for host, output in outputs.iteritems():
writer.writerow([host, output])

The changes are that msg is returned (instead of out) and that no writing
is attempted in parallel, an operation is done on the return dictionary of
execute() as explained in my first email.

On Thu, Sep 10, 2015 at 7:03 PM Felix Almeida <***@rci.rogers.com>
wrote:

> Thanks Brandon!
>
> But that's what I thought I was doing when I decorated my master task with
> "runs _once" and used the "parallel" decorator only on my subtask (which I
> call with "execute")... since I open the CSV file in the master task I
> assumed (incorrectly, I guess) it would work.
>
> Please, what do you think it would be a good approach here? Create a queue
> in the master task and make the subtask send messages to it?
>
>
>
> -------- Original message --------
> From: Brandon Whaley <***@gmail.com>
> Date: 2015-09-10 5:10 PM (GMT-05:00)
> To: Felix Almeida <***@rci.rogers.com>, fab-***@nongnu.org
> Subject: Re: [Fab-user] Can't save output to a CSV file if running task in
> parallel
>
> Parallel uses multiprocessing, so you're opening output.csv in write mode
> once per host. You'll need to have a master task consolidate the return
> values of your parallel subtask (use execute) and write the csv once
> everything is done. Remember that execute returns a dictionary whose keys
> are the host_strings of each host and the values are the return value of
> the function you're executing.
>
> On Thu, Sep 10, 2015 at 4:46 PM Felix Almeida <
> ***@rci.rogers.com> wrote:
>
>> Hi all,
>>
>>
>> I'm trying to run a simple task in parallel that captures the OS type
>> (Linux, AIX, HP-UX, etc.) of a set of UNIX servers and save this
>> information into a CSV file.
>>
>>
>> However, if I add the *@parallel* decorator to the task the CSV file is
>> left empty (only the header is saved), but if I remove the *@parallel*
>> decorator then everything goes well. Note that the output to the screen via
>> *puts* works fine in both cases.
>>
>>
>> Please, any ideas of what I'm doing wrong?
>>
>> Perhaps this is not even a fabric question but a Python one (I'm not
>> sure, sorry).
>>
>>
>> Here is the complete code:
>>
>>
>> import csv
>> from fabric.api import env, task, runs_once, parallel, execute, run,
>> puts, hide
>>
>> env.abort_on_prompts = True
>> env.always_use_pty = False
>> env.command_timeout = 3
>> env.disable_known_hosts = True
>> env.eagerly_disconnect = True
>> env.timeout = 3
>> env.use_shell = False
>> env.warn_only = True
>>
>> *@parallel(pool_size=5)*
>> def _run(csv_output):
>> try:
>> with hide('output', 'running'):
>> out = run('uname')
>> except SystemExit:
>> msg = 'error: a password is being requested'
>> else:
>> msg = str(out)
>> puts(msg)
>> csv_output.writerow([env.host, msg])
>> return str(out)
>>
>> @task
>> @runs_once
>> def run_uname(hosts_file=None):
>> if hosts_file:
>> with open(hosts_file) as input_:
>> host_list = [line.strip() for line in input_]
>> env.hosts.extend(host_list)
>> with open('output.csv', 'wb') as output:
>> writer = csv.writer(output)
>> writer.writerow(['HOSTNAME', 'UNAME'])
>> with hide('running', 'status'):
>> execute(_run, writer)
>>
>>
>> I run it like this: fab run_uname:test_hosts.txt
>>
>> My environment is this: RHEL4, Python 2.7.10, Fabric 1.10.2, Paramiko
>> 1.15.2
>>
>> Thank you!
>>
>> Felix
>>
>>
>>
>>
>>
>>
>> ------------------------------
>> This communication is confidential. We only send and receive email on the
>> basis of the terms set out at www.rogers.com/web/content/emailnotice
>>
>>
>>
>> Ce message est confidentiel. Notre transmission et réception de courriels
>> se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel
>>
>> ------------------------------
>> _______________________________________________
>> Fab-user mailing list
>> Fab-***@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/fab-user
>>
>
>
>
>
> ------------------------------
> This communication is confidential. We only send and receive email on the
> basis of the terms set out at www.rogers.com/web/content/emailnotice
>
>
>
> Ce message est confidentiel. Notre transmission et réception de courriels
> se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel
>
> ------------------------------
>

[Felix Almeida]

Thank you Brandon.
It worked perfectly. :)


From: Brandon Whaley [mailto:***@gmail.com]
Sent: Thursday, September 10, 2015 7:10 PM
To: Felix Almeida; fab-***@nongnu.org
Subject: Re: [Fab-user] Can't save output to a CSV file if running task in parallel

I'm fairly sure this is what you want:

import csv
from fabric.api import env, task, runs_once, parallel, execute, run, puts, hide

env.abort_on_prompts = True
env.always_use_pty = False
env.command_timeout = 3
env.disable_known_hosts = True
env.eagerly_disconnect = True
env.timeout = 3
env.use_shell = False
env.warn_only = True

@parallel(pool_size=5)
def _run():
try:
with hide('output', 'running'):
out = run('uname')
except SystemExit:
msg = 'error: a password is being requested'
else:
msg = str(out)
puts(msg)
return msg

@task
@runs_once
def run_uname(hosts_file=None):
if hosts_file:
with open(hosts_file) as input_:
host_list = [line.strip() for line in input_]
env.hosts.extend(host_list)
with open('output.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerow(['HOSTNAME', 'UNAME'])
with hide('running', 'status'):
outputs = execute(run)
for host, output in outputs.iteritems():
writer.writerow([host, output])

The changes are that msg is returned (instead of out) and that no writing is attempted in parallel, an operation is done on the return dictionary of execute() as explained in my first email.

On Thu, Sep 10, 2015 at 7:03 PM Felix Almeida <***@rci.rogers.com<mailto:***@rci.rogers.com>> wrote:
Thanks Brandon!

But that's what I thought I was doing when I decorated my master task with "runs _once" and used the "parallel" decorator only on my subtask (which I call with "execute")... since I open the CSV file in the master task I assumed (incorrectly, I guess) it would work.

Please, what do you think it would be a good approach here? Create a queue in the master task and make the subtask send messages to it?



-------- Original message --------
From: Brandon Whaley <***@gmail.com<mailto:***@gmail.com>>
Date: 2015-09-10 5:10 PM (GMT-05:00)
To: Felix Almeida <***@rci.rogers.com<mailto:***@rci.rogers.com>>, fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Subject: Re: [Fab-user] Can't save output to a CSV file if running task in parallel
Parallel uses multiprocessing, so you're opening output.csv in write mode once per host. You'll need to have a master task consolidate the return values of your parallel subtask (use execute) and write the csv once everything is done. Remember that execute returns a dictionary whose keys are the host_strings of each host and the values are the return value of the function you're executing.

On Thu, Sep 10, 2015 at 4:46 PM Felix Almeida <***@rci.rogers.com<mailto:***@rci.rogers.com>> wrote:

Hi all,



I'm trying to run a simple task in parallel that captures the OS type (Linux, AIX, HP-UX, etc.) of a set of UNIX servers and save this information into a CSV file.



However, if I add the @parallel decorator to the task the CSV file is left empty (only the header is saved), but if I remove the @parallel decorator then everything goes well. Note that the output to the screen via puts works fine in both cases.



Please, any ideas of what I'm doing wrong?

Perhaps this is not even a fabric question but a Python one (I'm not sure, sorry).



Here is the complete code:



import csv
from fabric.api import env, task, runs_once, parallel, execute, run, puts, hide

env.abort_on_prompts = True
env.always_use_pty = False
env.command_timeout = 3
env.disable_known_hosts = True
env.eagerly_disconnect = True
env.timeout = 3
env.use_shell = False
env.warn_only = True

@parallel(pool_size=5)
def _run(csv_output):
try:
with hide('output', 'running'):
out = run('uname')
except SystemExit:
msg = 'error: a password is being requested'
else:
msg = str(out)
puts(msg)
csv_output.writerow([env.host, msg])
return str(out)

@task
@runs_once
def run_uname(hosts_file=None):
if hosts_file:
with open(hosts_file) as input_:
host_list = [line.strip() for line in input_]
env.hosts.extend(host_list)
with open('output.csv', 'wb') as output:
writer = csv.writer(output)
writer.writerow(['HOSTNAME', 'UNAME'])
with hide('running', 'status'):
execute(_run, writer)



I run it like this: fab run_uname:test_hosts.txt

My environment is this: RHEL4, Python 2.7.10, Fabric 1.10.2, Paramiko 1.15.2

Thank you!

Felix





________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et réception de courriels se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________
_______________________________________________
Fab-user mailing list
Fab-***@nongnu.org<mailto:Fab-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/fab-user



________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et réception de courriels se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________




________________________________
This communication is confidential. We only send and receive email on the basis of the terms set out at www.rogers.com/web/content/emailnotice<http://www.rogers.com/web/content/emailnotice>



Ce message est confidentiel. Notre transmission et réception de courriels se fait strictement suivant les modalités énoncées dans l’avis publié à www.rogers.com/aviscourriel <http://www.rogers.com/aviscourriel>
________________________________