[Fab-user] Hi there, two questions from newcomer.

[李鸥]

If I can re-express that: what is the recommended value of pool_size? - Leon

From: Lee, Leon
Time: Dec 4th, 2016 10:45 A.M.
To: ***@bitprophet.org; fab-***@nongnu.org
Cc: Lee, Leon
Subject: Hi there, two questions from newcomer.

Dear Jeff and All,

This is Leon from Beijing, I installed fabric for my Linux server on Sep 2nd, 2016 in order to manage over ten thousand infrastructure devices (such as the server, storage and switch¡­.etc) by SSH protocol. Just two questions at this time:
1.) how many ¡°@parallel(pool_size)¡± should I specify in my fabfile.py file and what is a proper value for this parameter?
2.) what is the maximum rows of the ip address list?
The following is my environment information, just waiting for your suggestions. Thanks in advance.

# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
Author-email: python-***@python.org<mailto:python-***@python.org>
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
Requires:
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto

Best Regards,
Leon Lee

[李鸥]

Dear Jeff and All,

This is Leon from Beijing, I installed fabric for my Linux server on Sep 2nd, 2016 in order to manage over ten thousand infrastructure devices (such as the server, storage and switch¡­.etc) by SSH protocol. Just two questions at this time:
1.) how many ¡°@parallel(pool_size)¡± should I specify in my fabfile.py file and what is a proper value for this parameter?
2.) what is the maximum rows of the ip address list?
The following is my environment information, just waiting for your suggestions. Thanks in advance.

# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
Author-email: python-***@python.org
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
Requires:
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
Author-email: ***@bitprophet.org
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: ***@bitprophet.org
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto

Best Regards,
Leon Lee

[Nitesh Nanotkar]

Hi,
Pool size depends on your central server bandwidth and ram
usable. You can specify n no. of ips in list.
In my scenario I use pool size to be 300 when I just want to connect to
servers and transfer files in kbs. I use 50 pool size when I have to
transfer installation scripts which is of 950MB.
My server's upload speed is 320Mbps and has RAM of 256GB.
Hope you understood on which factors pool size depends now.

Thanks!

Post by 李鸥
If I can re-express that: what is the recommended value of pool_size? - Leon
*From:* Lee, Leon
*Time:* Dec 4th, 2016 10:45 A.M.
*Cc:* Lee, Leon
*Subject:* Hi there, two questions from newcomer.
Dear Jeff and All,
This is Leon from Beijing, I installed fabric for my Linux server on Sep 2
nd, 2016 in order to manage over ten thousand infrastructure devices
(such as the server, storage and switch
.etc) by SSH protocol. Just two
and what is a proper value for this parameter?
2.) what is the maximum rows of the ip address list?
The following is my environment information, just waiting for your
suggestions. Thanks in advance.
# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto
Best Regards,
Leon Lee
_______________________________________________
Fab-user mailing list
https://lists.nongnu.org/mailman/listinfo/fab-user

[李鸥]

Hi Nitesh,

Thanks for your reply, got it, my server has 128GB RAM and 1Gbps bandwidth in data center. In addition, I noticed that the fabric performance will drop drastically if my ip list over 5000 lines, fabric output was very slow at that moment, how to optimize it?

Best Regards,
Leon from smart phone

From: Nitesh Nanotkar [mailto:***@fecdirect.net]
Time: Dec 4th, 2016 2:31 P.M.
To: Lee, Leon [mailto:***@jd.com]
Cc: fab-***@nongnu.org; ***@bitprophet.org
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

Hi,
Pool size depends on your central server bandwidth and ram usable. You can specify n no. of ips in list.
In my scenario I use pool size to be 300 when I just want to connect to servers and transfer files in kbs. I use 50 pool size when I have to transfer installation scripts which is of 950MB.
My server's upload speed is 320Mbps and has RAM of 256GB.
Hope you understood on which factors pool size depends now.
Thanks!

On 4 Dec 2016 11:14 am, "李鞥" <***@jd.com<mailto:***@jd.com>> wrote:
If I can re-express that: what is the recommended value of pool_size? – Leon

From: Lee, Leon
Time: Dec 4th, 2016 10:45 A.M.
To: ***@bitprophet.org<mailto:***@bitprophet.org>; fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Cc: Lee, Leon [mailto:***@jd.com]
Subject: Hi there, two questions from newcomer.
Dear Jeff and All,

This is Leon from Beijing, I already installed fabric for my Linux server on Sep 2nd, 2016 in order to manage over ten thousand infrastructure devices (such as the server, storage and switch
.etc) by SSH protocol. Just two questions at this time:
1.) how many “@parallel(pool_size)” should I specify in fabfile.py and what is a proper value for this parameter?
2.) what is the maximum number of rows in ip address list?
The following is my environment information, just waiting for your suggestions. Thanks in advance.

# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
Author-email: python-***@python.org<mailto:python-***@python.org>
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
Requires:
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto

Best Regards,
Leon Lee

[Nitesh Nanotkar]

It may happen due to some slow and not connecting servers. As the pool size
is constant, over a time not connecting or slow responding servers get
accumulated in that pool. Leaving less count for good servers. which drops
performance.
Use try and except in your fabric for connection. Keep connection timeout
10sec and connection attempts 2.

On 4 Dec 2016 1:03 pm, "李鞥" <***@jd.com> wrote:

Hi Nitesh,



Thanks for your reply, got it, my server has 128GB RAM and 1Gbps bandwidth
in data center. In addition, I noticed that the fabric performance will
drop drastically if my ip list over 5000 lines, fabric output was very slow
at that moment, how to optimize it?



Best Regards,

Leon from smart phone



*From:* Nitesh Nanotkar [mailto:***@fecdirect.net]
*Time:* Dec 4th, 2016 2:31 P.M.
*To:* Lee, Leon [mailto:***@jd.com]
*Cc:* fab-***@nongnu.org; ***@bitprophet.org
*Subject:* Re: [Fab-user] Hi there, two questions from newcomer.



Hi,

Pool size depends on your central server bandwidth and ram usable. You
can specify n no. of ips in list.

In my scenario I use pool size to be 300 when I just want to connect to
servers and transfer files in kbs. I use 50 pool size when I have to
transfer installation scripts which is of 950MB.

My server's upload speed is 320Mbps and has RAM of 256GB.

Hope you understood on which factors pool size depends now.

Thanks!



On 4 Dec 2016 11:14 am, "李鞥" <***@jd.com> wrote:

If I can re-express that: what is the recommended value of pool_size? – Leon

*From:* Lee, Leon

*Time:* Dec 4th, 2016 10:45 A.M.
*To:* ***@bitprophet.org; fab-***@nongnu.org
*Cc:* Lee, Leon [mailto:***@jd.com]

*Subject:* Hi there, two questions from newcomer.

Dear Jeff and All,



This is Leon from Beijing, I already installed fabric for my Linux server
on Sep 2nd, 2016 in order to manage over ten thousand infrastructure
devices (such as the server, storage and switch
.etc) by SSH protocol. Just
two questions at this time:

1.) how many “@parallel(pool_size)” should I specify in fabfile.py and what
is a proper value for this parameter?

2.) what is the maximum number of rows in ip address list?

The following is my environment information, just waiting for your
suggestions. Thanks in advance.



# ulimit -n

65535

# pip show python

Name: Python

Version: 2.7.12

Summary: A high-level object-oriented programming language

Home-page: http://www.python.org/2.7

Author: Guido van Rossum and the Python community

Author-email: python-***@python.org

License: PSF license

Location: /usr/local/python27/lib/python2.7/lib-dynload

Requires:

# pip show fabric

Name: Fabric

Version: 1.12.0

Summary: Fabric is a simple, Pythonic tool for remote execution and
deployment.

Home-page: http://fabfile.org

Author: Jeff Forcier

Author-email: ***@bitprophet.org

License: UNKNOWN

Location: /usr/local/python27/lib/python2.7/site-packages

Requires: paramiko

# pip show paramiko

Name: paramiko

Version: 1.17.2

Summary: SSH2 protocol library

Home-page: https://github.com/paramiko/paramiko/

Author: Jeff Forcier

Author-email: ***@bitprophet.org

License: LGPL

Location: /usr/local/python27/lib/python2.7/site-packages

Requires: ecdsa, pycrypto



Best Regards,
Leon Lee

[李鸥]

Sorry, I forgot to tell you that I disabled “@parallel(pool_size)” when I put over 5000 ip to the list, just used a single thread to run my task. Is there any other good idea?

Regards,
Leon

From: Nitesh Nanotkar [mailto:***@fecdirect.net]
Time: Dec 4th, 2016 3:40 P.M.
To: Lee, Leon [mailto:***@jd.com]
Cc: ***@bitprophet.org; fab-***@nongnu.org
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

It may happen due to some slow and not connecting servers. As the pool size is constant, over a time not connecting or slow responding servers get accumulated in that pool. Leaving less count for good servers. which drops performance.
Use try and except in your fabric for connection. Keep connection timeout 10sec and connection attempts 2.

On 4 Dec 2016 1:03 pm, "李鞥" <***@jd.com<mailto:***@jd.com>> wrote:

Hi Nitesh,

Thanks for your reply, got it, my server has 128GB RAM and 1Gbps bandwidth in data center. In addition, I noticed that the fabric performance will drop drastically if my ip list over 5000 lines, fabric output was very slow at that moment, how to optimize it?

Best Regards,
Leon from smart phone

From: Nitesh Nanotkar [mailto:***@fecdirect.net<mailto:***@fecdirect.net>]
Time: Dec 4th, 2016 2:31 P.M.
To: Lee, Leon [mailto:***@jd.com<mailto:***@jd.com>]
Cc: fab-***@nongnu.org<mailto:fab-***@nongnu.org>; ***@bitprophet.org<mailto:***@bitprophet.org>
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

Hi,
Pool size depends on your central server bandwidth and ram usable. You can specify n no. of ips in list.
In my scenario I use pool size to be 300 when I just want to connect to servers and transfer files in kbs. I use 50 pool size when I have to transfer installation scripts which is of 950MB.
My server's upload speed is 320Mbps and has RAM of 256GB.
Hope you understood on which factors pool size depends now.
Thanks!

On 4 Dec 2016 11:14 am, "李鞥" <***@jd.com<mailto:***@jd.com>> wrote:

If I can re-express that: what is the recommended value of pool_size?

Regards,
Leon
From: Lee, Leon

Time: Dec 4th, 2016 10:45 A.M.
To: ***@bitprophet.org<mailto:***@bitprophet.org>; fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Cc: Lee, Leon [mailto:***@jd.com<mailto:***@jd.com>]

Subject: Hi there, two questions from newcomer.

Dear Jeff and All,

This is Leon from Beijing, I already installed fabric for my Linux server on Sep 2nd, 2016 in order to manage over ten thousand infrastructure devices (such as the server, storage and switch
.etc) by SSH protocol. Just two questions at this time:
1.) how many “@parallel(pool_size)” should I specify in fabfile.py and what is a proper value for this parameter?
2.) what is the maximum number of rows in ip address list?
The following is my environment information, just waiting for your suggestions. Thanks in advance.

# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
Author-email: python-***@python.org<mailto:python-***@python.org>
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
Requires:
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto

Best Regards,
Leon Lee

[李鸥]

I have to reduce the ip list and run fabric one more time. ☹
# sed -n “1, 3000p” total > ip
# fab task_name
# sed -n “3000, 6000p” total > ip
# fab task_name

Regards,
Leon

From: Lee, Leon
Time: Dec 4th, 2016 4:53 P.M.
To: Nitesh Nanotkar
Cc: ***@bitprophet.org; fab-***@nongnu.org
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

Sorry, I forgot to tell you that I disabled “@parallel(pool_size)” when I put over 5000 ip to the list, just used a single thread to run my task. Is there any other good idea?

Regards,
Leon

From: Nitesh Nanotkar [mailto:***@fecdirect.net]
Time: Dec 4th, 2016 3:40 P.M.
To: Lee, Leon [mailto:***@jd.com]
Cc: ***@bitprophet.org<mailto:***@bitprophet.org>; fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

It may happen due to some slow and not connecting servers. As the pool size is constant, over a time not connecting or slow responding servers get accumulated in that pool. Leaving less count for good servers. which drops performance.
Use try and except in your fabric for connection. Keep connection timeout 10sec and connection attempts 2.

On 4 Dec 2016 1:03 pm, "李鞥" <***@jd.com<mailto:***@jd.com>> wrote:

Hi Nitesh,

Thanks for your reply, got it, my server has 128GB RAM and 1Gbps bandwidth in data center. In addition, I noticed that the fabric performance will drop drastically if my ip list over 5000 lines, fabric output was very slow at that moment, how to optimize it?

Best Regards,
Leon from smart phone

From: Nitesh Nanotkar [mailto:***@fecdirect.net<mailto:***@fecdirect.net>]
Time: Dec 4th, 2016 2:31 P.M.
To: Lee, Leon [mailto:***@jd.com<mailto:***@jd.com>]
Cc: fab-***@nongnu.org<mailto:fab-***@nongnu.org>; ***@bitprophet.org<mailto:***@bitprophet.org>
Subject: Re: [Fab-user] Hi there, two questions from newcomer.

Hi,
Pool size depends on your central server bandwidth and ram usable. You can specify n no. of ips in list.
In my scenario I use pool size to be 300 when I just want to connect to servers and transfer files in kbs. I use 50 pool size when I have to transfer installation scripts which is of 950MB.
My server's upload speed is 320Mbps and has RAM of 256GB.
Hope you understood on which factors pool size depends now.
Thanks!

On 4 Dec 2016 11:14 am, "李鞥" <***@jd.com<mailto:***@jd.com>> wrote:

If I can re-express that: what is the recommended value of pool_size?

Regards,
Leon
From: Lee, Leon

Time: Dec 4th, 2016 10:45 A.M.
To: ***@bitprophet.org<mailto:***@bitprophet.org>; fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Cc: Lee, Leon [mailto:***@jd.com<mailto:***@jd.com>]

Subject: Hi there, two questions from newcomer.

Dear Jeff and All,

This is Leon from Beijing, I already installed fabric for my Linux server on Sep 2nd, 2016 in order to manage over ten thousand infrastructure devices (such as the server, storage and switch
.etc) by SSH protocol. Just two questions at this time:
1.) how many “@parallel(pool_size)” should I specify in fabfile.py and what is a proper value for this parameter?
2.) what is the maximum number of rows in ip address list?
The following is my environment information, just waiting for your suggestions. Thanks in advance.

# ulimit -n
65535
# pip show python
Name: Python
Version: 2.7.12
Summary: A high-level object-oriented programming language
Home-page: http://www.python.org/2.7
Author: Guido van Rossum and the Python community
Author-email: python-***@python.org<mailto:python-***@python.org>
License: PSF license
Location: /usr/local/python27/lib/python2.7/lib-dynload
Requires:
# pip show fabric
Name: Fabric
Version: 1.12.0
Summary: Fabric is a simple, Pythonic tool for remote execution and deployment.
Home-page: http://fabfile.org
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: UNKNOWN
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: paramiko
# pip show paramiko
Name: paramiko
Version: 1.17.2
Summary: SSH2 protocol library
Home-page: https://github.com/paramiko/paramiko/
Author: Jeff Forcier
Author-email: ***@bitprophet.org<mailto:***@bitprophet.org>
License: LGPL
Location: /usr/local/python27/lib/python2.7/site-packages
Requires: ecdsa, pycrypto

Best Regards,
Leon Lee