Discussion:
[Fab-user] New to fabric 2. Using groups with Connection?
Christian MacNevin
2018-07-23 22:45:16 UTC
Permalink
Hi all,
I'm new to Fabric 2, though in the past I'd used 1 a lot. I can find docs on using SerialGroup, but only with run directly. I'm using interactive mode authentication, so it seems like I need to gather a password (that's fine, I'm doing it via getpass) and also pass kwargs into Connection. I have a large list of hosts, and also was planning to use @task decorators.

What I have right now is a muddle between the two which obviously won't work, but I can't figure out how to glue the three
concepts of having a defined group, a task which can be specified from the cli, and an interactive user/pass sequence together.


import getpass
from fabric import Connection, group
from invoke import task

switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')

pw = getpass.getpass("Password for netuser?")

@task
def inventory(c): <-- This is purely here because tasks will error out if a kwarg 'Context(..?)' isn't specified
with Connection(user = 'user', connect_kwargs = {'password' : pw }) as c:
switch_group.run('show chassis hardware')

It's failing out with 'TypeError: __init__() takes at least 2 arguments (3 given)'
Jeff Forcier
2018-07-24 04:55:31 UTC
Permalink
Hi Christian,

First, do note you can always fall back to Fabric 1 for a while if Fabric 2
isn't yet up to snuff for your particular use case. We're still feeling out
some of the APIs in terms of ease of use. Unfortunately, Group is one of
those - it's real basic right now.

I just looked and there's no great way to set global connect_kwargs in such
an object, without creating your own Connections to hand to its alternate
constructor 'from_connections()':

```
hosts = ['switch1', 'switch2', 'switch3']
connections = [Connection(host=host, user='user',
connect_kwargs={'password': 'pw'}) for host in hosts]
switch_group = group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group.run('show chassis hardware')
```

That's not the end of the world, but it's not ideal, so I just made
https://github.com/fabric/fabric/issues/1831 and then (since I was thinking
about it and it was very easy for once) implemented it. It'll be out in the
next feature release!

The rest of your issues are more high level, so here's a bit of a wall of
text (sorry!):

Version 2 is much less implicit/magic than v1 – all objects need to fit
together in obvious, Pythonic ways (such as being handed to one another in
constructors or method calls). For example, context managers don't change
any object besides the one that they yield, so trying to do things to
switch_group inside a context manager about some other Connection object
won't do anything. (This is as opposed to e.g. 'with settings()' in v1,
which I assume you were thinking of; it, like the rest of that API, is all
about magically twiddling data behind the scenes.)

Connections can't be created without some 'host' argument, which I think is
triggering your TypeError. This ought to be in its __init__ docs, FWIW :)
See the code example above; you either need to make your Group from
explicitly-created Connections, where a host arg is being given, or (after
the next release) you can tell the Group's constructor about connect_kwargs.

Depending on your use case, there's a CLI flag that may replace your
getpass code:
http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-for-login-password

Which brings us to that context argument in the task signature: that's how
`fab` explicitly transmits CLI and config info to your tasks, including the
connection password. However, this introduces some ordering problems with
your code - you need to pass that context's config into your Group or
Connections.

Should be easily solved by moving away from module-level code exec (which
is honestly an antipattern anyways). Based on my above snippet:

```
hosts = ['switch1', 'switch2', 'switch3']

def make_group(c):
connections = [
Connection(host=host, user='user', config=c.config)
for host in hosts
]
return group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group = make_group(c)
switch_group.run('show chassis hardware')
```

Or, after 2.3 comes out, you could arguably nix make_group() again, though
hopefully you can see the various opportunities for refactoring, given that
everything is explicit:

```
hosts = ['switch1', 'switch2', 'switch3']

@task
def inventory(c):
switch_group = SerialGroup(*hosts, config=c.config)
switch_group.run('show chassis hardware')
```

Finally: if you're thinking "ok, but it'd be so nice to just plop my switch
hostnames and the fact that I want password prompting all the time, into a
config file and call it done": that sort of thing is definitely in the
pipeline! As I said up top, we're still hashing out a lot of the
'convenience' angles here, even though the core APIs are mostly in good
shape.

Hope that all helps,
Jeff


On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com>
wrote:

> Hi all,
>
> I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs
> on using SerialGroup, but only with run directly. I’m using interactive
> mode authentication, so it seems like I need to gather a password (that’s
> fine, I’m doing it via getpass) and also pass kwargs into Connection. I
> have a large list of hosts, and also was planning to use @task decorators.
>
>
>
> What I have right now is a muddle between the two which obviously won’t
> work, but I can’t figure out how to glue the three
> concepts of having a defined group, a task which can be specified from the
> cli, and an interactive user/pass sequence together.
>
>
>
>
>
> import getpass
>
> from fabric import Connection, group
>
> from invoke import task
>
>
>
> switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')
>
>
>
> pw = getpass.getpass("Password for netuser?")
>
>
>
> @task
>
> def inventory(c): ß This is purely here because tasks will error out
> if a kwarg ‘Context(..?)’ isn’t specified
>
> with Connection(user = 'user', connect_kwargs = {'password' : pw }) as
> c:
>
> switch_group.run('show chassis hardware')
>
>
>
> It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3
> given)’
>
>
>
>
>
> _______________________________________________
> Fab-user mailing list
> Fab-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/fab-user
>
>


--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Christian MacNevin
2018-07-24 17:38:57 UTC
Permalink
Thanks so much for the epic response 😊 So far Fabric 2 seems more sensible in a lot of ways, and since my old codebase was so integrated into a former employer’s infrastructure, I’m going to keep trying to operate in the new world.

I’ll work through the workaround for now. When is 2.3 likely to drop?



From: ***@gmail.com <***@gmail.com> On Behalf Of Jeff Forcier
Sent: Monday, July 23, 2018 9:56 PM
To: Christian MacNevin <***@nvidia.com>
Cc: fab-***@nongnu.org
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

Hi Christian,

First, do note you can always fall back to Fabric 1 for a while if Fabric 2 isn't yet up to snuff for your particular use case. We're still feeling out some of the APIs in terms of ease of use. Unfortunately, Group is one of those - it's real basic right now.

I just looked and there's no great way to set global connect_kwargs in such an object, without creating your own Connections to hand to its alternate constructor 'from_connections()':

```
hosts = ['switch1', 'switch2', 'switch3']
connections = [Connection(host=host, user='user', connect_kwargs={'password': 'pw'}) for host in hosts]
switch_group = group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group.run('show chassis hardware')
```

That's not the end of the world, but it's not ideal, so I just made https://github.com/fabric/fabric/issues/1831 and then (since I was thinking about it and it was very easy for once) implemented it. It'll be out in the next feature release!

The rest of your issues are more high level, so here's a bit of a wall of text (sorry!):

Version 2 is much less implicit/magic than v1 – all objects need to fit together in obvious, Pythonic ways (such as being handed to one another in constructors or method calls). For example, context managers don't change any object besides the one that they yield, so trying to do things to switch_group inside a context manager about some other Connection object won't do anything. (This is as opposed to e.g. 'with settings()' in v1, which I assume you were thinking of; it, like the rest of that API, is all about magically twiddling data behind the scenes.)

Connections can't be created without some 'host' argument, which I think is triggering your TypeError. This ought to be in its __init__ docs, FWIW :) See the code example above; you either need to make your Group from explicitly-created Connections, where a host arg is being given, or (after the next release) you can tell the Group's constructor about connect_kwargs.

Depending on your use case, there's a CLI flag that may replace your getpass code: http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-for-login-password

Which brings us to that context argument in the task signature: that's how `fab` explicitly transmits CLI and config info to your tasks, including the connection password. However, this introduces some ordering problems with your code - you need to pass that context's config into your Group or Connections.

Should be easily solved by moving away from module-level code exec (which is honestly an antipattern anyways). Based on my above snippet:

```
hosts = ['switch1', 'switch2', 'switch3']

def make_group(c):
connections = [
Connection(host=host, user='user', config=c.config)
for host in hosts
]
return group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group = make_group(c)
switch_group.run('show chassis hardware')
```

Or, after 2.3 comes out, you could arguably nix make_group() again, though hopefully you can see the various opportunities for refactoring, given that everything is explicit:

```
hosts = ['switch1', 'switch2', 'switch3']

@task
def inventory(c):
switch_group = SerialGroup(*hosts, config=c.config)
switch_group.run('show chassis hardware')
```

Finally: if you're thinking "ok, but it'd be so nice to just plop my switch hostnames and the fact that I want password prompting all the time, into a config file and call it done": that sort of thing is definitely in the pipeline! As I said up top, we're still hashing out a lot of the 'convenience' angles here, even though the core APIs are mostly in good shape.

Hope that all helps,
Jeff


On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
Hi all,
I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs on using SerialGroup, but only with run directly. I’m using interactive mode authentication, so it seems like I need to gather a password (that’s fine, I’m doing it via getpass) and also pass kwargs into Connection. I have a large list of hosts, and also was planning to use @task decorators.

What I have right now is a muddle between the two which obviously won’t work, but I can’t figure out how to glue the three
concepts of having a defined group, a task which can be specified from the cli, and an interactive user/pass sequence together.


import getpass
from fabric import Connection, group
from invoke import task

switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')

pw = getpass.getpass("Password for netuser?")

@task
def inventory(c): <-- This is purely here because tasks will error out if a kwarg ‘Context(..?)’ isn’t specified
with Connection(user = 'user', connect_kwargs = {'password' : pw }) as c:
switch_group.run('show chassis hardware')

It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3 given)’



_______________________________________________
Fab-user mailing list
Fab-***@nongnu.org<mailto:Fab-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/fab-user



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Jeff Forcier
2018-07-24 18:14:43 UTC
Permalink
I'd like to bang out another 1 or 2 nice new features before I drop 2.3,
but even if I don't, I'll make sure to push it out by early next week.

On Tue, Jul 24, 2018 at 10:38 AM, Christian MacNevin <***@nvidia.com>
wrote:

> Thanks so much for the epic response [image: 😊] So far Fabric 2 seems
> more sensible in a lot of ways, and since my old codebase was so integrated
> into a former employer’s infrastructure, I’m going to keep trying to
> operate in the new world.
>
> I’ll work through the workaround for now. When is 2.3 likely to drop?
>
>
>
>
>
>
>
> *From:* ***@gmail.com <***@gmail.com> * On Behalf Of *Jeff
> Forcier
> *Sent:* Monday, July 23, 2018 9:56 PM
> *To:* Christian MacNevin <***@nvidia.com>
> *Cc:* fab-***@nongnu.org
> *Subject:* Re: [Fab-user] New to fabric 2. Using groups with Connection?
>
>
>
> Hi Christian,
>
>
>
> First, do note you can always fall back to Fabric 1 for a while if Fabric
> 2 isn't yet up to snuff for your particular use case. We're still feeling
> out some of the APIs in terms of ease of use. Unfortunately, Group is one
> of those - it's real basic right now.
>
>
>
> I just looked and there's no great way to set global connect_kwargs in
> such an object, without creating your own Connections to hand to its
> alternate constructor 'from_connections()':
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
> connections = [Connection(host=host, user='user',
> connect_kwargs={'password': 'pw'}) for host in hosts]
>
> switch_group = group.SerialGroup.from_connections(connections)
>
>
>
> @task
>
> def inventory(c):
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> That's not the end of the world, but it's not ideal, so I just made
> https://github.com/fabric/fabric/issues/1831 and then (since I was
> thinking about it and it was very easy for once) implemented it. It'll be
> out in the next feature release!
>
>
>
> The rest of your issues are more high level, so here's a bit of a wall of
> text (sorry!):
>
>
>
> Version 2 is much less implicit/magic than v1 – all objects need to fit
> together in obvious, Pythonic ways (such as being handed to one another in
> constructors or method calls). For example, context managers don't change
> any object besides the one that they yield, so trying to do things to
> switch_group inside a context manager about some other Connection object
> won't do anything. (This is as opposed to e.g. 'with settings()' in v1,
> which I assume you were thinking of; it, like the rest of that API, is all
> about magically twiddling data behind the scenes.)
>
>
>
> Connections can't be created without some 'host' argument, which I think
> is triggering your TypeError. This ought to be in its __init__ docs, FWIW
> :) See the code example above; you either need to make your Group from
> explicitly-created Connections, where a host arg is being given, or (after
> the next release) you can tell the Group's constructor about connect_kwargs.
>
>
>
> Depending on your use case, there's a CLI flag that may replace your
> getpass code: http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-
> for-login-password
>
>
>
> Which brings us to that context argument in the task signature: that's how
> `fab` explicitly transmits CLI and config info to your tasks, including the
> connection password. However, this introduces some ordering problems with
> your code - you need to pass that context's config into your Group or
> Connections.
>
>
>
> Should be easily solved by moving away from module-level code exec (which
> is honestly an antipattern anyways). Based on my above snippet:
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
>
>
> def make_group(c):
>
> connections = [
>
> Connection(host=host, user='user', config=c.config)
>
> for host in hosts
>
> ]
>
> return group.SerialGroup.from_connections(connections)
>
>
>
> @task
>
> def inventory(c):
>
> switch_group = make_group(c)
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> Or, after 2.3 comes out, you could arguably nix make_group() again, though
> hopefully you can see the various opportunities for refactoring, given that
> everything is explicit:
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
>
>
> @task
>
> def inventory(c):
>
> switch_group = SerialGroup(*hosts, config=c.config)
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> Finally: if you're thinking "ok, but it'd be so nice to just plop my
> switch hostnames and the fact that I want password prompting all the time,
> into a config file and call it done": that sort of thing is definitely in
> the pipeline! As I said up top, we're still hashing out a lot of the
> 'convenience' angles here, even though the core APIs are mostly in good
> shape.
>
>
>
> Hope that all helps,
>
> Jeff
>
>
>
>
>
> On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com>
> wrote:
>
> Hi all,
>
> I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs
> on using SerialGroup, but only with run directly. I’m using interactive
> mode authentication, so it seems like I need to gather a password (that’s
> fine, I’m doing it via getpass) and also pass kwargs into Connection. I
> have a large list of hosts, and also was planning to use @task decorators.
>
>
>
> What I have right now is a muddle between the two which obviously won’t
> work, but I can’t figure out how to glue the three
> concepts of having a defined group, a task which can be specified from the
> cli, and an interactive user/pass sequence together.
>
>
>
>
>
> import getpass
>
> from fabric import Connection, group
>
> from invoke import task
>
>
>
> switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')
>
>
>
> pw = getpass.getpass("Password for netuser?")
>
>
>
> @task
>
> def inventory(c): ß This is purely here because tasks will error out
> if a kwarg ‘Context(..?)’ isn’t specified
>
> with Connection(user = 'user', connect_kwargs = {'password' : pw }) as
> c:
>
> switch_group.run('show chassis hardware')
>
>
>
> It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3
> given)’
>
>
>
>
>
>
> _______________________________________________
> Fab-user mailing list
> Fab-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/fab-user
>
>
>
>
>
> --
>
> Jeff Forcier
> Unix sysadmin; Python engineer
> http://bitprophet.org
> ------------------------------
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Christian MacNevin
2018-07-24 20:53:09 UTC
Permalink
So, I implemented it as you suggested. It works for most of the hosts (35 out of 42), but it finishes with:

File "/home/cmacnevin/fab/local/lib/python2.7/site-packages/fabric/group.py", line 153, in run
raise GroupException(results)
fabric.exceptions.GroupException

I’d like to be able to try and view the GroupResults.failed to see which hosts didn’t make it, as with this method,
the output is printed to STDOUT but the hostnames aren’t. So without a dict(like) object to query and without the
hostname explicitly stated, I don’t know who’s who or who isn’t who 😊


From: ***@gmail.com <***@gmail.com> On Behalf Of Jeff Forcier
Sent: Tuesday, July 24, 2018 11:15 AM
To: Christian MacNevin <***@nvidia.com>
Cc: fab-***@nongnu.org
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

I'd like to bang out another 1 or 2 nice new features before I drop 2.3, but even if I don't, I'll make sure to push it out by early next week.

On Tue, Jul 24, 2018 at 10:38 AM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
Thanks so much for the epic response So far Fabric 2 seems more sensible in a lot of ways, and since my old codebase was so integrated into a former employer’s infrastructure, I’m going to keep trying to operate in the new world.

I’ll work through the workaround for now. When is 2.3 likely to drop?



From: ***@gmail.com<mailto:***@gmail.com> <***@gmail.com<mailto:***@gmail.com>> On Behalf Of Jeff Forcier
Sent: Monday, July 23, 2018 9:56 PM
To: Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>>
Cc: fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

Hi Christian,

First, do note you can always fall back to Fabric 1 for a while if Fabric 2 isn't yet up to snuff for your particular use case. We're still feeling out some of the APIs in terms of ease of use. Unfortunately, Group is one of those - it's real basic right now.

I just looked and there's no great way to set global connect_kwargs in such an object, without creating your own Connections to hand to its alternate constructor 'from_connections()':

```
hosts = ['switch1', 'switch2', 'switch3']
connections = [Connection(host=host, user='user', connect_kwargs={'password': 'pw'}) for host in hosts]
switch_group = group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group.run('show chassis hardware')
```

That's not the end of the world, but it's not ideal, so I just made https://github.com/fabric/fabric/issues/1831 and then (since I was thinking about it and it was very easy for once) implemented it. It'll be out in the next feature release!

The rest of your issues are more high level, so here's a bit of a wall of text (sorry!):

Version 2 is much less implicit/magic than v1 – all objects need to fit together in obvious, Pythonic ways (such as being handed to one another in constructors or method calls). For example, context managers don't change any object besides the one that they yield, so trying to do things to switch_group inside a context manager about some other Connection object won't do anything. (This is as opposed to e.g. 'with settings()' in v1, which I assume you were thinking of; it, like the rest of that API, is all about magically twiddling data behind the scenes.)

Connections can't be created without some 'host' argument, which I think is triggering your TypeError. This ought to be in its __init__ docs, FWIW :) See the code example above; you either need to make your Group from explicitly-created Connections, where a host arg is being given, or (after the next release) you can tell the Group's constructor about connect_kwargs.

Depending on your use case, there's a CLI flag that may replace your getpass code: http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-for-login-password

Which brings us to that context argument in the task signature: that's how `fab` explicitly transmits CLI and config info to your tasks, including the connection password. However, this introduces some ordering problems with your code - you need to pass that context's config into your Group or Connections.

Should be easily solved by moving away from module-level code exec (which is honestly an antipattern anyways). Based on my above snippet:

```
hosts = ['switch1', 'switch2', 'switch3']

def make_group(c):
connections = [
Connection(host=host, user='user', config=c.config)
for host in hosts
]
return group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group = make_group(c)
switch_group.run('show chassis hardware')
```

Or, after 2.3 comes out, you could arguably nix make_group() again, though hopefully you can see the various opportunities for refactoring, given that everything is explicit:

```
hosts = ['switch1', 'switch2', 'switch3']

@task
def inventory(c):
switch_group = SerialGroup(*hosts, config=c.config)
switch_group.run('show chassis hardware')
```

Finally: if you're thinking "ok, but it'd be so nice to just plop my switch hostnames and the fact that I want password prompting all the time, into a config file and call it done": that sort of thing is definitely in the pipeline! As I said up top, we're still hashing out a lot of the 'convenience' angles here, even though the core APIs are mostly in good shape.

Hope that all helps,
Jeff


On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
Hi all,
I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs on using SerialGroup, but only with run directly. I’m using interactive mode authentication, so it seems like I need to gather a password (that’s fine, I’m doing it via getpass) and also pass kwargs into Connection. I have a large list of hosts, and also was planning to use @task decorators.

What I have right now is a muddle between the two which obviously won’t work, but I can’t figure out how to glue the three
concepts of having a defined group, a task which can be specified from the cli, and an interactive user/pass sequence together.


import getpass
from fabric import Connection, group
from invoke import task

switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')

pw = getpass.getpass("Password for netuser?")

@task
def inventory(c): <-- This is purely here because tasks will error out if a kwarg ‘Context(..?)’ isn’t specified
with Connection(user = 'user', connect_kwargs = {'password' : pw }) as c:
switch_group.run('show chassis hardware')

It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3 given)’



_______________________________________________
Fab-user mailing list
Fab-***@nongnu.org<mailto:Fab-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/fab-user



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
________________________________
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Jeff Forcier
2018-07-24 22:54:17 UTC
Permalink
​Last I checked, GroupException should be a wrapper around what would've
been the GroupResult, tho I see its API docs don't actually mention HOW to
get that result object; it's simply the `.result` attribute on the
exception instance. I'm amending the docs for this now, so cheers :)

(GroupResult's API docs are fleshed out already:
http://docs.fabfile.org/en/2.2/api/group.html#fabric.group.GroupResult -
tl;dr it's a dict and the hosts which had problems will have that exception
as their corresponding value, instead of a return value.)

On Tue, Jul 24, 2018 at 1:53 PM, Christian MacNevin <***@nvidia.com>
wrote:

> So, I implemented it as you suggested. It works for most of the hosts (35
> out of 42), but it finishes with:
>
>
>
> File "/home/cmacnevin/fab/local/lib/python2.7/site-packages/fabric/group.py",
> line 153, in run
>
> raise GroupException(results)
>
> fabric.exceptions.GroupException
>
>
>
> I’d like to be able to try and view the GroupResults.failed to see which
> hosts didn’t make it, as with this method,
> the output is printed to STDOUT but the hostnames aren’t. So without a
> dict(like) object to query and without the
> hostname explicitly stated, I don’t know who’s who or who isn’t who 😊
>
>
>
>
>
> *From:* ***@gmail.com <***@gmail.com> * On Behalf Of *Jeff
> Forcier
> *Sent:* Tuesday, July 24, 2018 11:15 AM
>
> *To:* Christian MacNevin <***@nvidia.com>
> *Cc:* fab-***@nongnu.org
> *Subject:* Re: [Fab-user] New to fabric 2. Using groups with Connection?
>
>
>
> I'd like to bang out another 1 or 2 nice new features before I drop 2.3,
> but even if I don't, I'll make sure to push it out by early next week.
>
>
>
> On Tue, Jul 24, 2018 at 10:38 AM, Christian MacNevin <***@nvidia.com>
> wrote:
>
> Thanks so much for the epic response So far Fabric 2 seems more sensible
> in a lot of ways, and since my old codebase was so integrated into a former
> employer’s infrastructure, I’m going to keep trying to operate in the new
> world.
>
> I’ll work through the workaround for now. When is 2.3 likely to drop?
>
>
>
>
>
>
>
> *From:* ***@gmail.com <***@gmail.com> *On Behalf Of *Jeff
> Forcier
> *Sent:* Monday, July 23, 2018 9:56 PM
> *To:* Christian MacNevin <***@nvidia.com>
> *Cc:* fab-***@nongnu.org
> *Subject:* Re: [Fab-user] New to fabric 2. Using groups with Connection?
>
>
>
> Hi Christian,
>
>
>
> First, do note you can always fall back to Fabric 1 for a while if Fabric
> 2 isn't yet up to snuff for your particular use case. We're still feeling
> out some of the APIs in terms of ease of use. Unfortunately, Group is one
> of those - it's real basic right now.
>
>
>
> I just looked and there's no great way to set global connect_kwargs in
> such an object, without creating your own Connections to hand to its
> alternate constructor 'from_connections()':
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
> connections = [Connection(host=host, user='user',
> connect_kwargs={'password': 'pw'}) for host in hosts]
>
> switch_group = group.SerialGroup.from_connections(connections)
>
>
>
> @task
>
> def inventory(c):
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> That's not the end of the world, but it's not ideal, so I just made
> https://github.com/fabric/fabric/issues/1831 and then (since I was
> thinking about it and it was very easy for once) implemented it. It'll be
> out in the next feature release!
>
>
>
> The rest of your issues are more high level, so here's a bit of a wall of
> text (sorry!):
>
>
>
> Version 2 is much less implicit/magic than v1 – all objects need to fit
> together in obvious, Pythonic ways (such as being handed to one another in
> constructors or method calls). For example, context managers don't change
> any object besides the one that they yield, so trying to do things to
> switch_group inside a context manager about some other Connection object
> won't do anything. (This is as opposed to e.g. 'with settings()' in v1,
> which I assume you were thinking of; it, like the rest of that API, is all
> about magically twiddling data behind the scenes.)
>
>
>
> Connections can't be created without some 'host' argument, which I think
> is triggering your TypeError. This ought to be in its __init__ docs, FWIW
> :) See the code example above; you either need to make your Group from
> explicitly-created Connections, where a host arg is being given, or (after
> the next release) you can tell the Group's constructor about connect_kwargs.
>
>
>
> Depending on your use case, there's a CLI flag that may replace your
> getpass code: http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-
> for-login-password
>
>
>
> Which brings us to that context argument in the task signature: that's how
> `fab` explicitly transmits CLI and config info to your tasks, including the
> connection password. However, this introduces some ordering problems with
> your code - you need to pass that context's config into your Group or
> Connections.
>
>
>
> Should be easily solved by moving away from module-level code exec (which
> is honestly an antipattern anyways). Based on my above snippet:
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
>
>
> def make_group(c):
>
> connections = [
>
> Connection(host=host, user='user', config=c.config)
>
> for host in hosts
>
> ]
>
> return group.SerialGroup.from_connections(connections)
>
>
>
> @task
>
> def inventory(c):
>
> switch_group = make_group(c)
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> Or, after 2.3 comes out, you could arguably nix make_group() again, though
> hopefully you can see the various opportunities for refactoring, given that
> everything is explicit:
>
>
>
> ```
>
> hosts = ['switch1', 'switch2', 'switch3']
>
>
>
> @task
>
> def inventory(c):
>
> switch_group = SerialGroup(*hosts, config=c.config)
>
> switch_group.run('show chassis hardware')
>
> ```
>
>
>
> Finally: if you're thinking "ok, but it'd be so nice to just plop my
> switch hostnames and the fact that I want password prompting all the time,
> into a config file and call it done": that sort of thing is definitely in
> the pipeline! As I said up top, we're still hashing out a lot of the
> 'convenience' angles here, even though the core APIs are mostly in good
> shape.
>
>
>
> Hope that all helps,
>
> Jeff
>
>
>
>
>
> On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com>
> wrote:
>
> Hi all,
>
> I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs
> on using SerialGroup, but only with run directly. I’m using interactive
> mode authentication, so it seems like I need to gather a password (that’s
> fine, I’m doing it via getpass) and also pass kwargs into Connection. I
> have a large list of hosts, and also was planning to use @task decorators.
>
>
>
> What I have right now is a muddle between the two which obviously won’t
> work, but I can’t figure out how to glue the three
> concepts of having a defined group, a task which can be specified from the
> cli, and an interactive user/pass sequence together.
>
>
>
>
>
> import getpass
>
> from fabric import Connection, group
>
> from invoke import task
>
>
>
> switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')
>
>
>
> pw = getpass.getpass("Password for netuser?")
>
>
>
> @task
>
> def inventory(c): ß This is purely here because tasks will error out
> if a kwarg ‘Context(..?)’ isn’t specified
>
> with Connection(user = 'user', connect_kwargs = {'password' : pw }) as
> c:
>
> switch_group.run('show chassis hardware')
>
>
>
> It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3
> given)’
>
>
>
>
>
>
> _______________________________________________
> Fab-user mailing list
> Fab-***@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/fab-user
>
>
>
>
>
> --
>
> Jeff Forcier
> Unix sysadmin; Python engineer
> http://bitprophet.org
> ------------------------------
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>
>
>
>
>
> --
>
> Jeff Forcier
> Unix sysadmin; Python engineer
> http://bitprophet.org
>



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Christian MacNevin
2018-07-25 15:53:04 UTC
Permalink
So far this has all been very reassuring. Apparently I’m not as dumb as I thought lol

The docs could definitely use examples, tbh. But then you’d hear less from me :)

Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________
From: 30471232320n behalf of
Sent: Tuesday, July 24, 2018 3:55 PM
To: Christian MacNevin
Cc: fab-***@nongnu.org
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

​Last I checked, GroupException should be a wrapper around what would've been the GroupResult, tho I see its API docs don't actually mention HOW to get that result object; it's simply the `.result` attribute on the exception instance. I'm amending the docs for this now, so cheers :)

(GroupResult's API docs are fleshed out already: http://docs.fabfile.org/en/2.2/api/group.html#fabric.group.GroupResult - tl;dr it's a dict and the hosts which had problems will have that exception as their corresponding value, instead of a return value.)

On Tue, Jul 24, 2018 at 1:53 PM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
So, I implemented it as you suggested. It works for most of the hosts (35 out of 42), but it finishes with:

File "/home/cmacnevin/fab/local/lib/python2.7/site-packages/fabric/group.py", line 153, in run
raise GroupException(results)
fabric.exceptions.GroupException

I’d like to be able to try and view the GroupResults.failed to see which hosts didn’t make it, as with this method,
the output is printed to STDOUT but the hostnames aren’t. So without a dict(like) object to query and without the
hostname explicitly stated, I don’t know who’s who or who isn’t who 😊


From: ***@gmail.com<mailto:***@gmail.com> <***@gmail.com<mailto:***@gmail.com>> On Behalf Of Jeff Forcier
Sent: Tuesday, July 24, 2018 11:15 AM

To: Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>>
Cc: fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

I'd like to bang out another 1 or 2 nice new features before I drop 2.3, but even if I don't, I'll make sure to push it out by early next week.

On Tue, Jul 24, 2018 at 10:38 AM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
Thanks so much for the epic response So far Fabric 2 seems more sensible in a lot of ways, and since my old codebase was so integrated into a former employer’s infrastructure, I’m going to keep trying to operate in the new world.

I’ll work through the workaround for now. When is 2.3 likely to drop?



From: ***@gmail.com<mailto:***@gmail.com> <***@gmail.com<mailto:***@gmail.com>> On Behalf Of Jeff Forcier
Sent: Monday, July 23, 2018 9:56 PM
To: Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>>
Cc: fab-***@nongnu.org<mailto:fab-***@nongnu.org>
Subject: Re: [Fab-user] New to fabric 2. Using groups with Connection?

Hi Christian,

First, do note you can always fall back to Fabric 1 for a while if Fabric 2 isn't yet up to snuff for your particular use case. We're still feeling out some of the APIs in terms of ease of use. Unfortunately, Group is one of those - it's real basic right now.

I just looked and there's no great way to set global connect_kwargs in such an object, without creating your own Connections to hand to its alternate constructor 'from_connections()':

```
hosts = ['switch1', 'switch2', 'switch3']
connections = [Connection(host=host, user='user', connect_kwargs={'password': 'pw'}) for host in hosts]
switch_group = group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group.run('show chassis hardware')
```

That's not the end of the world, but it's not ideal, so I just made https://github.com/fabric/fabric/issues/1831 and then (since I was thinking about it and it was very easy for once) implemented it. It'll be out in the next feature release!

The rest of your issues are more high level, so here's a bit of a wall of text (sorry!):

Version 2 is much less implicit/magic than v1 – all objects need to fit together in obvious, Pythonic ways (such as being handed to one another in constructors or method calls). For example, context managers don't change any object besides the one that they yield, so trying to do things to switch_group inside a context manager about some other Connection object won't do anything. (This is as opposed to e.g. 'with settings()' in v1, which I assume you were thinking of; it, like the rest of that API, is all about magically twiddling data behind the scenes.)

Connections can't be created without some 'host' argument, which I think is triggering your TypeError. This ought to be in its __init__ docs, FWIW :) See the code example above; you either need to make your Group from explicitly-created Connections, where a host arg is being given, or (after the next release) you can tell the Group's constructor about connect_kwargs.

Depending on your use case, there's a CLI flag that may replace your getpass code: http://docs.fabfile.org/en/2.2/cli.html#cmdoption-prompt-for-login-password

Which brings us to that context argument in the task signature: that's how `fab` explicitly transmits CLI and config info to your tasks, including the connection password. However, this introduces some ordering problems with your code - you need to pass that context's config into your Group or Connections.

Should be easily solved by moving away from module-level code exec (which is honestly an antipattern anyways). Based on my above snippet:

```
hosts = ['switch1', 'switch2', 'switch3']

def make_group(c):
connections = [
Connection(host=host, user='user', config=c.config)
for host in hosts
]
return group.SerialGroup.from_connections(connections)

@task
def inventory(c):
switch_group = make_group(c)
switch_group.run('show chassis hardware')
```

Or, after 2.3 comes out, you could arguably nix make_group() again, though hopefully you can see the various opportunities for refactoring, given that everything is explicit:

```
hosts = ['switch1', 'switch2', 'switch3']

@task
def inventory(c):
switch_group = SerialGroup(*hosts, config=c.config)
switch_group.run('show chassis hardware')
```

Finally: if you're thinking "ok, but it'd be so nice to just plop my switch hostnames and the fact that I want password prompting all the time, into a config file and call it done": that sort of thing is definitely in the pipeline! As I said up top, we're still hashing out a lot of the 'convenience' angles here, even though the core APIs are mostly in good shape.

Hope that all helps,
Jeff


On Mon, Jul 23, 2018 at 3:45 PM, Christian MacNevin <***@nvidia.com<mailto:***@nvidia.com>> wrote:
Hi all,
I’m new to Fabric 2, though in the past I’d used 1 a lot. I can find docs on using SerialGroup, but only with run directly. I’m using interactive mode authentication, so it seems like I need to gather a password (that’s fine, I’m doing it via getpass) and also pass kwargs into Connection. I have a large list of hosts, and also was planning to use @task decorators.

What I have right now is a muddle between the two which obviously won’t work, but I can’t figure out how to glue the three
concepts of having a defined group, a task which can be specified from the cli, and an interactive user/pass sequence together.


import getpass
from fabric import Connection, group
from invoke import task

switch_group = group.SerialGroup(switch1', 'switch2', 'switch3')

pw = getpass.getpass("Password for netuser?")

@task
def inventory(c): <-- This is purely here because tasks will error out if a kwarg ‘Context(..?)’ isn’t specified
with Connection(user = 'user', connect_kwargs = {'password' : pw }) as c:
switch_group.run('show chassis hardware')

It’s failing out with ’TypeError: __init__() takes at least 2 arguments (3 given)’



_______________________________________________
Fab-user mailing list
Fab-***@nongnu.org<mailto:Fab-***@nongnu.org>
https://lists.nongnu.org/mailman/listinfo/fab-user



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
________________________________
This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org



--
Jeff Forcier
Unix sysadmin; Python engineer
http://bitprophet.org
Loading...