Node Management
===============

managed node behavior
---------------------

node "stalling"
^^^^^^^^^^^^^^^

There is one difference in how nodes behave when they're managed
(i.e. in `MANAGED` mode), compared to their base behavior in `AUTO`
mode.

In `MANAGED` mode, nodes don't automatically recover after jump
transitions.  They instead hold in the state they jumped to.  This is
called a "STALL".

This allows the manager to see that there has been a jump and
coordinate it's recovery as needed.

NodeManager interface
---------------------

The :class:`~guardian.NodeManager` provides an interface whereby one
guardian node can "manage" other nodes.  The
:class:`~guardian.NodeManager` object has methods for fully
controlling subordinate nodes, as well as monitoring their state,
status, and progress towards achieving their requests.

The :class:`~guardian.NodeManager` is instantiated in the main body of
the module by passing it a list of nodes to be managed::

  from guardian import NodeManager

  nodes = NodeManager(['SUS_MC1', 'SUS_MC2', 'SUS_MC3'])

Guardian will initialize connections to the nodes automatically.
The `nodes` object is then usable throughout the system to manage the
specified nodes.

managing nodes
--------------

If the manager is going to be setting the requests of the
subordinates, it should set the nodes to be in `MANAGED` mode in the
`INIT` state::

  class INIT(GuardState):
      def main(self):
          nodes.set_managed()
          ...

Requests can be made of the nodes, and their progress can be
monitored by inspecting their state::

  # set the request
  nodes['SUS_MC2'] = 'ALIGNED'
  # check the current state
  if nodes['SUS_MC2'] == 'ALIGNED':
      ...

The arrived property is `True` if all nodes have arrived at their
requested states::

  if nodes . arrived :
      ...

reviving stalled nodes
^^^^^^^^^^^^^^^^^^^^^^

If a managed node has "stalled", i.e. experienced a jump
transition, there are two ways to revive it:

* issue a new request::

    if nodes['SUS_MC2'].stalled:
        nodes['SUS_MC2'] = 'ALIGNED'

* issue a :py:meth:`guardian.Node.revive` command, which
  re-requests the last requested state::

    for node in nodes.get_stalled_nodes():
        node.revive()

checking node status
^^^^^^^^^^^^^^^^^^^^

The checker method returns a decorator that looks for faults
in the nodes. It will report if there are connection errors, node
errors, notifications, or if the node mode has been changed::

  @nodes.checker()
  def main(self):
      ...

It only reports via the `NOTIFICATION` interface, unless specifically
told to jump if there is a fault::

  @nodes.checker(fail_return='DOWN')
  def main(self):
      ...

The node checker should be run in all states.

Node and NodeManager classes
----------------------------

.. autoclass:: guardian.NodeManager
   :members:

.. autoclass:: guardian.Node
   :members: