Updating HA GUEST API OVERVIEW DOCUMENT based on Bertrand Souville's comments.

- adding standard Doctor / Vitrage Fault Mgmt Architecture Diagram, and
- refering to Doctor SB API

-------------

Updating HA GUEST API OVERVIEW Proposal after review with OPNFV DOCTOR Team.

Minutes from meeting, that were addressed as part of this update are below.

- VM Heartbeating & Health Checking
  * libvirt watchdog and its integration / use with OpenStack
     > https://blueprints.launchpad.net/nova/+spec/libvirt-watchdog
     > https://wiki.openstack.org/wiki/LibvirtWatchdog#Notifications
     > Need to review behaviour of this capability as compared to the
       proposed VM Heartbeating/Health-checking
     > i believe there is a bit of overlap
       but believe that VM Heartbeating / Health-checking provides a
       more complete solution
     > i'll update document with a NOTE on comparing the proposal with
       the libvirt watchdog
  * Update the architecture diagram to be consistent with the most current
    DOCTOR Architecture Diagrams
     > e.g. Vitrage & Congress are not necessrily deployed at same time
     > use DOCTOR terminology where applicable, e.g. "inspector modules"
     > OPNFV DOCTOR includes patches/components of AODH, NOVA, NEUTRON
       ... not just Vitrage and Congress, as shown in diagram
       - although also had comment that should remove OPNFV DOCTOR outline
         from diagram as OPNFV DOCTOR is a reuirement specification and
         not an implementation
     > indicate (possibly just in text below diagram) that the "Guest
       Heartbeat / Health-check Server' on Controller Node is possibly not
       required, as the Vitrage data source interface can be remotely
       reached by "Guest Heartbeat / Health-check Compute" on the Compute
       Node
  * In text, provide a little more detail on content of actual messaging
    e.g. PDUs and rough message content
  * OVERALL
     > believe there was a general agreement in the way that VM Heartbeating
       & Health-checking was integrated / inter-worked with OPNFV DOCTOR's
       Vitrage / Congress and overall OPNFV fault reporting architecture
     > key feedback was to understand and highlight the additional value of
       the VM Heartbeating & Health-checking functionality over the existing
       libvirt watchdog integration into OpenStack.

- Server Group Messaging
  * suggestion that Rabbit MQ pub/sub messaging could be an alternative for
    routing of messages
     > this is an implementation detail though
  * general discussions on "HA use cases" for how this messaging could be
    leveraged
     > e.g. split-brain avoidance, faster peer VM state change notifications
  * OVERALL
     > agreement that the Server Group Messaging Architecture did NOT conflict
       with Doctor Architecture
     > need further review with OPNFV MANO Team as to how they would position
       this functionality
       e.g.
           - position it as an alternative for various HA use cases ?
           - versus
           - mandating that this service group messaging be used for specific
             HA user cases

Change-Id: Icd54bbf8889017cfe3f617656ddf483cbb171e63
Signed-off-by: gwaines <greg.waines@windriver.com>

4 files changed, 56 insertions, 314 deletions

diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png
index ccfd249..c394763 100644
--- a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png
+++ b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png
diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png
new file mode 100644
index 0000000..3f2491a
--- /dev/null
+++ b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png
diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst
index 0bc0537..7356f4c 100644
--- a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst
+++ b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst
@@ -5,16 +5,16 @@ Overview
 :organization: Wind River Systems
 :organization: OPNFV - High Availability
 :status: Draft - PROPOSAL
-:date: March 2017
-:revision: 1.5
+:date: April 2017
+:revision: 1.6
 
-:abstract: This document presents a PROPOSAL for a set of new
-   optional capabilities where the OpenStack Cloud messages
-   into the Guest VMs in order to provide improved Availability
-   of the hosted VMs.  The initial set of new capabilities
-   include: enabling the detection of and recovery from internal
-   VM faults and providing a simple out-of-band messaging service
-   to prevent scenarios such as split brain.
+:abstract: This document describes a set of new optional
+   capabilities where the OpenStack Cloud messages into the Guest
+   VMs in order to provide improved Availability of the hosted VMs.
+   The initial set of new capabilities include: enabling the
+   detection of and recovery from internal VM faults and providing
+   a simple out-of-band messaging service to prevent scenarios such
+   as split brain.
 
 
 .. sectnum::
@@ -26,8 +26,8 @@ Overview
 Introduction
 =====================================================================
 
-   This document provides an overview and rationale for a PROPOSAL
-   of a set of new capabilities where the OpenStack Cloud messages
+   This document provides an overview and rationale for a
+   set of new capabilities where the OpenStack Cloud messages
    into the Guest VMs in order to provide improved Availability
    of the hosted VMs.
 
@@ -49,10 +49,6 @@ Introduction
    with VNFM), specific OPNFV HA Team deliverables, and the use cases
    for how availability of the hosted VM is improved.
 
-   The intent is for the OPNFV HA Team to review the proposals of this
-   document with the related other teams in OPNFV (Doctor and Management
-   & Orchestration (MANO)) and OpenStack (Nova).
-
 
 
 
@@ -92,18 +88,23 @@ VM Heartbeating and Health Checking
 
    Normally OpenStack monitoring of the health of a Guest VM is limited
    to a black-box approach of simply monitoring the presence of the
-   QEMU/KVM PID containing the VM.
-
-   VM Heartbeating and Health Checking provides a heartbeat service to monitor
-   the health of guest application(s) within a VM running under the OpenStack
-   Cloud.  Loss of heartbeat or a failed health check status will result in a
-   fault event being reported to OPNFV's DOCTOR infrastructure for alarm
-   identification, impact analysis and reporting.  This would then enable VNF
-   Managers (VNFMs) listening to OPNFV's DOCTOR External Alarm Reporting through
+   QEMU/KVM PID containing the VM, and/or by enabling libvirt's emulated
+   hardware watchdog.
+
+   VM Heartbeating and Health Checking provides a heartbeat service to enhance
+   the monitoring of the health of guest application(s) within a VM running
+   under the OpenStack Cloud.  Loss of heartbeat or a failed health check status
+   will result in a fault event being reported to OPNFV's DOCTOR infrastructure
+   for alarm identification, impact analysis and reporting.  This would then enable
+   VNF Managers (VNFMs) listening to OPNFV's DOCTOR External Alarm Reporting through
    Telemetry's AODH, to initiate any required fault recovery actions.
 
    .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png
 
+   Or, in the context of the OPNFV DOCTOR's Fault Management Architecture:
+
+   .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png
+
    The VM Heartbeating and Health Checking functionality is enabled on
    a VM through a new flavor extraspec indicating that the VM supports
    and wants to enable Guest Heartbeating.  An extension to Nova Compute uses
@@ -115,8 +116,8 @@ VM Heartbeating and Health Checking
    (i.e. the Guest VM).  The OpenStack Compute Node will start heartbeating the
    Guest VM, and if the heartbeat fails, the OpenStack Compute Node will report
    the VM Fault thru DOCTOR and ultimately VNFM will see this thru NOVA VM
-   State Chagne Notifications thru AODH.  I.e. VNFM wouild see the VM Heartbeat
-   Failure events in teh same way it sees all other VM Faults, thru DOCTOR
+   State Change Notifications thru AODH.  I.e. VNFM wouild see the VM Heartbeat
+   Failure events in the same way it sees all other VM Faults, thru DOCTOR
    initiated VM state changes.
 
    Part of the Guest VM's registration process is the specification of the
@@ -130,8 +131,13 @@ VM Heartbeating and Health Checking
    If the OpenStack Host does not receive a valid response, or if the response
    specifies that the VM is in ill health, then a fault event for the Guest VM
    is reported to the OpenStack Guest Heartbeat Service on the controller node which
-   will report the event to OPNFV's DOCTOR (i.e. thru the OpenStack Vitrage data
-   source APIs).
+   will report the event to OPNFV's DOCTOR (i.e. thru the Doctor SouthBound (SB)
+   APIs).
+
+   In summary, the Guest Heartbeating Messaging Specification is quite simple,
+   including the following PDUs: Init, Init-Ack, Challenge-Request,
+   Challenge-Response, Exit.  The Challenge-Response returning a healthy /
+   not-healthy boolean.
 
    The registered Guest VM daemon's response to the challenge can be as simple
    as just immediately responding with OK.  This alone allows for detection of
@@ -177,8 +183,19 @@ VM Heartbeating and Health Checking
       - will provide code and compile instructions,
       - Guest will compile based on its specific OS.
 
-   This proposal requires review with OPNFV's Doctor and Management & Orchestration
-   teams, and OpenStack's Nova Team.
+   NOTE that the described VM Heartbeating and Healthchecking functionality provides
+   enhanced monitoring over and above libvirt's emulated hardware watchdog.  VM
+   Heartbeating and Healthchecking can detect a wider range of issues than simply
+   lack of cpu time scheduling for a lower priority process feeding the hardware
+   watchdog.  VM Heartbeating and Healthchecking can ensure that specific key processes
+   within the application are not blocked, kernel resources for basic IO within
+   the Guest VM are available, and/or ensure the application-specific health of the VM
+   is good.
+
+   This proposal has been reviewed with both the OPNFV's Doctor and Management
+   and Orchestration teams, and general agreement was that the proposal integrated
+   / inter-worked correctly with the OPNFV DOCTOR's Vitrage, Congress and the overall
+   OPNFV fault reporting architecture.
 
 
 
@@ -257,19 +274,23 @@ VM Peer State Notification and Messaging
      Server Group Messaging containing the peer protocol layers
      and Guest Application hooks within the Guest.
 
-   This proposal requires review with OPNFV's Management & Orchestration team and
-   OpenStack's Nova Team.
+   This proposal has been reviewed with both the OPNFV's Doctor and Management
+   and Orchestration teams, and general agreement was that the proposal did not
+   conflict with the OPNFV Doctor Architecture, and provided, at the very least,
+   an alternative messaging and state-change-notification mechanism for hosted
+   VMs in various HA use cases.
+
 
 
 Conclusion
 ======================================================================================
 
-   The PROPOSAL of Reach-thru Guest Monitoring and Services described in this document
+   The Reach-thru Guest Monitoring and Services described in this document
    leverage Host-to-Guest messaging to provide a number of extended capabilities
    that improve the Availability of the hosted VMs.  These new capabilities
    enable detection of and recovery from internal VM faults and provides a simple
    out-of-band messaging service to prevent scenarios such as split brain.
 
-   The integration of these proposed new capabilities into the larger OpenStack and OPNFV
-   Architectures need to be reviewed with the other related teams in OPNFV (Doctor and
-   Management & Orchestration (MANO)) and OpenStack (Nova).
+   The next steps in progressing this proposal will be to submit blueprints to
+   the appropriate OpenStack working groups;  Vitrage for VM Heartbeating and
+   Healthchecking and Nova for VM Server Group Messaging.
diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst.bak b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst.bak
deleted file mode 100644
index f7d5284..0000000
--- a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst.bak
+++ /dev/null
@@ -1,279 +0,0 @@
-*********************************************************************
-PROPOSAL:  Reach-thru Guest Monitoring and Services for High Availability
-*********************************************************************
-
-Overview
-=====================================================================
-
-:author: Greg Waines
-:organization: Wind River Systems
-:organization: OPNFV - High Availability
-:status: Draft - PROPOSAL
-:date: March 2017
-:revision: 1.5
-
-:abstract: This document presents a PROPOSAL for a set of new
-   optional capabilities where the OpenStack Cloud messages
-   into the Guest VMs in order to provide improved Availability
-   of the hosted VMs.  The initial set of new capabilities
-   include: enabling the detection of and recovery from internal
-   VM faults and providing a simple out-of-band messaging service
-   to prevent scenarios such as split brain.
-
-
-.. sectnum::
-
-.. contents:: Table of Contents
-
-
-
-Introduction
-=====================================================================
-
-   This document provides an overview and rationale for a PROPOSAL
-   of a set of new capabilities where the OpenStack Cloud messages
-   into the Guest VMs in order to provide improved Availability
-   of the hosted VMs.
-
-   The initial set of new capabilities specifically include:
-
-        - VM Heartbeating and Health Checking
-        - VM Peer State Notification and Messaging
-
-   All of these capabilities leverage Host-to-Guest Messaging
-   Interfaces / APIs which are built on a messaging service between the
-   OpenStack Host and the Guest VM that uses a simple low-bandwidth
-   datagram messaging capability in the hypervisor and therefore has no
-   requirements on OpenStack Networking, and is available very early
-   after spawning the VM.
-
-   For each capability, the document outlines the interaction with
-   the Guest VM, any key technologies involved, the integration into
-   the larger OpenStack and OPNFV Architectures (e.g. interactions
-   with VNFM), specific OPNFV HA Team deliverables, and the use cases
-   for how availability of the hosted VM is improved.
-
-   The intent is for the OPNFV HA Team to review the proposals of this
-   document with the related other teams in OPNFV (Doctor and Management
-   & Orchestration (MANO)) and OpenStack (Nova).
-
-
-
-
-Messaging Layer
-========================================================================
-
-   The Host-to-Guest messaging APIs used by the services discussed
-   in this document use a JSON-formatted application messaging layer
-   on top of a ‘virtio serial device’ between QEMU on the OpenStack Host
-   and the Guest VM.  JSON formatting provides a simple, humanly readable
-   messaging format which can be easily parsed and formatted using any
-   high level programming language being used in the Guest VM (e.g. C/C++,
-   Python, Java, etc.).  Use of the ‘virtio serial device’ provides a
-   simple, direct communication channel between host and guest which is
-   independent of the Guest’s L2/L3 networking.
-
-   The upper layer JSON messaging format is actually structured as a
-   hierarchical JSON format containing a Base JSON Message Layer and an
-   Application JSON Message Layer:
-
-        - the Base Layer provides the ability to multiplex different groups
-          of message types on top of a single ‘virtio serial device’
-          e.g.
-
-           + heartbeating and healthchecks,
-           + server group messaging,
-
-          and
-
-        - the Application Layer provides the specific message types and
-          fields of a particular group of message types.
-
-
-
-VM Heartbeating and Health Checking
-============================================================================
-
-   Normally OpenStack monitoring of the health of a Guest VM is limited
-   to a black-box approach of simply monitoring the presence of the
-   QEMU/KVM PID containing the VM.
-
-   VM Heartbeating and Health Checking provides a heartbeat service to monitor
-   the health of guest application(s) within a VM running under the OpenStack
-   Cloud.  Loss of heartbeat or a failed health check status will result in a
-   fault event being reported to OPNFV's DOCTOR infrastructure for alarm
-   identification, impact analysis and reporting.  This would then enable VNF
-   Managers (VNFMs) listening to OPNFV's DOCTOR External Alarm Reporting through
-   Telemetry's AODH, to initiate any required fault recovery actions.
-
-   .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png
-
-   The VM Heartbeating and Health Checking functionality is enabled on
-   a VM through a new flavor extraspec indicating that the VM supports
-   and wants to enable Guest Heartbeating.  An extension to Nova Compute uses
-   this extraspec to setup the required 'virtio serial device' for Host-to-Guest
-   messaging, on the QEMU/KVM instance created for the VM.
-
-   A daemon within the Guest VM will register with the OpenStack Guest
-   Heartbeat Service on the compute node to initiate the heartbeating on itself
-   (i.e. the Guest VM).  The OpenStack Compute Node will start heartbeating the
-   Guest VM, and if the heartbeat fails, the OpenStack Compute Node will report
-   the VM Fault thru DOCTOR and ultimately VNFM will see this thru NOVA VM
-   State Chagne Notifications thru AODH.  I.e. VNFM wouild see the VM Heartbeat
-   Failure events in teh same way it sees all other VM Faults, thru DOCTOR
-   initiated VM state changes.
-
-   Part of the Guest VM's registration process is the specification of the
-   heartbeat interval in msecs.  I.e. the registering Guest VM specifies the
-   heartbeating interval.
-
-   Guest heartbeat works on a challenge response model.  The OpenStack
-   Guest Heartbeat Service on the compute node will challenge the registered
-   Guest VM daemon with a message each interval.  The registered Guest VM daemon
-   must respond prior to the next interval with a message indicating good health.
-   If the OpenStack Host does not receive a valid response, or if the response
-   specifies that the VM is in ill health, then a fault event for the Guest VM
-   is reported to the OpenStack Guest Heartbeat Service on the controller node which
-   will report the event to OPNFV's DOCTOR (i.e. thru the OpenStack Vitrage data
-   source APIs).
-
-   The registered Guest VM daemon's response to the challenge can be as simple
-   as just immediately responding with OK.  This alone allows for detection of
-   a failed or hung QEMU/KVM instance, or a failure of the OS within the VM to
-   schedule the registered Guest VM's daemon or failure to route basic IO within
-   the Guest VM.
-
-   However the registered Guest VM daemon's response to the challenge can be more
-   complex, running anything from a quick simple sanity check of the health of
-   applications running in the Guest VM, to a more thorough audit of the
-   application state and data.  In either case returning the status of the
-   health check enables the OpenStack host to detect and report the event in order
-   to initiate recovery from application level errors or failures within the Guest VM.
-
-   In summary, the deliverables of this activity would be:
-
-   - Host Deliverables:    (OpenStack and OPNFV blueprints and implementation)
-
-   + an OpenStack Nova or libvirt extension to interpret the new flavor extraspec and
-     if present setup the required 'virtio serial device' for Host-to-Guest
-     heartbeat / health-check messaging, on the QEMU/KVM instance created
-     for the VM,
-   + an OPNFV Base Host-to-Guest Msging Layer Agent for multiplexing of Application
-     Layer messaging over the 'virtio serial device' to the VM,
-   + an OPNFV Heartbeat / Health-Check Compute Agent for local heartbeating of VM
-     and reporting of failures to the OpenStack Controller,
-   + an OPNFV Heartbeat / Health-check Server on the OpenStack Controller for
-     receiving VM failure notifications and reporting these to Vitrage thru
-     Vitrage's Data Source API,
-
-   - Guest Deliverables:
-
-   + a Heartbeat / Health-Check Message Specification covering
- 
-      - Heartbeat / Health-Check Application Layer JSON Protocol,
-      - Base Host-to-Guest JSON Protocol,
-      - Details on the use of the underlying 'virtio serial device',
-
-   + a Reference Implementation of the Guest-side support of
-     Heartbeat / Health-check containing the peer protocol layers
-     within the Guest.
-
-      - will provide code and compile instructions,
-      - Guest will compile based on its specific OS.
-
-   This proposal requires review with OPNFV's Doctor and Management & Orchestration
-   teams, and OpenStack's Nova Team.
-
-
-
-VM Peer State Notification and Messaging
-===================================================================================
-
-   Server Group State Notification and Messaging is a service to provide
-   simple low-bandwidth datagram messaging and notifications for servers that
-   are part of the same server group.  This messaging channel is available
-   regardless of whether IP networking is functional within the server, and
-   it requires no knowledge within the server about the other members of the group.
-
-   NOTE: A Server Group here is the OpenStack Nova Server Group concept where VMs
-   are grouped together for purposes of scheduling.  E.g. A specific Server Group
-   instance can specify whether the VMs within the group should be scheduled to
-   run on the same compute host or different compute hosts.  A 'peer' VM in the
-   context of this section refers to a VM within the same Nova Server Group.
-
-   This Server Group Messaging service provides three types of messaging:
-
-        - Broadcast: this allows a server to send a datagram (size of up to 3050 bytes)
-          to all other servers within the server group.
-        - Notification: this provides servers with information about changes to the
-          (Nova) state of other servers within the server group.
-        - Status: this allows a server to query the current (Nova) state of all servers within
-          the server group (including itself).
-
-   A Server Group Messaging entity on both the controller node and the compute nodes
-   manage the routing of of VM-to-VM messages through the platform, leveraging Nova
-   to determine Server Group membership and compute node locations of VMs.  The Server
-   Group Messaging entity on the controller also listens to Nova VM state change notifications
-   and querys VM state data from Nova, in order to provide the VM query and notification
-   functionality of this service.
-
-   .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png
-
-   This service is not intended for high bandwidth or low-latency operations.  It
-   is best-effort, not reliable.  Applications should do end-to-end acks and
-   retries if they care about reliability.
-
-   This service provides building block type capabilities for the Guest VMs that
-   contribute to higher availability of the VMs in the Guest VM Server Group.  Notifications
-   of VM Status changes potentially provide a faster and more accurate notification
-   of failed peer VMs than traditional peer VM monitoring over Tenant Networks.  While
-   the Broadcast Messaging mechanism provides an out-of-band messaging mechanism to
-   monitor and control a peer VM under fault conditions; e.g. providing the ability to
-   avoid potential split brain scenarios between 1:1 VMs when faults in Tenant
-   Networking occur.
-
-   In summary, the deliverables for Server Group Messaging would be:
-
-   - Host Deliverables:
-
-   + a Nova or libvirt extension to interpret the new flavor extraspec and
-     if present setup the required 'virtio serial device' for Host-to-Guest
-     Server Group Messaging, on the QEMU/KVM instance created
-     for the VM,
-   + [ leveraging the Base Host-to-Guest Msging Layer Agent from previous section ],
-   + a Server Group Messaging Compute Agent for implementing the Application Layer
-     Server Group Messaging JSON Protocol with the VM, and forwarding the
-     messages to/from the Server Group Messaging Server on the Controller,
-   + a Server Group Messaging Server on the Controller for routing broadcast
-     messages to the proper Computes and VMs, as well as listening for Nova
-     VM State Change Notifications and forwarding these to applicable Computes
-     and VMs,
-
-   - Guest Deliverables:
-
-   + a Server Group Messaging Message Specification covering
- 
-      - Server Group Messaging Application Layer JSON Protocol,
-      - [ leveraging Base Host-to-Guest JSON Protocol from previous section ],
-      - [ leveraging Details on the use of the underlying 'virtio serial device' from previous section ],
-
-   + a Reference Implementation of the Guest-side support of
-     Server Group Messaging containing the peer protocol layers
-     and Guest Application hooks within the Guest.
-
-   This proposal requires review with OPNFV's Management & Orchestration team and
-   OpenStack's Nova Team.
-
-
-Conclusion
-======================================================================================
-
-   The PROPOSAL of Reach-thru Guest Monitoring and Services described in this document
-   leverage Host-to-Guest messaging to provide a number of extended capabilities
-   that improve the Availability of the hosted VMs.  These new capabilities
-   enable detection of and recovery from internal VM faults and provides a simple
-   out-of-band messaging service to prevent scenarios such as split brain.
-
-   The integration of these proposed new capabilities into the larger OpenStack and OPNFV
-   Architectures need to be reviewed with the other related teams in OPNFV (Doctor and
-   Management & Orchestration (MANO)) and OpenStack (Nova).