From be0a877555d619fb72e888a65cc8e5159c35f953 Mon Sep 17 00:00:00 2001 From: fuqiao Date: Tue, 6 Nov 2018 16:13:32 +0800 Subject: Gambia Release Doc Add HA analysis doc for Gambia Release JIRA: HA-32 Change-Id: I67a9e8b62b509b9d0b70bbe5613a90e3e7a18b9e Signed-off-by: fuqiao@chinamobile.com --- docs/development/overview/HA_Analysis-Gambia.rst | 670 +++++++++++++++++++++ ..._APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png | Bin 110866 -> 0 bytes ...APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png | Bin 126072 -> 0 bytes ...t_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png | Bin 88884 -> 0 bytes .../overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst | 289 --------- .../overview/images/Active_Active_Redundancy.png | Bin 47984 -> 0 bytes .../overview/images/Active_Passive_Redundancy.png | Bin 40769 -> 0 bytes docs/development/overview/images/KAOS_Sample.png | Bin 61999 -> 0 bytes docs/development/overview/images/Message_Queue.png | Bin 37164 -> 0 bytes .../overview/images/Total_Framework.png | Bin 108646 -> 0 bytes docs/development/overview/images/VIM_Analysis.png | Bin 55324 -> 0 bytes .../overview/images/fig10_Message_Queue.png | Bin 0 -> 37164 bytes .../images/fig11_VIM_HA_analysis_in_K8s.png | Bin 0 -> 42522 bytes .../images/fig12_VIM_HA_analysis_in_K8s_2.png | Bin 0 -> 45280 bytes .../overview/images/fig1_KAOS_Sample.png | Bin 0 -> 61999 bytes .../overview/images/fig2_Total_Framework.png | Bin 0 -> 108646 bytes .../overview/images/fig3_VM_HA_Analysis.png | Bin 0 -> 37675 bytes .../images/fig4_Heartbeating_and_Healthchecks.png | Bin 0 -> 74066 bytes ...g5_VM_Peer_State_Notification_and_Messaging.png | Bin 0 -> 40788 bytes .../images/fig6_Container_HA_analysis_in_K8s.png | Bin 0 -> 46624 bytes .../overview/images/fig7_VIM_Analysis.png | Bin 0 -> 55324 bytes .../images/fig8_Active_Active_Redundancy.png | Bin 0 -> 47984 bytes .../images/fig9_Active_Passive_Redundancy.png | Bin 0 -> 40769 bytes docs/development/overview/index.rst | 4 +- docs/development/overview/index.rst.bak | 15 + 25 files changed, 687 insertions(+), 291 deletions(-) create mode 100644 docs/development/overview/HA_Analysis-Gambia.rst delete mode 100644 docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png delete mode 100644 docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png delete mode 100644 docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png delete mode 100644 docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst delete mode 100644 docs/development/overview/images/Active_Active_Redundancy.png delete mode 100644 docs/development/overview/images/Active_Passive_Redundancy.png delete mode 100644 docs/development/overview/images/KAOS_Sample.png delete mode 100644 docs/development/overview/images/Message_Queue.png delete mode 100644 docs/development/overview/images/Total_Framework.png delete mode 100644 docs/development/overview/images/VIM_Analysis.png create mode 100644 docs/development/overview/images/fig10_Message_Queue.png create mode 100644 docs/development/overview/images/fig11_VIM_HA_analysis_in_K8s.png create mode 100644 docs/development/overview/images/fig12_VIM_HA_analysis_in_K8s_2.png create mode 100644 docs/development/overview/images/fig1_KAOS_Sample.png create mode 100644 docs/development/overview/images/fig2_Total_Framework.png create mode 100644 docs/development/overview/images/fig3_VM_HA_Analysis.png create mode 100644 docs/development/overview/images/fig4_Heartbeating_and_Healthchecks.png create mode 100644 docs/development/overview/images/fig5_VM_Peer_State_Notification_and_Messaging.png create mode 100644 docs/development/overview/images/fig6_Container_HA_analysis_in_K8s.png create mode 100644 docs/development/overview/images/fig7_VIM_Analysis.png create mode 100644 docs/development/overview/images/fig8_Active_Active_Redundancy.png create mode 100644 docs/development/overview/images/fig9_Active_Passive_Redundancy.png create mode 100644 docs/development/overview/index.rst.bak (limited to 'docs') diff --git a/docs/development/overview/HA_Analysis-Gambia.rst b/docs/development/overview/HA_Analysis-Gambia.rst new file mode 100644 index 0000000..46dc121 --- /dev/null +++ b/docs/development/overview/HA_Analysis-Gambia.rst @@ -0,0 +1,670 @@ +.. image:: opnfv-logo.png + :height: 40 + :width: 200 + :alt: OPNFV + :align: left + +============ +High Availability Requirement Analysis in OPNFV +============ + +****************** +1 Introduction +****************** +This High Availability Requirement Analysis Document is used for eliciting High Availability +Requirements of OPNFV. The document will refine high-level High Availability goals, into +detailed HA mechanism design. And HA mechanisms are related with potential failures on +different layers in OPNFV. Moreover, this document can be used as reference for HA Testing +scenarios design. +A requirement engineering model KAOS is used in this document. + +****************** +2 Terminologies and Symbols +****************** +The following concepts in KAOS will be used in the diagrams of this document. + +- **Goal**: The objective to be met by the target system. + +- **Obstacle**: Condition whose satisfaction may prevent some goals from being achieved. + +- **Agent**: Active Object performing operations to achieve goals. + +- **Requirement**: Goal assigned to an agent of the software being studied. + +- **Domain Property**: Descriptive assertion about objects in the environment of the software. + +- **Refinement**: Relationship linking a goal to other goals that are called its subgoals. + Each subgoal contributes to the satisfaction of the goal it refines. There are two types of + refinements: AND refinement and OR refinement, which means whether the goal can be archived by + satisfying all of its sub goals or any one of its sub goals. + +- **Conflict**: Relationship linking an obstacle to a goal if the obstacle obstructs the goal + from being satisfied. + +- **Resolution**: Relationship linking a goal to an obstacle if the goal can resolve the + obstacle. + +- **Responsibility**: Relationship between an agent and a requirement. Holds when an agent is + assigned the responsibility of achieving the linked requirement. + +Figure 1 shows how these concepts are displayed in a KAOS diagram. + +.. figure:: images/fig1_KAOS_Sample.png + :alt: KAOS Sample + :figclass: align-center + + Fig 1. A KAOS Sample Diagram + +****************** +3 High Availability Goals of OPNFV +****************** + +3.1 Overall Goals +>>>>>>>>>>>>>>>>>> + +The Final Goal of OPNFV High Availability is to provide high available VNF services. And the +following objectives are required to meet: + +- There should be no single point of failure in the NFV framework. + +- All resiliency mechanisms shall be designed for a multi-vendor environment, where for example + the NFVI, NFV-MANO, and VNFs may be supplied by different vendors. + +- Resiliency related information shall always be explicitly specified and communicated using + the reference interfaces (including policies/templates) of the NFV framework. + + + +3.2 Service Level Agreements of OPNFV HA +>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + +Service Level Agreements of OPNFV HA are mainly focused on time constraints of service outage, +failure detection, failure recovery. The following table outlines the SLA metrics of different +service availability levels described in ETSI GS NFV-REL 001 V1.1.1 (2015-01). Table 1 shows +time constraints of different Service Availability Levels. In this document, SAL1 is the +default benchmark value required to meet. + +*Table 1. Time Constraints for Different Service Availability Levels* + ++--------------------------------+----------------------------+------------------------+ +| Service Availability Level | Failure Detection Time | Failure Recovery Time | ++================================+============================+========================+ +| SAL1 | <1s | 5-6s | ++--------------------------------+----------------------------+------------------------+ +| SAL2 | <5s | 10-15s | ++--------------------------------+----------------------------+------------------------+ +| SAL3 | <10s | 20-25s | ++--------------------------------+----------------------------+------------------------+ + + +****************** +4 Overall Analysis +****************** +Figure 2 shows the overall decomposition of high availability goals. The high availability of +VNF Services can be refined to high availability of VNFs, MANO, and the NFVI where VNFs are +deployed; the high availability of NFVI Service can be refined to high availability of Virtual +Compute Instances, Virtual Storage and Virtual Network Services; the high availability of +virtual instance is either the high availability of containers or the high availability of VMs, +and these high availability goals can be further decomposed by how the NFV environment is +deployed. + +.. figure:: images/fig2_Total_Framework.png + :alt: Overall HA Analysis of OPNFV + :figclass: align-center + + Fig 2. Overall HA Analysis of OPNFV + +Thus the high availability requirement of VNF services can be classified into high availability +requirements on different layers in OPNFV. The following layers are mainly discussed in this +document: + +- VNF HA + +- MANO HA + +- Virtual Infrastructure HA (container HA or VM HA) + +- VIM HA + +- SDN HA + +- Hypervisor HA + +- Host OS HA + +- Hardware HA + +The next section will illustrate detailed analysis of HA requirements on these layers. + +****************** +5 Detailed Analysis +****************** + +5.1 VNF HA +>>>>>>>>>>>>>>>>>> + +.. TBD + +5.2 MANO HA +>>>>>>>>>>>>>>>>>> + +.. TBD + +5.3 Virtual Infrastructure HA +>>>>>>>>>>>>>>>>>> + +The Virtual Infrastructure HA in OPNFV includes container HA and VM HA. + +VM HA +:::::::::::::::::::::::::::::::::::::: + +This part describes a set of new optional capabilities where the OpenStack Cloud messages into the Guest +VMs in order to provide improved Availability of the Host VMs. + +Table 2 shows the potential faults of VMs and corresponding initial solution capabilities or methods. + +*Table 2. Potential Faults of VMs and the initial solution capabilities* + ++---------------------------+------------------------------------+--------------------------------------------+ +| Fault | Description | solution capabilities | ++===========================+====================================+============================================+ +| VM faults | General internal VM faults | VM Heartbeating and Health Checking | ++---------------------------+------------------------------------+--------------------------------------------+ +| VM Server Group faults | such as split brain | VM Peer State Notification and Messaging | ++---------------------------+------------------------------------+--------------------------------------------+ + + +.. figure:: images/fig3_VM_HA_Analysis.png + :alt: VM HA + :figclass: align-center + + Fig 3. VM HA Analysis + +NOTE: A Server Group here is the OpenStack Nova Server Group concept where VMs +are grouped together for purposes of scheduling. E.g. A specific Server Group +instance can specify whether the VMs within the group should be scheduled to +run on the same compute host or different compute hosts. A 'peer' VM in the +context of this section refers to a VM within the same Nova Server Group. + +The initial set of new capabilities include: enabling the +detection of and recovery from internal VM faults and providing +a simple out-of-band messaging service to prevent scenarios such +as split brain. + +More detailed description is located in R5_HA_API/OPNFV_HA_Guest_APIs-Overview_HLD.rst in this project. + +The Host-to-Guest messaging APIs used by the services discussed +in this Virtual Infrastructure HA part use a JSON-formatted application messaging layer +on top of a virtio serial device between QEMU on the OpenStack Host +and the Guest VM. Use of the virtio serial device provides a +simple, direct communication channel between host and guest which is +independent of the Guest's L2/L3 networking. + +The upper layer JSON messaging format is actually structured as a +hierarchical JSON format containing a Base JSON Message Layer and an +Application JSON Message Layer: + +- the Base Layer provides the ability to multiplex different groups of message types on top of a single virtio serial device +e.g. + + + heartbeating and healthchecks, + + server group messaging, + +and + +- the Application Layer provides the specific message types and fields of a particular group of message types. + + +A) VM Heartbeating and Health Checking + + +.. figure:: images/fig4_Heartbeating_and_Healthchecks.png + :alt: Heartbeating and Healthchecks + :figclass: align-center + + Fig 4. Heartbeating and Healthchecks + +VM Heartbeating and Health Checking provides a heartbeat service to enhance +the monitoring of the health of guest application(s) within a VM running +under the OpenStack Cloud. Loss of heartbeat or a failed health check status +will result in a fault event being reported to OPNFV's DOCTOR infrastructure +for alarm identification, impact analysis and reporting. This would then enable +VNF Managers (VNFMs) listening to OPNFV's DOCTOR External Alarm Reporting through +Telemetry's AODH, to initiate any required fault recovery actions. + +Guest heartbeat works on a challenge response model. The OpenStack Guest Heartbeat +Service on the compute node will challenge the registered Guest VM daemon with a +message each interval. The registered Guest VM daemon must respond prior to the +next interval with a message indicating good health. If the OpenStack Host does +not receive a valid response, or if the response specifies that the VM is in ill +health, then a fault event for the Guest VM is reported to the OpenStack Guest +Heartbeat Service on the controller node which will report the event to OPNFV's +DOCTOR (i.e. thru the Doctor SouthBound (SB) APIs). + +In summary, the Guest Heartbeating Messaging Specification is quite simple, +including the following PDUs: Init, Init-Ack, Challenge-Request, +Challenge-Response, Exit. The Challenge-Response returning a healthy / +not-healthy boolean. + +The registered Guest VM daemon's response to the challenge can be as simple +as just immediately responding with OK. This alone allows for detection of +a failed or hung QEMU/KVM instance, or a failure of the OS within the VM to +schedule the registered Guest VM's daemon or failure to route basic IO within +the Guest VM. + +However the registered Guest VM daemon's response to the challenge can be more +complex, running anything from a quick simple sanity check of the health of +applications running in the Guest VM, to a more thorough audit of the +application state and data. In either case returning the status of the +health check enables the OpenStack host to detect and report the event in order +to initiate recovery from application level errors or failures within the Guest VM. + + +B) VM Peer State Notification and Messaging + + +.. figure:: images/fig5_VM_Peer_State_Notification_and_Messaging.png + :alt: VM Peer State Notification and Messaging + :figclass: align-center + + Fig 5. VM Peer State Notification and Messaging + +Server Group State Notification and Messaging is a service to provide +simple low-bandwidth datagram messaging and notifications for servers that +are part of the same server group. This messaging channel is available +regardless of whether IP networking is functional within the server, and +it requires no knowledge within the server about the other members of the group. + +This Server Group Messaging service provides three types of messaging: + +- Broadcast: this allows a server to send a datagram (size of up to 3050 bytes) + to all other servers within the server group. +- Notification: this provides servers with information about changes to the + (Nova) state of other servers within the server group. +- Status: this allows a server to query the current (Nova) state of all servers within + the server group (including itself). + +A Server Group Messaging entity on both the controller node and the compute nodes manage +the routing of of VM-to-VM messages through the platform, leveraging Nova to determine +Server Group membership and compute node locations of VMs. The Server Group Messaging +entity on the controller also listens to Nova VM state change notifications and querys +VM state data from Nova, in order to provide the VM query and notification functionality +of this service. + +This service is not intended for high bandwidth or low-latency operations. It is best-effort, +not reliable. Applications should do end-to-end acks and retries if they care about reliability. + +This service provides building block type capabilities for the Guest VMs that +contribute to higher availability of the VMs in the Guest VM Server Group. Notifications +of VM Status changes potentially provide a faster and more accurate notification +of failed peer VMs than traditional peer VM monitoring over Tenant Networks. While +the Broadcast Messaging mechanism provides an out-of-band messaging mechanism to +monitor and control a peer VM under fault conditions; e.g. providing the ability to +avoid potential split brain scenarios between 1:1 VMs when faults in Tenant +Networking occur. + +Container HA +:::::::::::::::::::::::::::: + +The container HA in OPNFV is mainly focus on Kubernetes(K8s) platform. And using the Pod as +the smallest unit of management, creation, and planning, the K8s' container HA actually means +the High Availability of running Pods. + +Table 3 shows the potential faults of running pods in K8s. when it happens, the ReplicationController +or ReplicaSet can prevent the services provided by the pod from being unavailable, as is shown in +figure 6. + +*Table 3. Potential Faults in VIM level* + ++------------+--------------+----------------------------------------------------+----------------+ +| Service | Fault | Description | Severity | ++============+==============+====================================================+================+ +| | | All Containers in the Pod have terminated, and | | +| Running by | Pod failure | at least one Container has terminated in failure. | Critical | +| pods | | That is, the Container either exited with non-zero | | +| | | status or was terminated by the system. | | ++------------+--------------+----------------------------------------------------+----------------+ + +.. figure:: images/fig6_Container_HA_analysis_in_K8s.png + :alt: VIM HA Analysis + :figclass: align-center + + Fig 6. Container HA analysis in K8s + + +The Replication Controller or ReplicaSet (ReplicaSet is the next-generation Replication Controller) +is a kind of K8s Master Components, which ensures that a specified number of pod replicas are running +at any one time. + +The following requirements are elicited for Pod HA: + +**[Req 5.3.1]** A pod or a homogeneous set of pods is always up and available until terminated properly. + +**[Req 5.3.2]** The ReplicationController or ReplicaSet should terminate the extra pods If there are +more pods than specified number. + +**[Req 5.3.3]** The ReplicationController or ReplicaSet should start more pods If there are fewer pods +than specified number. + +**[Req 5.3.4]** The new Pod should be scheduled to other Nodes, if detecting the failure state of the +host or container. + + + +5.4 VIM HA +>>>>>>>>>>>>>>>>>> + + +OpenStack High Availability +:::::::::::::::::::::::::::: + +The VIM in the NFV reference architecture contains different components of Openstack, SDN +controllers and other virtual resource controllers. VIM components can be classified into three +types: + +- **Entry Point Components**: Components that give VIM service interfaces to users, like nova- + api, neutron-server. + +- **Middlewares**: Components that provide load balancer services, messaging queues, cluster + management services, etc. + +- **Subcomponents**: Components that implement VIM functions, which are called by Entry Point + Components but not by users directly. + +Table 4 shows the potential faults that may happen on VIM layer. Currently the main focus of +VIM HA is the service crash of VIM components, which may occur on all types of VIM components. +To prevent VIM services from being unavailable, Active/Active Redundancy, Active/Passive +Redundancy and Message Queue are used for different types of VIM components, as is shown in +figure 7. + +*Table 4. Potential Faults in VIM level* + ++------------+------------------+-------------------------------------------------+----------------+ +| Service | Fault | Description | Severity | ++============+==================+=================================================+================+ +| General | Service Crash | The processes of a service crashed unnormally. | Critical | ++------------+------------------+-------------------------------------------------+----------------+ + +.. figure:: images/VIM_Analysis.png + :alt: VIM HA Analysis + :figclass: align-center + + Fig 7. VIM HA Analysis + + +A) Active/Active Redundancy + +Active/Active Redundancy manages both the main and redundant systems concurrently. If there is +a failure happens on a component, the backups are already online and users are unlikely to +notice that the failed VIM component is under fixing. A typical Active/Active Redundancy will +have redundant instances, and these instances are load balanced via a virtual IP address and a +load balancer such as HAProxy. + +When one of the redundant VIM component fails, the load balancer should be aware of the +instance failure, and then isolate the failed instance from being called until it is recovered. +The requirement decomposition of Active/Active Redundancy is shown in Figure 8. + +.. figure:: images/fig8_Active_Active_Redundancy.png + :alt: Active/Active Redundancy Requirement Decomposition + :figclass: align-center + + Fig 8. Active/Active Redundancy Requirement Decomposition + +The following requirements are elicited for VIM Active/Active Redundancy: + +**[Req 5.4.1]** Redundant VIM components should be load balanced by a load balancer. + +**[Req 5.4.2]** The load balancer should check the health status of VIM component instances. + +**[Req 5.4.3]** The load balancer should isolate the failed VIM component instance until it is +recovered. + +**[Req 5.4.4]** The alarm information of VIM component failure should be reported. + +**[Req 5.4.5]** Failed VIM component instances should be recovered by a cluster manager. + +Table 5 shows the current VIM components using Active/Active Redundancy and the corresponding +HA test cases to verify them. + +*Table 5. VIM Components using Active/Active Redundancy* + ++-------------------+-------------------------------------------------------+----------------------+ +| Component | Description | Related HA Test Case | ++===================+=======================================================+======================+ +| nova-api | endpoint component of Openstack Compute Service Nova | yardstick_tc019 | ++-------------------+-------------------------------------------------------+----------------------+ +| nova-novncproxy | server daemon that serves the Nova noVNC Websocket | | +| | Proxy service, which provides a websocket proxy that | | +| | is compatible with OpenStack Nova noVNC consoles. | | ++-------------------+-------------------------------------------------------+----------------------+ +| neeutron-server | endpoint component of Openstack Networking Service | yardstick_tc045 | +| | Neutron | | ++-------------------+-------------------------------------------------------+----------------------+ +| keystone | component of Openstack Identity Service Service | yardstick_tc046 | +| | Keystone | | ++-------------------+-------------------------------------------------------+----------------------+ +| glance-api | endpoint component of Openstack Image Service Glance | yardstick_tc047 | ++-------------------+-------------------------------------------------------+----------------------+ +| glance-registry | server daemon that serves image metadata through a | | +| | REST-like API. | | ++-------------------+-------------------------------------------------------+----------------------+ +| cinder-api | endpoint component of Openstack Block Storage Service | yardstick_tc048 | +| | Service Cinder | | ++-------------------+-------------------------------------------------------+----------------------+ +| swift-proxy | endpoint component of Openstack Object Storage | yardstick_tc049 | +| | Swift | | ++-------------------+-------------------------------------------------------+----------------------+ +| horizon | component of Openstack Dashboard Service Horizon | | ++-------------------+-------------------------------------------------------+----------------------+ +| heat-api | endpoint component of Openstack Stack Service Heat | yardstick_tc091 | ++-------------------+-------------------------------------------------------+----------------------+ +| mysqld | database service of VIM components | yardstick_tc090 | ++-------------------+-------------------------------------------------------+----------------------+ + +B)Active/Passive Redundancy + + +Active/Passive Redundancy maintains a redundant instance that can be brought online when the +active service fails. A typical Active/Passive Redundancy maintains replacement resources that +can be brought online when required. Requests are handled using a virtual IP address (VIP) that +facilitates returning to service with minimal reconfiguration. A cluster manager (such as +Pacemaker or Corosync) monitors these components, bringing the backup online as necessary. + +When the main instance of a VIM component is failed, the cluster manager should be aware of the +failure and switch the backup instance online. And the failed instance should also be recovered +to another backup instance. The requirement decomposition of Active/Passive Redundancy is shown +in Figure 9. + +.. figure:: images/fig9_Active_Passive_Redundancy.png + :alt: Active/Passive Redundancy Requirement Decomposition + :figclass: align-center + + Fig 9. Active/Passive Redundancy Requirement Decomposition + +The following requirements are elicited for VIM Active/Passive Redundancy: + +**[Req 5.4.6]** The cluster manager should replace the failed main VIM component instance with +a backup instance. + +**[Req 5.4.7]** The cluster manager should check the health status of VIM component instances. + +**[Req 5.4.8]** Failed VIM component instances should be recovered by the cluster manager. + +**[Req 5.4.9]** The alarm information of VIM component failure should be reported. + + +Table 6 shows the current VIM components using Active/Passive Redundancy and the corresponding +HA test cases to verify them. + +*Table 6. VIM Components using Active/Passive Redundancy* + ++-------------------+-------------------------------------------------------+----------------------+ +| Component | Description | Related HA Test Case | ++===================+=======================================================+======================+ +| haproxy | load balancer component of VIM components | yardstick_tc053 | ++-------------------+-------------------------------------------------------+----------------------+ +| rabbitmq-server | messaging queue service of VIM components | yardstick_tc056 | ++-------------------+-------------------------------------------------------+----------------------+ +| corosync | cluster management component of VIM components | yardstick_tc057 | ++-------------------+-------------------------------------------------------+----------------------+ + +C) Message Queue + +Message Queue provides an asynchronous communication protocol. In Openstack, some projects ( +like Nova, Cinder) use Message Queue to call their sub components. Although Message Queue +itself is not an HA mechanism, how it works ensures the high availaibility when redundant +components subscribe to the Messsage Queue. When a VIM sub component fails, since there are +other redundant components are subscribing to the Message Queue, requests still can be processed. +And fault isolation can also be archived since failed components won't fetch requests actively. +Also, the recovery of failed components is required. Figure 10 shows the requirement +decomposition of Message Queue. + +.. figure:: images/fig10_Message_Queue.png + :alt: Message Queue Requirement Decomposition + :figclass: align-center + + Fig 10. Message Queue Redundancy Requirement Decomposition + +The following requirements are elicited for Message Queue: + +**[Req 5.4.10]** Redundant component instances should subscribe to the Message Queue, which is +implemented by the installer. + +**[Req 5.4.11]** Failed VIM component instances should be recovered by the cluster manager. + +**[Req 5.4.12]** The alarm information of VIM component failure should be reported. + +Table 7 shows the current VIM components using Message Queue and the corresponding HA test cases +to verify them. + +*Table 7. VIM Components using Messaging Queue* + ++-------------------+-------------------------------------------------------+----------------------+ +| Component | Description | Related HA Test Case | ++===================+=======================================================+======================+ +| nova-scheduler | Openstack compute component determines how to | yardstick_tc088 | +| | dispatch compute requests | | ++-------------------+-------------------------------------------------------+----------------------+ +| nova-cert | Openstack compute component that serves the Nova Cert | | +| | service for X509 certificates. Used to generate | | +| | certificates for euca-bundle-image. | | ++-------------------+-------------------------------------------------------+----------------------+ +| nova-conductor | server daemon that serves the Nova Conductor service, | yardstick_tc089 | +| | which provides coordination and database query | | +| | support for Nova. | | ++-------------------+-------------------------------------------------------+----------------------+ +| nova-compute | Handles all processes relating to instances (guest | | +| | vms). nova-compute is responsible for building a disk | | +| | image, launching it via the underlying virtualization | | +| | driver, responding to calls to check its state, | | +| | attaching persistent storage, and terminating it. | | ++-------------------+-------------------------------------------------------+----------------------+ +| nova-consoleauth | Openstack compute component for Authentication of | | +| | nova consoles. | | ++-------------------+-------------------------------------------------------+----------------------+ +| cinder-scheduler | Openstack volume storage component decides on | | +| | placement for newly created volumes and forwards the | | +| | request to cinder-volume. | | ++-------------------+-------------------------------------------------------+----------------------+ +| cinder-volume | Openstack volume storage component receives volume | | +| | management requests from cinder-api and | | +| | cinder-scheduler, and routes them to storage backends | | +| | using vendor-supplied drivers. | | ++-------------------+-------------------------------------------------------+----------------------+ +| heat-engine | Openstack Heat project server with an internal RPC | | +| | api called by the heat-api server. | | ++-------------------+-------------------------------------------------------+----------------------+ + + +VIM HA in K8s +:::::::::::::::::::::::::::: + +The VIM HA in K8s can be generally analyzed from the following two concepts: + +- **Master Components HA**: the HA of k8s components in Master. (for example, Kube-apiserver, + Kube-scheduler, Kube-controller-manager) + +- **Data Storage HA**: the HA of etcd cluster. Actually etcd is a master component used as + Kubernetes' backing store for all cluster data. Considering that etcd is the only stateful service + in k8s and that its HA policy can be deployed independent on K8s, it is necessary to discuss the + HA of etcd separately. + +Table 8 shows the potential faults that may happen in K8s. + +*Table 8. Potential Faults in K8s* + ++--------------------+------------------+----------------------------------------+----------------+ +| Service | Fault | Description | Severity | ++====================+==================+========================================+================+ +| Provided by Master | Master | A Master component crashed and can't | Critical | +| Components | Component crash | provide normal service. | | ++--------------------+------------------+----------------------------------------+----------------+ +| Data storage | Etcd Crash | The Etcd cluster crashed unnormally. | Critical | ++--------------------+------------------+----------------------------------------+----------------+ + + +.. figure:: images/fig11_VIM_HA_analysis_in_K8s.png + :alt: Message Queue Requirement Decomposition + :figclass: align-center + + Fig 11. VIM HA analysis in K8s + +Master components can be run on any machine in the cluster. However, for simplicity, all master +components are typically started on the same machine, and do not run user containers on this machine. +In this case, the K8s is based on a single Master, and only has container HA on application layer +realized by ReplicationController or ReplicaSet Master Component as mentioned in the container HA +part above. + +The HA of Mater and its components in K8s must depend on the multi-master setup. + +The Data Storage HA can use an existing Etcd HA cluster to realize, or can be realized as a master +component through multiple master implementation. + +.. figure:: images/fig12_VIM_HA_analysis_in_K8s_2.png + :alt: Message Queue Requirement Decomposition + :figclass: align-center + + Fig 12. VIM HA analysis in K8s(2) + +In Multi-Master K8s, the Master Components HA is mainly based on the Leader Election function of Etcd +cluster. And load balancer is used to realize the HA of Kube-apiserver Master component. + +The following requirements are elicited for Master components HA: + +**[Req 5.4.13]** The Load Balancer should always forward the request to an available Kube-apiserver +instance. + +**[Req 5.4.14]** The Master Component in the Leader state should confirm its Leader state to all +follower Components regularly through Heatbeat. + +**[Req 5.4.15]** When a Master Component in the Leader state crashed, an available Master Component +should be elected as Leader. + +5.5 Hypervisor HA +>>>>>>>>>>>>>>>>>> + +.. TBD + +5.6 Host OS HA +>>>>>>>>>>>>>>>>>> + +.. TBD + +5.7 Hardware HA +>>>>>>>>>>>>>>>>>> + +.. TBD + + +****************** +6 References +****************** + +- A KAOS Tutorial: http://www.objectiver.com/fileadmin/download/documents/KaosTutorial.pdf + +- ETSI GS NFV-REL 001 V1.1.1(2015-01): + http://www.etsi.org/deliver/etsi_gs/NFV-REL/001_099/001/01.01.01_60/gs_NFV-REL001v010101p.pdf + +- Openstack High Availability Guide: https://docs.openstack.org/ha-guide/ + +- Highly Available (Mirrored) Queues: https://www.rabbitmq.com/ha.html diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png deleted file mode 100644 index c394763..0000000 Binary files a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png and /dev/null differ diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png deleted file mode 100644 index 3f2491a..0000000 Binary files a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png and /dev/null differ diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png deleted file mode 100644 index 7147445..0000000 Binary files a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png and /dev/null differ diff --git a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst b/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst deleted file mode 100644 index b634a6b..0000000 --- a/docs/development/overview/OPNFV_HA_Guest_APIs-Overview_HLD.rst +++ /dev/null @@ -1,289 +0,0 @@ -Overview -===================================================================== - -:abstract: This document describes a set of new optional - capabilities where the OpenStack Cloud messages into the Guest - VMs in order to provide improved Availability of the hosted VMs. - The initial set of new capabilities include: enabling the - detection of and recovery from internal VM faults and providing - a simple out-of-band messaging service to prevent scenarios such - as split brain. - - -.. sectnum:: - -.. contents:: Table of Contents - - - -Introduction -===================================================================== - - This document provides an overview and rationale for a - set of new capabilities where the OpenStack Cloud messages - into the Guest VMs in order to provide improved Availability - of the hosted VMs. - - The initial set of new capabilities specifically include: - - - VM Heartbeating and Health Checking - - VM Peer State Notification and Messaging - - All of these capabilities leverage Host-to-Guest Messaging - Interfaces / APIs which are built on a messaging service between the - OpenStack Host and the Guest VM that uses a simple low-bandwidth - datagram messaging capability in the hypervisor and therefore has no - requirements on OpenStack Networking, and is available very early - after spawning the VM. - - For each capability, the document outlines the interaction with - the Guest VM, any key technologies involved, the integration into - the larger OpenStack and OPNFV Architectures (e.g. interactions - with VNFM), specific OPNFV HA Team deliverables, and the use cases - for how availability of the hosted VM is improved. - - - - -Messaging Layer -======================================================================== - - The Host-to-Guest messaging APIs used by the services discussed - in this document use a JSON-formatted application messaging layer - on top of a ‘virtio serial device�between QEMU on the OpenStack Host - and the Guest VM. JSON formatting provides a simple, humanly readable - messaging format which can be easily parsed and formatted using any - high level programming language being used in the Guest VM (e.g. C/C++, - Python, Java, etc.). Use of the ‘virtio serial device�provides a - simple, direct communication channel between host and guest which is - independent of the Guest’s L2/L3 networking. - - The upper layer JSON messaging format is actually structured as a - hierarchical JSON format containing a Base JSON Message Layer and an - Application JSON Message Layer: - - - the Base Layer provides the ability to multiplex different groups - of message types on top of a single ‘virtio serial device� - e.g. - - + heartbeating and healthchecks, - + server group messaging, - - and - - - the Application Layer provides the specific message types and - fields of a particular group of message types. - - - -VM Heartbeating and Health Checking -============================================================================ - - Normally OpenStack monitoring of the health of a Guest VM is limited - to a black-box approach of simply monitoring the presence of the - QEMU/KVM PID containing the VM, and/or by enabling libvirt's emulated - hardware watchdog. - - VM Heartbeating and Health Checking provides a heartbeat service to enhance - the monitoring of the health of guest application(s) within a VM running - under the OpenStack Cloud. Loss of heartbeat or a failed health check status - will result in a fault event being reported to OPNFV's DOCTOR infrastructure - for alarm identification, impact analysis and reporting. This would then enable - VNF Managers (VNFMs) listening to OPNFV's DOCTOR External Alarm Reporting through - Telemetry's AODH, to initiate any required fault recovery actions. - - .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png - - Or, in the context of the OPNFV DOCTOR's Fault Management Architecture: - - .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1b.png - - The VM Heartbeating and Health Checking functionality is enabled on - a VM through a new flavor extraspec indicating that the VM supports - and wants to enable Guest Heartbeating. An extension to Nova Compute uses - this extraspec to setup the required 'virtio serial device' for Host-to-Guest - messaging, on the QEMU/KVM instance created for the VM. - - A daemon within the Guest VM will register with the OpenStack Guest - Heartbeat Service on the compute node to initiate the heartbeating on itself - (i.e. the Guest VM). The OpenStack Compute Node will start heartbeating the - Guest VM, and if the heartbeat fails, the OpenStack Compute Node will report - the VM Fault thru DOCTOR and ultimately VNFM will see this thru NOVA VM - State Change Notifications thru AODH. I.e. VNFM wouild see the VM Heartbeat - Failure events in the same way it sees all other VM Faults, thru DOCTOR - initiated VM state changes. - - Part of the Guest VM's registration process is the specification of the - heartbeat interval in msecs. I.e. the registering Guest VM specifies the - heartbeating interval. - - Guest heartbeat works on a challenge response model. The OpenStack - Guest Heartbeat Service on the compute node will challenge the registered - Guest VM daemon with a message each interval. The registered Guest VM daemon - must respond prior to the next interval with a message indicating good health. - If the OpenStack Host does not receive a valid response, or if the response - specifies that the VM is in ill health, then a fault event for the Guest VM - is reported to the OpenStack Guest Heartbeat Service on the controller node which - will report the event to OPNFV's DOCTOR (i.e. thru the Doctor SouthBound (SB) - APIs). - - In summary, the Guest Heartbeating Messaging Specification is quite simple, - including the following PDUs: Init, Init-Ack, Challenge-Request, - Challenge-Response, Exit. The Challenge-Response returning a healthy / - not-healthy boolean. - - The registered Guest VM daemon's response to the challenge can be as simple - as just immediately responding with OK. This alone allows for detection of - a failed or hung QEMU/KVM instance, or a failure of the OS within the VM to - schedule the registered Guest VM's daemon or failure to route basic IO within - the Guest VM. - - However the registered Guest VM daemon's response to the challenge can be more - complex, running anything from a quick simple sanity check of the health of - applications running in the Guest VM, to a more thorough audit of the - application state and data. In either case returning the status of the - health check enables the OpenStack host to detect and report the event in order - to initiate recovery from application level errors or failures within the Guest VM. - - In summary, the deliverables of this activity would be: - - - Host Deliverables: (OpenStack and OPNFV blueprints and implementation) - - + an OpenStack Nova or libvirt extension to interpret the new flavor extraspec and - if present setup the required 'virtio serial device' for Host-to-Guest - heartbeat / health-check messaging, on the QEMU/KVM instance created - for the VM, - + an OPNFV Base Host-to-Guest Msging Layer Agent for multiplexing of Application - Layer messaging over the 'virtio serial device' to the VM, - + an OPNFV Heartbeat / Health-Check Compute Agent for local heartbeating of VM - and reporting of failures to the OpenStack Controller, - + an OPNFV Heartbeat / Health-check Server on the OpenStack Controller for - receiving VM failure notifications and reporting these to Vitrage thru - Vitrage's Data Source API, - - - Guest Deliverables: - - + a Heartbeat / Health-Check Message Specification covering - - - Heartbeat / Health-Check Application Layer JSON Protocol, - - Base Host-to-Guest JSON Protocol, - - Details on the use of the underlying 'virtio serial device', - - + a Reference Implementation of the Guest-side support of - Heartbeat / Health-check containing the peer protocol layers - within the Guest. - - - will provide code and compile instructions, - - Guest will compile based on its specific OS. - - NOTE that the described VM Heartbeating and Healthchecking functionality provides - enhanced monitoring over and above libvirt's emulated hardware watchdog. VM - Heartbeating and Healthchecking can detect a wider range of issues than simply - lack of cpu time scheduling for a lower priority process feeding the hardware - watchdog. VM Heartbeating and Healthchecking can ensure that specific key processes - within the application are not blocked, kernel resources for basic IO within - the Guest VM are available, and/or ensure the application-specific health of the VM - is good. - - This proposal has been reviewed with both the OPNFV's Doctor and Management - and Orchestration teams, and general agreement was that the proposal integrated - / inter-worked correctly with the OPNFV DOCTOR's Vitrage, Congress and the overall - OPNFV fault reporting architecture. - - - -VM Peer State Notification and Messaging -=================================================================================== - - Server Group State Notification and Messaging is a service to provide - simple low-bandwidth datagram messaging and notifications for servers that - are part of the same server group. This messaging channel is available - regardless of whether IP networking is functional within the server, and - it requires no knowledge within the server about the other members of the group. - - NOTE: A Server Group here is the OpenStack Nova Server Group concept where VMs - are grouped together for purposes of scheduling. E.g. A specific Server Group - instance can specify whether the VMs within the group should be scheduled to - run on the same compute host or different compute hosts. A 'peer' VM in the - context of this section refers to a VM within the same Nova Server Group. - - This Server Group Messaging service provides three types of messaging: - - - Broadcast: this allows a server to send a datagram (size of up to 3050 bytes) - to all other servers within the server group. - - Notification: this provides servers with information about changes to the - (Nova) state of other servers within the server group. - - Status: this allows a server to query the current (Nova) state of all servers within - the server group (including itself). - - A Server Group Messaging entity on both the controller node and the compute nodes - manage the routing of of VM-to-VM messages through the platform, leveraging Nova - to determine Server Group membership and compute node locations of VMs. The Server - Group Messaging entity on the controller also listens to Nova VM state change notifications - and querys VM state data from Nova, in order to provide the VM query and notification - functionality of this service. - - .. image:: OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png - - This service is not intended for high bandwidth or low-latency operations. It - is best-effort, not reliable. Applications should do end-to-end acks and - retries if they care about reliability. - - This service provides building block type capabilities for the Guest VMs that - contribute to higher availability of the VMs in the Guest VM Server Group. Notifications - of VM Status changes potentially provide a faster and more accurate notification - of failed peer VMs than traditional peer VM monitoring over Tenant Networks. While - the Broadcast Messaging mechanism provides an out-of-band messaging mechanism to - monitor and control a peer VM under fault conditions; e.g. providing the ability to - avoid potential split brain scenarios between 1:1 VMs when faults in Tenant - Networking occur. - - In summary, the deliverables for Server Group Messaging would be: - - - Host Deliverables: - - + a Nova or libvirt extension to interpret the new flavor extraspec and - if present setup the required 'virtio serial device' for Host-to-Guest - Server Group Messaging, on the QEMU/KVM instance created - for the VM, - + [ leveraging the Base Host-to-Guest Msging Layer Agent from previous section ], - + a Server Group Messaging Compute Agent for implementing the Application Layer - Server Group Messaging JSON Protocol with the VM, and forwarding the - messages to/from the Server Group Messaging Server on the Controller, - + a Server Group Messaging Server on the Controller for routing broadcast - messages to the proper Computes and VMs, as well as listening for Nova - VM State Change Notifications and forwarding these to applicable Computes - and VMs, - - - Guest Deliverables: - - + a Server Group Messaging Message Specification covering - - - Server Group Messaging Application Layer JSON Protocol, - - [ leveraging Base Host-to-Guest JSON Protocol from previous section ], - - [ leveraging Details on the use of the underlying 'virtio serial device' from previous section ], - - + a Reference Implementation of the Guest-side support of - Server Group Messaging containing the peer protocol layers - and Guest Application hooks within the Guest. - - This proposal has been reviewed with both the OPNFV's Doctor and Management - and Orchestration teams, and general agreement was that the proposal did not - conflict with the OPNFV Doctor Architecture, and provided, at the very least, - an alternative messaging and state-change-notification mechanism for hosted - VMs in various HA use cases. - - - -Conclusion -====================================================================================== - - The Reach-thru Guest Monitoring and Services described in this document - leverage Host-to-Guest messaging to provide a number of extended capabilities - that improve the Availability of the hosted VMs. These new capabilities - enable detection of and recovery from internal VM faults and provides a simple - out-of-band messaging service to prevent scenarios such as split brain. - - The next steps in progressing this proposal will be to submit blueprints to - the appropriate OpenStack working groups; Vitrage for VM Heartbeating and - Healthchecking and Nova for VM Server Group Messaging. diff --git a/docs/development/overview/images/Active_Active_Redundancy.png b/docs/development/overview/images/Active_Active_Redundancy.png deleted file mode 100644 index 1863de1..0000000 Binary files a/docs/development/overview/images/Active_Active_Redundancy.png and /dev/null differ diff --git a/docs/development/overview/images/Active_Passive_Redundancy.png b/docs/development/overview/images/Active_Passive_Redundancy.png deleted file mode 100644 index e94c82f..0000000 Binary files a/docs/development/overview/images/Active_Passive_Redundancy.png and /dev/null differ diff --git a/docs/development/overview/images/KAOS_Sample.png b/docs/development/overview/images/KAOS_Sample.png deleted file mode 100644 index 0d35cd7..0000000 Binary files a/docs/development/overview/images/KAOS_Sample.png and /dev/null differ diff --git a/docs/development/overview/images/Message_Queue.png b/docs/development/overview/images/Message_Queue.png deleted file mode 100644 index 3ebedbe..0000000 Binary files a/docs/development/overview/images/Message_Queue.png and /dev/null differ diff --git a/docs/development/overview/images/Total_Framework.png b/docs/development/overview/images/Total_Framework.png deleted file mode 100644 index c900908..0000000 Binary files a/docs/development/overview/images/Total_Framework.png and /dev/null differ diff --git a/docs/development/overview/images/VIM_Analysis.png b/docs/development/overview/images/VIM_Analysis.png deleted file mode 100644 index 5b0f579..0000000 Binary files a/docs/development/overview/images/VIM_Analysis.png and /dev/null differ diff --git a/docs/development/overview/images/fig10_Message_Queue.png b/docs/development/overview/images/fig10_Message_Queue.png new file mode 100644 index 0000000..3ebedbe Binary files /dev/null and b/docs/development/overview/images/fig10_Message_Queue.png differ diff --git a/docs/development/overview/images/fig11_VIM_HA_analysis_in_K8s.png b/docs/development/overview/images/fig11_VIM_HA_analysis_in_K8s.png new file mode 100644 index 0000000..70252a7 Binary files /dev/null and b/docs/development/overview/images/fig11_VIM_HA_analysis_in_K8s.png differ diff --git a/docs/development/overview/images/fig12_VIM_HA_analysis_in_K8s_2.png b/docs/development/overview/images/fig12_VIM_HA_analysis_in_K8s_2.png new file mode 100644 index 0000000..205cd2e Binary files /dev/null and b/docs/development/overview/images/fig12_VIM_HA_analysis_in_K8s_2.png differ diff --git a/docs/development/overview/images/fig1_KAOS_Sample.png b/docs/development/overview/images/fig1_KAOS_Sample.png new file mode 100644 index 0000000..0d35cd7 Binary files /dev/null and b/docs/development/overview/images/fig1_KAOS_Sample.png differ diff --git a/docs/development/overview/images/fig2_Total_Framework.png b/docs/development/overview/images/fig2_Total_Framework.png new file mode 100644 index 0000000..c900908 Binary files /dev/null and b/docs/development/overview/images/fig2_Total_Framework.png differ diff --git a/docs/development/overview/images/fig3_VM_HA_Analysis.png b/docs/development/overview/images/fig3_VM_HA_Analysis.png new file mode 100644 index 0000000..e263e60 Binary files /dev/null and b/docs/development/overview/images/fig3_VM_HA_Analysis.png differ diff --git a/docs/development/overview/images/fig4_Heartbeating_and_Healthchecks.png b/docs/development/overview/images/fig4_Heartbeating_and_Healthchecks.png new file mode 100644 index 0000000..cd7a551 Binary files /dev/null and b/docs/development/overview/images/fig4_Heartbeating_and_Healthchecks.png differ diff --git a/docs/development/overview/images/fig5_VM_Peer_State_Notification_and_Messaging.png b/docs/development/overview/images/fig5_VM_Peer_State_Notification_and_Messaging.png new file mode 100644 index 0000000..7614e19 Binary files /dev/null and b/docs/development/overview/images/fig5_VM_Peer_State_Notification_and_Messaging.png differ diff --git a/docs/development/overview/images/fig6_Container_HA_analysis_in_K8s.png b/docs/development/overview/images/fig6_Container_HA_analysis_in_K8s.png new file mode 100644 index 0000000..5fdb8a7 Binary files /dev/null and b/docs/development/overview/images/fig6_Container_HA_analysis_in_K8s.png differ diff --git a/docs/development/overview/images/fig7_VIM_Analysis.png b/docs/development/overview/images/fig7_VIM_Analysis.png new file mode 100644 index 0000000..5b0f579 Binary files /dev/null and b/docs/development/overview/images/fig7_VIM_Analysis.png differ diff --git a/docs/development/overview/images/fig8_Active_Active_Redundancy.png b/docs/development/overview/images/fig8_Active_Active_Redundancy.png new file mode 100644 index 0000000..1863de1 Binary files /dev/null and b/docs/development/overview/images/fig8_Active_Active_Redundancy.png differ diff --git a/docs/development/overview/images/fig9_Active_Passive_Redundancy.png b/docs/development/overview/images/fig9_Active_Passive_Redundancy.png new file mode 100644 index 0000000..e94c82f Binary files /dev/null and b/docs/development/overview/images/fig9_Active_Passive_Redundancy.png differ diff --git a/docs/development/overview/index.rst b/docs/development/overview/index.rst index 03de9ec..3e69259 100644 --- a/docs/development/overview/index.rst +++ b/docs/development/overview/index.rst @@ -6,10 +6,10 @@ ********************************************************************************* -High Availability Requirement Analysis and Detailed Mechanism Design +High Availability Requirement Analysis in OPNFV ********************************************************************************* .. toctree:: :maxdepth: 4 - HA_Analysis.rst + HA_Analysis-Gambia.rst diff --git a/docs/development/overview/index.rst.bak b/docs/development/overview/index.rst.bak new file mode 100644 index 0000000..1114613 --- /dev/null +++ b/docs/development/overview/index.rst.bak @@ -0,0 +1,15 @@ +.. _availability-overview: + +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. SPDX-License-Identifier: CC-BY-4.0 +.. (c) + + +********************************************************************************* +Reach-thru Guest Monitoring and Services for High Availability +********************************************************************************* + +.. toctree:: + :maxdepth: 4 + + OPNFV_HA_Guest_APIs-Overview_HLD.rst -- cgit 1.2.3-korg