Colorado Release Doc: High Available Deployment Guideline of OPNFV

HA deployment Guideline of OPNFV

JIRA: HA-21

Signed-off-by: fuqiao@chinamobile.com

10 files changed, 462 insertions, 0 deletions

diff --git a/R3_Deployment/Deployment_Guideline.rst b/R3_Deployment/Deployment_Guideline.rst
new file mode 100644
index 0000000..4bb7088
--- /dev/null
+++ b/R3_Deployment/Deployment_Guideline.rst
@@ -0,0 +1,462 @@
+.. image:: opnfv-logo.png
+  :height: 40
+  :width: 200
+  :alt: OPNFV
+  :align: left
+
+=====================================================================
+HA Deployment Framework Guideline
+=====================================================================
+
+This document will provide an overall framework for the high availability
+deployment of NFV system. It will also continiously update to include HA
+deployment guidelines and suggestions for the releases of OPNFV.
+
+*********************************************************************
+1. Overview of High Available Deployment of OPNFV
+*********************************************************************
+
+In this section, we would like to discuss the overall HA deployment of NFV system.
+Different modules, such as hardware,VIM,VMs and etc, will be included, and HA
+deployment of each single module will be discussed. However, not all of these HA
+schemes should be deployed in on system at the same time. For the HA deployment of
+a single system, we should consider the tradeoff between high availability and the
+cost and resource to leverage.
+
+
+1.1 Architecture of HA deployment
+==================================================================
+
+This section intends to introduce the different modules we should consider
+when talking about HA deployment. These moduels include the Hardware
+(compute, network, storage hardware), the VIM, the hypervisor, VMs and VNFs.
+HA schemes for these different moduels should all be considered when deploying
+an NFV system. And the schemes should be coordinated so that the system can make
+sure to react in its best way when facing failure.
+
+The following picture shows the the architecture of HA deployment based on the
+framework from ETSI NFV ISG.
+
+.. figure:: images/Overview.jpg
+    :alt: Architecture for HA Deployment
+    :figclass: align-center
+
+    Fig 1. Architecture of HA Deployment based on the Framework of ETSI NFV ISG
+
+1.2 HA deployment topology
+==================================================================
+
+This section will introduce the HA deployment topology for an NFV system.
+The topology explained in this section is to support the software
+cluster of OPNFV platform, which we will discuss in detail in section1.3.
+
+The typical topology of deployment OPNFV platform should include at
+least the controller nodes, and the compute nodes. Depend on the request of
+the users, standalone network nodes or storage nodes can be added into this
+topology. The simplest HA deployment of OPNFV only include the control nodes. Further
+HA schemes can be provided to the compute nodes, the network nodes and the storage
+nodes, according to the requirement of services deployed on the NFV system.
+Figure 2 shows the deployment topology, in which the controller nodes are all in
+a cluster, and the compute nodes can be in another cluster.
+
+The control node cluster here is to provide HA for the controller services, so
+that the services on the control node can successfully failover when failure
+happens and the service can continue. The cluster service should also provide
+automatic recovery for the control nodes. For OPNFV, the control node cluster
+should include at least 3 nodes, and should be an odd number if the cluster
+management system use quorum. This may change if we use different cluster
+management schemes though.
+
+The compute node clusters is responsible for providing HA for the services running
+on the compute nodes. These services may include agents for openstack, host os,
+hypervisors. Such cluster is responsible for the recovery and repair
+of the services. However, compute node cluster will certainly bring complexity to
+the whole system, and would increase the cost. There could be multiple solutions
+for the compute cluster, e.g., senlin from openstack.
+
+There could be other HA solutions for the compute nodes except for cluster. Combination
+of congress and doctor can be one of them, in which doctor provides quickly notification
+of failure to the VIM, and congress provides proper recovery procedure. In such scheme,
+the compute nodes are not recovered by the cluster scheme, but recovered under the
+supervision of VIM.
+
+.. figure:: images/topology_control_compute.jpg
+    :alt: HA Deployment Topology of Control Nodes and Compute Nodes
+    :figclass: align-center
+
+    Fig 2. HA Deployment Topology of Control Nodes and Compute Nodes
+
+When the cloud is supporting heavy network traffic, which is often the case for the data
+plane services in the Telecom scenarios, it is necessary to deploy standalone network
+nodes for openstack, so that the large amont of traffic switching and routing will not
+bring extra load to the controller nodes. In figure 3, we add network nodes into the
+topology and shows how to deploy it in a high available way. In this figure, the
+network nodes are deployed in a cluster. The cluster will provide HA for the services
+runing on the network nodes. Such cluster scheme could be the same with that of the
+compute nodes.
+
+On thing to be notify is that all hosts in the NFV system should have at least two NICs
+that are bonded via LACP.
+
+.. figure:: images/topology_control_compute_network.jpg
+    :alt: HA Deployment Topology of Control Nodes and Compute Nodes and Network Nodes
+    :figclass: align-center
+
+    Fig 3. HA Deployment Topology of Control Nodes, Compute Nodes and network Nodes
+
+The HA deployment for storage can be different for all different storage schemes. We will
+discuss the detail of the storage HA deployment in section 1.3.3
+
+1.3 Software HA Framework
+==================================================================
+
+In this section, we introduce more details about the HA schemes for a complete NFV system. 
+
+1.3.1 Openstack Controller services (Openstack services)
+--------------------------------------------------------
+
+For the High Availability of OpenStack Controller nodes, Pacemaker and Corosync are
+often used. The following texts are refering from the HA guideline of OpenStack, which
+gives an example of solution of HA deployment.(http://docs.openstack.org/ha-guide/)
+
+At its core, a cluster is a distributed finite state machine capable of co-ordinating the startup and recovery
+of inter-related services across a set of machines. For OpenStack Controller nodes, a cluster management system,
+such as Pacemaker, is recommended to use to provide the following metrics.
+
+1, Awareness of other applications in the stack
+
+2, Awareness of instances on other machines
+
+3, A shared implementation and calculation of quorum.
+
+4, Data integrity through fencing (a non-responsive process does not imply it is not doing anything)
+
+5, Automated recovery of failed instances
+
+Figure 4 shows the details of HA schemes for Openstack controller nodes with Pacemaker.
+
+.. figure:: images/HA_control.jpg
+    :alt: HA Deployment of Openstack Control Nodes based on Pacemaker
+    :figclass: align-center
+
+    Fig 4. HA Deployment of Openstack Control Nodes based on Pacemaker
+
+High availability of all stateless services are provided by pacemaker and HAProxy.
+
+Pacemaker cluster stack is the state-of-the-art high availability and load
+balancing stack for the Linux platform. Pacemaker is useful to make OpenStack
+infrastructure highly available. Also, it is storage and application-agnostic,
+and in no way specific to OpenStack.
+
+Pacemaker relies on the Corosync messaging layer for reliable cluster
+communications. Corosync implements the Totem single-ring ordering and
+membership protocol. It also provides UDP and InfiniBand based messaging,
+quorum, and cluster membership to Pacemaker.
+
+Pacemaker does not inherently (need or want to) understand the applications
+it manages. Instead, it relies on resource agents (RAs), scripts that
+encapsulate the knowledge of how to start, stop, and check the health
+of each application managed by the cluster.These agents must conform
+to one of the OCF, SysV Init, Upstart, or Systemd standards.Pacemaker
+ships with a large set of OCF agents (such as those managing MySQL
+databases, virtual IP addresses, and RabbitMQ), but can also use any
+agents already installed on your system and can be extended with your
+own (see the developer guide).
+
+After deployment of Pacemaker, HAProxy is used to provide VIP for all the
+OpenStack services and act as load balancer. HAProxy provides a fast and
+reliable HTTP reverse proxy and load balancer for TCP or HTTP applications.
+It is particularly suited for web crawling under very high loads while
+needing persistence or Layer 7 processing. It realistically supports tens
+of thousands of connections with recent hardware.
+
+Each instance of HAProxy configures its front end to accept connections
+only from the virtual IP (VIP) address and to terminate them as a list
+of all instances of the corresponding service under load balancing, such
+as any OpenStack API service. This makes the instances of HAProxy act
+independently and fail over transparently together with the network endpoints
+(VIP addresses) failover and, therefore, shares the same SLA.
+
+We can alternatively use a commercial load balancer, which is a hardware or
+software. A hardware load balancer generally has good performance.
+
+Galera Cluster, or other database cluster service, should also be deployed
+to provide data replication and  synchronization between data base. Galera
+Cluster is a synchronous multi-master database cluster, based on MySQL and
+the InnoDB storage engine. It is a high-availability service that provides
+high system uptime, no data loss, and scalability for growth. The selection
+of DB also will have potential influence on the behaviour on the application
+code. For instance using Galera Clusterl may give you higher concurrent write
+perfomance but may require a more complex conflict resolution.
+
+We can also achieve high availability for the OpenStack database in many different
+ways, depending on the type of database that we are using. There are three
+implementations of Galera Cluster available:
+
+1, Galera Cluster for MySQL The MySQL reference implementation from Codership;
+
+2, MariaDB Galera Cluster The MariaDB implementation of Galera Cluster, which is
+commonly supported in environments based on Red Hat distributions;
+
+3, Percona XtraDB Cluster The XtraDB implementation of Galera Cluster from Percona.
+
+In addition to Galera Cluster, we can also achieve high availability through other
+database options, such as PostgreSQL, which has its own replication system.
+
+To make the RabbitMQ high available, Rabbit HA queue should be configued, and all
+openstack services should be configurd to use the Rabbit HA queue.
+
+In the meantime, specific schemes should also be provided to avoid single point of
+failure of Pacemaker. And services failed should be automaticly repaired.
+
+Note that the scheme we described above is just one possible scheme for the HA
+deployment of the controller nodes. Other schemes can also be used to provide cluster
+management and monitoring.
+
+1.3.2 SDN controller services
+---------------------------------------
+
+SDN controller software is data intensive application. All static and dynamic data has
+one or more duplicates distributed to other physical nodes in cluster. Built-in HA schema
+always be concordant with data distribution and built-in mechanism will select or
+re-select master nodes in cluster. In deployment stage software of SDN controller
+should be deployed to at least two or more physical nodes regardless whether the
+software is deployed inside VM or containner. Dual management network plane should
+be provided for SDN controller cluster to support built-in HA schema.
+
+1.3.3 Storage
+----------------------------------------
+Depending on what storage scheme deployed, different HA schemes should be used. The following
+text are refering from the Mirantis OpenStack reference architecture, which provides suggestions
+on the HA deployment of different storage schemes.
+
+1, Ceph
+
+Ceph implements its own HA. When deploying it, enough controller nodes running the Ceph Monitor
+service to form a quarum, and enough Ceph OSD nodes to satisfy the object replication factor are
+needed.
+
+2, Swift
+
+Swift API relies on the same HAProxy setup with VIP on controller nodes as the other REST
+APIs. For small scale deployment, swift storage and Proxy services can be deployed on the
+controller nodes. However, for a larger production environment, dedicated storage nodes, in
+which two for swift proxy and at least three for swift storage, are needed.
+
+
+
+1.3.4 Host OS and Hypervisor
+---------------------------------------
+
+The Host OS and Hypervisor should be supervised and monitored for failure, and should be
+repaired when failure happens. Such supervision can based on a cluster scheme, or can
+just simply use controller to constantly monitor the computer host. Figure 6 shows a
+simplified framework for hypervisor cluster.
+
+When host/hypervisor failure happens, VMs on that host should be evacuated. However,
+such scheme should coordinate with the VM HA scheme, so that when both the host and the
+VM detect the failure, they should know who should take responsibility for the evacuation. 
+
+
+.. figure:: images/HA_Hypervisor.jpg
+    :alt: HA Deployment of Host OS and Hypervisor
+    :figclass: align-center
+
+    Fig 5. HA Deployment of Host OS and Hypervisor
+
+1.3.5 Virtual Machine (VM)
+---------------------------------------
+
+VM should be supervised and monitored for failure, and should be repaired when failure
+happens. We can rely on the hypervisor to monitor the VM failure. Another scheme can be
+used is a cluster for the VM, in which failure of VMs in one cluster can be supervised
+and will be repaired by the cluster manager. Pacemaker and other cluster management
+schemes can be considered for the VM cluster.
+
+In case when VNFs do not have HA schemes, extra HA scheme for VM should be taken into
+consideration. Such approach is kind of best effort for the NFV platform to provide HA
+for the VNF service, and may lead to failure copy between VMs when VNF fails. Since the
+NFVI can hardly know of the service runing in the VNF, it is imporssible for the NFVI
+level to provide overall HA solution for the VNF services. Therefore, even though we
+mention this scheme here, we strongly suggest the VNF should have its own HA schemes.
+
+Figure 6 gives an example for the VM active/standby deployment. In this case, both the
+active VM and the standby VM are deployed with the same VNF image. When failure happens
+to the active VM, the standby VM should take the traffic and replace the active VM. Such
+scheme is the best effort of the NFVI when VNFs do not have HA schemes and would only
+rely on VMs to provide redundancy. However, for stateful VNFs, there should be data copy
+between the active VM and standby VM. In this case, fault for the active VM can also be
+copied to the standby VM, leading to failure of the new active VM.
+
+.. figure:: images/HA_VM.jpg
+    :alt: VM Active/Standby Deployment
+    :figclass: align-center
+
+    Fig 6. VM Active/Standby Deployment
+
+1.3.6 Virtual Network Functions (VNF)
+---------------------------------------
+
+For telecom services, it is suggested that VNFs should have its own built-in HA schemes
+or HA schemes implemented in VNF Managerhave to provide high available services to
+the customers. HA schemes for the VNFs can based on cluster. In this case, OpenSAF,
+pacemaker and other cluster management services can be used.
+
+HA schemes for the VNFs should be coordinate with the lower layer. For example, it
+should be clear which level will take responsibility for VM restart. A suggested
+schemes could be, the VNF layer should be responsible for the redundancy and failover
+of the VNFs when failure happens. Such failover should take place in quite a short
+time (less then seconds). The repairing procedure will then take place from upper
+layer to lower layer, that is, the VNF layer will first check if the failure is at
+its layer, and should try to repair itself. If it fails to repaire the failure,
+the failure should escalate to lower layers and let the NFVI layer to do the repair
+work. There could also be cases that the NFVI layer has detected the failure and will
+repair it before the escalation. These functions should be complished by the coordination
+of all different component, including the VNFM, VIM, VNFs and NFVI.
+
+In the meantime, the VNFs can take advantage of API the hypervisor can provide to
+them to enhance HA. Such API may include constant health check from the hypervisor,
+affinity/inaffinity deployment support. example about watchdog
+
+Figure 7 gives an example for the VNF HA scheme.
+
+.. figure:: images/HA_VNF.jpg
+    :alt: HA Deployment of VNFs
+    :figclass: align-center
+
+    Fig 7. HA Deployment of VNFs
+
+*********************************************************************************
+2. HA deployment guideline for OPNFV releases
+*********************************************************************************
+
+In this section, we will continiously update the HA deployment guideline for the releases
+of OPNFV.
+
+2.1 HA deployment guideline for Arno
+==============================================
+
+2.1.1 Deployment Framework
+-----------------------------------------------
+
+Figure 8 shows an overall architecture for the HA deployment of ARNO.
+
+For OPNFV Arno release, HA deployment of Openstack Control Node (Openstack Juno) and ODL
+controller (ODL Helium) is supported. Both deployment tools (fuel and forman)support
+such HA deployment.
+
+For such HA deployment, the following components¡¯ failure is protected
+
+Software:
+* Nova scheduler
+* Nova conductor
+* Cinder scheduler
+* Neutron server
+* Heat engine
+
+Controller hardware:
+* dead server
+* dead switch
+* dead port
+* dead disk
+* full disk
+
+Figure 9 gives an example for the VNF HA scheme.
+
+.. figure:: images/HA_ARNO.jpg
+    :alt: HA Deployment of OPNFV ARNO release
+    :figclass: align-center
+
+    Fig 9. HA Deployment of OPNFV ARNO release
+
+2.1.2 HA test result for ARNO
+-------------------------------------------------
+
+Two specific High Availability testcases are done on the ARNO release. These test cases
+are collaboratively developed by the High Availability project and the Yardstick project.
+
+Both cases are excuted in the China Mobile's Lab, where ARNO SR1 release is deployed with
+Fuel.
+
+The two testcases respectively test the following two aspects:
+
+1, Controll Node Service HA
+
+In this test, HA of "nova-api" is tested. According to the result, the service can
+successfully failover to the other controller nodes  within 2.36s, once failure happens
+at the active node. However, the service can't repair itself automatically. more
+explaination about the repair, other services are not tested yet.
+
+2, Control Node Hardware HA
+
+In this test, HA of the controller node hardware is tested. One of the hardware is
+abnormally shutdown, and the service of "nova-api" is monitored. According to the test
+results, the service can failover to the other controller node within 10.71 secondes.
+However, the failed hardware can't automatically repair itself.
+
+See more details about these test cases in the Yardstick doc of "Test Results for
+yardstick-opnfv-ha"(https://gerrit.opnfv.org/gerrit/#/c/7543/).
+
+From these basic test cases we can see that OPNFV ARNO has integrated with some HA
+schemes in its controller nodes. However, its capability of self repair should be
+enhanced.
+
+2.2 HA deployment guideline for Brahmaputra
+==============================================
+In the Brahmaputra release, 4 installers are provided. We will discuss about the HA
+deployment of each installer.
+
+2.2.1 Apex
+----------------------------------------------------
+
+For the installer of Apex, all of the OpenStack services are in HA on all 3 controllers.
+The services are monitored by pacemaker and load balanced by HA Proxy with VIPs.  The
+SDN controllers usually only run as a single instance on the first controller with no
+HA scheme.
+
+Database is clustered with galera in an active passive failover via pacemaker and the
+message bus is rabbitHA and the services are managed by pacemaker.
+
+Storage is using ceph, clustered across the control nodes.
+
+In the future, more work is on the way to provide HA for the SDN controller. The Apex
+team has already finished a demo that runs ODL on each controller, load balanced to
+neutron via a VIP + HA Proxy, but is not using pacemaker. Meanwhile, they are also
+working to include ceph storage HA for compute nodes as well.
+
+2.2.2 Compass
+---------------------------------------------------------
+
+2.2.3 Fuel
+-------------------------------------------------------------
+
+At moment Fuel installer support the following HA schemes.
+
+1)Openstackcontrollers: N-way redundant (1,3,5, etc)
+2)OpenDaylight:No redundancy
+3)Cephstorage OSD: N-way redundant (1,3,5, etc)
+4)Networkingattachment redundancy: LAG
+5)NTPredundancy: N-way relays, up to 3 upstream sources
+6)DNSredundancy: N-way relays, up to 3 upstream sources
+7)DHCP:1+1
+
+2.2.4 JOID
+---------------------------------------------------------
+
+JOID provides HA based on openstack services. Individual service charms have been
+deployed in a container within a host, and each charms are distributed in a way each
+service which meant for HA will go into container on individual nodes. For example
+keystone service, there are three containers on each control node and VIP has been
+assigned to use by the front end API to use keystone. So in case any of the container
+fails VIP will keep responding to via the other two services. As HA can be maintainer
+with odd units at least one service container is required to response.
+
+
+Reference
+==========
+
+* https://www.rdoproject.org/ha/ha-architecture/
+* http://docs.openstack.org/ha-guide/
+* https://wiki.opnfv.org/display/availability?preview=/2926706/2926714/scenario_analysis_for_high_availability_in_nfv.pdf
+* https://wiki.opnfv.org/display/availability?preview=/2926706/2926708/ha_requirement.pdf
+
diff --git a/R3_Deployment/images/HA_ARNO.JPG b/R3_Deployment/images/HA_ARNO.JPG
new file mode 100644
index 0000000..58f9eca
--- /dev/null
+++ b/R3_Deployment/images/HA_ARNO.JPG
diff --git a/R3_Deployment/images/HA_Hypervisor.JPG b/R3_Deployment/images/HA_Hypervisor.JPG
new file mode 100644
index 0000000..4c1d04b
--- /dev/null
+++ b/R3_Deployment/images/HA_Hypervisor.JPG
diff --git a/R3_Deployment/images/HA_VM.JPG b/R3_Deployment/images/HA_VM.JPG
new file mode 100644
index 0000000..fdfd759
--- /dev/null
+++ b/R3_Deployment/images/HA_VM.JPG
diff --git a/R3_Deployment/images/HA_VNF.JPG b/R3_Deployment/images/HA_VNF.JPG
new file mode 100644
index 0000000..f3154ff
--- /dev/null
+++ b/R3_Deployment/images/HA_VNF.JPG
diff --git a/R3_Deployment/images/HA_control.JPG b/R3_Deployment/images/HA_control.JPG
new file mode 100644
index 0000000..c14866a
--- /dev/null
+++ b/R3_Deployment/images/HA_control.JPG
diff --git a/R3_Deployment/images/Overview.JPG b/R3_Deployment/images/Overview.JPG
new file mode 100644
index 0000000..ed7c6ab
--- /dev/null
+++ b/R3_Deployment/images/Overview.JPG
diff --git a/R3_Deployment/images/topology_control_compute.JPG b/R3_Deployment/images/topology_control_compute.JPG
new file mode 100644
index 0000000..107b5f0
--- /dev/null
+++ b/R3_Deployment/images/topology_control_compute.JPG
diff --git a/R3_Deployment/images/topology_control_compute_network.JPG b/R3_Deployment/images/topology_control_compute_network.JPG
new file mode 100644
index 0000000..57c384a
--- /dev/null
+++ b/R3_Deployment/images/topology_control_compute_network.JPG
diff --git a/R3_Deployment/images/topology_control_compute_network_storage.JPG b/R3_Deployment/images/topology_control_compute_network_storage.JPG
new file mode 100644
index 0000000..2b0937c
--- /dev/null
+++ b/R3_Deployment/images/topology_control_compute_network_storage.JPG