.. This work is licensed under a Creative Commons Attribution 4.0 International
.. License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Ericsson and others.

*************************************
Yardstick Test Case Description TC092
*************************************

+-----------------------------------------------------------------------------+
|SDN Controller resilience in HA configuration                                |
|                                                                             |
+--------------+--------------------------------------------------------------+
|test case id  | OPNFV_YARDSTICK_TC092: SDN controller resilience and high    |
|              | availability HA configuration                                |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|test purpose  | This test validates SDN controller node high availability by |
|              | verifying there is no impact on the data plane connectivity  |
|              | when one SDN controller fails in a HA configuration,         |
|              | i.e. all existing configured network services DHCP, ARP, L2, |
|              | L3VPN, Security Groups should continue to operate            |
|              | between the existing VMs while one SDN controller instance   |
|              | is offline and rebooting.                                    |
|              |                                                              |
|              | The test also validates that network service operations such |
|              | as creating a new VM in an existing or new L2 network        |
|              | network remain operational while one instance of the         |
|              | SDN controller is offline and recovers from the failure.     |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|test method   | This test case:                                              |
|              |  1. fails one instance of a SDN controller cluster running   |
|              |     in a HA configuration on the OpenStack controller node   |
|              |                                                              |
|              |  2. checks if already configured L2 connectivity between     |
|              |     existing VMs is not impacted                             |
|              |                                                              |
|              |  3. verifies that the system never loses the ability to      |
|              |     execute virtual network operations, even when the        |
|              |     failed SDN Controller is still recovering                |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|attackers     | In this test case, an attacker called “kill-process” is      |
|              | needed. This attacker includes three parameters:             |
|              |  1. ``fault_type``: which is used for finding the attacker's |
|              |     scripts. It should be set to 'kill-process' in this test |
|              |                                                              |
|              |  2. ``process_name``: should be set to sdn controller        |
|              |     process                                                  |
|              |                                                              |
|              |  3. ``host``: which is the name of a control node where      |
|              |     opendaylight process is running                          |
|              |                                                              |
|              | example:                                                     |
|              |   - ``fault_type``: “kill-process”                           |
|              |   - ``process_name``: “opendaylight-karaf” (TBD)             |
|              |   - ``host``: node1                                          |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|monitors      | In this test case, the following monitors are needed         |
|              |  1. ``ping_same_network_l2``: monitor pinging traffic        |
|              |     between the VMs in same neutron network                  |
|              |                                                              |
|              |  2. ``ping_external_snat``: monitor ping traffic from VMs to |
|              |     external destinations (e.g. google.com)                  |
|              |                                                              |
|              |  3. ``SDN controller process monitor``: a monitor checking   |
|              |     the state of a specified SDN controller process. It      |
|              |     measures the recovery time of the given process.         |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|operations    | In this test case, the following operations are needed:      |
|              |  1. "nova-create-instance-in_network": create a VM instance  |
|              |     in one of the existing neutron network.                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|metrics       | In this test case, there are two metrics:                    |
|              |  1. process_recover_time: which indicates the maximun        |
|              |     time (seconds) from the process being killed to          |
|              |     recovered                                                |
|              |                                                              |
|              |  2. packet_drop: measure the packets that have been dropped  |
|              |     by the monitors using pktgen.                            |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|test tool     | Developed by the project. Please see folder:                 |
|              | "yardstick/benchmark/scenarios/availability/ha_tools"        |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|references    | TBD                                                          |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|configuration | This test case needs two configuration files:                |
|              |  1. test case file: opnfv_yardstick_tc092.yaml               |
|              |     - Attackers: see above “attackers” discription           |
|              |     - Monitors: see above “monitors” discription             |
|              |       - waiting_time: which is the time (seconds) from the   |
|              |         process being killed to stoping monitors the         |
|              |         monitors                                             |
|              |     - SLA: see above “metrics” discription                   |
|              |                                                              |
|              |  2. POD file: pod.yaml The POD configuration should record   |
|              |     on pod.yaml first. the “host” item in this test case     |
|              |     will use the node name in the pod.yaml.                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|test sequence | Description and expected result                              |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|pre-action    |  1. The OpenStack cluster is set up with an SDN controller   |
|              |     running in a three node cluster configuration.           |
|              |                                                              |
|              |  2. One or more neutron networks are created with two or     |
|              |     more VMs attached to each of the neutron networks.       |
|              |                                                              |
|              |  3. The neutron networks are attached to a neutron router    |
|              |     which is attached to an external network the towards     |
|              |     DCGW.                                                    |
|              |                                                              |
|              |  4. The master node of SDN controller cluster is known.      |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 1        | Start ip connectivity monitors:                              |
|              |  1. Check the L2 connectivity between the VMs in the same    |
|              |     neutron network.                                         |
|              |                                                              |
|              |  2. Check the external connectivity of the VMs.              |
|              |                                                              |
|              | Each monitor runs in an independent process.                 |
|              |                                                              |
|              | Result: The monitor info will be collected.                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 2        | Start attacker:                                              |
|              | SSH to the VIM node and kill the SDN controller process      |
|              | determined in step 2.                                        |
|              |                                                              |
|              | Result: One SDN controller service will be shut down         |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 3        | Restart the SDN controller.                                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 4        | Create a new VM in the existing Neutron network while the    |
|              | SDN controller is offline or still recovering.               |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 5        | Stop IP connectivity monitors after a period of time         |
|              | specified by “waiting_time”                                  |
|              |                                                              |
|              | Result: The monitor info will be aggregated                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 6        | Verify the IP connectivity monitor result                    |
|              |                                                              |
|              | Result: IP connectivity monitor should not have any packet   |
|              | drop failures reported                                       |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 7        | Verify process_recover_time, which indicates the maximun     |
|              | time (seconds) from the process being killed to recovered,   |
|              | is within the SLA. This step blocks until either the         |
|              | process has recovered or a timeout occurred.                 |
|              |                                                              |
|              | Result: process_recover_time is within SLA limits, if not,   |
|              | test case failed and stopped.                                |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 8        | Start IP connectivity monitors for the  new VM:              |
|              |  1. Check the L2 connectivity from the existing VMs to the   |
|              |     new VM in the Neutron network.                           |
|              |                                                              |
|              |  2. Check connectivity from one VM to an external host on    |
|              |     the Internet to verify SNAT functionality.               |
|              |                                                              |
|              | Result: The monitor info will be collected.                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 9        | Stop IP connectivity monitors after a period of time         |
|              | specified by “waiting_time”                                  |
|              |                                                              |
|              | Result: The monitor info will be aggregated                  |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|step 10       | Verify the IP connectivity monitor result                    |
|              |                                                              |
|              | Result: IP connectivity monitor should not have any packet   |
|              | drop failures reported                                       |
|              |                                                              |
+--------------+--------------------------------------------------------------+
|test verdict  | Fails only if SLA is not passed, or if there is a test case  |
|              | execution problem.                                           |
|              |                                                              |
+--------------+--------------------------------------------------------------+