Merge "Docs: Adding preliminary documentation for Network Service Benchmarking."

9 files changed, 452 insertions, 11 deletions

diff --git a/docs/userguide/01-introduction.rst b/docs/userguide/01-introduction.rst
index 9d9cf0fb5..0e0eea002 100755
--- a/docs/userguide/01-introduction.rst
+++ b/docs/userguide/01-introduction.rst
@@ -46,18 +46,27 @@ This document consists of the following chapters:
 * Chapter :doc:`04-vtc-overview` provides information on the :term:`VTC`.
 
 * Chapter :doc:`05-apexlake_installation` provides instructions to install the
-  experimental framework *ApexLake* and chapter :doc:`06-apexlake_api` explains
-  how this framework is integrated in *Yardstick*.
+  experimental framework *ApexLake*
 
-* Chapter :doc:`07-installation` provides instructions to install *Yardstick*.
+* Chapter :doc:`06-apexlake_api` explains how this framework is integrated in
+  *Yardstick*.
 
-* Chapter :doc:`08-yardstick_plugin` provides information on how to integrate
+* Chapter :doc:`07-nsb-overview` describes the methodology implemented by the
+  yardstick - Network service benchmarking to test real world usecase for a
+  given VNF
+
+* Chapter :doc:`08-nsb_installation` provides instructions to install
+  *Yardstick - Network service benchmarking testing*.
+
+* Chapter :doc:`09-installation` provides instructions to install *Yardstick*.
+
+* Chapter :doc:`10-yardstick_plugin` provides information on how to integrate
   other OPNFV testing projects into *Yardstick*.
 
-* Chapter :doc:`09-result-store-InfluxDB` provides inforamtion on how to run
+* Chapter :doc:`11-result-store-InfluxDB` provides inforamtion on how to run
   plug-in test cases and store test results into community's InfluxDB.
 
-* Chapter :doc:`10-list-of-tcs` includes a list of available Yardstick test
+* Chapter :doc:`12-list-of-tcs` includes a list of available Yardstick test
   cases.
 
 
diff --git a/docs/userguide/07-nsb-overview.rst b/docs/userguide/07-nsb-overview.rst
new file mode 100644
index 000000000..19719f1a7
--- /dev/null
+++ b/docs/userguide/07-nsb-overview.rst
@@ -0,0 +1,177 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, 2016-2017 Intel Corporation.
+
+=====================================
+Network Services Benchmarking (NSB)
+=====================================
+
+Abstract
+========
+
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+
+This chapter provides an overview of the NSB, a contribution to OPNFV
+Yardstick_ from Intel.
+
+Overview
+========
+
+GOAL: Extend Yardstick to perform real world VNFs and NFVi Characterization and
+benchmarking with repeatable and deterministic methods.
+
+The Network Service Benchmarking (NSB) extends the yardstick framework to do
+VNF characterization and benchmarking in three different execution
+environments viz., bare metal i.e. native Linux environment, standalone virtual
+environment and managed virtualized environment (e.g. Open stack etc.).
+It also brings in the capability to interact with external traffic generators
+both hardware & software based for triggering and validating the traffic
+according to user defined profiles.
+
+NSB extension includes:
+    • Generic data models of Network Services, based on ETSI specs
+    • New Standalone context for VNF testing like SRIOV, OVS, OVS-DPDK etc
+    • Generic VNF configuration models and metrics implemented with Python
+      classes
+    • Traffic generator features and traffic profiles
+        • L1-L3 state-less traffic profiles
+        • L4-L7 state-full  traffic  profiles
+        • Tunneling protocol / network overlay support
+    • Test case samples
+        • Ping
+        • Trex
+        • vPE,vCGNAT, vFirewall etc - ipv4 throughput, latency etc
+    • Traffic generators like Trex, ab/nginx, ixia, iperf etc
+    • KPIs for a given use case:
+        • System agent support for collecting NFvi KPI. This includes:
+            o CPU statistic
+            o Memory BW
+            o OVS-DPDK Stats
+        • Network KPIs – eg, inpackets, outpackets, thoughput, latency etc
+        • VNF KPIs – packet_in, packet_drop, packet_fwd etc
+
+Architecture
+============
+The Network Service (NS) defines a set of Virtual Network Functions (VNF)
+connected together using NFV infrastructure.
+
+The Yardstick NSB extension can support multiple VNFs created by different
+vendors including traffic generators. Every VNF being tested has its
+own data model. The Network service defines a VNF modelling on base of performed
+network functionality. The part of the data model is a set of the configuration
+parameters, number of connection points used and flavor including core and
+memory amount.
+
+The ETSI defines a Network Service as a set of configurable VNFs working in
+some NFV Infrastructure connecting each other using Virtual Links available
+through Connection Points. The ETSI MANO specification defines a set of
+management entities called Network Service Descriptors (NSD) and
+VNF Descriptors (VNFD) that define real Network Service. The picture below
+makes an example how the real Network Operator use-case can map into ETSI
+Network service definition
+
+Network Service framework performs the necessary test steps. It may involve
+    o Interacting with traffic generator and providing the inputs on traffic
+      type / packet structure to generate the required traffic as per the
+      test case. Traffic profiles will be used for this.
+    o Executing the commands required for the test procedure and analyses the
+      command output for confirming whether the command got executed correctly
+      or not. E.g. As per the test case, run the traffic for the given
+      time period / wait for the necessary time delay
+    o Verify the test result.
+    o Validate the traffic flow from SUT
+    o Fetch the table / data from SUT and verify the value as per the test case
+    o Upload the logs from SUT onto the Test Harness server
+    o Read the KPI’s provided by particular VNF
+
+Components of Network Service
+------------------------------
+
+* *Models for Network Service benchmarking*: The Network Service benchmarking
+  requires the proper modelling approach. The NSB provides models using Python
+  files and defining of NSDs and VNFDs.
+
+The benchmark control application being a part of OPNFV yardstick can call
+that python models to instantiate and configure the VNFs. Depending on
+infrastructure type (bare-metal or fully virtualized) that calls could be
+made directly or using MANO system.
+
+* *Traffic generators in NSB*: Any benchmark application requires a set of
+  traffic generator and traffic profiles defining the method in which traffic
+  is generated.
+
+The Network Service benchmarking model extends the Network Service
+definition with a set of Traffic Generators (TG) that are treated
+same way as other VNFs being a part of benchmarked network service.
+Same as other VNFs the traffic generator are instantiated and terminated.
+
+Every traffic generator has own configuration defined as a traffic profile and
+a set of KPIs supported. The python models for TG is extended by specific calls
+to listen and generate traffic.
+
+* *The stateless TREX traffic generator*: The main traffic generator used as
+  Network Service stimulus is open source TREX tool.
+
+The TREX tool can generate any kind of stateless traffic.
+
+.. code-block:: console
+
+        +--------+      +-------+      +--------+
+        |        |      |       |      |        |
+        |  Trex  | ---> |  VNF  | ---> |  Trex  |
+        |        |      |       |      |        |
+        +--------+      +-------+      +--------+
+
+Supported testcases scenarios:
+• Correlated UDP traffic using TREX traffic generator and replay VNF.
+    o using different IMIX configuration like pure voice, pure video traffic etc
+    o using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows
+    o Using different number of rules configured like 1 rule, 1K, 10K rules
+
+For UDP correlated traffic following Key Performance Indicators are collected
+for every combination of test case parameters:
+        • RFC2544 throughput for various loss rate defined (1% is a default)
+
+Graphical Overview
+==================
+
+NSB Testing with yardstick framework  facilitate performance testing of various
+VNFs provided.
+
+.. code-block:: console
+  +-----------+
+  |           |                                                     +-----------+
+  |   vPE     |                                                   ->|TGen Port 0|
+  | TestCase  |                                                   | +-----------+
+  |           |                                                   |
+  +-----------+     +------------------+            +-------+     |
+                    |                  | -- API --> |  VNF  | <--->
+  +-----------+     |     Yardstick    |            +-------+     |
+  | Test Case | --> |    NSB Testing   |                          |
+  +-----------+     |                  |                          |
+        |           |                  |                          |
+        |           +------------------+                          |
+  +-----------+                                                   | +-----------+
+  |   Traffic |                                                   ->|TGen Port 1|
+  |  patterns |                                                     +-----------+
+  +-----------+
+              Figure 1: Network Service - 2 server configuration
+
+
+Install
+=======
+
+run the nsb_install.sh with root privileges
+
+Run
+===
+
+source ~/.bash_profile
+cd <yardstick_repo>/yardstick/cmd
+sudo -E ./NSBperf.py --vnf vpe --test tc_baremetal_rfc2544_ipv4_1flow_64B.yaml
+
+Development Environment
+=======================
+
+Ubuntu 14.04, Ubuntu 16.04
diff --git a/docs/userguide/08-nsb_installation.rst b/docs/userguide/08-nsb_installation.rst
new file mode 100644
index 000000000..a390bb7d7
--- /dev/null
+++ b/docs/userguide/08-nsb_installation.rst
@@ -0,0 +1,253 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, 2016-2017 Intel Corporation.
+
+Yardstick - NSB Testing -Installation
+=====================================
+
+Abstract
+--------
+
+Yardstick supports installation on Ubuntu 14.04 or via a Docker image. The
+installation procedure on Ubuntu 14.04 or via the docker image are detailed in
+the section below.
+
+The Network Service Benchmarking (NSB) extends the yardstick framework to do
+VNF characterization and benchmarking in three different execution
+environments viz., bare metal i.e. native Linux environment, standalone virtual
+environment and managed virtualized environment (e.g. Open stack etc.).
+It also brings in the capability to interact with external traffic generators
+both hardware & software based for triggering and validating the traffic
+according to user defined profiles.
+
+The steps needed to run Yardstick with NSB testing are:
+
+* Install Yardstick (NSB Testing).
+* Setup pod.yaml describing Test topology
+* Create the test configuration yaml file.
+* Run the test case.
+
+
+Prerequisites
+-------------
+
+Refer chapter 08-instalaltion.rst for more information on yardstick
+prerequisites
+
+Several prerequisites are needed for Yardstick(VNF testing):
+* Python Modules: pyzmq, pika.
+* flex
+* bison
+* build-essential
+* automake
+* libtool
+* librabbitmq-dev
+* rabbitmq-server
+* collectd
+* intel-cmt-cat
+
+Installing Yardstick on Ubuntu 14.04
+------------------------------------
+
+.. _install-framework:
+
+You can install Yardstick framework directly on Ubuntu 14.04 or in an Ubuntu
+14.04 Docker image. No matter which way you choose to install Yardstick
+framework, the following installation steps are identical.
+
+If you choose to use the Ubuntu 14.04 Docker image, You can pull the Ubuntu
+14.04 Docker image from Docker hub:
+
+::
+
+  docker pull ubuntu:14.04
+
+Installing Yardstick framework
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Download source code and install python dependencies:
+
+::
+
+  git clone https://gerrit.opnfv.org/gerrit/yardstick
+  cd yardstick
+  ./nsb_setup.sh
+
+It will automatically download all the packages needed for NSB Testing setup.
+
+System Topology:
+-----------------
+
+.. code-block:: console
+
+  +----------+              +----------+
+  |          |              |          |
+  |          | (0)----->(0) |   Ping/  |
+  |    TG1   |              |   vPE/   |
+  |          |              |   2Trex  |
+  |          | (1)<-----(1) |          |
+  +----------+              +----------+
+  trafficgen_1                   vnf
+
+
+OpenStack parameters and credentials
+------------------------------------
+
+Environment variables
+^^^^^^^^^^^^^^^^^^^^^
+Before running Yardstick (NSB Testing) it is necessary to export traffic
+generator libraries.
+
+::
+  source ~/.bash_profile
+
+Config yardstick conf
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
+
+vi /etc/yardstick/yardstick.conf
+
+Config yardstick.conf
+::
+
+  [DEFAULT]
+  debug = True
+  dispatcher = influxdb
+
+  [dispatcher_influxdb]
+  timeout = 5
+  target = http://{YOUR_IP_HERE}:8086
+  db_name = yardstick
+  username = root
+  password = root
+
+  [nsb]
+  trex_path=/opt/nsb_bin/trex/scripts
+  bin_path=/opt/nsb_bin
+
+
+Config pod.yaml describing Topology
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Before executing Yardstick test cases, make sure that pod.yaml reflects the
+topology and update all the required fields.
+
+copy /etc/yardstick/nodes/pod.yaml.nsb.example to /etc/yardstick/nodes/pod.yaml
+
+Config pod.yaml
+::
+    nodes:
+    -
+        name: trafficgen_1
+        role: TrafficGen
+        ip: 1.1.1.1
+        user: root
+        password: r00t
+        interfaces:
+            xe0:  # logical name from topology.yaml and vnfd.yaml
+                vpci:      "0000:07:00.0"
+                driver:    i40e # default kernel driver
+                dpdk_port_num: 0
+                local_ip: "152.16.100.20"
+                netmask:   "255.255.255.0"
+                local_mac: "00:00:00:00:00:01"
+            xe1:  # logical name from topology.yaml and vnfd.yaml
+                vpci:      "0000:07:00.1"
+                driver:    i40e # default kernel driver
+                dpdk_port_num: 1
+                local_ip: "152.16.40.20"
+                netmask:   "255.255.255.0"
+                local_mac: "00:00.00:00:00:02"
+
+    -
+        name: vnf
+        role: vnf
+        ip: 1.1.1.2
+        user: root
+        password: r00t
+        host: 1.1.1.2 #BM - host == ip, virtualized env - Host - compute node
+        interfaces:
+            xe0:  # logical name from topology.yaml and vnfd.yaml
+                vpci:      "0000:07:00.0"
+                driver:    i40e # default kernel driver
+                dpdk_port_num: 0
+                local_ip: "152.16.100.19"
+                netmask:   "255.255.255.0"
+                local_mac: "00:00:00:00:00:03"
+
+            xe1:  # logical name from topology.yaml and vnfd.yaml
+                vpci:      "0000:07:00.1"
+                driver:    i40e # default kernel driver
+                dpdk_port_num: 1
+                local_ip: "152.16.40.19"
+                netmask:   "255.255.255.0"
+                local_mac: "00:00:00:00:00:04"
+        routing_table:
+        - network: "152.16.100.20"
+          netmask: "255.255.255.0"
+          gateway: "152.16.100.20"
+          if: "xe0"
+        - network: "152.16.40.20"
+          netmask: "255.255.255.0"
+          gateway: "152.16.40.20"
+          if: "xe1"
+        nd_route_tbl:
+        - network: "0064:ff9b:0:0:0:0:9810:6414"
+          netmask: "112"
+          gateway: "0064:ff9b:0:0:0:0:9810:6414"
+          if: "xe0"
+        - network: "0064:ff9b:0:0:0:0:9810:2814"
+          netmask: "112"
+          gateway: "0064:ff9b:0:0:0:0:9810:2814"
+          if: "xe1"
+
+Enable yardstick virtual environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Before executing yardstick test cases, make sure to activate yardstick
+python virtual environment
+
+::
+    source /opt/nsb_bin/yardstick_venv/bin/activate
+
+
+Examples and verifying the install
+----------------------------------
+
+It is recommended to verify that Yardstick was installed successfully
+by executing some simple commands and test samples. Before executing yardstick
+test cases make sure yardstick flavor and building yardstick-trusty-server
+image can be found in glance and openrc file is sourced. Below is an example
+invocation of yardstick help command and ping.py test sample:
+::
+
+  yardstick –h
+  yardstick task start samples/ping.yaml
+
+Each testing tool supported by Yardstick has a sample configuration file.
+These configuration files can be found in the **samples** directory.
+
+Default location for the output is ``/tmp/yardstick.out``.
+
+
+Run Yardstick - Network Service Testcases
+-----------------------------------------
+
+NS testing - using NSBperf CLI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  source /opt/nsb_setup/yardstick_venv/bin/activate
+  PYTHONPATH: ". ~/.bash_profile"
+  cd <yardstick_repo>/yardstick/cmd
+  Execute command: ./NSPerf.py -h
+      ./NSBperf.py --vnf <selected vnf> --test <rfc test>
+      eg: ./NSBperf.py --vnf vpe --test tc_baremetal_rfc2544_ipv4_1flow_64B.yaml
+
+NS testing - using yardstick CLI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+  source /opt/nsb_setup/yardstick_venv/bin/activate
+  PYTHONPATH: ". ~/.bash_profile"
+  Go to test case forlder type we want to execute.
+      e.g. <yardstick repo>/samples/vnf_samples/nsut/<vnf>/
+      run: yardstick --debug task start <test_case.yaml>
diff --git a/docs/userguide/07-installation.rst b/docs/userguide/09-installation.rst
index 9c2082a27..9c2082a27 100644
--- a/docs/userguide/07-installation.rst
+++ b/docs/userguide/09-installation.rst
diff --git a/docs/userguide/08-yardstick_plugin.rst b/docs/userguide/10-yardstick_plugin.rst
index f16dedd02..f16dedd02 100644
--- a/docs/userguide/08-yardstick_plugin.rst
+++ b/docs/userguide/10-yardstick_plugin.rst
diff --git a/docs/userguide/09-result-store-InfluxDB.rst b/docs/userguide/11-result-store-InfluxDB.rst
index a0bb48a80..a0bb48a80 100644
--- a/docs/userguide/09-result-store-InfluxDB.rst
+++ b/docs/userguide/11-result-store-InfluxDB.rst
diff --git a/docs/userguide/10-grafana.rst b/docs/userguide/12-grafana.rst
index 416857b71..416857b71 100644
--- a/docs/userguide/10-grafana.rst
+++ b/docs/userguide/12-grafana.rst
diff --git a/docs/userguide/11-list-of-tcs.rst b/docs/userguide/13-list-of-tcs.rst
index 1b5806cd9..1b5806cd9 100644
--- a/docs/userguide/11-list-of-tcs.rst
+++ b/docs/userguide/13-list-of-tcs.rst
diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst
index 60e1340ac..826a9d9bf 100644
--- a/docs/userguide/index.rst
+++ b/docs/userguide/index.rst
@@ -16,10 +16,12 @@ Yardstick Overview
    04-vtc-overview
    05-apexlake_installation
    06-apexlake_api
-   07-installation
-   08-yardstick_plugin
-   09-result-store-InfluxDB
-   10-grafana
-   11-list-of-tcs
+   07-nsb-overview
+   08-nsb_installation
+   09-installation
+   10-yardstick_plugin
+   11-result-store-InfluxDB
+   12-grafana
+   13-list-of-tcs
    glossary
    references