Merge "Addition of NSB Prox developer guide."

1 files changed, 1226 insertions, 0 deletions

diff --git a/docs/testing/developer/devguide/devguide_nsb_prox.rst b/docs/testing/developer/devguide/devguide_nsb_prox.rst
new file mode 100755
index 000000000..fc533b2cf
--- /dev/null
+++ b/docs/testing/developer/devguide/devguide_nsb_prox.rst
@@ -0,0 +1,1226 @@
+Introduction
+=============
+
+This document describes the steps to create a new NSB PROX test based on
+existing PROX functionalities. NSB PROX provides is a simple approximation
+of an operation and can be used to develop best practices and TCO models
+for Telco customers, investigate the impact of new Intel compute,
+network and storage technologies, characterize performance, and develop
+optimal system architectures and configurations.
+
+.. contents::
+
+Prerequisites
+=============
+
+In order to integrate PROX tests into NSB, the following prerequisites are required.
+
+.. _`dpdk wiki page`: http://dpdk.org/
+.. _`yardstick wiki page`: https://wiki.opnfv.org/display/yardstick/
+.. _`Prox documentation`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation
+.. _`openstack wiki page`: https://wiki.openstack.org/wiki/Main_Page
+.. _`grafana getting started`: http://docs.grafana.org/guides/gettingstarted/
+.. _`opnfv grafana dashboard`: https://wiki.opnfv.org/display/yardstick/How+to+work+with+grafana+dashboard
+.. _`Prox command line`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation#Command_line_options
+.. _`grafana deployment`: https://wiki.opnfv.org/display/yardstick/How+to+deploy+InfluxDB+and+Grafana+locally
+.. _`Prox options`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation#.5Beal_options.5D
+.. _`NSB Installation`: http://artifacts.opnfv.org/yardstick/docs/userguide/index.html#document-09-installation
+
+* A working knowledge of Yardstick. See `yardstick wiki page`_.
+* A working knowledge of PROX. See `Prox documentation`_.
+* Knowledge of Openstack. See `openstack wiki page`_.
+* Knowledge of how to use Grafana. See `grafana getting started`_.
+* How to Deploy InfluxDB & Grafana. See `grafana deployment`_.
+* How to use Grafana in OPNFV/Yardstick. See `opnfv grafana dashboard`_.
+* How to install NSB. See `NSB Installation`_
+
+Sample Prox Test Hardware Architecture
+======================================
+
+The following is a diagram of a sample NSB PROX Hardware Architecture
+for both NSB PROX on Bare metal and on Openstack.
+
+In this example when running yardstick on baremetal, yardstick will
+run on the deployment node, the generator will run on the deployment node
+and the SUT(SUT) will run on the Controller Node.
+
+
+.. image:: images/PROX_Hardware_Arch.png
+   :width: 800px
+   :alt: Sample NSB PROX Hard Architecture
+
+Prox Test Architecture
+======================
+
+In order to create a new test, one must understand the architecture of
+the test.
+
+A NSB Prox test architecture is composed of:
+
+* A traffic generator. This provides blocks of data on 1 or more ports
+  to the SUT.
+  The traffic generator also consumes the result packets from the system
+  under test.
+* A SUT consumes the packets generated by the packet
+  generator, and applies one or more tasks to the packets and return the
+  modified packets to the traffic generator.
+
+  This is an example of a sample NSB PROX test architecture.
+
+.. image:: images/PROX_Software_Arch.png
+   :width: 800px
+   :alt: NSB PROX test Architecture
+
+This diagram is of a sample NSB PROX test application.
+
+* Traffic Generator
+
+  * Generator Tasks - Composted of 1 or more tasks (It is possible to
+    have multiple tasks sending packets to same port No. See Tasks Ai and Aii
+    plus Di and Dii)
+
+    * Task Ai - Generates Packets on Port 0 of Traffic Generator
+      and send to Port 0 of SUT Port 0
+    * Task Aii - Generates Packets on Port 0 of Traffic Generator
+      and send to Port 0 of SUT Port 0
+    * Task B - Generates Packets on Port 1 of Traffic Generator
+      and send to Port 1 of SUT Port 1
+    * Task C - Generates Packets on Port 2 of Traffic Generator
+      and send to Port 2 of SUT Port 2
+    * Task Di - Generates Packets on Port 3 of Traffic Generator
+      and send to Port 3 of SUT Port 3
+    * Task Dii - Generates Packets on Port 0 of Traffic Generator
+      and send to Port 0 of SUT Port 0
+
+  * Verifier Tasks - Composed of 1 or more tasks which receives
+    packets from SUT
+
+    * Task E - Receives packets on Port 0 of Traffic Generator sent
+      from Port 0 of SUT Port 0
+    * Task F - Receives packets on Port 1 of Traffic Generator sent
+      from Port 1 of SUT Port 1
+    * Task G - Receives packets on Port 2 of Traffic Generator sent
+      from Port 2 of SUT Port 2
+    * Task H - Receives packets on Port 3 of Traffic Generator sent
+      from Port 3 of SUT Port 3
+
+* SUT
+
+  * Receiver Tasks - Receives packets from generator - Composed on 1 or
+    more tasks which consume the packs sent from Traffic Generator
+
+    * Task A - Receives Packets on Port 0 of System-Under-Test from
+      Traffic Generator Port 0, and forwards packets to Task E
+    * Task B - Receives Packets on Port 1 of System-Under-Test from
+      Traffic Generator Port 1, and forwards packets to Task E
+    * Task C - Receives Packets on Port 2 of System-Under-Test from
+      Traffic Generator Port 2, and forwards packets to Task E
+    * Task D - Receives Packets on Port 3 of System-Under-Test from
+      Traffic Generator Port 3, and forwards packets to Task E
+
+  * Processing Tasks - Composed of multiple tasks in series which carry
+    out some processing on received packets before forwarding to the
+    task.
+
+    * Task E - This receives packets from the Receiver Tasks,
+      carries out some operation on the data and forwards to result
+      packets to the next task in the sequence - Task F
+    * Task F - This receives packets from the previous Task - Task
+      E, carries out some operation on the data and forwards to result
+      packets to the next task in the sequence - Task G
+    * Task G - This receives packets from the previous Task - Task F
+      and distributes the result packages to the Transmitter tasks
+
+  * Transmitter Tasks - Composed on 1 or more tasks which send the
+    processed packets back to the Traffic Generator
+
+    * Task H - Receives Packets from Task G of System-Under-Test and
+      sends packets to Traffic Generator Port 0
+    * Task I - Receives Packets from Task G of System-Under-Test and
+      sends packets to Traffic Generator Port 1
+    * Task J - Receives Packets from Task G of System-Under-Test and
+      sends packets to Traffic Generator Port 2
+    * Task K - Receives Packets From Task G of System-Under-Test and
+      sends packets to Traffic Generator Port 3
+
+NSB Prox Test
+=============
+
+A NSB Prox test is composed of the following components :-
+
+* Test Description File. Usually called
+  ``tc_prox_<context>_<test>-<ports>.yaml`` where
+
+  * <context> is either ``baremetal`` or ``heat_context``
+  * <test> is the a one or 2 word description of the test.
+  * <ports> is the number of ports used
+
+  Example tests ``tc_prox_baremetal_l2fwd-2.yaml`` or
+  ``tc_prox_heat_context_vpe-4.yaml``. This file describes the components
+  of the test, in the case of openstack the network description and
+  server descriptions, in the case of baremetal the hardware
+  description location. It also contains the name of the Traffic Generator, the SUT config file
+  and the traffic profile description, all described below. See nsb-test-description-label_
+
+* Traffic Profile file. Example ``prox_binsearch.yaml``. This describes the packet size, tolerated
+  loss, initial line rate to start traffic at, test interval etc See nsb-traffic-profile-label_
+
+* Traffic Generator Config file. Usually called ``gen_<test>-<ports>.cfg``.
+
+  This describes the activity of the traffic generator
+
+  * What each core of the traffic generator does,
+  * The packet of data sent by a core on a port of the traffic generator
+    to the system under test
+  * What core is used to wait on what port for data from the system
+    under test.
+
+  Example traffic generator config file  ``gen_l2fwd-4.cfg``
+  See nsb-traffic-generator-label_
+
+* SUT Config file. Usually called ``handle_<test>-<ports>.cfg``.
+
+  This describes the activity of the SUTs
+
+  * What each core of the  does,
+  * What cores receives packets from what ports
+  * What cores perform operations on the packets and pass the packets onto
+    another core
+  * What cores receives packets from what cores and transmit the packets on
+    the ports to the Traffic Verifier tasks of the Traffic Generator.
+
+  Example traffic generator config file  ``handle_l2fwd-4.cfg``
+  See nsb-sut-generator-label_
+
+* NSB PROX Baremetal Configuration file. Usually called
+  ``prox-baremetal-<ports>.yaml``
+
+  * <ports> is the number of ports used
+
+  This is required for baremetal only.  This describes hardware, NICs,
+  IP addresses, Network drivers, usernames and passwords.
+  See baremetal-config-label_
+
+* Grafana Dashboard. Usually called
+  ``Prox_<context>_<test>-<port>-<DateAndTime>.json`` where
+
+  * <context> Is either ``BM`` or ``heat``
+  * <test> Is the a one or 2 word description of the test.
+  * <port> is the number of ports used express as ``2Port`` or ``4Port``
+  * <DateAndTime> is the Date and Time expressed as a string.
+
+  Example grafana dashboard ``Prox_BM_L2FWD-4Port-1507804504588.json``
+
+Other files may be required. These are test specific files and will be
+covered later.
+
+.. _nsb-test-description-label:
+
+**Test Description File**
+
+Here we will discuss the test description for both
+baremetal and openstack.
+
+*Test Description File for Baremetal*
+-------------------------------------
+
+This section will introduce the meaning of the Test case description
+file. We will use ``tc_prox_baremetal_l2fwd-2.yaml`` as an example to
+show you how to understand the test description file.
+
+.. image:: images/PROX_Test_BM_Script.png
+   :width: 800px
+   :alt: NSB PROX Test Description File
+
+Now let's examine the components of the file in detail
+
+1. ``traffic_profile`` - This specifies the traffic profile for the
+   test. In this case ``prox_binsearch.yaml`` is used. See nsb-traffic-profile-label_
+
+2. ``topology`` - This is either ``prox-tg-topology-1.yaml`` or
+    ``prox-tg-topology-2.yaml`` or ``prox-tg-topology-4.yaml``
+    depending on number of ports required.
+
+3. ``nodes`` - This names the Traffic Generator and the System
+   under Test. Does not need to change.
+
+4. ``prox_path`` - Location of the Prox executable on the traffic
+   generator (Either baremetal or Openstack Virtual Machine)
+
+5. ``prox_config`` - This is the ``SUT Config File``.
+   In this case it is ``handle_l2fwd-2.cfg``
+
+   A number of additional parameters can be added. This example
+   is taken from VPE::
+
+    options:
+      vnf__0:
+        prox_path: /opt/nsb_bin/prox
+        prox_config: ``configs/handle_vpe-4.cfg``
+        prox_args:
+          ``-t``: ````
+        prox_files:
+          ``configs/vpe_ipv4.lua`` : ````
+          ``configs/vpe_dscp.lua`` : ````
+          ``configs/vpe_cpe_table.lua`` : ````
+          ``configs/vpe_user_table.lua`` : ````
+          ``configs/vpe_rules.lua`` : ````
+        prox_generate_parameter: True
+
+   ``prox_files`` - this specified that a number of addition files
+   need to be provided for the test to run correctly. This files
+   could provide routing information,hashing information or a
+   hashing algorithm and ip/mac information.
+
+   ``prox_generate_parameter`` - this specifies that the NSB application
+   is required to provide information to the nsb Prox in the form
+   of a file called ``parameters.lua``, which contains information
+   retrieved from either the hardware or the openstack configuration.
+
+6. ``prox_args`` - this specifies the command line arguments to start
+   prox. See `prox command line`_.
+
+7. ``prox_config`` - This specifies the Traffic Generator config file.
+
+8. ``runner`` - This is set to ``Duration`` - This specified that the
+   test run for a set duration. Other runner types are available
+   but it is recommend to use ``Duration``
+
+9. ``context`` - This is ``context`` for a 2 port Baremetal configuration.
+   If a 4 port configuration was required then file
+   ``prox-baremetal-4.yaml`` would be used. This is the NSB Prox
+   baremetal configuration file.
+
+.. _nsb-traffic-profile-label:
+
+*Traffic Profile file*
+----------------------
+
+This describes the details of the traffic flow. In this case ``prox_binsearch.yaml`` is used.
+
+.. image:: images/PROX_Traffic_profile.png
+   :width: 800px
+   :alt: NSB PROX Traffic Profile
+
+
+1. ``name`` - The name of the traffic profile. This name should match the name specified in the
+   ``traffic_profile`` field in the Test Description File.
+
+2. ``traffic_type`` - This specifies the type of traffic pattern generated, This name matches
+   class name of the traffic generator See::
+
+      network_services/traffic_profile/prox_binsearch.py class ProxBinSearchProfile(ProxProfile)
+
+   In this case it lowers the traffic rate until the number of packets
+   sent is equal to the number of packets received (plus a
+   tolerated loss). Once it achieves this it increases the traffic
+   rate in order to find the highest rate with no traffic loss.
+
+   Custom traffic types can be created by creating a new traffic profile class.
+
+3. ``tolerated_loss`` - This specifies the percentage of packets that can be lost/dropped before
+   we declare success or failure. Success is Transmitted-Packets from Traffic Generator is greater than or equal to
+   packets received by Traffic Generator plus tolerated loss.
+
+4. ``test_precision`` - This specifies the precision of the test results. For some tests the success criteria
+   may never be achieved because the test precision may be greater than the successful throughput. For finer
+   results increase the precision by making this value smaller.
+
+5. ``packet_sizes`` - This specifies the range of packets size this test is run for.
+
+6. ``duration`` - This specifies the sample duration that the test uses to check for success or failure.
+
+7. ``lower_bound`` - This specifies the test initial lower bound sample rate. On success this value is increased.
+
+8. ``upper_bound`` - This specifies the test initial upper bound sample rate. On success this value is decreased.
+
+Other traffic profiles exist eg prox_ACL.yaml which does not
+compare what is received with what is transmitted. It just
+sends packet at max rate.
+
+It is possible to create custom traffic profiles with by
+creating new file in the same folder as prox_binsearch.yaml.
+See this prox_vpe.yaml as example::
+
+     schema: ``nsb:traffic_profile:0.1``
+
+     name:            prox_vpe
+     description:     Prox vPE traffic profile
+
+     traffic_profile:
+       traffic_type: ProxBinSearchProfile
+       tolerated_loss: 100.0 #0.001
+       test_precision: 0.01
+     # The minimum size of the Ethernet frame for the vPE test is 68 bytes.
+       packet_sizes: [68]
+       duration: 5
+       lower_bound: 0.0
+       upper_bound: 100.0
+
+*Test Description File for Openstack*
+-------------------------------------
+
+We will use ``tc_prox_heat_context_l2fwd-2.yaml`` as a example to show
+you how to understand the test description file.
+
+.. image:: images/PROX_Test_HEAT_Script.png
+   :width: 800px
+   :alt: NSB PROX Test Description File
+
+Now lets examine the components of the file in detail
+
+Sections 1 to 8 are exactly the same in Baremetal and in Heat. Section
+``9`` is replaced with sections A to F. Section 9 was for a baremetal
+configuration file. This has no place in a heat configuration.
+
+A. ``image`` - yardstick-samplevnfs. This is the name of the image
+   created during the installation of NSB. This is fixed.
+
+B. ``flavor`` - The flavor is created dynamically. However we could
+   use an already existing flavor if required. In that case the
+   flavor would be named::
+
+    flavor: yardstick-flavor
+
+C. ``extra_specs`` - This allows us to specify the number of
+   cores sockets and hyperthreading assigned to it. In this case
+   we have 1 socket with 10 codes and no hyperthreading enabled.
+
+D. ``placement_groups`` - default. Do not change for NSB PROX.
+
+E. ``servers`` - ``tg_0`` is the traffic generator and ``vnf_0``
+   is the system under test.
+
+F. ``networks`` - is composed of a management network labeled ``mgmt``
+   and one uplink network labeled ``uplink_0``  and one downlink
+   network labeled ``downlink_0`` for 2 ports. If this was a 4 port
+   configuration there would be 2 extra downlink ports. See this
+   example from a 4 port l2fwd test.::
+
+    networks:
+      mgmt:
+        cidr: '10.0.1.0/24'
+      uplink_0:
+        cidr: '10.0.2.0/24'
+        gateway_ip: 'null'
+        port_security_enabled: False
+        enable_dhcp: 'false'
+      downlink_0:
+        cidr: '10.0.3.0/24'
+        gateway_ip: 'null'
+        port_security_enabled: False
+        enable_dhcp: 'false'
+      downlink_1:
+        cidr: '10.0.4.0/24'
+        gateway_ip: 'null'
+        port_security_enabled: False
+        enable_dhcp: 'false'
+      downlink_2:
+        cidr: '10.0.5.0/24'
+        gateway_ip: 'null'
+        port_security_enabled: False
+        enable_dhcp: 'false'
+
+.. _nsb-traffic-generator-label:
+
+*Traffic Generator Config file*
+-------------------------------
+
+This section will describe the traffic generator config file.
+This is the same for both baremetal and heat. See this example
+of ``gen_l2fwd_multiflow-2.cfg`` to explain the options.
+
+.. image:: images/PROX_Gen_2port_cfg.png
+   :width: 1400px
+   :alt: NSB PROX Gen Config File
+
+The configuration file is divided into multiple sections, each
+of which is used to define some parameters and options.::
+
+  [eal options]
+  [variables]
+  [port 0]
+  [port 1]
+  [port .]
+  [port Z]
+  [defaults]
+  [global]
+  [core 0]
+  [core 1]
+  [core 2]
+  [core .]
+  [core Z]
+
+See `prox options`_ for details
+
+Now let's examine the components of the file in detail
+
+1. ``[eal options]`` - This specified the EAL (Environmental
+   Abstraction Layer) options. These are default values and
+   are not changed. See `dpdk wiki page`_.
+
+2. ``[variables]`` - This section contains variables, as
+   the name suggests. Variables for Core numbers, mac
+   addresses, ip addresses etc. They are assigned as a
+   ``key = value`` where the key is used in place of the value.
+
+   .. caution::
+     A special case for valuables with a value beginning with
+     ``@@``. These values are dynamically updated by the NSB
+     application at run time. Values like MAC address,
+     IP Address etc.
+
+3. ``[port 0]`` - This section describes the DPDK Port. The number
+   following the keyword ``port`` usually refers to the DPDK Port
+   Id. usually starting from ``0``. Because you can have multiple
+   ports this entry usually repeated. Eg. For a 2 port setup
+   ``[port0]`` and ``[port 1]`` and for a 4 port setup ``[port 0]``,
+   ``[port 1]``, ``[port 2]`` and ``[port 3]``::
+
+      [port 0]
+      name=p0
+      mac=hardware
+      rx desc=2048
+      tx desc=2048
+      promiscuous=yes
+
+   a. In this example ``name = p0`` assigned the name ``p0`` to the
+      port. Any name can be assigned to a port.
+   b. ``mac=hardware`` sets the MAC address assigned by the hardware
+      to data from this port.
+   c. ``rx desc=2048`` sets the number of available descriptors to
+      allocate for receive packets. This can be changed and can
+      effect performance.
+   d. ``tx desc=2048`` sets the number of available descriptors to
+      allocate for transmit packets. This can be changed and can
+      effect performance.
+   e. ``promiscuous=yes`` this enables promiscuous mode for this port.
+
+4. ``[defaults]`` - Here default operations and settings can be over
+   written. In this example ``mempool size=4K`` the number of mbufs
+   per task is altered. Altering this value could effect
+   performance. See `prox options`_ for details.
+
+5. ``[global]`` - Here application wide setting are supported. Things
+   like application name, start time, duration and memory
+   configurations can be set here. In this example.::
+
+      [global]
+      start time=5
+      name=Basic Gen
+
+    a. ``start time=5`` Time is seconds after which average
+       stats will be started.
+    b. ``name=Basic Gen`` Name of the configuration.
+
+6. ``[core 0]`` - This core is designated the master core. Every
+   Prox application must have a master core. The master mode must
+   be assigned to exactly one task, running alone on one core.::
+
+    [core 0]
+    mode=master
+
+7. ``[core 1]`` - This describes the activity on core 1. Cores can
+   be configured by means of a set of [core #] sections, where
+   # represents either:
+
+   a. an absolute core number: e.g. on a 10-core, dual socket
+      system with hyper-threading,
+      cores are numbered from 0 to 39.
+
+   b. PROX allows a core to be identified by a core number, the
+      letter 's', and a socket number.
+
+      It is possible to write a baremetal and an openstack test which use
+      the same traffic generator config file and SUT config file.
+      In this case it is advisable not to use physical
+      core numbering.
+
+      However it is also possible to write NSB Prox tests that
+      have been optimized for a particular hardware configuration.
+      In this case it is advisable to use the core numbering.
+      It is up to the user to make sure that cores from
+      the right sockets are used (i.e. from the socket on which the NIC
+      is attached to), to ensure good performance (EPA).
+
+   Each core can be assigned with a set of tasks, each running
+   one of the implemented packet processing modes.::
+
+     [core 1]
+     name=p0
+     task=0
+     mode=gen
+     tx port=p0
+     bps=1250000000
+     ; Ethernet + IP + UDP
+     pkt inline=${sut_mac0} 70 00 00 00 00 01 08 00 45 00 00 1c 00 01 00 00 40 11 f7 7d 98 10 64 01 98 10 64 02 13 88 13 88 00 08 55 7b
+     ; src_ip: 152.16.100.0/8
+     random=0000XXX1
+     rand_offset=29
+     ; dst_ip: 152.16.100.0/8
+     random=0000XXX0
+     rand_offset=33
+     random=0001001110001XXX0001001110001XXX
+     rand_offset=34
+
+   a. ``name=p0`` - Name assigned to the core.
+   b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``.
+      Task 1 can be defined later in this core or
+      can be defined in another ``[core 1]`` section with ``task=1``
+      later in configuration file. Sometimes running
+      multiple task related to the same packet on the same physical
+      core improves performance, however sometimes it
+      is optimal to move task to a separate core. This is best
+      decided by checking performance.
+   c. ``mode=gen`` - Specifies the action carried out by this task on
+      this core. Supported modes are: classify, drop, gen, lat, genl4, nop, l2fwd, gredecap,
+      greencap, lbpos, lbnetwork, lbqinq, lb5tuple, ipv6_decap, ipv6_encap,
+      qinqdecapv4, qinqencapv4, qos, routing, impair,
+      mirror, unmpls, tagmpls, nat, decapnsh, encapnsh, police, acl
+      Which are :-
+
+       * Classify
+       * Drop
+       * Basic Forwarding (no touch)
+       * L2 Forwarding (change MAC)
+       * GRE encap/decap
+       * Load balance based on packet fields
+       * Symmetric load balancing
+       * QinQ encap/decap IPv4/IPv6
+       * ARP
+       * QoS
+       * Routing
+       * Unmpls
+       * Nsh encap/decap
+       * Policing
+       * ACL
+
+      In the traffic generator we expect a core to generate packets (``gen``)
+      and to receive packets & calculate latency (``lat``)
+      This core does ``gen`` . ie it is a traffic generator.
+
+      To understand what each of the modes support please see
+      `prox documentation`_.
+
+   d. ``tx port=p0`` - This specifies that the packets generated are
+      transmitted to port ``p0``
+   e. ``bps=1250000000`` - This indicates Bytes Per Second to
+      generate packets.
+   f. ``; Ethernet + IP + UDP`` - This is a comment. Items starting with
+      ``;`` are ignored.
+   g. ``pkt inline=${sut_mac0} 70 00 00 00 ...`` - Defines the packet
+      format as a sequence of bytes (each
+      expressed in hexadecimal notation). This defines the packet
+      that is generated. This packets begins
+      with the hexadecimal sequence assigned to ``sut_mac`` and the
+      remainder of the bytes in the string.
+      This packet could now be sent or modified by ``random=..``
+      described below before being sent to target.
+   h. ``; src_ip: 152.16.100.0/8`` - Comment
+   i. ``random=0000XXX1`` - This describes a field of the packet
+      containing random data. This string can be
+      8,16,24 or 32 character long and represents 1,2,3 or 4
+      bytes of data. In this case it describes a byte of
+      data. Each character in string can be 0,1 or ``X``. 0 or 1
+      are fixed bit values in the data packet and ``X`` is a
+      random bit. So random=0000XXX1 generates 00000001(1),
+      00000011(3), 00000101(5), 00000111(7),
+      00001001(9), 00001011(11), 00001101(13) and 00001111(15)
+      combinations.
+   j. ``rand_offset=29`` - Defines where to place the previously
+      defined random field.
+   k. ``; dst_ip: 152.16.100.0/8`` - Comment
+   l. ``random=0000XXX0`` - This is another random field which
+      generates a byte of 00000000(0), 00000010(2),
+      00000100(4), 00000110(6), 00001000(8), 00001010(10),
+      00001100(12) and 00001110(14) combinations.
+   m. ``rand_offset=33`` - Defines where to place the previously
+      defined random field.
+   n. ``random=0001001110001XXX0001001110001XXX`` - This is
+      another random field which generates 4 bytes.
+   o. ``rand_offset=34`` - Defines where to place the previously
+      defined 4 byte random field.
+
+   Core 2 executes same scenario as Core 1. The only difference
+   in this case is that the packets are generated
+   for Port 1.
+
+8. ``[core 3]`` - This defines the activities on core 3. The purpose
+   of ``core 3`` and ``core 4`` is to receive packets
+   sent by the SUT.::
+
+     [core 3]
+     name=rec 0
+     task=0
+     mode=lat
+     rx port=p0
+     lat pos=42
+
+   a. ``name=rec 0`` - Name assigned to the core.
+   b. ``task=0`` - Each core can run a set of tasks. Starting with
+      ``0``. Task 1 can be defined later in this core or
+      can be defined in another ``[core 1]`` section with
+      ``task=1`` later in configuration file. Sometimes running
+      multiple task related to the same packet on the same
+      physical core improves performance, however sometimes it
+      is optimal to move task to a separate core. This is
+      best decided by checking performance.
+   c. ``mode=lat`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
+      classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
+      police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
+      nat, decapnsh, encapnsh, gen, genl4 and lat. This task(0) per core(3) receives packets on port.
+   d. ``rx port=p0`` - The port to receive packets on ``Port 0``. Core 4 will receive packets on ``Port 1``.
+   e. ``lat pos=42`` - Describes where to put a 4-byte timestamp in the packet. Note that the packet length should
+      be longer than ``lat pos`` + 4 bytes to avoid truncation of the timestamp. It defines where the timestamp is
+      to be read from. Note that the SUT workload might cause the position of the timestamp to change
+      (i.e. due to encapsulation).
+
+.. _nsb-sut-generator-label:
+
+*SUT Config file*
+-------------------------------
+
+This section will describes the SUT(VNF) config file. This is the same for both
+baremetal and heat. See this example of ``handle_l2fwd_multiflow-2.cfg`` to explain the options.
+
+.. image:: images/PROX_Handle_2port_cfg.png
+   :width: 1400px
+   :alt: NSB PROX Handle Config File
+
+See `prox options`_ for details
+
+Now let's examine the components of the file in detail
+
+1. ``[eal options]`` - same as the Generator config file. This specified the EAL (Environmental Abstraction Layer)
+   options. These are default values and are not changed.
+   See `dpdk wiki page`_.
+
+2. ``[port 0]`` - This section describes the DPDK Port. The number following the keyword ``port`` usually refers to the DPDK Port Id. usually starting from ``0``.
+   Because you can have multiple ports this entry usually repeated. Eg. For a 2 port setup ``[port0]`` and ``[port 1]`` and for a 4 port setup ``[port 0]``, ``[port 1]``,
+   ``[port 2]`` and ``[port 3]``::
+
+      [port 0]
+      name=if0
+      mac=hardware
+      rx desc=2048
+      tx desc=2048
+      promiscuous=yes
+
+   a. In this example ``name =if0`` assigned the name ``if0`` to the port. Any name can be assigned to a port.
+   b. ``mac=hardware`` sets the MAC address assigned by the hardware to data from this port.
+   c. ``rx desc=2048`` sets the number of available descriptors to allocate for receive packets. This can be changed and can effect performance.
+   d. ``tx desc=2048`` sets the number of available descriptors to allocate for transmit packets. This can be changed and can effect performance.
+   e. ``promiscuous=yes`` this enables promiscuous mode for this port.
+
+3. ``[defaults]`` - Here default operations and settings can be over written.::
+
+     [defaults]
+     mempool size=8K
+     memcache size=512
+
+   a. In this example ``mempool size=8K`` the number of mbufs per task is altered. Altering this value could effect performance. See `prox options`_ for details.
+   b. ``memcache size=512`` - number of mbufs cached per core, default is 256 this is the cache_size. Altering this value could effect performance.
+
+4. ``[global]`` - Here application wide setting are supported. Things like application name, start time, duration and memory configurations can be set here.
+   In this example.::
+
+      [global]
+      start time=5
+      name=Basic Gen
+
+    a. ``start time=5`` Time is seconds after which average stats will be started.
+    b. ``name=Handle L2FWD Multiflow (2x)`` Name of the configuration.
+
+5. ``[core 0]`` - This core is designated the master core. Every Prox application must have a master core. The master mode must be assigned to
+   exactly one task, running alone on one core.::
+
+    [core 0]
+    mode=master
+
+6. ``[core 1]`` - This describes the activity on core 1. Cores can be configured by means of a set of [core #] sections,   where # represents either:
+
+   a. an absolute core number: e.g. on a 10-core, dual socket system with hyper-threading,
+      cores are numbered from 0 to 39.
+
+   b. PROX allows a core to be identified by a core number, the letter 's', and a socket number.
+      However NSB PROX is hardware agnostic (physical and virtual configurations are the same) it
+      is advisable no to use physical core numbering.
+
+   Each core can be assigned with a set of tasks, each running one of the implemented packet processing modes.::
+
+     [core 1]
+     name=none
+     task=0
+     mode=l2fwd
+     dst mac=@@tester_mac1
+     rx port=if0
+     tx port=if1
+
+   a. ``name=none`` - No name assigned to the core.
+   b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``. Task 1 can be defined later in this core or
+      can be defined in another ``[core 1]`` section with ``task=1`` later in configuration file. Sometimes running
+      multiple task related to the same packet on the same physical core improves performance, however sometimes it
+      is optimal to move task to a separate core. This is best decided by checking performance.
+   c. ``mode=l2fwd`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
+      classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
+      police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
+      nat, decapnsh, encapnsh, gen, genl4 and lat. This code does ``l2fwd`` .. ie it does the L2FWD.
+
+   d. ``dst mac=@@tester_mac1`` - The destination mac address of the packet will be set to the MAC address of ``Port 1`` of destination device. (The Traffic Generator/Verifier)
+   e. ``rx port=if0`` - This specifies that the packets are received from ``Port 0`` called if0
+   f. ``tx port=if1`` - This specifies that the packets are transmitted to ``Port 1``  called if1
+
+   If this example we receive a packet on core on a port, carry out operation on the packet on the core and transmit it on on another port still using the same task on the same core.
+
+   On some implementation you may wish to use multiple tasks, like this.::
+
+     [core 1]
+     name=rx_task
+     task=0
+     mode=l2fwd
+     dst mac=@@tester_p0
+     rx port=if0
+     tx cores=1t1
+     drop=no
+
+     name=l2fwd_if0
+     task=1
+     mode=nop
+     rx ring=yes
+     tx port=if0
+     drop=no
+
+   In this example you can see Core 1/Task 0 called ``rx_task`` receives the packet from if0 and perform the l2fwd. However instead of sending the packet to a
+   port it sends it to a core see ``tx cores=1t1``. In this case it sends it to Core 1/Task 1.
+
+   Core 1/Task 1 called ``l2fwd_if0``, receives the packet, not from a port but from the ring. See ``rx ring=yes``. It does not perform any operation on the packet See ``mode=none``
+   and sends the packets to ``if0`` see ``tx port=if0``.
+
+   It is also possible to implement more complex operations be chaining multiple operations in sequence and using rings to pass packets from one core to another.
+
+   In thus example we show a Broadband Network Gateway (BNG) with Quality of Service (QoS).  Communication from task to task is via rings.
+
+   .. image:: images/PROX_BNG_QOS.png
+      :width: 1000px
+      :alt: NSB PROX Config File for BNG_QOS
+
+*Baremetal Configuration file*
+------------------------------
+
+.. _baremetal-config-label:
+
+This is required for baremetal testing. It describes the IP address of the various ports, the Network devices drivers and MAC addresses and the network
+configuration.
+
+In this example we will describe a 2 port configuration. This file is the same for all 2 port NSB Prox tests on the same platforms/configuration.
+
+  .. image:: images/PROX_Baremetal_config.png
+     :width: 1000px
+     :alt: NSB PROX Yardstick Config
+
+Now lets describe the sections of the file.
+
+  1. ``TrafficGen`` - This section describes the Traffic Generator node of the test configuration. The name of the node ``trafficgen_1`` must match the node name
+     in the ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
+     can remain as default settings.
+  2. ``interfaces`` - This defines the DPDK interfaces on the Traffic Generator.
+  3. ``xe0`` is DPDK Port 0. ``lspci`` and `` ./dpdk-devbind.py -s`` can be used to provide the interface information. ``netmask`` and ``local_ip`` should not be changed
+  4. ``xe1`` is DPDK Port 1. If more than 2 ports are required then ``xe1`` section needs to be repeated and modified accordingly.
+  5. ``vnf`` - This section describes the SUT of the test configuration. The name of the node ``vnf`` must match the node name in the
+     ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
+     can remain as default settings
+  6. ``interfaces`` - This defines the DPDK interfaces on the SUT
+  7. ``xe0`` - Same as 3 but for the ``SUT``.
+  8. ``xe1`` - Same as 4 but for the ``SUT`` also.
+  9. ``routing_table`` - All parameters should remain unchanged.
+  10. ``nd_route_tbl`` - All parameters should remain unchanged.
+
+*Grafana Dashboard*
+-------------------
+
+The grafana dashboard visually displays the results of the tests. The steps required to produce a grafana dashboard are described here.
+
+.. _yardstick-config-label:
+
+  a. Configure ``yardstick`` to use influxDB to store test results. See file ``/etc/yardstick/yardstick.conf``.
+
+     .. image:: images/PROX_Yardstick_config.png
+        :width: 1000px
+        :alt: NSB PROX Yardstick Config
+
+     1. Specify the dispatcher to use influxDB to store results.
+     2. "target = .. " - Specify location of influxDB to store results.
+        "db_name = yardstick" - name of database. Do not change
+        "username = root" - username to use to store result. (Many tests are run as root)
+        "password = ... " - Please set to root user password
+
+  b. Deploy InfludDB & Grafana. See how to Deploy InfluxDB & Grafana. See `grafana deployment`_.
+  c. Generate the test data. Run the tests as follows .::
+
+       yardstick --debug task start tc_prox_<context>_<test>-ports.yaml
+
+     eg.::
+
+       yardstick --debug task start tc_prox_heat_context_l2fwd-4.yaml
+
+  d. Now build the dashboard for the test you just ran. The easiest way to do this is to copy an existing dashboard and rename the
+     test and the field names. The procedure to do so is described here. See `opnfv grafana dashboard`_.
+
+How to run NSB Prox Test on an baremetal environment
+====================================================
+
+In order to run the NSB PROX test.
+
+  1. Install NSB on Traffic Generator node and Prox in SUT. See `NSB Installation`_
+
+  2. To enter container::
+
+       docker exec -it yardstick /bin/bash
+
+  3. Install baremetal configuration file (POD files)
+
+     a. Go to location of PROX tests in container ::
+
+          cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox
+
+     b. Install prox-baremetal-2.yam and prox-baremetal-4.yaml for that topology
+        into this directory as per baremetal-config-label_
+
+     c. Install and configure ``yardstick.conf`` ::
+
+            cd /etc/yardstick/
+
+        Modify /etc/yardstick/yardstick.conf as per yardstick-config-label_
+
+  4. Execute the test. Eg.::
+
+        yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml
+
+How to run NSB Prox Test on an Openstack environment
+====================================================
+
+In order to run the NSB PROX test.
+
+  1. Install NSB on Openstack deployment node. See  `NSB Installation`_
+
+  2. To enter container::
+
+       docker exec -it yardstick /bin/bash
+
+  3. Install configuration file
+
+     a. Goto location of PROX tests in container ::
+
+          cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox
+
+     b. Install and configure ``yardstick.conf`` ::
+
+            cd /etc/yardstick/
+
+        Modify /etc/yardstick/yardstick.conf as per yardstick-config-label_
+
+
+  4. Execute the test. Eg.::
+
+        yardstick --debug task start ./tc_prox_heat_context_l2fwd-4.yaml
+
+Frequently Asked Questions
+==========================
+
+Here is a list of frequently asked questions.
+
+*NSB Prox does not work on Baremetal, How do I resolve this?*
+-------------------------------------------------------------
+
+If PROX NSB does not work on baremetal, problem is either in network configuration or test file.
+
+*Solution*
+
+1. Verify network configuration. Execute existing baremetal test.::
+
+       yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml
+
+   If test does not work then error in network configuration.
+
+      a. Check DPDK on Traffic Generator and SUT via:- ::
+
+           /root/dpdk-17./usertools/dpdk-devbind.py
+
+      b. Verify MAC addresses match ``prox-baremetal-<ports>.yaml`` via ``ifconfig`` and ``dpdk-devbind``
+
+      c. Check your eth port is what you expect. You would not be the first person to think that
+         the port your cable is plugged into is ethX when in fact it is ethY. Use
+         ethtool to visually confirm that the eth is where you expect.::
+
+            ethtool -p ethX
+
+         A led should start blinking on port. (On both System-Under-Test and Traffic Generator)
+
+      d. Check cable.
+
+         Install Linux kernel network driver and ensure your ports are
+         ``bound`` to the driver via ``dpdk-devbind``. Bring up port on both
+         SUT and Traffic Generator and check connection.
+
+         i) On SUT and on Traffic Generator::
+
+              ifconfig ethX/enoX up
+
+         ii) Check link
+
+               ethtool ethX/enoX
+
+             See ``Link detected`` if ``yes`` .... Cable is good. If ``no`` you have an issue with your cable/port.
+
+2. If existing baremetal works then issue is with your test. Check the traffic generator gen_<test>-<ports>.cfg to ensure
+   it is producing a valid packet.
+
+*How do I debug NSB Prox on Baremetal?*
+---------------------------------------
+
+*Solution*
+
+1. Execute the test as follows::
+
+     yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml
+
+2. Login to Traffic Generator as ``root``.::
+
+     cd
+     /opt/nsb_bin/prox -f /tmp/gen_<test>-<ports>.cfg
+
+3. Login to SUT as ``root``.::
+
+     cd
+     /opt/nsb_bin/prox -f /tmp/handle_<test>-<ports>.cfg
+
+4. Now let's examine the Generator Output. In this case the output of gen_l2fwd-4.cfg.
+
+     .. image:: images/PROX_Gen_GUI.png
+        :width: 1000px
+        :alt: NSB PROX Traffic Generator GUI
+
+   Now let's examine the output
+
+     1. Indicates the amount of data successfully transmitted on Port 0
+     2. Indicates the amount of data successfully received on port 1
+     3. Indicates the amount of data successfully handled for port 1
+
+   It appears what is transmitted is received.
+
+   .. Caution::
+      The number of packets MAY not exactly match because the ports are read in sequence.
+
+   .. Caution::
+      What is transmitted on PORT X may not always be received on same port. Please check the Test scenario.
+
+5. Now lets examine the SUT Output
+
+     .. image:: images/PROX_SUT_GUI.png
+        :width: 1400px
+        :alt: NSB PROX SUT GUI
+
+   Now lets examine the output
+
+     1. What is received on 0 is transmitted on 1, received on 1 transmitted on 0,
+        received on 2 transmitted on 3 and received on 3 transmitted on 2.
+     2. No packets are Failed.
+     3. No Packets are discarded.
+
+  We can also dump the packets being received or transmitted via the following commands. ::
+
+       dump                   Arguments: <core id> <task id> <nb packets>
+                              Create a hex dump of <nb_packets> from <task_id> on <core_id> showing how
+                              packets have changed between RX and TX.
+       dump_rx                Arguments: <core id> <task id> <nb packets>
+                              Create a hex dump of <nb_packets> from <task_id> on <core_id> at RX
+       dump_tx                Arguments: <core id> <task id> <nb packets>
+                              Create a hex dump of <nb_packets> from <task_id> on <core_id> at TX
+
+  eg.::
+
+       dump_tx 1 0 1
+
+*NSB Prox works on Baremetal but not in Openstack. How do I resolve this?*
+--------------------------------------------------------------------------
+
+NSB Prox on Baremetal is a lot more forgiving than NSB Prox on Openstack. A badly
+formed packed may still work with PROX on Baremetal. However on
+Openstack the packet must be correct and all fields of the header correct.
+Eg A packet with an invalid Protocol ID would still work in Baremetal
+but this packet would be rejected by openstack.
+
+*Solution*
+
+ 1. Check the validity of the packet.
+ 2. Use a known good packet in your test
+ 3. If using ``Random`` fields in the traffic generator, disable them and retry.
+
+
+*How do I debug NSB Prox on Openstack?*
+---------------------------------------
+
+*Solution*
+
+1. Execute the test as follows::
+
+     yardstick --debug task start --keep-deploy ./tc_prox_heat_context_l2fwd-4.yaml
+
+2. Access docker image if required via::
+
+      docker exec -it yardstick /bin/bash
+
+3. Install openstack credentials.
+
+   Depending on your openstack deployment, the location of these credentials may vary.
+   On this platform I do this via::
+
+     scp root@10.237.222.55:/etc/kolla/admin-openrc.sh .
+     source ./admin-openrc.sh
+
+4. List Stack details
+
+   a. Get the name of the Stack.
+
+         .. image:: images/PROX_Openstack_stack_list.png
+            :width: 1000px
+            :alt: NSB PROX openstack stack list
+
+   b. Get the Floating IP of the Traffic Generator & SUT
+
+      This generates a lot of information. Please not the floating IP of the VNF and
+      the Traffic Generator.
+
+         .. image:: images/PROX_Openstack_stack_show_a.png
+            :width: 1000px
+            :alt: NSB PROX openstack stack show (Top)
+
+      From here you can see the floating IP Address of the SUT / VNF
+
+         .. image:: images/PROX_Openstack_stack_show_b.png
+            :width: 1000px
+            :alt: NSB PROX openstack stack show (Top)
+
+      From here you can see the floating IP Address of the Traffic Generator
+
+   c. Get ssh identity file
+
+      In the docker container locate the identity file.::
+
+        cd /home/opnfv/repos/yardstick/yardstick/resources/files
+        ls -lt
+
+5. Login to SUT as ``Ubuntu``.::
+
+     ssh -i ./yardstick_key-01029d1d ubuntu@172.16.2.158
+
+   Change to root::
+
+     sudo su
+
+    Now continue as baremetal.
+
+6. Login to SUT as ``Ubuntu``.::
+
+     ssh -i ./yardstick_key-01029d1d ubuntu@172.16.2.156
+
+   Change to root::
+
+     sudo su
+
+    Now continue as baremetal.
+
+*How do I resolve "Quota exceeded for resources"*
+-------------------------------------------------
+
+*Solution*
+
+This usually occurs due to 2 reasons when executing an openstack test.
+
+1. One or more stacks already exists and are consuming all resources. To resolve ::
+
+     openstack stack list
+
+   Response::
+
+     +--------------------------------------+--------------------+-----------------+----------------------+--------------+
+     | ID                                   | Stack Name         | Stack Status    | Creation Time        | Updated Time |
+     +--------------------------------------+--------------------+-----------------+----------------------+--------------+
+     | acb559d7-f575-4266-a2d4-67290b556f15 | yardstick-e05ba5a4 | CREATE_COMPLETE | 2017-12-06T15:00:05Z | None         |
+     | 7edf21ce-8824-4c86-8edb-f7e23801a01b | yardstick-08bda9e3 | CREATE_COMPLETE | 2017-12-06T14:56:43Z | None         |
+     +--------------------------------------+--------------------+-----------------+----------------------+--------------+
+
+   In this case 2 stacks already exist.
+
+   To remove stack::
+
+     openstack stack delete yardstick-08bda9e3
+     Are you sure you want to delete this stack(s) [y/N]? y
+
+2. The openstack configuration quotas are too small.
+
+   The solution is to increase the quota. Use below to query existing quotas::
+
+     openstack quota show
+
+   And to set quota::
+
+     openstack quota set <resource>
+
+*Openstack Cli fails or hangs. How do I resolve this?*
+------------------------------------------------------
+
+*Solution*
+
+If it fails due to ::
+
+   Missing value auth-url required for auth plugin password
+
+Check your shell environment for Openstack variables. One of them should contain the authentication URL ::
+
+
+   OS_AUTH_URL=``https://192.168.72.41:5000/v3``
+
+Or similar. Ensure that openstack configurations are exported. ::
+
+   cat  /etc/kolla/admin-openrc.sh
+
+Result ::
+
+   export OS_PROJECT_DOMAIN_NAME=default
+   export OS_USER_DOMAIN_NAME=default
+   export OS_PROJECT_NAME=admin
+   export OS_TENANT_NAME=admin
+   export OS_USERNAME=admin
+   export OS_PASSWORD=BwwSEZqmUJA676klr9wa052PFjNkz99tOccS9sTc
+   export OS_AUTH_URL=http://193.168.72.41:35357/v3
+   export OS_INTERFACE=internal
+   export OS_IDENTITY_API_VERSION=3
+   export EXTERNAL_NETWORK=yardstick-public
+
+and visible.
+
+If the Openstack Cli appears to hang, then verify the proxys and no_proxy are set correctly.
+They should be similar to ::
+
+   FTP_PROXY="http://proxy.ir.intel.com:911/"
+   HTTPS_PROXY="http://proxy.ir.intel.com:911/"
+   HTTP_PROXY="http://proxy.ir.intel.com:911/"
+   NO_PROXY="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"
+   ftp_proxy="http://proxy.ir.intel.com:911/"
+   http_proxy="http://proxy.ir.intel.com:911/"
+   https_proxy="http://proxy.ir.intel.com:911/"
+   no_proxy="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"
+
+Where
+
+    1) 10.237.222.55 = IP Address of deployment node
+    2) 10.237.223.80 = IP Address of Controller node
+    3) 10.237.222.134 = IP Address of Compute Node
+    4) ir.intel.com = local no proxy
+
+
+
+
+
+