summaryrefslogtreecommitdiffstats
path: root/docs/configguide
diff options
context:
space:
mode:
Diffstat (limited to 'docs/configguide')
-rw-r--r--docs/configguide/yardstick_testcases/01-introduction.rst38
-rw-r--r--docs/configguide/yardstick_testcases/02-methodology.rst181
-rw-r--r--docs/configguide/yardstick_testcases/03-list-of-tcs.rst72
-rw-r--r--docs/configguide/yardstick_testcases/04-vtc-overview.rst114
-rwxr-xr-xdocs/configguide/yardstick_testcases/Yardstick_task_templates.rst155
-rw-r--r--docs/configguide/yardstick_testcases/glossary.rst33
-rw-r--r--docs/configguide/yardstick_testcases/index.rst12
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst79
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst78
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst72
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst139
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst157
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst85
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst84
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst77
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst81
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst69
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst136
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst152
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst67
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst99
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst99
-rw-r--r--docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst60
-rw-r--r--docs/configguide/yardstick_testcases/testcase_description_v2_template.rst64
24 files changed, 2203 insertions, 0 deletions
diff --git a/docs/configguide/yardstick_testcases/01-introduction.rst b/docs/configguide/yardstick_testcases/01-introduction.rst
new file mode 100644
index 000000000..6cca2875e
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/01-introduction.rst
@@ -0,0 +1,38 @@
+============
+Introduction
+============
+
+**Welcome to Yardstick's documentation !**
+
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+
+Yardstick_ is an OPNFV Project.
+
+The project's goal is to verify infrastructure compliance, from the perspective
+of a :term:`VNF`.
+
+The Project's scope is the development of a test framework, *Yardstick*, test
+cases and test stimuli to enable :term:`NFVI` verification.
+The Project also includes a sample :term:`VNF`, the :term:`VTC` and its
+experimental framework, *ApexLake* !
+
+The chapter :doc:`02-methodology` describes the methodology implemented by the
+Yardstick Project for :term:`NFVI` verification. The chapter
+:doc:`03-list-of-tcs` includes a list of available Yardstick test cases.
+
+Yardstick is used for verifying the OPNFV infrastructure and some of the OPNFV
+features, listed in :doc:`03-list-of-tcs`.
+
+The *Yardstick* framework is deployed in several OPNFV community labs. It is
+installer, infrastructure and application independent.
+
+.. _Pharos: https://wiki.opnfv.org/pharos
+
+.. seealso:: Pharos_ for information on OPNFV community labs.
+
+Contact Yardstick
+=================
+
+Feedback? `Contact us`_
+
+.. _Contact us: opnfv-users@lists.opnfv.org
diff --git a/docs/configguide/yardstick_testcases/02-methodology.rst b/docs/configguide/yardstick_testcases/02-methodology.rst
new file mode 100644
index 000000000..5097c566b
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/02-methodology.rst
@@ -0,0 +1,181 @@
+===========
+Methodology
+===========
+
+Abstract
+========
+
+This chapter describes the methodology implemented by the Yardstick project for
+verifying the NFV Infrastructure from the perspective of a VNF.
+
+ETSI-NFV
+========
+
+.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/
+
+The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation
+of NFV Environments and Services", recommends methods for pre-deployment
+testing of the functional components of an NFV environment.
+
+The Yardstick project implements the methodology described in chapter 6, "Pre-
+deployment validation of NFV infrastructure".
+
+The methodology consists in decomposing the typical VNF work-load performance
+metrics into a number of characteristics/performance vectors, which each can be
+represented by distinct test-cases.
+
+The methodology includes five steps:
+
+* *Step1:* Define Infrastruture - the HW, SW and corresponding configuration
+ target for validation; the OPNFV infrastructure, in OPNFV community labs.
+
+* *Step2:* Identify VNF type - the application for which the infrastructure is
+ to be validated, and its requirements on the underlying infrastructure.
+
+* *Step3:* Select test cases - depending on the workload that represents the
+ application for which the infrastruture is to be validated, the relevant
+ test cases amongst the list of available Yardstick test cases.
+
+* *Step4:* Execute tests - define the duration and number of iterations for the
+ selected test cases, tests runs are automated via OPNFV Jenkins Jobs.
+
+* *Step5:* Collect results - using the common API for result collection.
+
+Metrics
+=======
+
+The metrics, as defined by ETSI GS NFV-TST001, are shown in
+:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and
+:ref:`Table3 <table2_3>`.
+
+In OPNFV Brahmaputra release, generic test cases covering aspects of the listed
+metrics are available; further OPNFV releases will provide extended testing of
+these metrics.
+The view of available Yardstick test cases cross ETSI definitions in
+:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>`
+is shown in :ref:`Table4 <table2_4>`.
+It shall be noticed that the Yardstick test cases are examples, the test
+duration and number of iterations are configurable, as are the System Under
+Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario
+options).
+
+.. _table2_1:
+
+**Table 1 - Performance/Speed Metrics**
+
++---------+-------------------------------------------------------------------+
+| Category| Performance/Speed |
+| | |
++---------+-------------------------------------------------------------------+
+| Compute | * Latency for random memory access |
+| | * Latency for cache read/write operations |
+| | * Processing speed (instructions per second) |
+| | * Throughput for random memory access (bytes per second) |
+| | |
++---------+-------------------------------------------------------------------+
+| Network | * Throughput per NFVI node (frames/byte per second) |
+| | * Throughput provided to a VM (frames/byte per second) |
+| | * Latency per traffic flow |
+| | * Latency between VMs |
+| | * Latency between NFVI nodes |
+| | * Packet delay variation (jitter) between VMs |
+| | * Packet delay variation (jitter) between NFVI nodes |
+| | |
++---------+-------------------------------------------------------------------+
+| Storage | * Sequential read/write IOPS |
+| | * Random read/write IOPS |
+| | * Latency for storage read/write operations |
+| | * Throughput for storage read/write operations |
+| | |
++---------+-------------------------------------------------------------------+
+
+.. _table2_2:
+
+**Table 2 - Capacity/Scale Metrics**
+
++---------+-------------------------------------------------------------------+
+| Category| Capacity/Scale |
+| | |
++---------+-------------------------------------------------------------------+
+| Compute | * Number of cores and threads- Available memory size |
+| | * Cache size |
+| | * Processor utilization (max, average, standard deviation) |
+| | * Memory utilization (max, average, standard deviation) |
+| | * Cache utilization (max, average, standard deviation) |
+| | |
++---------+-------------------------------------------------------------------+
+| Network | * Number of connections |
+| | * Number of frames sent/received |
+| | * Maximum throughput between VMs (frames/byte per second) |
+| | * Maximum throughput between NFVI nodes (frames/byte per second) |
+| | * Network utilization (max, average, standard deviation) |
+| | * Number of traffic flows |
+| | |
++---------+-------------------------------------------------------------------+
+| Storage | * Storage/Disk size |
+| | * Capacity allocation (block-based, object-based) |
+| | * Block size |
+| | * Maximum sequential read/write IOPS |
+| | * Maximum random read/write IOPS |
+| | * Disk utilization (max, average, standard deviation) |
+| | |
++---------+-------------------------------------------------------------------+
+
+.. _table2_3:
+
+**Table 3 - Availability/Reliability Metrics**
+
++---------+-------------------------------------------------------------------+
+| Category| Availability/Reliability |
+| | |
++---------+-------------------------------------------------------------------+
+| Compute | * Processor availability (Error free processing time) |
+| | * Memory availability (Error free memory time) |
+| | * Processor mean-time-to-failure |
+| | * Memory mean-time-to-failure |
+| | * Number of processing faults per second |
+| | |
++---------+-------------------------------------------------------------------+
+| Network | * NIC availability (Error free connection time) |
+| | * Link availability (Error free transmission time) |
+| | * NIC mean-time-to-failure |
+| | * Network timeout duration due to link failure |
+| | * Frame loss rate |
+| | |
++---------+-------------------------------------------------------------------+
+| Storage | * Disk availability (Error free disk access time) |
+| | * Disk mean-time-to-failure |
+| | * Number of failed storage read/write operations per second |
+| | |
++---------+-------------------------------------------------------------------+
+
+.. _table2_4:
+
+**Table 4 - Yardstick Generic Test Cases**
+
++---------+-------------------+----------------+------------------------------+
+| Category| Performance/Speed | Capacity/Scale | Availability/Reliability |
+| | | | |
++---------+-------------------+----------------+------------------------------+
+| Compute | TC003 | TC003 | TC013 [1]_ |
+| | TC004 | TC004 | TC015 [1]_ |
+| | TC014 | TC010 | |
+| | TC024 | TC012 | |
+| | | | |
++---------+-------------------+----------------+------------------------------+
+| Network | TC002 | TC001 | TC016 [1]_ |
+| | TC011 | TC008 | TC018 [1]_ |
+| | | TC009 | |
+| | | | |
++---------+-------------------+----------------+------------------------------+
+| Storage | TC005 | TC005 | TC017 [1]_ |
+| | | | |
++---------+-------------------+----------------+------------------------------+
+
+.. note:: The description in this OPNFV document is intended as a reference for
+ users to understand the scope of the Yardstick Project and the
+ deliverables of the Yardstick framework. For complete description of
+ the methodology, refer to the ETSI document.
+
+.. rubric:: Footnotes
+.. [1] To be included in future deliveries.
diff --git a/docs/configguide/yardstick_testcases/03-list-of-tcs.rst b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst
new file mode 100644
index 000000000..f72d80b75
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/03-list-of-tcs.rst
@@ -0,0 +1,72 @@
+====================
+Yardstick Test Cases
+====================
+
+Abstract
+========
+
+This chapter lists available Yardstick test cases.
+Yardstick test cases are divided in two main categories:
+
+* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology
+described in :doc:`02-methodology`
+
+* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more
+aspect of a feature delivered by an OPNFV Project, including the test cases
+developed for the :term:`VTC`.
+
+Generic NFVI Test Case Descriptions
+===================================
+
+.. toctree::
+ :maxdepth: 1
+
+ opnfv_yardstick_tc001.rst
+ opnfv_yardstick_tc002.rst
+ opnfv_yardstick_tc005.rst
+ opnfv_yardstick_tc008.rst
+ opnfv_yardstick_tc009.rst
+ opnfv_yardstick_tc010.rst
+ opnfv_yardstick_tc012.rst
+ opnfv_yardstick_tc014.rst
+ opnfv_yardstick_tc037.rst
+ opnfv_yardstick_tc038.rst
+
+OPNFV Feature Test Cases
+========================
+
+IPv6
+----
+
+.. toctree::
+ :maxdepth: 1
+
+ opnfv_yardstick_tc027.rst
+
+Parser
+------
+
+.. toctree::
+ :maxdepth: 1
+
+ opnfv_yardstick_tc040.rst
+
+virtual Traffic Classifier
+--------------------------
+
+.. toctree::
+ :maxdepth: 1
+
+ opnfv_yardstick_tc006.rst
+ opnfv_yardstick_tc007.rst
+ opnfv_yardstick_tc020.rst
+ opnfv_yardstick_tc021.rst
+
+Templates
+=========
+
+.. toctree::
+ :maxdepth: 1
+
+ testcase_description_v2_template
+ Yardstick_task_templates
diff --git a/docs/configguide/yardstick_testcases/04-vtc-overview.rst b/docs/configguide/yardstick_testcases/04-vtc-overview.rst
new file mode 100644
index 000000000..95159a9bc
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/04-vtc-overview.rst
@@ -0,0 +1,114 @@
+==========================
+Virtual Traffic Classifier
+==========================
+
+Abstract
+========
+
+.. _TNOVA: http://www.t-nova.eu/
+.. _TNOVAresults: http://www.t-nova.eu/results/
+.. _Yardstick: https://wiki.opnfv.org/yardstick
+
+This chapter provides an overview of the virtual Traffic Classifier, a
+contribution to OPNFV Yardstick_ from the EU Project TNOVA_.
+Additional documentation is available in TNOVAresults_.
+
+Overview
+========
+
+The virtual Traffic Classifier :term:`VNF`, the :term:`VTC`, comprises of a
+:term:`VNFC`. The :term:`VNFC` contains both the Traffic Inspection module, and
+the Traffic forwarding module, needed to run the VNF. The exploitation of
+:term:`DPI` methods for traffic classification is built around two basic
+assumptions:
+
+* third parties unaffiliated with either source or recipient are able to
+inspect each IP packet’s payload
+
+* the classifier knows the relevant syntax of each application’s packet
+payloads (protocol signatures, data patterns, etc.).
+
+The proposed :term:`DPI` based approach will only use an indicative, small
+number of the initial packets from each flow in order to identify the content
+and not inspect each packet.
+
+In this respect it follows the :term:`PBFS`. This method uses a table to track
+each session based on the 5-tuples (src address, dest address, src port,dest
+port, transport protocol) that is maintained for each flow.
+
+Concepts
+========
+
+* *Traffic Inspection*: The process of packet analysis and application
+identification of network traffic that passes through the :term:`VTC`.
+
+* *Traffic Forwarding*: The process of packet forwarding from an incoming
+network interface to a pre-defined outgoing network interface.
+
+* *Traffic Rule Application*: The process of packet tagging, based on a
+predefined set of rules. Packet tagging may include e.g. :term:`ToS` field
+modification.
+
+Architecture
+============
+
+The Traffic Inspection module is the most computationally intensive component
+of the :term:`VNF`. It implements filtering and packet matching algorithms in
+order to support the enhanced traffic forwarding capability of the :term:`VNF`.
+The component supports a flow table (exploiting hashing algorithms for fast
+indexing of flows) and an inspection engine for traffic classification.
+
+The implementation used for these experiments exploits the nDPI library.
+The packet capturing mechanism is implemented using libpcap. When the
+:term:`DPI` engine identifies a new flow, the flow register is updated with the
+appropriate information and transmitted across the Traffic Forwarding module,
+which then applies any required policy updates.
+
+The Traffic Forwarding moudle is responsible for routing and packet forwarding.
+It accepts incoming network traffic, consults the flow table for classification
+information for each incoming flow and then applies pre-defined policies
+marking e.g. :term:`ToS`/:term:`DSCP` multimedia traffic for :term:`QoS`
+enablement on the forwarded traffic.
+It is assumed that the traffic is forwarded using the default policy until it
+is identified and new policies are enforced.
+
+The expected response delay is considered to be negligible, as only a small
+number of packets are required to identify each flow.
+
+Graphical Overview
+==================
+
+.. code-block:: console
+
+ +----------------------------+
+ | |
+ | Virtual Traffic Classifier |
+ | |
+ | Analysing/Forwarding |
+ | ------------> |
+ | ethA ethB |
+ | |
+ +----------------------------+
+ | ^
+ | |
+ v |
+ +----------------------------+
+ | |
+ | Virtual Switch |
+ | |
+ +----------------------------+
+
+Install
+=======
+
+run the build.sh with root privileges
+
+Run
+===
+
+sudo ./pfbridge -a eth1 -b eth2
+
+Development Environment
+=======================
+
+Ubuntu 14.04
diff --git a/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst
new file mode 100755
index 000000000..d2c2b7ec9
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/Yardstick_task_templates.rst
@@ -0,0 +1,155 @@
+Task Template Syntax
+====================
+
+Basic template syntax
+---------------------
+A nice feature of the input task format used in Yardstick is that it supports
+the template syntax based on Jinja2.
+This turns out to be extremely useful when, say, you have a fixed structure of
+your task but you want to parameterize this task in some way.
+For example, imagine your input task file (task.yaml) runs a set of Ping
+scenarios:
+
+::
+
+ # Sample benchmark task config file
+ # measure network latency using ping
+ schema: "yardstick:task:0.1"
+
+ scenarios:
+ -
+ type: Ping
+ options:
+ packetsize: 200
+ host: athena.demo
+ target: ares.demo
+
+ runner:
+ type: Duration
+ duration: 60
+ interval: 1
+
+ sla:
+ max_rtt: 10
+ action: monitor
+
+ context:
+ ...
+
+Let's say you want to run the same set of scenarios with the same runner/
+context/sla, but you want to try another packetsize to compare the performance.
+The most elegant solution is then to turn the packetsize name into a template
+variable:
+
+::
+
+ # Sample benchmark task config file
+ # measure network latency using ping
+
+ schema: "yardstick:task:0.1"
+ scenarios:
+ -
+ type: Ping
+ options:
+ packetsize: {{packetsize}}
+ host: athena.demo
+ target: ares.demo
+
+ runner:
+ type: Duration
+ duration: 60
+ interval: 1
+
+ sla:
+ max_rtt: 10
+ action: monitor
+
+ context:
+ ...
+
+and then pass the argument value for {{packetsize}} when starting a task with
+this configuration file.
+Yardstick provides you with different ways to do that:
+
+1.Pass the argument values directly in the command-line interface (with either
+a JSON or YAML dictionary):
+
+::
+
+ yardstick task start samples/ping-template.yaml
+ --task-args'{"packetsize":"200"}'
+
+2.Refer to a file that specifies the argument values (JSON/YAML):
+
+::
+
+ yardstick task start samples/ping-template.yaml --task-args-file args.yaml
+
+Using the default values
+------------------------
+Note that the Jinja2 template syntax allows you to set the default values for
+your parameters.
+With default values set, your task file will work even if you don't
+parameterize it explicitly while starting a task.
+The default values should be set using the {% set ... %} clause (task.yaml).
+For example:
+
+::
+
+ # Sample benchmark task config file
+ # measure network latency using ping
+ schema: "yardstick:task:0.1"
+ {% set packetsize = packetsize or "100" %}
+ scenarios:
+ -
+ type: Ping
+ options:
+ packetsize: {{packetsize}}
+ host: athena.demo
+ target: ares.demo
+
+ runner:
+ type: Duration
+ duration: 60
+ interval: 1
+ ...
+
+If you don't pass the value for {{packetsize}} while starting a task, the
+default one will be used.
+
+Advanced templates
+------------------
+
+Yardstick makes it possible to use all the power of Jinja2 template syntax,
+including the mechanism of built-in functions.
+As an example, let us make up a task file that will do a block storage
+performance test.
+The input task file (fio-template.yaml) below uses the Jinja2 for-endfor
+construct to accomplish that:
+
+::
+
+ #Test block sizes of 4KB, 8KB, 64KB, 1MB
+ #Test 5 workloads: read, write, randwrite, randread, rw
+ schema: "yardstick:task:0.1"
+
+ scenarios:
+ {% for bs in ['4k', '8k', '64k', '1024k' ] %}
+ {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %}
+ -
+ type: Fio
+ options:
+ filename: /home/ec2-user/data.raw
+ bs: {{bs}}
+ rw: {{rw}}
+ ramp_time: 10
+ host: fio.demo
+ runner:
+ type: Duration
+ duration: 60
+ interval: 60
+
+ {% endfor %}
+ {% endfor %}
+ context
+ ...
diff --git a/docs/configguide/yardstick_testcases/glossary.rst b/docs/configguide/yardstick_testcases/glossary.rst
new file mode 100644
index 000000000..8ce9a6ba0
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/glossary.rst
@@ -0,0 +1,33 @@
+==================
+Yardstick Glossary
+==================
+
+.. glossary::
+ :sorted:
+
+ DPI
+ Deep Packet Inspection
+
+ DSCP
+ Differentiated Services Code Point
+
+ PBFS
+ Packet Based per Flow State
+
+ QoS
+ Quality of Service
+
+ VNF
+ Virtual Network Function
+
+ VNFC
+ Virtual Network Function Component
+
+ NFVI
+ Network Function Virtualization Infrastructure
+
+ ToS
+ Type of Service
+
+ VTC
+ Virtual Traffic Classifier
diff --git a/docs/configguide/yardstick_testcases/index.rst b/docs/configguide/yardstick_testcases/index.rst
new file mode 100644
index 000000000..55d4ea3e1
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/index.rst
@@ -0,0 +1,12 @@
+==================
+Yardstick Overview
+==================
+
+.. toctree::
+ :maxdepth: 2
+
+ 01-introduction
+ 02-methodology
+ 04-vtc-overview
+ 03-list-of-tcs
+ glossary
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst
new file mode 100644
index 000000000..810bad489
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc001.rst
@@ -0,0 +1,79 @@
+*************************************
+Yardstick Test Case Description TC001
+*************************************
+
+.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
+
++-----------------------------------------------------------------------------+
+|Network Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC001_NW PERF |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Number of flows and throughput |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS network performance with regards to |
+| | flows and throughput, such as if and how different amounts |
+| | of flows matter for the throughput between hosts on different|
+| | compute blades. Typically e.g. the performance of a vSwitch |
+| | depends on the number of flows running through it. Also |
+| | performance of other equipment or entities can depend |
+| | on the number of flows or the packet sizes used. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs ans similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc001.yaml |
+| | |
+| | Packet size: 60 bytes |
+| | Number of ports: 10, 50, 100, 500 and 1000, where each |
+| | runs for 20 seconds. The whole sequence is run |
+| | twice. The client and server are distributed on different HW.|
+| | For SLA max_ppm is set to 1000. The amount of configured |
+| | ports map to between 110 up to 1001000 flows, respectively. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | pktgen |
+| | |
+| | (Pktgen is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Docker |
+| | image. |
+| | As an example see the /yardstick/tools/ directory for how |
+| | to generate a Linux image with pktgen included.) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | pktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different packet sizes, amount |
+| | of flows and test duration. Default values exist. |
+| | |
+| | SLA (optional): max_ppm: The number of packets per million |
+| | packets sent that are acceptable to loose, not received. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with pktgen included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. pktgen is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst
new file mode 100644
index 000000000..56350f5bb
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc002.rst
@@ -0,0 +1,78 @@
+*************************************
+Yardstick Test Case Description TC002
+*************************************
+
+.. _cirros-image: https://download.cirros-cloud.net
+
++-----------------------------------------------------------------------------+
+|Network Latency |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | RTT, Round Trip Time |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To do a basic verification that network latency is within |
+| | acceptable boundaries when packets travel between hosts |
+| | located on same or different compute blades. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs and similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc002.yaml |
+| | |
+| | Packet size 100 bytes. Total test duration 600 seconds. |
+| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | ping |
+| | |
+| | Ping is normally part of any Linux distribution, hence it |
+| | doesn't need to be installed. It is also part of the |
+| | Yardstick Docker image. |
+| | (For example also a Cirros image can be downloaded from |
+| | cirros-image_, it includes ping) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | Ping man page |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test case can be configured with different packet sizes, |
+| | burst sizes, ping intervals and test duration. |
+| | SLA is optional. The SLA in this test case serves as an |
+| | example. Considerably lower RTT is expected, and |
+| | also normal to achieve in balanced L2 environments. However, |
+| | to cover most configurations, both bare metal and fully |
+| | virtualized ones, this value should be possible to achieve |
+| | and acceptable for black box testing. Many real time |
+| | applications start to suffer badly if the RTT time is higher |
+| | than this. Some may suffer bad also close to this RTT, while |
+| | others may not suffer at all. It is a compromise that may |
+| | have to be tuned for different configuration purposes. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with ping included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. Ping is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Test should not PASS if any RTT is above the optional SLA |
+| | value, or if there is a test case execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst
new file mode 100644
index 000000000..8b7474696
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc005.rst
@@ -0,0 +1,72 @@
+*************************************
+Yardstick Test Case Description TC005
+*************************************
+
+.. _fio: http://www.bluestop.org/fio/HOWTO.txt
+
++-----------------------------------------------------------------------------+
+|Storage Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC005_Storage Performance |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | IOPS, throughput and latency |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS storage performance with regards to |
+| | IOPS, throughput and latency. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs and similar shall be stored for comparison reasons |
+| | and product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc005.yaml |
+| | |
+| | IO types: read, write, randwrite, randread, rw |
+| | IO block size: 4KB, 64KB, 1024KB, where each |
+| | runs for 30 seconds(10 for ramp time, 20 for runtime). |
+| | |
+| | For SLA minimum read/write iops is set to 100, minimum |
+| | read/write throughput is set to 400 KB/s, and maximum |
+| | read/write latency is set to 20000 usec. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | fio |
+| | |
+| | (fio is not always part of a Linux distribution, hence it |
+| | needs to be installed. As an example see the |
+| | /yardstick/tools/ directory for how to generate a Linux |
+| | image with fio included.) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | fio_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different read/write types, IO |
+| | block size, IO depth, ramp time (runtime required for stable |
+| | results) and test duration. Default values exist. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with fio included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. fio is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst
new file mode 100644
index 000000000..b68315078
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc006.rst
@@ -0,0 +1,139 @@
+*************************************
+Yardstick Test Case Description TC006
+*************************************
+
+.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/
+.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt
+
++-----------------------------------------------------------------------------+
+|Network Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC006_Virtual Traffic Classifier Data Plane |
+| | Throughput Benchmarking Test. |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Throughput |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To measure the throughput supported by the virtual Traffic |
+| | Classifier according to the RFC2544 methodology for a |
+| | user-defined set of vTC deployment configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: file: opnfv_yardstick_tc006.yaml |
+| | |
+| | packet_size: size of the packets to be used during the |
+| | throughput calculation. |
+| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] |
+| | |
+| | vnic_type: type of VNIC to be used. |
+| | Allowed values are: |
+| | - normal: for default OvS port configuration |
+| | - direct: for SR-IOV port configuration |
+| | Default value: None |
+| | |
+| | vtc_flavor: OpenStack flavor to be used for the vTC |
+| | Default available values are: m1.small, m1.medium, |
+| | and m1.large, but the user can create his/her own |
+| | flavor and give it as input |
+| | Default value: None |
+| | |
+| | vlan_sender: vlan tag of the network on which the vTC will |
+| | receive traffic (VLAN Network 1). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | vlan_receiver: vlan tag of the network on which the vTC |
+| | will send traffic back to the packet generator |
+| | (VLAN Network 2). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | default_net_name: neutron name of the defaul network that |
+| | is used for access to the internet from the vTC |
+| | (vNIC 1). |
+| | |
+| | default_subnet_name: subnet name for vNIC1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_1_name: Neutron Name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_2_name: Neutron Name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | DPDK pktgen |
+| | |
+| | DPDK Pktgen is not part of a Linux distribution, |
+| | hence it needs to be installed by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | DPDK Pktgen: DPDKpktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
+| | RFC 2544: rfc2544_ |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different flavors, vNIC type |
+| | and packet sizes. Default values exist as specified above. |
+| | The vNIC type and flavor MUST be specified by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The vTC has been successfully instantiated and configured. |
+| | The user has correctly assigned the values to the deployment |
+| | configuration parameters. |
+| | |
+| | - Multicast traffic MUST be enabled on the network. |
+| | The Data network switches need to be configured in |
+| | order to manage multicast traffic. |
+| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs |
+| | must be used on the compute node. |
+| | - Yarsdtick needs to be installed on a host connected to the |
+| | data network and the host must have 2 DPDK-compatible |
+| | NICs. Proper configuration of DPDK and DPDK pktgen is |
+| | required before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | Description and expected results |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The vTC is deployed, according to the user-defined |
+| | configuration |
+| | |
++--------------+--------------------------------------------------------------+
+|step 2 | The vTC is correctly deployed and configured as necessary |
+| | The initialization script has been correctly executed and |
+| | vTC is ready to receive and process the traffic. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 3 | Test case is executed with the selected parameters: |
+| | - vTC flavor |
+| | - vNIC type |
+| | - packet size |
+| | The traffic is sent to the vTC using the maximum available |
+| | traffic rate for 60 seconds. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 4 | The vTC instance forwards all the packets back to the packet |
+| | generator for 60 seconds, as specified by RFC 2544. |
+| | |
+| | Steps 3 and 4 are executed different times, with different |
+| | rates in order to find the maximum supported traffic rate |
+| | according to the current definition of throughput in RFC |
+| | 2544. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | The result of the test is a number between 0 and 100 which |
+| | represents the throughput in terms of percentage of the |
+| | available pktgen NIC bandwidth. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst
new file mode 100644
index 000000000..a7a4776d5
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc007.rst
@@ -0,0 +1,157 @@
+*************************************
+Yardstick Test Case Description TC007
+*************************************
+
+.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/
+.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt
+
++-----------------------------------------------------------------------------+
+|Network Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC007_Virtual Traffic Classifier Data Plane |
+| | Throughput Benchmarking Test in Presence of Noisy |
+| | neighbours |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Throughput |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To measure the throughput supported by the virtual Traffic |
+| | Classifier according to the RFC2544 methodology for a |
+| | user-defined set of vTC deployment configurations in the |
+| | presence of noisy neighbours. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc007.yaml |
+| | |
+| | packet_size: size of the packets to be used during the |
+| | throughput calculation. |
+| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] |
+| | |
+| | vnic_type: type of VNIC to be used. |
+| | Allowed values are: |
+| | - normal: for default OvS port configuration |
+| | - direct: for SR-IOV port configuration |
+| | |
+| | vtc_flavor: OpenStack flavor to be used for the vTC |
+| | Default available values are: m1.small, m1.medium, |
+| | and m1.large, but the user can create his/her own |
+| | flavor and give it as input |
+| | |
+| | num_of_neighbours: Number of noisy neighbours (VMs) to be |
+| | instantiated during the experiment. |
+| | Allowed values: range (1, 10) |
+| | |
+| | amount_of_ram: RAM to be used by each neighbor. |
+| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', |
+| | '6G', '7G', '8G', '9G', '10G'] |
+| | Deault value: 256M |
+| | |
+| | number_of_cores: Number of noisy neighbours (VMs) to be |
+| | instantiated during the experiment. |
+| | Allowed values: range (1, 10) |
+| | Default value: 1 |
+| | |
+| | vlan_sender: vlan tag of the network on which the vTC will |
+| | receive traffic (VLAN Network 1). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | vlan_receiver: vlan tag of the network on which the vTC |
+| | will send traffic back to the packet generator |
+| | (VLAN Network 2). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | default_net_name: neutron name of the defaul network that |
+| | is used for access to the internet from the vTC |
+| | (vNIC 1). |
+| | |
+| | default_subnet_name: subnet name for vNIC1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_1_name: Neutron Name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_2_name: Neutron Name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | DPDK pktgen |
+| | |
+| | DPDK Pktgen is not part of a Linux distribution, |
+| | hence it needs to be installed by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | DPDKpktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
+| | rfc2544_ |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different flavors, vNIC type |
+| | and packet sizes. Default values exist as specified above. |
+| | The vNIC type and flavor MUST be specified by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The vTC has been successfully instantiated and configured. |
+| | The user has correctly assigned the values to the deployment |
+| | configuration parameters. |
+| | |
+| | - Multicast traffic MUST be enabled on the network. |
+| | The Data network switches need to be configured in |
+| | order to manage multicast traffic. |
+| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs |
+| | must be used on the compute node. |
+| | - Yarsdtick needs to be installed on a host connected to the |
+| | data network and the host must have 2 DPDK-compatible |
+| | NICs. Proper configuration of DPDK and DPDK pktgen is |
+| | required before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | Description and expected results |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The noisy neighbours are deployed as required by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 2 | The vTC is deployed, according to the configuration required |
+| | by the user |
+| | |
++--------------+--------------------------------------------------------------+
+|step 3 | The vTC is correctly deployed and configured as necessary. |
+| | The initialization script has been correctly executed and |
+| | the vTC is ready to receive and process the traffic. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 4 | Test case is executed with the parameters specified by the |
+| | user: |
+| | - vTC flavor |
+| | - vNIC type |
+| | - packet size |
+| | The traffic is sent to the vTC using the maximum available |
+| | traffic rate |
+| | |
++--------------+--------------------------------------------------------------+
+|step 5 | The vTC instance forwards all the packets back to the |
+| | packet generator for 60 seconds, as specified by RFC 2544. |
+| | |
+| | Steps 4 and 5 are executed different times with different |
+| | with different traffic rates, in order to find the maximum |
+| | supported traffic rate, accoring to the current definition |
+| | of throughput in RFC 2544. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | The result of the test is a number between 0 and 100 which |
+| | represents the throughput in terms of percentage of the |
+| | available pktgen NIC bandwidth. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst
new file mode 100644
index 000000000..e176e633a
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc008.rst
@@ -0,0 +1,85 @@
+*************************************
+Yardstick Test Case Description TC008
+*************************************
+
+.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
+
++-----------------------------------------------------------------------------+
+|Packet Loss Extended Test |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Number of flows, packet size and throughput |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS network performance with regards to |
+| | flows and throughput, such as if and how different amounts |
+| | of packet sizes and flows matter for the throughput between |
+| | VMs on different compute blades. Typically e.g. the |
+| | performance of a vSwitch |
+| | depends on the number of flows running through it. Also |
+| | performance of other equipment or entities can depend |
+| | on the number of flows or the packet sizes used. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs ans similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc008.yaml |
+| | |
+| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. |
+| | |
+| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of |
+| | configured ports map from 2 up to 1001000 flows, |
+| | respectively. Each packet_size/port_amount combination is run|
+| | ten times, for 20 seconds each. Then the next |
+| | packet_size/port_amount combination is run, and so on. |
+| | |
+| | The client and server are distributed on different HW. |
+| | |
+| | For SLA max_ppm is set to 1000. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | pktgen |
+| | |
+| | (Pktgen is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Docker |
+| | image. |
+| | As an example see the /yardstick/tools/ directory for how |
+| | to generate a Linux image with pktgen included.) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | pktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different packet sizes, amount |
+| | of flows and test duration. Default values exist. |
+| | |
+| | SLA (optional): max_ppm: The number of packets per million |
+| | packets sent that are acceptable to loose, not received. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with pktgen included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. pktgen is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst
new file mode 100644
index 000000000..e4002a884
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc009.rst
@@ -0,0 +1,84 @@
+*************************************
+Yardstick Test Case Description TC009
+*************************************
+
+.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
+
++-----------------------------------------------------------------------------+
+|Packet Loss |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Number of flows, packets lost and throughput |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS network performance with regards to |
+| | flows and throughput, such as if and how different amounts |
+| | of flows matter for the throughput between VMs on different |
+| | compute blades. |
+| | Typically e.g. the performance of a vSwitch |
+| | depends on the number of flows running through it. Also |
+| | performance of other equipment or entities can depend |
+| | on the number of flows or the packet sizes used. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs ans similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc009.yaml |
+| | |
+| | Packet size: 64 bytes |
+| | |
+| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of |
+| | configured ports map from 2 up to 1001000 flows, |
+| | respectively. Each port amount is run ten times, for 20 |
+| | seconds each. Then the next port_amount is run, and so on. |
+| | |
+| | The client and server are distributed on different HW. |
+| | |
+| | For SLA max_ppm is set to 1000. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | pktgen |
+| | |
+| | (Pktgen is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Docker |
+| | image. |
+| | As an example see the /yardstick/tools/ directory for how |
+| | to generate a Linux image with pktgen included.) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | pktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different packet sizes, amount |
+| | of flows and test duration. Default values exist. |
+| | |
+| | SLA (optional): max_ppm: The number of packets per million |
+| | packets sent that are acceptable to loose, not received. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with pktgen included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. pktgen is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst
new file mode 100644
index 000000000..ebb74ea30
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc010.rst
@@ -0,0 +1,77 @@
+*************************************
+Yardstick Test Case Description TC010
+*************************************
+
+.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html
+
++-----------------------------------------------------------------------------+
+|Memory Latency |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC010_Memory Latency |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Latency in nanoseconds |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | Measure the memory read latency for varying memory sizes and |
+| | strides. Whole memory hierarchy is measured including all |
+| | levels of cache. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | File: opnfv_yardstick_tc010.yaml |
+| | |
+| | * SLA (max_latency): 30 nanoseconds |
+| | * Stride - 128 bytes |
+| | * Stop size - 64 megabytes |
+| | * Iterations: 10 - test is run 10 times iteratively. |
+| | * Interval: 1 - there is 1 second delay between each |
+| | iteration. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | Lmbench |
+| | |
+| | Lmbench is a suite of operating system microbenchmarks. This |
+| | test uses lat_mem_rd tool from that suite. |
+| | Lmbench is not always part of a Linux distribution, hence it |
+| | needs to be installed in the test image |
+| | |
++--------------+--------------------------------------------------------------+
+|references | man-pages_ |
+| | |
+| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools |
+| | for Performance Analysis." USENIX annual technical |
+| | conference 1996. |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different: |
+| | |
+| | * strides; |
+| | * stop_size; |
+| | * iterations and intervals. |
+| | |
+| | There are default values for each above-mentioned option. |
+| | |
+| | SLA (optional) : max_latency: The maximum memory latency |
+| | that is accepted. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with Lmbench included in the image. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool |
+| | is invoked and logs are produced and stored. |
+| | |
+| | Result: logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Test fails if the measured memory latency is above the SLA |
+| | value or if there is a test case execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst
new file mode 100644
index 000000000..e7889c14e
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc012.rst
@@ -0,0 +1,81 @@
+*************************************
+Yardstick Test Case Description TC012
+*************************************
+
+.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html
+
++-----------------------------------------------------------------------------+
+|Memory Bandwidth |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Megabyte per second (MBps) |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | Measure the rate at which data can be read from and written |
+| | to the memory (this includes all levels of memory). |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | File: opnfv_yardstick_tc012.yaml |
+| | |
+| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of |
+| | memory bandwidth that is accepted. |
+| | * Size: 10 240 kB - test allocates twice that size (20 480kB)|
+| | zeros it and then measures the time it takes to copy from |
+| | one side to another. |
+| | * Benchmark: rdwr - measures the time to read data into |
+| | memory and then write data to the same location. |
+| | * Warmup: 0 - the number of iterations to perform before |
+| | taking actual measurements. |
+| | * Iterations: 10 - test is run 10 times iteratively. |
+| | * Interval: 1 - there is 1 second delay between each |
+| | iteration. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | Lmbench |
+| | |
+| | Lmbench is a suite of operating system microbenchmarks. This |
+| | test uses bw_mem tool from that suite. |
+| | Lmbench is not always part of a Linux distribution, hence it |
+| | needs to be installed in the test image. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | man-pages_ |
+| | |
+| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools |
+| | for Performance Analysis." USENIX annual technical |
+| | conference. 1996. |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different: |
+| | |
+| | * memory sizes; |
+| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, |
+| | fcp, bzero, bcopy); |
+| | * number of warmup iterations; |
+| | * iterations and intervals. |
+| | |
+| | There are default values for each above-mentioned option. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with Lmbench included in the image. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The host is installed as client. Lmbench's bw_mem tool is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Test fails if the measured memory bandwidth is below the SLA |
+| | value or if there is a test case execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst
new file mode 100644
index 000000000..68d36ecd2
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc014.rst
@@ -0,0 +1,69 @@
+*************************************
+Yardstick Test Case Description TC014
+*************************************
+
+.. _unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench
+
++-----------------------------------------------------------------------------+
+|Processing speed |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC014_Processing speed |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | score of single cpu running, score of parallel running |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS processing speed with regards to score |
+| | of single cpu running and parallel running |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs and similar shall be stored for comparison reasons |
+| | and product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc014.yaml |
+| | |
+| | run_mode: Run unixbench in quiet mode or verbose mode |
+| | test_type: dhry2reg, whetstone and so on |
+| | |
+| | For SLA with single_score and parallel_score, both can be |
+| | set by user, default is NA |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | unixbench |
+| | |
+| | (unixbench is not always part of a Linux distribution, hence |
+| | it needs to be installed. As an example see the |
+| | /yardstick/tools/ directory for how to generate a Linux |
+| | image with unixbench included.) |
+| | |
++--------------+--------------------------------------------------------------+
+|references | unixbench_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different test types, dhry2reg, |
+| | whetstone and so on. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with unixbench included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as a client. unixbench is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst
new file mode 100644
index 000000000..9a5130f71
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc020.rst
@@ -0,0 +1,136 @@
+*************************************
+Yardstick Test Case Description TC020
+*************************************
+
+.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/
+.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt
+
++-----------------------------------------------------------------------------+
+|Network Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC0020_Virtual Traffic Classifier |
+| | Instantiation Test |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Failure |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To verify that a newly instantiated vTC is 'alive' and |
+| | functional and its instantiation is correctly supported by |
+| | the infrastructure. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc020.yaml |
+| | |
+| | vnic_type: type of VNIC to be used. |
+| | Allowed values are: |
+| | - normal: for default OvS port configuration |
+| | - direct: for SR-IOV port configuration |
+| | Default value: None |
+| | |
+| | vtc_flavor: OpenStack flavor to be used for the vTC |
+| | Default available values are: m1.small, m1.medium, |
+| | and m1.large, but the user can create his/her own |
+| | flavor and give it as input |
+| | Default value: None |
+| | |
+| | vlan_sender: vlan tag of the network on which the vTC will |
+| | receive traffic (VLAN Network 1). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | vlan_receiver: vlan tag of the network on which the vTC |
+| | will send traffic back to the packet generator |
+| | (VLAN Network 2). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | default_net_name: neutron name of the defaul network that |
+| | is used for access to the internet from the vTC |
+| | (vNIC 1). |
+| | |
+| | default_subnet_name: subnet name for vNIC1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_1_name: Neutron Name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_2_name: Neutron Name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | DPDK pktgen |
+| | |
+| | DPDK Pktgen is not part of a Linux distribution, |
+| | hence it needs to be installed by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | DPDKpktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
+| | rfc2544_ |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different flavors, vNIC type |
+| | and packet sizes. Default values exist as specified above. |
+| | The vNIC type and flavor MUST be specified by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The vTC has been successfully instantiated and configured. |
+| | The user has correctly assigned the values to the deployment |
+| | configuration parameters. |
+| | |
+| | - Multicast traffic MUST be enabled on the network. |
+| | The Data network switches need to be configured in |
+| | order to manage multicast traffic. |
+| | Installation and configuration of smcroute is required |
+| | before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs |
+| | must be used on the compute node. |
+| | - Yarsdtick needs to be installed on a host connected to the |
+| | data network and the host must have 2 DPDK-compatible |
+| | NICs. Proper configuration of DPDK and DPDK pktgen is |
+| | required before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | Description and expected results |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The vTC is deployed, according to the configuration provided |
+| | by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 2 | The vTC is correctly deployed and configured as necessary. |
+| | The initialization script has been correctly executed and |
+| | the vTC is ready to receive and process the traffic. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 3 | Test case is executed with the parameters specified by the |
+| | the user: |
+| | - vTC flavor |
+| | - vNIC type |
+| | A constant rate traffic is sent to the vTC for 10 seconds. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 4 | The vTC instance tags all the packets and sends them back to |
+| | the packet generator for 10 seconds. |
+| | |
+| | The framework checks that the packet generator receives |
+| | back all the packets with the correct tag from the vTC. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | The vTC is deemed to be successfully instantiated if all |
+| | packets are sent back with the right tag as requested, |
+| | else it is deemed DoA (Dead on arrival) |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst
new file mode 100644
index 000000000..a493ddfc0
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc021.rst
@@ -0,0 +1,152 @@
+*************************************
+Yardstick Test Case Description TC021
+*************************************
+
+.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/
+.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt
+
++-----------------------------------------------------------------------------+
+|Network Performance |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC0021_Virtual Traffic Classifier |
+| | Instantiation Test in Presence of Noisy Neighbours |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Failure |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To verify that a newly instantiated vTC is 'alive' and |
+| | functional and its instantiation is correctly supported by |
+| | the infrastructure in the presence of noisy neighbours. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc021.yaml |
+| | |
+| | vnic_type: type of VNIC to be used. |
+| | Allowed values are: |
+| | - normal: for default OvS port configuration |
+| | - direct: for SR-IOV port configuration |
+| | Default value: None |
+| | |
+| | vtc_flavor: OpenStack flavor to be used for the vTC |
+| | Default available values are: m1.small, m1.medium, |
+| | and m1.large, but the user can create his/her own |
+| | flavor and give it as input |
+| | Default value: None |
+| | |
+| | num_of_neighbours: Number of noisy neighbours (VMs) to be |
+| | instantiated during the experiment. |
+| | Allowed values: range (1, 10) |
+| | |
+| | amount_of_ram: RAM to be used by each neighbor. |
+| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', |
+| | '6G', '7G', '8G', '9G', '10G'] |
+| | Deault value: 256M |
+| | |
+| | number_of_cores: Number of noisy neighbours (VMs) to be |
+| | instantiated during the experiment. |
+| | Allowed values: range (1, 10) |
+| | Default value: 1 |
+| | |
+| | vlan_sender: vlan tag of the network on which the vTC will |
+| | receive traffic (VLAN Network 1). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | vlan_receiver: vlan tag of the network on which the vTC |
+| | will send traffic back to the packet generator |
+| | (VLAN Network 2). |
+| | Allowed values: range (1, 4096) |
+| | |
+| | default_net_name: neutron name of the defaul network that |
+| | is used for access to the internet from the vTC |
+| | (vNIC 1). |
+| | |
+| | default_subnet_name: subnet name for vNIC1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_1_name: Neutron Name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 |
+| | (information available through Neutron). |
+| | |
+| | vlan_net_2_name: Neutron Name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
+| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 |
+| | (information available through Neutron). |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | DPDK pktgen |
+| | |
+| | DPDK Pktgen is not part of a Linux distribution, |
+| | hence it needs to be installed by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | DPDK Pktgen: DPDK Pktgen: DPDKpktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
+| | RFC 2544: rfc2544_ |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different flavors, vNIC type |
+| | and packet sizes. Default values exist as specified above. |
+| | The vNIC type and flavor MUST be specified by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The vTC has been successfully instantiated and configured. |
+| | The user has correctly assigned the values to the deployment |
+| | configuration parameters. |
+| | |
+| | - Multicast traffic MUST be enabled on the network. |
+| | The Data network switches need to be configured in |
+| | order to manage multicast traffic. |
+| | Installation and configuration of smcroute is required |
+| | before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs |
+| | must be used on the compute node. |
+| | - Yarsdtick needs to be installed on a host connected to the |
+| | data network and the host must have 2 DPDK-compatible |
+| | NICs. Proper configuration of DPDK and DPDK pktgen is |
+| | required before to run the test case. |
+| | (For further instructions please refer to the ApexLake |
+| | documentation). |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | Description and expected results |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The noisy neighbours are deployed as required by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 2 | The vTC is deployed, according to the configuration provided |
+| | by the user. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 3 | The vTC is correctly deployed and configured as necessary. |
+| | The initialization script has been correctly executed and |
+| | the vTC is ready to receive and process the traffic. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 4 | Test case is executed with the selected parameters: |
+| | - vTC flavor |
+| | - vNIC type |
+| | A constant rate traffic is sent to the vTC for 10 seconds. |
+| | |
++--------------+--------------------------------------------------------------+
+|step 5 | The vTC instance tags all the packets and sends them back to |
+| | the packet generator for 10 seconds. |
+| | |
+| | The framework checks if the packet generator receives back |
+| | all the packets with the correct tag from the vTC. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | The vTC is deemed to be successfully instantiated if all |
+| | packets are sent back with the right tag as requested, |
+| | else it is deemed DoA (Dead on arrival) |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst
new file mode 100644
index 000000000..56c8227df
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc027.rst
@@ -0,0 +1,67 @@
+*************************************
+Yardstick Test Case Description TC027
+*************************************
+
+.. _ipv6: https://wiki.opnfv.org/ipv6_opnfv_project
+
++-----------------------------------------------------------------------------+
+|IPv6 connectivity between nodes on the tenant network |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC002_IPv6 connectivity |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | RTT, Round Trip Time |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To do a basic verification that IPv6 connectivity is within |
+| | acceptable boundaries when ipv6 packets travel between hosts |
+| | located on same or different compute blades. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs and similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc027.yaml |
+| | |
+| | Packet size 56 bytes. |
+| | SLA RTT is set to maximum 10 ms. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | ping6 |
+| | |
+| | Ping6 is normally part of Linux distribution, hence it |
+| | doesn't need to be installed. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | ipv6_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test case can be configured with different run step |
+| | you can run setup, run benchmakr, teardown independently |
+| | SLA is optional. The SLA in this test case serves as an |
+| | example. Considerably lower RTT is expected. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with ping6 included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. Ping is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Test should not PASS if any RTT is above the optional SLA |
+| | value, or if there is a test case execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst
new file mode 100644
index 000000000..5c91f6bf1
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc037.rst
@@ -0,0 +1,99 @@
+*************************************
+Yardstick Test Case Description TC037
+*************************************
+
+.. _cirros: https://download.cirros-cloud.net
+.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
+
++-----------------------------------------------------------------------------+
+|Latency, CPU Load, Throughput, Packet Loss |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss|
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Number of flows, latency, throughput, CPU load, packet loss |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS network performance with regards to |
+| | flows and throughput, such as if and how different amounts |
+| | of flows matter for the throughput between hosts on different|
+| | compute blades. Typically e.g. the performance of a vSwitch |
+| | depends on the number of flows running through it. Also |
+| | performance of other equipment or entities can depend |
+| | on the number of flows or the packet sizes used. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs ans similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc037.yaml |
+| | |
+| | Packet size: 64 bytes |
+| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. |
+| | The amount configured ports map from 2 up to 1001000 flows, |
+| | respectively. Each port amount is run two times, for 20 |
+| | seconds each. Then the next port_amount is run, and so on. |
+| | During the test CPU load on both client and server, and the |
+| | network latency between the client and server are measured. |
+| | The client and server are distributed on different HW. |
+| | For SLA max_ppm is set to 1000. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | pktgen |
+| | |
+| | (Pktgen is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Glance |
+| | image. |
+| | As an example see the /yardstick/tools/ directory for how |
+| | to generate a Linux image with pktgen included.) |
+| | |
+| | ping |
+| | |
+| | Ping is normally part of any Linux distribution, hence it |
+| | doesn't need to be installed. It is also part of the |
+| | Yardstick Glance image. |
+| | (For example also a cirros_ image can be downloaded, it |
+| | includes ping) |
+| | |
+| | mpstat |
+| | |
+| | (Mpstat is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Glance |
+| | image. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | Ping and Mpstat man pages |
+| | |
+| | pktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different packet sizes, amount |
+| | of flows and test duration. Default values exist. |
+| | |
+| | SLA (optional): max_ppm: The number of packets per million |
+| | packets sent that are acceptable to loose, not received. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with pktgen included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. pktgen is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst
new file mode 100644
index 000000000..93c2cf3d8
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc038.rst
@@ -0,0 +1,99 @@
+*************************************
+Yardstick Test Case Description TC038
+*************************************
+
+.. _cirros: https://download.cirros-cloud.net
+.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt
+
++-----------------------------------------------------------------------------+
+|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss|
+| | |
++--------------+--------------------------------------------------------------+
+|metric | Number of flows, latency, throughput, CPU load, packet loss |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | To evaluate the IaaS network performance with regards to |
+| | flows and throughput, such as if and how different amounts |
+| | of flows matter for the throughput between hosts on different|
+| | compute blades. Typically e.g. the performance of a vSwitch |
+| | depends on the number of flows running through it. Also |
+| | performance of other equipment or entities can depend |
+| | on the number of flows or the packet sizes used. |
+| | The purpose is also to be able to spot trends. Test results, |
+| | graphs ans similar shall be stored for comparison reasons and|
+| | product evolution understanding between different OPNFV |
+| | versions and/or configurations. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc038.yaml |
+| | |
+| | Packet size: 64 bytes |
+| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. |
+| | The amount configured ports map from 2 up to 1001000 flows, |
+| | respectively. Each port amount is run ten times, for 20 |
+| | seconds each. Then the next port_amount is run, and so on. |
+| | During the test CPU load on both client and server, and the |
+| | network latency between the client and server are measured. |
+| | The client and server are distributed on different HW. |
+| | For SLA max_ppm is set to 1000. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | pktgen |
+| | |
+| | (Pktgen is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Glance |
+| | image. |
+| | As an example see the /yardstick/tools/ directory for how |
+| | to generate a Linux image with pktgen included.) |
+| | |
+| | ping |
+| | |
+| | Ping is normally part of any Linux distribution, hence it |
+| | doesn't need to be installed. It is also part of the |
+| | Yardstick Glance image. |
+| | (For example also a cirros_ image can be downloaded, it |
+| | includes ping) |
+| | |
+| | mpstat |
+| | |
+| | (Mpstat is not always part of a Linux distribution, hence it |
+| | needs to be installed. It is part of the Yardstick Glance |
+| | image. |
+| | |
++--------------+--------------------------------------------------------------+
+|references | Ping and Mpstat man pages |
+| | |
+| | pktgen_ |
+| | |
+| | ETSI-NFV-TST001 |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different packet sizes, amount |
+| | of flows and test duration. Default values exist. |
+| | |
+| | SLA (optional): max_ppm: The number of packets per million |
+| | packets sent that are acceptable to loose, not received. |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | The test case image needs to be installed into Glance |
+|conditions | with pktgen included in it. |
+| | |
+| | No POD specific requirements have been identified. |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | The hosts are installed, as server and client. pktgen is |
+| | invoked and logs are produced and stored. |
+| | |
+| | Result: Logs are stored. |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if SLA is not passed, or if there is a test case |
+| | execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst
new file mode 100644
index 000000000..044ccf193
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/opnfv_yardstick_tc040.rst
@@ -0,0 +1,60 @@
+*************************************
+Yardstick Test Case Description TC040
+*************************************
+
+.. _Parser: https://wiki.opnfv.org/parser
+
++-----------------------------------------------------------------------------+
+|Verify Parser Yang-to-Tosca |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | OPNFV_YARDSTICK_TC040 Verify Parser Yang-to-Tosca |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | 1. tosca file which is converted from yang file by Parser |
+| | 2. result whether the output is same with expected outcome |
++--------------+--------------------------------------------------------------+
+|test purpose | To verify the function of Yang-to-Tosca in Parser. |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | file: opnfv_yardstick_tc040.yaml |
+| | |
+| | yangfile: the path of the yangfile which you want to convert |
+| | toscafile: the path of the toscafile which is your expected |
+| | outcome. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | Parser |
+| | |
+| | (Parser is not part of a Linux distribution, hence it |
+| | needs to be installed. As an example see the |
+| | /yardstick/benchmark/scenarios/parser/parser_setup.sh for |
+| | how to install it manual. Of course, it will be installed |
+| | and uninstalled automatically when you run this test case |
+| | by yardstick) |
++--------------+--------------------------------------------------------------+
+|references | Parser_ |
+| | |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | Test can be configured with different path of yangfile and |
+| | toscafile to fit your real environment to verify Parser |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | No POD specific requirements have been identified. |
+|conditions | it can be run without VM |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | parser is installed without VM, running Yang-to-Tosca module |
+| | to convert yang file to tosca file, validating output against|
+| | expected outcome. |
+| | |
+| | Result: Logs are stored. |
++--------------+--------------------------------------------------------------+
+|test verdict | Fails only if output is different with expected outcome |
+| | or if there is a test case execution problem. |
+| | |
++--------------+--------------------------------------------------------------+
diff --git a/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst
new file mode 100644
index 000000000..1b8754b05
--- /dev/null
+++ b/docs/configguide/yardstick_testcases/testcase_description_v2_template.rst
@@ -0,0 +1,64 @@
+.. Template to be used for test case descriptions in Yardstick Project.
+ Write one .rst per test case.
+ Upload the .rst for the test case in /docs/source/yardstick directory.
+ Review in Gerrit.
+
+*************************************
+Yardstick Test Case Description TCXXX
+*************************************
+
++-----------------------------------------------------------------------------+
+|test case slogan e.g. Network Latency |
+| |
++--------------+--------------------------------------------------------------+
+|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency |
+| | |
++--------------+--------------------------------------------------------------+
+|metric | what will be measured, e.g. latency |
+| | |
++--------------+--------------------------------------------------------------+
+|test purpose | describe what is the purpose of the test case |
+| | |
++--------------+--------------------------------------------------------------+
+|configuration | what .yaml file to use, state SLA if applicable, state |
+| | test duration, list and describe the scenario options used in|
+| | this TC and also list the options using default values. |
+| | |
++--------------+--------------------------------------------------------------+
+|test tool | e.g. ping |
+| | |
++--------------+--------------------------------------------------------------+
+|references | e.g. RFCxxx, ETSI-NFVyyy |
+| | |
++--------------+--------------------------------------------------------------+
+|applicability | describe variations of the test case which can be |
+| | performend, e.g. run the test for different packet sizes |
+| | |
++--------------+--------------------------------------------------------------+
+|pre-test | describe configuration in the tool(s) used to perform |
+|conditions | the measurements (e.g. fio, pktgen), POD-specific |
+| | configuration required to enable running the test |
+| | |
++--------------+--------------------------------------------------------------+
+|test sequence | description and expected result |
+| | |
++--------------+--------------------------------------------------------------+
+|step 1 | use this to describe tests that require sveveral steps e.g |
+| | collect logs. |
+| | |
+| | Result: what happens in this step e.g. logs collected |
+| | |
++--------------+--------------------------------------------------------------+
+|step 2 | remove interface |
+| | |
+| | Result: interface down. |
+| | |
++--------------+--------------------------------------------------------------+
+|step N | what is done in step N |
+| | |
+| | Result: what happens |
+| | |
++--------------+--------------------------------------------------------------+
+|test verdict | expected behavior, or SLA, pass/fail criteria |
+| | |
++--------------+--------------------------------------------------------------+