diff options
Diffstat (limited to 'docs/userguide')
35 files changed, 3020 insertions, 67 deletions
diff --git a/docs/userguide/01-introduction.rst b/docs/userguide/01-introduction.rst new file mode 100644 index 000000000..90c112a47 --- /dev/null +++ b/docs/userguide/01-introduction.rst @@ -0,0 +1,61 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +============ +Introduction +============ + +**Welcome to Yardstick's documentation !** + +.. _Pharos: https://wiki.opnfv.org/pharos +.. _Yardstick: https://wiki.opnfv.org/yardstick +.. _Presentation: https://wiki.opnfv.org/_media/opnfv_summit_-_yardstick_project.pdf + +Yardstick_ is an OPNFV Project. + +The project's goal is to verify infrastructure compliance, from the perspective +of a Virtual Network Function (:term:`VNF`). + +The Project's scope is the development of a test framework, *Yardstick*, test +cases and test stimuli to enable Network Function Virtualization Infrastructure +(:term:`NFVI`) verification. +The Project also includes a sample :term:`VNF`, the Virtual Traffic Classifier +(:term:`VTC`) and its experimental framework, *ApexLake* ! + +*Yardstick* is used in OPNFV for verifying the OPNFV infrastructure and some of +the OPNFV features. The *Yardstick* framework is deployed in several OPNFV +community labs. It is *installer*, *infrastructure* and *application* +independent. + +.. seealso:: Pharos_ for information on OPNFV community labs and this + Presentation_ for an overview of *Yardstick* + + +About This Document +=================== + +This document consists of the following chapters: + +* Chapter :doc:`02-methodology` describes the methodology implemented by the + Yardstick Project for :term:`NFVI` verification. + +* Chapter :doc:`04-vtc-overview` provides information on the :term:`VTC`. + +* Chapter :doc:`apexlake_installation` provides instructions to install the + experimental framework *ApexLake* and chapter :doc:`apexlake_api` explains + how this framework is integrated in *Yardstick*. + +* Chapter :doc:`03-installation` provides instructions to install *Yardstick*. + +* Chapter :doc:`03-list-of-tcs` includes a list of available Yardstick + test cases. + + +Contact Yardstick +================= + +Feedback? `Contact us`_ + +.. _Contact us: opnfv-users@lists.opnfv.org diff --git a/docs/userguide/02-methodology.rst b/docs/userguide/02-methodology.rst new file mode 100644 index 000000000..7fdb5f64f --- /dev/null +++ b/docs/userguide/02-methodology.rst @@ -0,0 +1,192 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +=========== +Methodology +=========== + +Abstract +======== + +This chapter describes the methodology implemented by the Yardstick project for +verifying the :term:`NFVI` from the perspective of a :term:`VNF`. + +ETSI-NFV +======== + +.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ +.. _Yardsticktst: https://wiki.opnfv.org/_media/opnfv_summit_-_bridging_opnfv_and_etsi.pdf + + +The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation +of NFV Environments and Services", recommends methods for pre-deployment +testing of the functional components of an NFV environment. + +The Yardstick project implements the methodology described in chapter 6, "Pre- +deployment validation of NFV infrastructure". + +The methodology consists in decomposing the typical :term:`VNF` work-load +performance metrics into a number of characteristics/performance vectors, which +each can be represented by distinct test-cases. + +The methodology includes five steps: + +* *Step1:* Define Infrastruture - the Hardware, Software and corresponding + configuration target for validation; the OPNFV infrastructure, in OPNFV + community labs. + +* *Step2:* Identify :term:`VNF` type - the application for which the + infrastructure is to be validated, and its requirements on the underlying + infrastructure. + +* *Step3:* Select test cases - depending on the workload that represents the + application for which the infrastruture is to be validated, the relevant + test cases amongst the list of available Yardstick test cases. + +* *Step4:* Execute tests - define the duration and number of iterations for the + selected test cases, tests runs are automated via OPNFV Jenkins Jobs. + +* *Step5:* Collect results - using the common API for result collection. + +.. seealso:: Yardsticktst_ for material on alignment ETSI TST001 and Yardstick. + +Metrics +======= + +The metrics, as defined by ETSI GS NFV-TST001, are shown in +:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and +:ref:`Table3 <table2_3>`. + +In OPNFV Brahmaputra release, generic test cases covering aspects of the listed +metrics are available; further OPNFV releases will provide extended testing of +these metrics. +The view of available Yardstick test cases cross ETSI definitions in +:ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>` +is shown in :ref:`Table4 <table2_4>`. +It shall be noticed that the Yardstick test cases are examples, the test +duration and number of iterations are configurable, as are the System Under +Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario +options). + +.. _table2_1: + +**Table 1 - Performance/Speed Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Performance/Speed | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Latency for random memory access | +| | * Latency for cache read/write operations | +| | * Processing speed (instructions per second) | +| | * Throughput for random memory access (bytes per second) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Throughput per NFVI node (frames/byte per second) | +| | * Throughput provided to a VM (frames/byte per second) | +| | * Latency per traffic flow | +| | * Latency between VMs | +| | * Latency between NFVI nodes | +| | * Packet delay variation (jitter) between VMs | +| | * Packet delay variation (jitter) between NFVI nodes | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Sequential read/write IOPS | +| | * Random read/write IOPS | +| | * Latency for storage read/write operations | +| | * Throughput for storage read/write operations | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_2: + +**Table 2 - Capacity/Scale Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Capacity/Scale | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Number of cores and threads- Available memory size | +| | * Cache size | +| | * Processor utilization (max, average, standard deviation) | +| | * Memory utilization (max, average, standard deviation) | +| | * Cache utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * Number of connections | +| | * Number of frames sent/received | +| | * Maximum throughput between VMs (frames/byte per second) | +| | * Maximum throughput between NFVI nodes (frames/byte per second) | +| | * Network utilization (max, average, standard deviation) | +| | * Number of traffic flows | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Storage/Disk size | +| | * Capacity allocation (block-based, object-based) | +| | * Block size | +| | * Maximum sequential read/write IOPS | +| | * Maximum random read/write IOPS | +| | * Disk utilization (max, average, standard deviation) | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_3: + +**Table 3 - Availability/Reliability Metrics** + ++---------+-------------------------------------------------------------------+ +| Category| Availability/Reliability | +| | | ++---------+-------------------------------------------------------------------+ +| Compute | * Processor availability (Error free processing time) | +| | * Memory availability (Error free memory time) | +| | * Processor mean-time-to-failure | +| | * Memory mean-time-to-failure | +| | * Number of processing faults per second | +| | | ++---------+-------------------------------------------------------------------+ +| Network | * NIC availability (Error free connection time) | +| | * Link availability (Error free transmission time) | +| | * NIC mean-time-to-failure | +| | * Network timeout duration due to link failure | +| | * Frame loss rate | +| | | ++---------+-------------------------------------------------------------------+ +| Storage | * Disk availability (Error free disk access time) | +| | * Disk mean-time-to-failure | +| | * Number of failed storage read/write operations per second | +| | | ++---------+-------------------------------------------------------------------+ + +.. _table2_4: + +**Table 4 - Yardstick Generic Test Cases** + ++---------+-------------------+----------------+------------------------------+ +| Category| Performance/Speed | Capacity/Scale | Availability/Reliability | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Compute | TC003 [1]_ | TC003 [1]_ | TC013 [1]_ | +| | TC004 [1]_ | TC004 [1]_ | TC015 [1]_ | +| | TC014 | TC010 | | +| | TC024 | TC012 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Network | TC002 | TC001 | TC016 [1]_ | +| | TC011 | TC008 | TC018 [1]_ | +| | | TC009 | | +| | | | | ++---------+-------------------+----------------+------------------------------+ +| Storage | TC005 | TC005 | TC017 [1]_ | +| | | | | ++---------+-------------------+----------------+------------------------------+ + +.. note:: The description in this OPNFV document is intended as a reference for + users to understand the scope of the Yardstick Project and the + deliverables of the Yardstick framework. For complete description of + the methodology, refer to the ETSI document. + +.. rubric:: Footnotes +.. [1] To be included in future deliveries. diff --git a/docs/userguide/yardstick_framework/03-installation.rst b/docs/userguide/03-installation.rst index 31f8a922e..47a3ea834 100644 --- a/docs/userguide/yardstick_framework/03-installation.rst +++ b/docs/userguide/03-installation.rst @@ -1,14 +1,13 @@ -.. - TODO As things will change, then this document has to be revised before the - next release. Steps: - 1. Verify that the instructions below are correct and have not been changed. - 2. Add everything that is currently missing and should be included in this document. - 3. Make sure each title has a paragraph or an introductory sentence under it. - 4. Make sure each sentence is grammatically correct and easily understandable. - 5. Remove this comment section. - -Installation -============== +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +Yardstick Installation +====================== + +Abstract +-------- Yardstick currently supports installation on Ubuntu 14.04 or by using a Docker image. Detailed steps about installing Yardstick using both of these options diff --git a/docs/userguide/03-list-of-tcs.rst b/docs/userguide/03-list-of-tcs.rst new file mode 100644 index 000000000..de48c7bc7 --- /dev/null +++ b/docs/userguide/03-list-of-tcs.rst @@ -0,0 +1,96 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +==================== +Yardstick Test Cases +==================== + +Abstract +======== + +This chapter lists available Yardstick test cases. +Yardstick test cases are divided in two main categories: + +* *Generic NFVI Test Cases* - Test Cases developed to realize the methodology +described in :doc:`02-methodology` + +* *OPNFV Feature Test Cases* - Test Cases developed to verify one or more +aspect of a feature delivered by an OPNFV Project, including the test cases +developed for the :term:`VTC`. + +Generic NFVI Test Case Descriptions +=================================== + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc001.rst + opnfv_yardstick_tc002.rst + opnfv_yardstick_tc005.rst + opnfv_yardstick_tc008.rst + opnfv_yardstick_tc009.rst + opnfv_yardstick_tc010.rst + opnfv_yardstick_tc011.rst + opnfv_yardstick_tc012.rst + opnfv_yardstick_tc014.rst + opnfv_yardstick_tc024.rst + opnfv_yardstick_tc037.rst + opnfv_yardstick_tc038.rst + +OPNFV Feature Test Cases +======================== + +H A +--- + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc019.rst + opnfv_yardstick_tc025.rst + +IPv6 +---- + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc027.rst + +KVM +--- + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc028.rst + +Parser +------ + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc040.rst + +virtual Traffic Classifier +-------------------------- + +.. toctree:: + :maxdepth: 1 + + opnfv_yardstick_tc006.rst + opnfv_yardstick_tc007.rst + opnfv_yardstick_tc020.rst + opnfv_yardstick_tc021.rst + +Templates +========= + +.. toctree:: + :maxdepth: 1 + + testcase_description_v2_template + Yardstick_task_templates diff --git a/docs/userguide/04-vtc-overview.rst b/docs/userguide/04-vtc-overview.rst new file mode 100644 index 000000000..82b20cad5 --- /dev/null +++ b/docs/userguide/04-vtc-overview.rst @@ -0,0 +1,122 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, National Center of Scientific Research "Demokritos" and others. + +========================== +Virtual Traffic Classifier +========================== + +Abstract +======== + +.. _TNOVA: http://www.t-nova.eu/ +.. _TNOVAresults: http://www.t-nova.eu/results/ +.. _Yardstick: https://wiki.opnfv.org/yardstick + +This chapter provides an overview of the virtual Traffic Classifier, a +contribution to OPNFV Yardstick_ from the EU Project TNOVA_. +Additional documentation is available in TNOVAresults_. + +Overview +======== + +The virtual Traffic Classifier (:term:`VTC`) :term:`VNF`, comprises of a +Virtual Network Function Component (:term:`VNFC`). The :term:`VNFC` contains +both the Traffic Inspection module, and the Traffic forwarding module, needed +to run the :term:`VNF`. The exploitation of Deep Packet Inspection +(:term:`DPI`) methods for traffic classification is built around two basic +assumptions: + +* third parties unaffiliated with either source or recipient are able to +inspect each IP packet’s payload + +* the classifier knows the relevant syntax of each application’s packet +payloads (protocol signatures, data patterns, etc.). + +The proposed :term:`DPI` based approach will only use an indicative, small +number of the initial packets from each flow in order to identify the content +and not inspect each packet. + +In this respect it follows the Packet Based per Flow State (term:`PBFS`). This +method uses a table to track each session based on the 5-tuples (src address, +dest address, src port,dest port, transport protocol) that is maintained for +each flow. + +Concepts +======== + +* *Traffic Inspection*: The process of packet analysis and application +identification of network traffic that passes through the :term:`VTC`. + +* *Traffic Forwarding*: The process of packet forwarding from an incoming +network interface to a pre-defined outgoing network interface. + +* *Traffic Rule Application*: The process of packet tagging, based on a +predefined set of rules. Packet tagging may include e.g. Type of Service +(:term:`ToS`) field modification. + +Architecture +============ + +The Traffic Inspection module is the most computationally intensive component +of the :term:`VNF`. It implements filtering and packet matching algorithms in +order to support the enhanced traffic forwarding capability of the :term:`VNF`. +The component supports a flow table (exploiting hashing algorithms for fast +indexing of flows) and an inspection engine for traffic classification. + +The implementation used for these experiments exploits the nDPI library. +The packet capturing mechanism is implemented using libpcap. When the +:term:`DPI` engine identifies a new flow, the flow register is updated with the +appropriate information and transmitted across the Traffic Forwarding module, +which then applies any required policy updates. + +The Traffic Forwarding moudle is responsible for routing and packet forwarding. +It accepts incoming network traffic, consults the flow table for classification +information for each incoming flow and then applies pre-defined policies +marking e.g. :term:`ToS`/Differentiated Services Code Point (:term:`DSCP`) +multimedia traffic for Quality of Service (:term:`QoS`) enablement on the +forwarded traffic. +It is assumed that the traffic is forwarded using the default policy until it +is identified and new policies are enforced. + +The expected response delay is considered to be negligible, as only a small +number of packets are required to identify each flow. + +Graphical Overview +================== + +.. code-block:: console + + +----------------------------+ + | | + | Virtual Traffic Classifier | + | | + | Analysing/Forwarding | + | ------------> | + | ethA ethB | + | | + +----------------------------+ + | ^ + | | + v | + +----------------------------+ + | | + | Virtual Switch | + | | + +----------------------------+ + +Install +======= + +run the build.sh with root privileges + +Run +=== + +sudo ./pfbridge -a eth1 -b eth2 + +Development Environment +======================= + +Ubuntu 14.04 diff --git a/docs/userguide/Yardstick_task_templates.rst b/docs/userguide/Yardstick_task_templates.rst new file mode 100755 index 000000000..e8130dd2a --- /dev/null +++ b/docs/userguide/Yardstick_task_templates.rst @@ -0,0 +1,160 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +Task Template Syntax +==================== + +Basic template syntax +--------------------- +A nice feature of the input task format used in Yardstick is that it supports +the template syntax based on Jinja2. +This turns out to be extremely useful when, say, you have a fixed structure of +your task but you want to parameterize this task in some way. +For example, imagine your input task file (task.yaml) runs a set of Ping +scenarios: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + + scenarios: + - + type: Ping + options: + packetsize: 200 + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +Let's say you want to run the same set of scenarios with the same runner/ +context/sla, but you want to try another packetsize to compare the performance. +The most elegant solution is then to turn the packetsize name into a template +variable: + +:: + + # Sample benchmark task config file + # measure network latency using ping + + schema: "yardstick:task:0.1" + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + + sla: + max_rtt: 10 + action: monitor + + context: + ... + +and then pass the argument value for {{packetsize}} when starting a task with +this configuration file. +Yardstick provides you with different ways to do that: + +1.Pass the argument values directly in the command-line interface (with either +a JSON or YAML dictionary): + +:: + + yardstick task start samples/ping-template.yaml + --task-args'{"packetsize":"200"}' + +2.Refer to a file that specifies the argument values (JSON/YAML): + +:: + + yardstick task start samples/ping-template.yaml --task-args-file args.yaml + +Using the default values +------------------------ +Note that the Jinja2 template syntax allows you to set the default values for +your parameters. +With default values set, your task file will work even if you don't +parameterize it explicitly while starting a task. +The default values should be set using the {% set ... %} clause (task.yaml). +For example: + +:: + + # Sample benchmark task config file + # measure network latency using ping + schema: "yardstick:task:0.1" + {% set packetsize = packetsize or "100" %} + scenarios: + - + type: Ping + options: + packetsize: {{packetsize}} + host: athena.demo + target: ares.demo + + runner: + type: Duration + duration: 60 + interval: 1 + ... + +If you don't pass the value for {{packetsize}} while starting a task, the +default one will be used. + +Advanced templates +------------------ + +Yardstick makes it possible to use all the power of Jinja2 template syntax, +including the mechanism of built-in functions. +As an example, let us make up a task file that will do a block storage +performance test. +The input task file (fio-template.yaml) below uses the Jinja2 for-endfor +construct to accomplish that: + +:: + + #Test block sizes of 4KB, 8KB, 64KB, 1MB + #Test 5 workloads: read, write, randwrite, randread, rw + schema: "yardstick:task:0.1" + + scenarios: + {% for bs in ['4k', '8k', '64k', '1024k' ] %} + {% for rw in ['read', 'write', 'randwrite', 'randread', 'rw' ] %} + - + type: Fio + options: + filename: /home/ubuntu/data.raw + bs: {{bs}} + rw: {{rw}} + ramp_time: 10 + host: fio.demo + runner: + type: Duration + duration: 60 + interval: 60 + + {% endfor %} + {% endfor %} + context + ... diff --git a/docs/userguide/apexlake_framework/apexlake_api.rst b/docs/userguide/apexlake_api.rst index 2ef3e43f5..35a1dbe3e 100644 --- a/docs/userguide/apexlake_framework/apexlake_api.rst +++ b/docs/userguide/apexlake_api.rst @@ -1,8 +1,18 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + + ================================= Apexlake API Interface Definition ================================= -The API interface provided by the framework to enable the execution of test cases is defined as follows. +Abstract +-------- + +The API interface provided by the framework to enable the execution of test +cases is defined as follows. init diff --git a/docs/userguide/apexlake_framework/index.rst b/docs/userguide/apexlake_framework/index.rst deleted file mode 100644 index 47ebfcdf4..000000000 --- a/docs/userguide/apexlake_framework/index.rst +++ /dev/null @@ -1,11 +0,0 @@ -******************************** -Apexlake Framework Documentation -******************************** - -.. toctree:: - :numbered: - :maxdepth: 2 - - apexlake_installation - apexlake_api - diff --git a/docs/userguide/apexlake_framework/apexlake_installation.rst b/docs/userguide/apexlake_installation.rst index 2e129bfa2..4870c2e21 100644 --- a/docs/userguide/apexlake_framework/apexlake_installation.rst +++ b/docs/userguide/apexlake_installation.rst @@ -1,3 +1,9 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + + .. _DPDK: http://dpdk.org/doc/nics .. _DPDK-pktgen: https://github.com/Pktgen/Pktgen-DPDK/ .. _SRIOV: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking @@ -7,35 +13,46 @@ ============================ Apexlake Installation Guide ============================ -ApexLake is a framework that provides automatic execution of experiments and related data collection to enable -a user validate infrastructure from the perspective of a Virtual Network Function (VNF). -In the context of Yardstick, a virtual Traffic Classifier (vTC) network function is utilized. + +Abstract +-------- + +ApexLake is a framework that provides automatic execution of experiments and +related data collection to enable a user validate infrastructure from the +perspective of a Virtual Network Function (:term:`VNF`). + +In the context of Yardstick, a virtual Traffic Classifier (:term:`VTC`) network +function is utilized. Framework Hardware Dependencies =============================== -In order to run the framework there are some hardware related dependencies for ApexLake. + +In order to run the framework there are some hardware related dependencies for +ApexLake. The framework needs to be installed on the same physical node where DPDK-pktgen_ is installed. -The installation requires the physical node hosting the packet generator must have 2 NICs -which are DPDK_ compatible. -The 2 NICs will be connected to the switch where the OpenStack VM network is managed. +The installation requires the physical node hosting the packet generator must +have 2 NICs which are DPDK_ compatible. -The switch used must support multicast traffic and IGMP snooping. +The 2 NICs will be connected to the switch where the OpenStack VM +network is managed. + +The switch used must support multicast traffic and :term:`IGMP` snooping. Further details about the configuration are provided at the following here_. -The corresponding ports to which the cables are connected need to be configured as VLAN trunks -using two of the VLAN IDs available for Neutron. +The corresponding ports to which the cables are connected need to be configured +as VLAN trunks using two of the VLAN IDs available for Neutron. Note the VLAN IDs used as they will be required in later configuration steps. Framework Software Dependencies =============================== Before starting the framework, a number of dependencies must first be installed. -The following describes the set of instructions to be executed via the Linux shell in order to install -and configure the required dependencies. +The following describes the set of instructions to be executed via the Linux +shell in order to install and configure the required dependencies. 1. Install Dependencies. @@ -66,10 +83,11 @@ After entering the Apexlake directory, run the following command. 4. Create Two Networks based on VLANs in Neutron. -To enable network communications between the packet generator and the compute node, -two networks must be created via Neutron and mapped to the VLAN IDs +To enable network communications between the packet generator and the compute +node, two networks must be created via Neutron and mapped to the VLAN IDs that were previously used in the configuration of the physical switch. -The following shows the typical set of commands required to configure Neutron correctly. +The following shows the typical set of commands required to configure Neutron +correctly. :: @@ -100,17 +118,24 @@ The following shows the typical set of commands required to configure Neutron co The VLAN tags must also be included in the test case Yardstick yaml file as parameters for the following test cases: - - TC 006 - - TC 007 - - TC 020 - - TC 021 + + * :doc:`opnfv_yardstick_tc006` + + * :doc:`opnfv_yardstick_tc007` + + * :doc:`opnfv_yardstick_tc020` + + * :doc:`opnfv_yardstick_tc021` Install and Configure DPDK Pktgen +++++++++++++++++++++++++++++++++ + Execution of the framework is based on DPDK Pktgen. -If DPDK Pktgen has not installed, it is necessary to download, install, compile and configure it. -The user can create a directory and download the dpdk packet generator source code: +If DPDK Pktgen has not installed, it is necessary to download, install, compile +and configure it. +The user can create a directory and download the dpdk packet generator source +code: :: @@ -118,17 +143,20 @@ The user can create a directory and download the dpdk packet generator source co mkdir dpdk_pktgen git clone https://github.com/pktgen/Pktgen-DPDK.git -For instructions on the installation and configuration of DPDK and DPDK Pktgen please follow the official -DPDK Pktgen README file. -Once the installation is completed, it is necessary to load the DPDK kernel driver, as follow: +For instructions on the installation and configuration of DPDK and DPDK Pktgen +please follow the official DPDK Pktgen README file. +Once the installation is completed, it is necessary to load the DPDK kernel +driver, as follow: :: insmod uio insmod DPDK_DIR/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko -It is necessary to set the configuration file to support the desired Pktgen configuration. -A description of the required configuration parameters and supporting examples is provided in the following: +It is necessary to set the configuration file to support the desired Pktgen +configuration. +A description of the required configuration parameters and supporting examples +is provided in the following: :: @@ -166,18 +194,21 @@ A description of the required configuration parameters and supporting examples i To find the parameters related to names of the NICs and the addresses of the PCI buses -the user may find it useful to run the DPDK tool nic_bind as follows: +the user may find it useful to run the :term:`DPDK` tool nic_bind as follows: :: DPDK_DIR/tools/dpdk_nic_bind.py --status Lists the NICs available on the system, and shows the available drivers and bus addresses for each interface. -Please make sure to select NICs which are DPDK compatible. +Please make sure to select NICs which are :term:`DPDK` compatible. Installation and Configuration of smcroute ++++++++++++++++++++++++++++++++++++++++++ -The user is required to install smcroute which is used by the framework to support multicast communications. + +The user is required to install smcroute which is used by the framework to +support multicast communications. + The following is the list of commands required to download and install smroute. :: @@ -193,7 +224,8 @@ The following is the list of commands required to download and install smroute. sudo make install cd .. -It is also requires the creation a configuration file using the following command: +It is also requires the creation a configuration file using the following +command: SMCROUTE_NIC=(name of the nic) @@ -211,7 +243,8 @@ Then create the smcroute configuration file /etc/smcroute.conf echo mgroup from $SMCROUTE_NIC group 224.192.16.1 > /etc/smcroute.conf -At the end of this procedure it will be necessary to perform the following actions to add the user to the sudoers: +At the end of this procedure it will be necessary to perform the following +actions to add the user to the sudoers: :: @@ -221,7 +254,9 @@ At the end of this procedure it will be necessary to perform the following actio Experiment using SR-IOV Configuration on the Compute Node +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -To enable SR-IOV interfaces on the physical NIC of the compute node, a compatible NIC is required. -NIC configuration depends on model and vendor. After proper configuration to support SR-IOV, -a proper configuration of OpenStack is required. -For further information, please refer to the _SRIOV configuration guide + +To enable :term:`SR-IOV` interfaces on the physical NIC of the compute node, a +compatible NIC is required. +NIC configuration depends on model and vendor. After proper configuration to +support :term:`SR-IOV`, a proper configuration of OpenStack is required. +For further information, please refer to the SRIOV_ configuration guide diff --git a/docs/userguide/glossary.rst b/docs/userguide/glossary.rst new file mode 100644 index 000000000..f8ff41887 --- /dev/null +++ b/docs/userguide/glossary.rst @@ -0,0 +1,65 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +======== +Glossary +======== + +.. glossary:: + :sorted: + + API + Application Programming Interface + + DPI + Deep Packet Inspection + + DPDK + Data Plane Development Kit + + DSCP + Differentiated Services Code Point + + IGMP + Internet Group Management Protocol + + IOPS + Input/Output Operations Per Second + + NIC + Network Interface Controller + + PBFS + Packet Based per Flow State + + QoS + Quality of Service + + VLAN + Virtual LAN + + VM + Virtual Machine + + VNF + Virtual Network Function + + VNFC + Virtual Network Function Component + + NFVI + Network Function Virtualization Infrastructure + + SR-IOV + Single Root IO Virtualization + + SUT + System Under Test + + ToS + Type of Service + + VTC + Virtual Traffic Classifier diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst new file mode 100644 index 000000000..3cad237e2 --- /dev/null +++ b/docs/userguide/index.rst @@ -0,0 +1,21 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +================== +Yardstick Overview +================== + +.. toctree:: + :maxdepth: 2 + + 01-introduction + 02-methodology + 04-vtc-overview + apexlake_installation + apexlake_api + 03-installation + 03-list-of-tcs + glossary + references diff --git a/docs/userguide/opnfv_yardstick_tc001.rst b/docs/userguide/opnfv_yardstick_tc001.rst new file mode 100644 index 000000000..4cf4b94a4 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc001.rst @@ -0,0 +1,84 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC001 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC001_NW PERF | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc001.yaml | +| | | +| | Packet size: 60 bytes | +| | Number of ports: 10, 50, 100, 500 and 1000, where each | +| | runs for 20 seconds. The whole sequence is run | +| | twice. The client and server are distributed on different HW.| +| | For SLA max_ppm is set to 1000. The amount of configured | +| | ports map to between 110 up to 1001000 flows, respectively. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc002.rst b/docs/userguide/opnfv_yardstick_tc002.rst new file mode 100644 index 000000000..193fc531f --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc002.rst @@ -0,0 +1,83 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC002 +************************************* + +.. _cirros-image: https://download.cirros-cloud.net + ++-----------------------------------------------------------------------------+ +|Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC002_NW LATENCY | +| | | ++--------------+--------------------------------------------------------------+ +|metric | RTT, Round Trip Time | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To do a basic verification that network latency is within | +| | acceptable boundaries when packets travel between hosts | +| | located on same or different compute blades. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc002.yaml | +| | | +| | Packet size 100 bytes. Total test duration 600 seconds. | +| | One ping each 10 seconds. SLA RTT is set to maximum 10 ms. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Docker image. | +| | (For example also a Cirros image can be downloaded from | +| | cirros-image_, it includes ping) | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping man page | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test case can be configured with different packet sizes, | +| | burst sizes, ping intervals and test duration. | +| | SLA is optional. The SLA in this test case serves as an | +| | example. Considerably lower RTT is expected, and | +| | also normal to achieve in balanced L2 environments. However, | +| | to cover most configurations, both bare metal and fully | +| | virtualized ones, this value should be possible to achieve | +| | and acceptable for black box testing. Many real time | +| | applications start to suffer badly if the RTT time is higher | +| | than this. Some may suffer bad also close to this RTT, while | +| | others may not suffer at all. It is a compromise that may | +| | have to be tuned for different configuration purposes. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with ping included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. Ping is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any RTT is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc005.rst b/docs/userguide/opnfv_yardstick_tc005.rst new file mode 100644 index 000000000..a181aa9f7 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc005.rst @@ -0,0 +1,77 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC005 +************************************* + +.. _fio: http://www.bluestop.org/fio/HOWTO.txt + ++-----------------------------------------------------------------------------+ +|Storage Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC005_Storage Performance | +| | | ++--------------+--------------------------------------------------------------+ +|metric | IOPS, throughput and latency | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS storage performance with regards to | +| | IOPS, throughput and latency. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons | +| | and product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc005.yaml | +| | | +| | IO types: read, write, randwrite, randread, rw | +| | IO block size: 4KB, 64KB, 1024KB, where each | +| | runs for 30 seconds(10 for ramp time, 20 for runtime). | +| | | +| | For SLA minimum read/write iops is set to 100, minimum | +| | read/write throughput is set to 400 KB/s, and maximum | +| | read/write latency is set to 20000 usec. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | fio | +| | | +| | (fio is not always part of a Linux distribution, hence it | +| | needs to be installed. As an example see the | +| | /yardstick/tools/ directory for how to generate a Linux | +| | image with fio included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | fio_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different read/write types, IO | +| | block size, IO depth, ramp time (runtime required for stable | +| | results) and test duration. Default values exist. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with fio included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed and fio is invoked and logs are | +| | produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc006.rst b/docs/userguide/opnfv_yardstick_tc006.rst new file mode 100644 index 000000000..2ccb417c1 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc006.rst @@ -0,0 +1,144 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + +************************************* +Yardstick Test Case Description TC006 +************************************* + +.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ +.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC006_Virtual Traffic Classifier Data Plane | +| | Throughput Benchmarking Test. | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To measure the throughput supported by the virtual Traffic | +| | Classifier according to the RFC2544 methodology for a | +| | user-defined set of vTC deployment configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: file: opnfv_yardstick_tc006.yaml | +| | | +| | packet_size: size of the packets to be used during the | +| | throughput calculation. | +| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | +| | | +| | vnic_type: type of VNIC to be used. | +| | Allowed values are: | +| | - normal: for default OvS port configuration | +| | - direct: for SR-IOV port configuration | +| | Default value: None | +| | | +| | vtc_flavor: OpenStack flavor to be used for the vTC | +| | Default available values are: m1.small, m1.medium, | +| | and m1.large, but the user can create his/her own | +| | flavor and give it as input | +| | Default value: None | +| | | +| | vlan_sender: vlan tag of the network on which the vTC will | +| | receive traffic (VLAN Network 1). | +| | Allowed values: range (1, 4096) | +| | | +| | vlan_receiver: vlan tag of the network on which the vTC | +| | will send traffic back to the packet generator | +| | (VLAN Network 2). | +| | Allowed values: range (1, 4096) | +| | | +| | default_net_name: neutron name of the defaul network that | +| | is used for access to the internet from the vTC | +| | (vNIC 1). | +| | | +| | default_subnet_name: subnet name for vNIC1 | +| | (information available through Neutron). | +| | | +| | vlan_net_1_name: Neutron Name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_net_2_name: Neutron Name for VLAN Network 2 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | +| | (information available through Neutron). | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | DPDK pktgen | +| | | +| | DPDK Pktgen is not part of a Linux distribution, | +| | hence it needs to be installed by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|references | DPDK Pktgen: DPDKpktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | +| | RFC 2544: rfc2544_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different flavors, vNIC type | +| | and packet sizes. Default values exist as specified above. | +| | The vNIC type and flavor MUST be specified by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The vTC has been successfully instantiated and configured. | +| | The user has correctly assigned the values to the deployment | +| | configuration parameters. | +| | | +| | - Multicast traffic MUST be enabled on the network. | +| | The Data network switches need to be configured in | +| | order to manage multicast traffic. | +| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | +| | must be used on the compute node. | +| | - Yarsdtick needs to be installed on a host connected to the | +| | data network and the host must have 2 DPDK-compatible | +| | NICs. Proper configuration of DPDK and DPDK pktgen is | +| | required before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected results | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The vTC is deployed, according to the user-defined | +| | configuration | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | The vTC is correctly deployed and configured as necessary | +| | The initialization script has been correctly executed and | +| | vTC is ready to receive and process the traffic. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Test case is executed with the selected parameters: | +| | - vTC flavor | +| | - vNIC type | +| | - packet size | +| | The traffic is sent to the vTC using the maximum available | +| | traffic rate for 60 seconds. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | The vTC instance forwards all the packets back to the packet | +| | generator for 60 seconds, as specified by RFC 2544. | +| | | +| | Steps 3 and 4 are executed different times, with different | +| | rates in order to find the maximum supported traffic rate | +| | according to the current definition of throughput in RFC | +| | 2544. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The result of the test is a number between 0 and 100 which | +| | represents the throughput in terms of percentage of the | +| | available pktgen NIC bandwidth. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc007.rst b/docs/userguide/opnfv_yardstick_tc007.rst new file mode 100644 index 000000000..87663f816 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc007.rst @@ -0,0 +1,162 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + +************************************* +Yardstick Test Case Description TC007 +************************************* + +.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ +.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC007_Virtual Traffic Classifier Data Plane | +| | Throughput Benchmarking Test in Presence of Noisy | +| | neighbours | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To measure the throughput supported by the virtual Traffic | +| | Classifier according to the RFC2544 methodology for a | +| | user-defined set of vTC deployment configurations in the | +| | presence of noisy neighbours. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc007.yaml | +| | | +| | packet_size: size of the packets to be used during the | +| | throughput calculation. | +| | Allowe values: [64, 128, 256, 512, 1024, 1280, 1518] | +| | | +| | vnic_type: type of VNIC to be used. | +| | Allowed values are: | +| | - normal: for default OvS port configuration | +| | - direct: for SR-IOV port configuration | +| | | +| | vtc_flavor: OpenStack flavor to be used for the vTC | +| | Default available values are: m1.small, m1.medium, | +| | and m1.large, but the user can create his/her own | +| | flavor and give it as input | +| | | +| | num_of_neighbours: Number of noisy neighbours (VMs) to be | +| | instantiated during the experiment. | +| | Allowed values: range (1, 10) | +| | | +| | amount_of_ram: RAM to be used by each neighbor. | +| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | +| | '6G', '7G', '8G', '9G', '10G'] | +| | Deault value: 256M | +| | | +| | number_of_cores: Number of noisy neighbours (VMs) to be | +| | instantiated during the experiment. | +| | Allowed values: range (1, 10) | +| | Default value: 1 | +| | | +| | vlan_sender: vlan tag of the network on which the vTC will | +| | receive traffic (VLAN Network 1). | +| | Allowed values: range (1, 4096) | +| | | +| | vlan_receiver: vlan tag of the network on which the vTC | +| | will send traffic back to the packet generator | +| | (VLAN Network 2). | +| | Allowed values: range (1, 4096) | +| | | +| | default_net_name: neutron name of the defaul network that | +| | is used for access to the internet from the vTC | +| | (vNIC 1). | +| | | +| | default_subnet_name: subnet name for vNIC1 | +| | (information available through Neutron). | +| | | +| | vlan_net_1_name: Neutron Name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_net_2_name: Neutron Name for VLAN Network 2 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | +| | (information available through Neutron). | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | DPDK pktgen | +| | | +| | DPDK Pktgen is not part of a Linux distribution, | +| | hence it needs to be installed by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|references | DPDKpktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | +| | rfc2544_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different flavors, vNIC type | +| | and packet sizes. Default values exist as specified above. | +| | The vNIC type and flavor MUST be specified by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The vTC has been successfully instantiated and configured. | +| | The user has correctly assigned the values to the deployment | +| | configuration parameters. | +| | | +| | - Multicast traffic MUST be enabled on the network. | +| | The Data network switches need to be configured in | +| | order to manage multicast traffic. | +| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | +| | must be used on the compute node. | +| | - Yarsdtick needs to be installed on a host connected to the | +| | data network and the host must have 2 DPDK-compatible | +| | NICs. Proper configuration of DPDK and DPDK pktgen is | +| | required before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected results | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The noisy neighbours are deployed as required by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | The vTC is deployed, according to the configuration required | +| | by the user | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | The vTC is correctly deployed and configured as necessary. | +| | The initialization script has been correctly executed and | +| | the vTC is ready to receive and process the traffic. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Test case is executed with the parameters specified by the | +| | user: | +| | - vTC flavor | +| | - vNIC type | +| | - packet size | +| | The traffic is sent to the vTC using the maximum available | +| | traffic rate | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | The vTC instance forwards all the packets back to the | +| | packet generator for 60 seconds, as specified by RFC 2544. | +| | | +| | Steps 4 and 5 are executed different times with different | +| | with different traffic rates, in order to find the maximum | +| | supported traffic rate, accoring to the current definition | +| | of throughput in RFC 2544. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The result of the test is a number between 0 and 100 which | +| | represents the throughput in terms of percentage of the | +| | available pktgen NIC bandwidth. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc008.rst b/docs/userguide/opnfv_yardstick_tc008.rst new file mode 100644 index 000000000..a4ecaf6ae --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc008.rst @@ -0,0 +1,90 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC008 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Packet Loss Extended Test | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC008_NW PERF, Packet loss Extended Test | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, packet size and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of packet sizes and flows matter for the throughput between | +| | VMs on different compute blades. Typically e.g. the | +| | performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc008.yaml | +| | | +| | Packet size: 64, 128, 256, 512, 1024, 1280 and 1518 bytes. | +| | | +| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | +| | configured ports map from 2 up to 1001000 flows, | +| | respectively. Each packet_size/port_amount combination is run| +| | ten times, for 20 seconds each. Then the next | +| | packet_size/port_amount combination is run, and so on. | +| | | +| | The client and server are distributed on different HW. | +| | | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc009.rst b/docs/userguide/opnfv_yardstick_tc009.rst new file mode 100644 index 000000000..d6f445361 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc009.rst @@ -0,0 +1,89 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC009 +************************************* + +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Packet Loss | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC009_NW PERF, Packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, packets lost and throughput | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between VMs on different | +| | compute blades. | +| | Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc009.yaml | +| | | +| | Packet size: 64 bytes | +| | | +| | Number of ports: 1, 10, 50, 100, 500 and 1000. The amount of | +| | configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run ten times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | | +| | The client and server are distributed on different HW. | +| | | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc010.rst b/docs/userguide/opnfv_yardstick_tc010.rst new file mode 100644 index 000000000..ab793de76 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc010.rst @@ -0,0 +1,82 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC010 +************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html + ++-----------------------------------------------------------------------------+ +|Memory Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC010_Memory Latency | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Latency in nanoseconds | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | Measure the memory read latency for varying memory sizes and | +| | strides. Whole memory hierarchy is measured including all | +| | levels of cache. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | File: opnfv_yardstick_tc010.yaml | +| | | +| | * SLA (max_latency): 30 nanoseconds | +| | * Stride - 128 bytes | +| | * Stop size - 64 megabytes | +| | * Iterations: 10 - test is run 10 times iteratively. | +| | * Interval: 1 - there is 1 second delay between each | +| | iteration. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Lmbench | +| | | +| | Lmbench is a suite of operating system microbenchmarks. This | +| | test uses lat_mem_rd tool from that suite. | +| | Lmbench is not always part of a Linux distribution, hence it | +| | needs to be installed in the test image | +| | | ++--------------+--------------------------------------------------------------+ +|references | man-pages_ | +| | | +| | McVoy, Larry W.,and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference 1996. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different: | +| | | +| | * strides; | +| | * stop_size; | +| | * iterations and intervals. | +| | | +| | There are default values for each above-mentioned option. | +| | | +| | SLA (optional) : max_latency: The maximum memory latency | +| | that is accepted. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with Lmbench included in the image. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's lat_mem_rd tool | +| | is invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test fails if the measured memory latency is above the SLA | +| | value or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc011.rst b/docs/userguide/opnfv_yardstick_tc011.rst new file mode 100644 index 000000000..1c643cd72 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc011.rst @@ -0,0 +1,88 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC011 +************************************* + +.. _iperf3: https://iperf.fr/ + ++-----------------------------------------------------------------------------+ +|Packet delay variation between VMs | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC011_Packet delay variation between VMs | +| | | ++--------------+--------------------------------------------------------------+ +|metric | jitter: packet delay variation (ms) | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | Measure the packet delay variation sending the packets from | +| | one VM to the other. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | File: opnfv_yardstick_tc011.yaml | +| | | +| | * options: | +| | protocol: udp # The protocol used by iperf3 tools | +| | bandwidth: 20m # It will send the given number of packets | +| | without pausing | +| | * runner: | +| | duration: 30 # Total test duration 30 seconds. | +| | | +| | * SLA (optional): | +| | jitter: 10 (ms) # The maximum amount of jitter that is | +| | accepted. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | iperf3 | +| | | +| | iPerf3 is a tool for active measurements of the maximum | +| | achievable bandwidth on IP networks. It supports tuning of | +| | various parameters related to timing, buffers and protocols. | +| | The UDP protocols can be used to measure jitter delay. | +| | | +| | (iperf3 is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Docker | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | iperf3_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different | +| | | +| | * bandwidth: Test case can be configured with different | +| | bandwidth | +| | | +| | * duration: The test duration can be configured | +| | | +| | * jitter: SLA is optional. The SLA in this test case | +| | serves as an example. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with iperf3 included in the image. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. iperf3 is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any jitter is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc012.rst b/docs/userguide/opnfv_yardstick_tc012.rst new file mode 100644 index 000000000..ffce06eb9 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc012.rst @@ -0,0 +1,86 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC012 +************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html + ++-----------------------------------------------------------------------------+ +|Memory Bandwidth | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC012_Memory Bandwidth | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Megabyte per second (MBps) | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | Measure the rate at which data can be read from and written | +| | to the memory (this includes all levels of memory). | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | File: opnfv_yardstick_tc012.yaml | +| | | +| | * SLA (optional): 15000 (MBps) min_bw: The minimum amount of | +| | memory bandwidth that is accepted. | +| | * Size: 10 240 kB - test allocates twice that size (20 480kB)| +| | zeros it and then measures the time it takes to copy from | +| | one side to another. | +| | * Benchmark: rdwr - measures the time to read data into | +| | memory and then write data to the same location. | +| | * Warmup: 0 - the number of iterations to perform before | +| | taking actual measurements. | +| | * Iterations: 10 - test is run 10 times iteratively. | +| | * Interval: 1 - there is 1 second delay between each | +| | iteration. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Lmbench | +| | | +| | Lmbench is a suite of operating system microbenchmarks. This | +| | test uses bw_mem tool from that suite. | +| | Lmbench is not always part of a Linux distribution, hence it | +| | needs to be installed in the test image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | man-pages_ | +| | | +| | McVoy, Larry W., and Carl Staelin. "lmbench: Portable Tools | +| | for Performance Analysis." USENIX annual technical | +| | conference. 1996. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different: | +| | | +| | * memory sizes; | +| | * memory operations (such as rd, wr, rdwr, cp, frd, fwr, | +| | fcp, bzero, bcopy); | +| | * number of warmup iterations; | +| | * iterations and intervals. | +| | | +| | There are default values for each above-mentioned option. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with Lmbench included in the image. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed as client. Lmbench's bw_mem tool is | +| | invoked and logs are produced and stored. | +| | | +| | Result: logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test fails if the measured memory bandwidth is below the SLA | +| | value or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc014.rst b/docs/userguide/opnfv_yardstick_tc014.rst new file mode 100644 index 000000000..27d390ac6 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc014.rst @@ -0,0 +1,74 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC014 +************************************* + +.. _unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench + ++-----------------------------------------------------------------------------+ +|Processing speed | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC014_Processing speed | +| | | ++--------------+--------------------------------------------------------------+ +|metric | score of single cpu running, score of parallel running | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS processing speed with regards to score | +| | of single cpu running and parallel running | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons | +| | and product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc014.yaml | +| | | +| | run_mode: Run unixbench in quiet mode or verbose mode | +| | test_type: dhry2reg, whetstone and so on | +| | | +| | For SLA with single_score and parallel_score, both can be | +| | set by user, default is NA | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | unixbench | +| | | +| | (unixbench is not always part of a Linux distribution, hence | +| | it needs to be installed. As an example see the | +| | /yardstick/tools/ directory for how to generate a Linux | +| | image with unixbench included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | unixbench_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different test types, dhry2reg, | +| | whetstone and so on. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with unixbench included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as a client. unixbench is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc019.rst b/docs/userguide/opnfv_yardstick_tc019.rst new file mode 100644 index 000000000..1af502253 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc019.rst @@ -0,0 +1,134 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC019 +************************************* + ++-----------------------------------------------------------------------------+ +|Control Node Openstack Service High Availability | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC019_HA: Control node Openstack service down| +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of the | +| | service provided by OpenStack (like nova-api, neutro-server) | +| | on control node. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of a specific Openstack | +| | service on a selected control node, then checks whether the | +| | request of the related Openstack command is OK and the killed| +| | processes are recovered. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the specified | +| | OpenStack service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | 3) host: which is the name of a control node being attacked. | +| | | +| | e.g. | +| | -fault_type: "kill-process" | +| | -process_name: "nova-api" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, two kinds of monitor are needed: | +| | 1. the "openstack-cmd" monitor constantly request a specific | +| | Openstack command, which needs two parameters: | +| | 1) monitor_type: which is used for finding the monitor class | +| | and related scritps. It should be always set to | +| | "openstack-cmd" for this monitor. | +| | 2) command_name: which is the command name used for request | +| | | +| | 2. the "process" monitor check whether a process is running | +| | on a specific node, which needs three parameters: | +| | 1) monitor_type: which used for finding the monitor class and| +| | related scritps. It should be always set to "process" | +| | for this monitor. | +| | 2) process_name: which is the process name for monitor | +| | 3) host: which is the name of the node runing the process | +| | | +| | e.g. | +| | monitor1: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "nova image-list" | +| | monitor2: | +| | -monitor_type: "process" | +| | -process_name: "nova-api" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1)service_outage_time: which indicates the maximum outage | +| | time (seconds) of the specified Openstack command request. | +| | 2)process_recover_time: which indicates the maximun time | +| | (seconds) from the process being killed to recovered | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc019.yaml | +| | -Attackers: see above "attackers" discription | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to stoping monitors the monitors | +| | -Monitors: see above "monitors" discription | +| | -SLA: see above "metrics" discription | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | stop monitors after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | verify the SLA | +| | | +| | Result: The test case is passed or not. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the specified process on the host, and restart the | +| | process if it is not running for next test cases | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc020.rst b/docs/userguide/opnfv_yardstick_tc020.rst new file mode 100644 index 000000000..f2f1d408b --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc020.rst @@ -0,0 +1,141 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + +************************************* +Yardstick Test Case Description TC020 +************************************* + +.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ +.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC0020_Virtual Traffic Classifier | +| | Instantiation Test | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Failure | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To verify that a newly instantiated vTC is 'alive' and | +| | functional and its instantiation is correctly supported by | +| | the infrastructure. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc020.yaml | +| | | +| | vnic_type: type of VNIC to be used. | +| | Allowed values are: | +| | - normal: for default OvS port configuration | +| | - direct: for SR-IOV port configuration | +| | Default value: None | +| | | +| | vtc_flavor: OpenStack flavor to be used for the vTC | +| | Default available values are: m1.small, m1.medium, | +| | and m1.large, but the user can create his/her own | +| | flavor and give it as input | +| | Default value: None | +| | | +| | vlan_sender: vlan tag of the network on which the vTC will | +| | receive traffic (VLAN Network 1). | +| | Allowed values: range (1, 4096) | +| | | +| | vlan_receiver: vlan tag of the network on which the vTC | +| | will send traffic back to the packet generator | +| | (VLAN Network 2). | +| | Allowed values: range (1, 4096) | +| | | +| | default_net_name: neutron name of the defaul network that | +| | is used for access to the internet from the vTC | +| | (vNIC 1). | +| | | +| | default_subnet_name: subnet name for vNIC1 | +| | (information available through Neutron). | +| | | +| | vlan_net_1_name: Neutron Name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_net_2_name: Neutron Name for VLAN Network 2 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | +| | (information available through Neutron). | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | DPDK pktgen | +| | | +| | DPDK Pktgen is not part of a Linux distribution, | +| | hence it needs to be installed by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|references | DPDKpktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | +| | rfc2544_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different flavors, vNIC type | +| | and packet sizes. Default values exist as specified above. | +| | The vNIC type and flavor MUST be specified by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The vTC has been successfully instantiated and configured. | +| | The user has correctly assigned the values to the deployment | +| | configuration parameters. | +| | | +| | - Multicast traffic MUST be enabled on the network. | +| | The Data network switches need to be configured in | +| | order to manage multicast traffic. | +| | Installation and configuration of smcroute is required | +| | before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | +| | must be used on the compute node. | +| | - Yarsdtick needs to be installed on a host connected to the | +| | data network and the host must have 2 DPDK-compatible | +| | NICs. Proper configuration of DPDK and DPDK pktgen is | +| | required before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected results | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The vTC is deployed, according to the configuration provided | +| | by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | The vTC is correctly deployed and configured as necessary. | +| | The initialization script has been correctly executed and | +| | the vTC is ready to receive and process the traffic. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Test case is executed with the parameters specified by the | +| | the user: | +| | - vTC flavor | +| | - vNIC type | +| | A constant rate traffic is sent to the vTC for 10 seconds. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | The vTC instance tags all the packets and sends them back to | +| | the packet generator for 10 seconds. | +| | | +| | The framework checks that the packet generator receives | +| | back all the packets with the correct tag from the vTC. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The vTC is deemed to be successfully instantiated if all | +| | packets are sent back with the right tag as requested, | +| | else it is deemed DoA (Dead on arrival) | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc021.rst b/docs/userguide/opnfv_yardstick_tc021.rst new file mode 100644 index 000000000..c7adc870a --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc021.rst @@ -0,0 +1,157 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation and others. + +************************************* +Yardstick Test Case Description TC021 +************************************* + +.. _DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ +.. _rfc2544: https://www.ietf.org/rfc/rfc2544.txt + ++-----------------------------------------------------------------------------+ +|Network Performance | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC0021_Virtual Traffic Classifier | +| | Instantiation Test in Presence of Noisy Neighbours | +| | | ++--------------+--------------------------------------------------------------+ +|metric | Failure | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To verify that a newly instantiated vTC is 'alive' and | +| | functional and its instantiation is correctly supported by | +| | the infrastructure in the presence of noisy neighbours. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc021.yaml | +| | | +| | vnic_type: type of VNIC to be used. | +| | Allowed values are: | +| | - normal: for default OvS port configuration | +| | - direct: for SR-IOV port configuration | +| | Default value: None | +| | | +| | vtc_flavor: OpenStack flavor to be used for the vTC | +| | Default available values are: m1.small, m1.medium, | +| | and m1.large, but the user can create his/her own | +| | flavor and give it as input | +| | Default value: None | +| | | +| | num_of_neighbours: Number of noisy neighbours (VMs) to be | +| | instantiated during the experiment. | +| | Allowed values: range (1, 10) | +| | | +| | amount_of_ram: RAM to be used by each neighbor. | +| | Allowed values: ['250M', '1G', '2G', '3G', '4G', '5G', | +| | '6G', '7G', '8G', '9G', '10G'] | +| | Deault value: 256M | +| | | +| | number_of_cores: Number of noisy neighbours (VMs) to be | +| | instantiated during the experiment. | +| | Allowed values: range (1, 10) | +| | Default value: 1 | +| | | +| | vlan_sender: vlan tag of the network on which the vTC will | +| | receive traffic (VLAN Network 1). | +| | Allowed values: range (1, 4096) | +| | | +| | vlan_receiver: vlan tag of the network on which the vTC | +| | will send traffic back to the packet generator | +| | (VLAN Network 2). | +| | Allowed values: range (1, 4096) | +| | | +| | default_net_name: neutron name of the defaul network that | +| | is used for access to the internet from the vTC | +| | (vNIC 1). | +| | | +| | default_subnet_name: subnet name for vNIC1 | +| | (information available through Neutron). | +| | | +| | vlan_net_1_name: Neutron Name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_1_name: Subnet Neutron name for VLAN Network 1 | +| | (information available through Neutron). | +| | | +| | vlan_net_2_name: Neutron Name for VLAN Network 2 | +| | (information available through Neutron). | +| | | +| | vlan_subnet_2_name: Subnet Neutron name for VLAN Network 2 | +| | (information available through Neutron). | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | DPDK pktgen | +| | | +| | DPDK Pktgen is not part of a Linux distribution, | +| | hence it needs to be installed by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|references | DPDK Pktgen: DPDK Pktgen: DPDKpktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | +| | RFC 2544: rfc2544_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different flavors, vNIC type | +| | and packet sizes. Default values exist as specified above. | +| | The vNIC type and flavor MUST be specified by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The vTC has been successfully instantiated and configured. | +| | The user has correctly assigned the values to the deployment | +| | configuration parameters. | +| | | +| | - Multicast traffic MUST be enabled on the network. | +| | The Data network switches need to be configured in | +| | order to manage multicast traffic. | +| | Installation and configuration of smcroute is required | +| | before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | - In the case of SR-IOV vNICs use, SR-IOV compatible NICs | +| | must be used on the compute node. | +| | - Yarsdtick needs to be installed on a host connected to the | +| | data network and the host must have 2 DPDK-compatible | +| | NICs. Proper configuration of DPDK and DPDK pktgen is | +| | required before to run the test case. | +| | (For further instructions please refer to the ApexLake | +| | documentation). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected results | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The noisy neighbours are deployed as required by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | The vTC is deployed, according to the configuration provided | +| | by the user. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | The vTC is correctly deployed and configured as necessary. | +| | The initialization script has been correctly executed and | +| | the vTC is ready to receive and process the traffic. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Test case is executed with the selected parameters: | +| | - vTC flavor | +| | - vNIC type | +| | A constant rate traffic is sent to the vTC for 10 seconds. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | The vTC instance tags all the packets and sends them back to | +| | the packet generator for 10 seconds. | +| | | +| | The framework checks if the packet generator receives back | +| | all the packets with the correct tag from the vTC. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The vTC is deemed to be successfully instantiated if all | +| | packets are sent back with the right tag as requested, | +| | else it is deemed DoA (Dead on arrival) | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc024.rst b/docs/userguide/opnfv_yardstick_tc024.rst new file mode 100644 index 000000000..ffdacb106 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc024.rst @@ -0,0 +1,69 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC024 +************************************* + +.. _man-pages: http://manpages.ubuntu.com/manpages/trusty/man1/mpstat.1.html + ++-----------------------------------------------------------------------------+ +| CPU Load | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC024_CPU Load | +| | | ++--------------+--------------------------------------------------------------+ +|metric | CPU load | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the CPU load performance of the IaaS. This test | +| | case should be run in parallel to other Yardstick test cases | +| | and not run as a stand-alone test case. | +| | | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: cpuload.yaml (in the 'samples' directory) | +| | | +| | There is are no additional configurations to be set for this | +| | TC. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | mpstat | +| | | +| | (mpstat is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. However, if mpstat is not present the TC instead uses | +| | /proc/stats as source to produce "mpstat" output. | +| | | ++--------------+--------------------------------------------------------------+ +|references | man-pages_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Run in background with other test cases. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with mpstat included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host is installed. The related TC, or TCs, is | +| | invoked and mpstat logs are produced and stored. | +| | | +| | Result: Stored logs | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | None. CPU load results are fetched and stored. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc025.rst b/docs/userguide/opnfv_yardstick_tc025.rst new file mode 100644 index 000000000..0e2e9a5f8 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc025.rst @@ -0,0 +1,123 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC025 +************************************* + ++-----------------------------------------------------------------------------+ +|OpenStack Controller Node abnormally shutdown High Availability | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC025_HA: OpenStack Controller Node | +| | abnormally shutdown | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of | +| | controller node. When one of the controller node abnormally | +| | shutdown, the service provided by it should be OK. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case shutdowns a specified controller node with | +| | some fault injection tools, then checks whether all services | +| | provided by the controller node are OK with some monitor | +| | tools. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "host-shutdown" is | +| | needed. This attacker includes two parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "host-shutdown" in | +| | this test case. | +| | 2) host: the name of a controller node being attacked. | +| | | +| | e.g. | +| | -fault_type: "host-shutdown" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, one kind of monitor are needed: | +| | 1. the "openstack-cmd" monitor constantly request a specific | +| | Openstack command, which needs two parameters | +| | 1) monitor_type: which is used for finding the monitor class | +| | and related scritps. It should be always set to | +| | "openstack-cmd" for this monitor. | +| | 2) command_name: which is the command name used for request | +| | | +| | There are four instance of the "openstack-cmd" monitor: | +| | monitor1: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "nova image-list" | +| | monitor2: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "neutron router-list" | +| | monitor3: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "heat stack-list" | +| | monitor4: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "cinder list" | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there is one metric: | +| | 1)service_outage_time: which indicates the maximum outage | +| | time (seconds) of the specified Openstack command request. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc019.yaml | +| | -Attackers: see above "attackers" discription | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to stoping monitors the monitors | +| | -Monitors: see above "monitors" discription | +| | -SLA: see above "metrics" discription | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | do attacker: connect the host through SSH, and then execute | +| | shutdown script on the host | +| | | +| | Result: The host will be shutdown. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | stop monitors after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: All monitor result will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | verify the SLA | +| | | +| | Result: The test case is passed or not. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It restarts the | +| | specified controller node if it is not restarted. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc027.rst b/docs/userguide/opnfv_yardstick_tc027.rst new file mode 100644 index 000000000..6215e6d2a --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc027.rst @@ -0,0 +1,72 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC027 +************************************* + +.. _ipv6: https://wiki.opnfv.org/ipv6_opnfv_project + ++-----------------------------------------------------------------------------+ +|IPv6 connectivity between nodes on the tenant network | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC027_IPv6 connectivity | +| | | ++--------------+--------------------------------------------------------------+ +|metric | RTT, Round Trip Time | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To do a basic verification that IPv6 connectivity is within | +| | acceptable boundaries when ipv6 packets travel between hosts | +| | located on same or different compute blades. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc027.yaml | +| | | +| | Packet size 56 bytes. | +| | SLA RTT is set to maximum 10 ms. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | ping6 | +| | | +| | Ping6 is normally part of Linux distribution, hence it | +| | doesn't need to be installed. | +| | | ++--------------+--------------------------------------------------------------+ +|references | ipv6_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test case can be configured with different run step | +| | you can run setup, run benchmakr, teardown independently | +| | SLA is optional. The SLA in this test case serves as an | +| | example. Considerably lower RTT is expected. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with ping6 included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. Ping is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any RTT is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc028.rst b/docs/userguide/opnfv_yardstick_tc028.rst new file mode 100644 index 000000000..24206f33f --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc028.rst @@ -0,0 +1,70 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co., Ltd and others. + +************************************* +Yardstick Test Case Description TC028 +************************************* + +.. _Cyclictest: https://rt.wiki.kernel.org/index.php/Cyclictest + ++-----------------------------------------------------------------------------+ +|KVM Latency measurements | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC028_KVM Latency measurements | +| | | ++--------------+--------------------------------------------------------------+ +|metric | min, avg and max latency | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS KVM virtualization capability with | +| | regards to min, avg and max latency. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs and similar shall be stored for comparison reasons | +| | and product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: samples/cyclictest-node-context.yaml | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Cyclictest | +| | | +| | (Cyclictest is not always part of a Linux distribution, | +| | hence it needs to be installed. As an example see the | +| | /yardstick/tools/ directory for how to generate a Linux | +| | image with cyclictest included.) | +| | | ++--------------+--------------------------------------------------------------+ +|references | Cyclictest_ | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | This test case is mainly for kvm4nfv project CI verify. | +| | Upgrade host linux kernel, boot a gust vm update it's linux | +| | kernel, and then run the cyclictest to test the new kernel | +| | is work well. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test kernel rpm, test sequence scripts and test guest | +|conditions | image need put the right folders as specified in the test | +| | case yaml file. | +| | The test guest image needs with cyclictest included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The host and guest os kernel is upgraded. Cyclictest is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc037.rst b/docs/userguide/opnfv_yardstick_tc037.rst new file mode 100644 index 000000000..3ed1fa529 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc037.rst @@ -0,0 +1,104 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC037 +************************************* + +.. _cirros: https://download.cirros-cloud.net +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Latency, CPU Load, Throughput, Packet Loss | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC037_Latency,CPU Load,Throughput,Packet Loss| +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, latency, throughput, CPU load, packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc037.yaml | +| | | +| | Packet size: 64 bytes | +| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | +| | The amount configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run two times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | During the test CPU load on both client and server, and the | +| | network latency between the client and server are measured. | +| | The client and server are distributed on different HW. | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | +| | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Glance image. | +| | (For example also a cirros_ image can be downloaded, it | +| | includes ping) | +| | | +| | mpstat | +| | | +| | (Mpstat is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping and Mpstat man pages | +| | | +| | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc038.rst b/docs/userguide/opnfv_yardstick_tc038.rst new file mode 100644 index 000000000..692c76819 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc038.rst @@ -0,0 +1,104 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TC038 +************************************* + +.. _cirros: https://download.cirros-cloud.net +.. _pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt + ++-----------------------------------------------------------------------------+ +|Latency, CPU Load, Throughput, Packet Loss (Extended measurements) | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC038_Latency,CPU Load,Throughput,Packet Loss| +| | | ++--------------+--------------------------------------------------------------+ +|metric | Number of flows, latency, throughput, CPU load, packet loss | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | To evaluate the IaaS network performance with regards to | +| | flows and throughput, such as if and how different amounts | +| | of flows matter for the throughput between hosts on different| +| | compute blades. Typically e.g. the performance of a vSwitch | +| | depends on the number of flows running through it. Also | +| | performance of other equipment or entities can depend | +| | on the number of flows or the packet sizes used. | +| | The purpose is also to be able to spot trends. Test results, | +| | graphs ans similar shall be stored for comparison reasons and| +| | product evolution understanding between different OPNFV | +| | versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc038.yaml | +| | | +| | Packet size: 64 bytes | +| | Number of ports: 1, 10, 50, 100, 300, 500, 750 and 1000. | +| | The amount configured ports map from 2 up to 1001000 flows, | +| | respectively. Each port amount is run ten times, for 20 | +| | seconds each. Then the next port_amount is run, and so on. | +| | During the test CPU load on both client and server, and the | +| | network latency between the client and server are measured. | +| | The client and server are distributed on different HW. | +| | For SLA max_ppm is set to 1000. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | pktgen | +| | | +| | (Pktgen is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | As an example see the /yardstick/tools/ directory for how | +| | to generate a Linux image with pktgen included.) | +| | | +| | ping | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Glance image. | +| | (For example also a cirros_ image can be downloaded, it | +| | includes ping) | +| | | +| | mpstat | +| | | +| | (Mpstat is not always part of a Linux distribution, hence it | +| | needs to be installed. It is part of the Yardstick Glance | +| | image. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping and Mpstat man pages | +| | | +| | pktgen_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different packet sizes, amount | +| | of flows and test duration. Default values exist. | +| | | +| | SLA (optional): max_ppm: The number of packets per million | +| | packets sent that are acceptable to loose, not received. | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case image needs to be installed into Glance | +|conditions | with pktgen included in it. | +| | | +| | No POD specific requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | The hosts are installed, as server and client. pktgen is | +| | invoked and logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/opnfv_yardstick_tc040.rst b/docs/userguide/opnfv_yardstick_tc040.rst new file mode 100644 index 000000000..d62fbf787 --- /dev/null +++ b/docs/userguide/opnfv_yardstick_tc040.rst @@ -0,0 +1,65 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC040 +************************************* + +.. _Parser: https://wiki.opnfv.org/parser + ++-----------------------------------------------------------------------------+ +|Verify Parser Yang-to-Tosca | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC040 Verify Parser Yang-to-Tosca | +| | | ++--------------+--------------------------------------------------------------+ +|metric | 1. tosca file which is converted from yang file by Parser | +| | 2. result whether the output is same with expected outcome | ++--------------+--------------------------------------------------------------+ +|test purpose | To verify the function of Yang-to-Tosca in Parser. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc040.yaml | +| | | +| | yangfile: the path of the yangfile which you want to convert | +| | toscafile: the path of the toscafile which is your expected | +| | outcome. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Parser | +| | | +| | (Parser is not part of a Linux distribution, hence it | +| | needs to be installed. As an example see the | +| | /yardstick/benchmark/scenarios/parser/parser_setup.sh for | +| | how to install it manual. Of course, it will be installed | +| | and uninstalled automatically when you run this test case | +| | by yardstick) | ++--------------+--------------------------------------------------------------+ +|references | Parser_ | +| | | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Test can be configured with different path of yangfile and | +| | toscafile to fit your real environment to verify Parser | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | No POD specific requirements have been identified. | +|conditions | it can be run without VM | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | parser is installed without VM, running Yang-to-Tosca module | +| | to convert yang file to tosca file, validating output against| +| | expected outcome. | +| | | +| | Result: Logs are stored. | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if output is different with expected outcome | +| | or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/references.rst b/docs/userguide/references.rst new file mode 100644 index 000000000..551926135 --- /dev/null +++ b/docs/userguide/references.rst @@ -0,0 +1,50 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +========== +References +========== + + +OPNFV +===== + +* Parser wiki: https://wiki.opnfv.org/parser +* Pharos wiki: https://wiki.opnfv.org/pharos +* VTC: https://wiki.opnfv.org/vtc +* Yardstick CI: https://build.opnfv.org/ci/view/yardstick/ +* Yardstick and ETSI TST001 presentation: https://wiki.opnfv.org/_media/opnfv_summit_-_bridging_opnfv_and_etsi.pdf +* Yardstick Project presentation: https://wiki.opnfv.org/_media/opnfv_summit_-_yardstick_project.pdf +* Yardstick wiki: https://wiki.opnfv.org/yardstick + +References used in Test Cases +============================= + +* cirros-image: https://download.cirros-cloud.net +* cyclictest: https://rt.wiki.kernel.org/index.php/Cyclictest +* DPDKpktgen: https://github.com/Pktgen/Pktgen-DPDK/ +* DPDK supported NICs: http://dpdk.org/doc/nics +* fio: http://www.bluestop.org/fio/HOWTO.txt +* iperf3: https://iperf.fr/ +* Lmbench man-pages: http://manpages.ubuntu.com/manpages/trusty/lat_mem_rd.8.html +* Memory bandwidth man-pages: http://manpages.ubuntu.com/manpages/trusty/bw_mem.8.html +* unixbench: https://github.com/kdlucas/byte-unixbench/blob/master/UnixBench +* mpstat man-pages: http://manpages.ubuntu.com/manpages/trusty/man1/mpstat.1.html +* pktgen: https://www.kernel.org/doc/Documentation/networking/pktgen.txt +* SR-IOV: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking + +Research +======== + +* NCSRD: http://www.demokritos.gr/?lang=en +* T-NOVA: http://www.t-nova.eu/ +* T-NOVA Results: http://www.t-nova.eu/results/ + +Standards +========= + +* ETSI NFV: http://www.etsi.org/technologies-clusters/technologies/nfv +* ETSI GS-NFV TST 001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/ +* RFC2544: https://www.ietf.org/rfc/rfc2544.txt diff --git a/docs/userguide/testcase_description_v2_template.rst b/docs/userguide/testcase_description_v2_template.rst new file mode 100644 index 000000000..91c2a7e33 --- /dev/null +++ b/docs/userguide/testcase_description_v2_template.rst @@ -0,0 +1,64 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson AB and others. + +************************************* +Yardstick Test Case Description TCXXX +************************************* + ++-----------------------------------------------------------------------------+ +|test case slogan e.g. Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | e.g. OPNFV_YARDSTICK_TC001_NW Latency | +| | | ++--------------+--------------------------------------------------------------+ +|metric | what will be measured, e.g. latency | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | describe what is the purpose of the test case | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | what .yaml file to use, state SLA if applicable, state | +| | test duration, list and describe the scenario options used in| +| | this TC and also list the options using default values. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | e.g. ping | +| | | ++--------------+--------------------------------------------------------------+ +|references | e.g. RFCxxx, ETSI-NFVyyy | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | describe variations of the test case which can be | +| | performend, e.g. run the test for different packet sizes | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | describe configuration in the tool(s) used to perform | +|conditions | the measurements (e.g. fio, pktgen), POD-specific | +| | configuration required to enable running the test | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | use this to describe tests that require sveveral steps e.g | +| | collect logs. | +| | | +| | Result: what happens in this step e.g. logs collected | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | remove interface | +| | | +| | Result: interface down. | +| | | ++--------------+--------------------------------------------------------------+ +|step N | what is done in step N | +| | | +| | Result: what happens | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | expected behavior, or SLA, pass/fail criteria | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/userguide/yardstick_framework/index.rst b/docs/userguide/yardstick_framework/index.rst deleted file mode 100644 index f982c30ff..000000000 --- a/docs/userguide/yardstick_framework/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -================================= -Yardstick Framework Documentation -================================= - -.. toctree:: - :numbered: - :maxdepth: 2 - - 03-installation |