diff options
Diffstat (limited to 'docs/requirements/vswitchperf_ltp.rst')
| -rw-r--r-- | docs/requirements/vswitchperf_ltp.rst | 1348 |
1 files changed, 0 insertions, 1348 deletions
diff --git a/docs/requirements/vswitchperf_ltp.rst b/docs/requirements/vswitchperf_ltp.rst deleted file mode 100644 index 2b74d676..00000000 --- a/docs/requirements/vswitchperf_ltp.rst +++ /dev/null @@ -1,1348 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -.. 3.1 - -***************************** -VSPERF LEVEL TEST PLAN (LTP) -***************************** - -=============== -Introduction -=============== - -The objective of the OPNFV project titled -**Characterize vSwitch Performance for Telco NFV Use Cases**, is to -evaluate the performance of virtual switches to identify its suitability for a -Telco Network Function Virtualization (NFV) environment. The intention of this -Level Test Plan (LTP) document is to specify the scope, approach, resources, -and schedule of the virtual switch performance benchmarking activities in -OPNFV. The test cases will be identified in a separate document called the -Level Test Design (LTD) document. - -This document is currently in draft form. - -.. 3.1.1 - - -.. _doc-id: - -Document identifier -========================= - -The document id will be used to uniquely identify versions of the LTP. The -format for the document id will be: OPNFV\_vswitchperf\_LTP\_REL\_STATUS, where -by the status is one of: draft, reviewed, corrected or final. The document id -for this version of the LTP is: OPNFV\_vswitchperf\_LTP\_Colorado\_REVIEWED. - -.. 3.1.2 - -.. _scope: - -Scope -========== - -The main purpose of this project is to specify a suite of -performance tests in order to objectively measure the current packet -transfer characteristics of a virtual switch in the NFVI. The intent of -the project is to facilitate the performance testing of any virtual switch. -Thus, a generic suite of tests shall be developed, with no hard dependencies to -a single implementation. In addition, the test case suite shall be -architecture independent. - -The test cases developed in this project shall not form part of a -separate test framework, all of these tests may be inserted into the -Continuous Integration Test Framework and/or the Platform Functionality -Test Framework - if a vSwitch becomes a standard component of an OPNFV -release. - -.. 3.1.3 - -References -=============== - -* `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ -* `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ -* `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ -* `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ -* `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ -* `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ -* `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ -* `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ - -.. 3.1.4 - -Level in the overall sequence -=============================== -The level of testing conducted by vswitchperf in the overall testing sequence (among -all the testing projects in OPNFV) is the performance benchmarking of a -specific component (the vswitch) in the OPNFV platfrom. It's expected that this -testing will follow on from the functional and integration testing conducted by -other testing projects in OPNFV, namely Functest and Yardstick. - -.. 3.1.5 - -Test classes and overall test conditions -========================================= -A benchmark is defined by the IETF as: A standardized test that serves as a -basis for performance evaluation and comparison. It's important to note that -benchmarks are not Functional tests. They do not provide PASS/FAIL criteria, -and most importantly ARE NOT performed on live networks, or performed with live -network traffic. - -In order to determine the packet transfer characteristics of a virtual switch, -the benchmarking tests will be broken down into the following categories: - -- **Throughput Tests** to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by `RFC1242 <https://www.rfc-editor.org/rfc/rfc1242.txt>`__) - without traffic loss. -- **Packet and Frame Delay Tests** to measure average, min and max - packet and frame delay for constant loads. -- **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the virtual switch. -- **Request/Response Performance** Tests (TCP, UDP) the measure the - transaction rate through the virtual switch. -- **Packet Delay Tests** to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers. -- **Scalability Tests** to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic's configuration... it has to deal with increases. -- **Control Path and Datapath Coupling** Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT. -- **CPU and Memory Consumption Tests** to understand the virtual - switch’s footprint on the system, this includes: - - * CPU core utilization. - * CPU cache utilization. - * Memory footprint. - * System bus (QPI, PCI, ..) utilization. - * Memory lanes utilization. - * CPU cycles consumed per packet. - * Time To Establish Flows Tests. - -- **Noisy Neighbour Tests**, to understand the effects of resource - sharing on the performance of a virtual switch. - -**Note:** some of the tests above can be conducted simultaneously where -the combined results would be insightful, for example Packet/Frame Delay -and Scalability. - - - -.. 3.2 - -.. _details-of-LTP: - -=================================== -Details of the Level Test Plan -=================================== - -This section describes the following items: -* Test items and their identifiers (TestItems_) -* Test Traceability Matrix (TestMatrix_) -* Features to be tested (FeaturesToBeTested_) -* Features not to be tested (FeaturesNotToBeTested_) -* Approach (Approach_) -* Item pass/fail criteria (PassFailCriteria_) -* Suspension criteria and resumption requirements (SuspensionResumptionReqs_) - -.. 3.2.1 - -.. _TestItems: - -Test items and their identifiers -================================== -The test item/application vsperf is trying to test are virtual switches and in -particular their performance in an nfv environment. vsperf will first try to -measure the maximum achievable performance by a virtual switch and then it will -focus in on usecases that are as close to real life deployment scenarios as -possible. - -.. 3.2.2 - -.. _TestMatrix: - -Test Traceability Matrix -========================== -vswitchperf leverages the "3x3" matrix (introduced in -https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02) to achieve test -traceability. The matrix was expanded to 3x4 to accommodate scale metrics when -displaying the coverage of many metrics/benchmarks). Test case covreage in the -LTD is tracked using the following catagories: - - -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| | SPEED | ACCURACY | RELIABILITY | SCALE | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| Activation | X | X | X | X | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| Operation | X | X | X | X | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ -| | | | | | -| De-activation | | | | | -| | | | | | -+---------------+-------------+------------+---------------+-------------+ - -X = denotes a test catagory that has 1 or more test cases defined. - -.. 3.2.3 - -.. _FeaturesToBeTested: - -Features to be tested -========================== - -Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) -includes measuring the following performance metrics: - -- **Throughput** as defined by `RFC1242 - <https://www.rfc-editor.org/rfc/rfc1242.txt>`__: The maximum rate at which - **none** of the offered frames are dropped by the DUT. The maximum frame - rate and bit rate that can be transmitted by the DUT without any error - should be recorded. Note there is an equivalent bit rate and a specific - layer at which the payloads contribute to the bits. Errors and - improperly formed frames or packets are dropped. -- **Packet delay** introduced by the DUT and its cumulative effect on - E2E networks. Frame delay can be measured equivalently. -- **Packet delay variation**: measured from the perspective of the - VNF/application. Packet delay variation is sometimes called "jitter". - However, we will avoid the term "jitter" as the term holds different - meaning to different groups of people. In this document we will - simply use the term packet delay variation. The preferred form for this - metric is the PDV form of delay variation defined in `RFC5481 - <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. The most relevant - measurement of PDV considers the delay variation of a single user flow, - as this will be relevant to the size of end-system buffers to compensate - for delay variation. The measurement system's ability to store the - delays of individual packets in the flow of interest is a key factor - that determines the specific measurement method. At the outset, it is - ideal to view the complete PDV distribution. Systems that can capture - and store packets and their delays have the freedom to calculate the - reference minimum delay and to determine various quantiles of the PDV - distribution accurately (in post-measurement processing routines). - Systems without storage must apply algorithms to calculate delay and - statistical measurements on the fly. For example, a system may store - temporary estimates of the mimimum delay and the set of (100) packets - with the longest delays during measurement (to calculate a high quantile, - and update these sets with new values periodically. - In some cases, a limited number of delay histogram bins will be - available, and the bin limits will need to be set using results from - repeated experiments. See section 8 of `RFC5481 - <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. -- **Packet loss** (within a configured waiting time at the receiver): All - packets sent to the DUT should be accounted for. -- **Burst behaviour**: measures the ability of the DUT to buffer packets. -- **Packet re-ordering**: measures the ability of the device under test to - maintain sending order throughout transfer to the destination. -- **Packet correctness**: packets or Frames must be well-formed, in that - they include all required fields, conform to length requirements, pass - integrity checks, etc. -- **Availability and capacity** of the DUT i.e. when the DUT is fully “up” - and connected, following measurements should be captured for - DUT without any network packet load: - - - Includes average power consumption of the CPUs (in various power states) and - system over specified period of time. Time period should not be less - than 60 seconds. - - Includes average per core CPU utilization over specified period of time. - Time period should not be less than 60 seconds. - - Includes the number of NIC interfaces supported. - - Includes headroom of VM workload processing cores (i.e. available - for applications). - -.. 3.2.4 - -.. _FeaturesNotToBeTested: - -Features not to be tested -========================== -vsperf doesn't intend to define or perform any functional tests. The aim is to -focus on performance. - -.. 3.2.5 - -.. _Approach: - -Approach -============== -The testing approach adoped by the vswitchperf project is black box testing, -meaning the test inputs can be generated and the outputs captured and -completely evaluated from the outside of the System Under Test. Some metrics -can be collected on the SUT, such as cpu or memory utilization if the -collection has no/minimal impact on benchmark. -This section will look at the deployment scenarios and the general methodology -used by vswitchperf. In addition, this section will also specify the details of -the Test Report that must be collected for each of the test cases. - -.. 3.2.5.1 - -Deployment Scenarios --------------------------- -The following represents possible deployment test scenarios which can -help to determine the performance of both the virtual switch and the -datapaths to physical ports (to NICs) and to logical ports (to VNFs): - -.. 3.2.5.1.1 - -.. _Phy2Phy: - -Physical port → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - _ - +--------------------------------------------------+ | - | +--------------------+ | | - | | | | | - | | v | | Host - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.2 - -.. _PVP: - -Physical port → vSwitch → VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ : | | - | | | | | Guest - | : v | | - | +---------------+ +---------------+ | | - | | logical port 0| | logical port 1| | | - +---+---------------+-----------+---------------+---+ _| - ^ : - | | - : v _ - +---+---------------+----------+---------------+---+ | - | | logical port 0| | logical port 1| | | - | +---------------+ +---------------+ | | - | ^ : | | - | | | | | Host - | : v | | - | +--------------+ +--------------+ | | - | | phy port | vSwitch | phy port | | | - +---+--------------+------------+--------------+---+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.3 - -.. _PVVP: - -Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+ _| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+---------+---------------+--+ | - | | 0 1 | | 3 4 | | | - | | logical ports | | logical ports | | | - | +---------------+ +---------------+ | | - | ^ | ^ | | | Host - | | L-----------------+ v | | - | +--------------+ +--------------+ | | - | | phy ports | vSwitch | phy ports | | | - +---+--------------+----------+--------------+---+ _| - ^ ^ : : - | | | | - : : v v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.4 - -Physical port → VNF → vSwitch → VNF → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - |+-------------------+ | | +-------------------+| | - || Application | | | | Application || | - |+-------------------+ | | +-------------------+| | - | ^ | | | ^ | | | Guests - | | v | | | v | | - |+-------------------+ | | +-------------------+| | - || logical ports | | | | logical ports || | - || 0 1 | | | | 0 1 || | - ++--------------------++ ++--------------------++ _| - ^ : ^ : - (PCI passthrough) | | (PCI passthrough) - | v : | _ - +--------++------------+-+------------++---------+ | - | | || 0 | | 1 || | | | - | | ||logical port| |logical port|| | | | - | | |+------------+ +------------+| | | | - | | | | ^ | | | | - | | | L-----------------+ | | | | - | | | | | | | Host - | | | vSwitch | | | | - | | +-----------------------------+ | | | - | | | | | - | | v | | - | +--------------+ +--------------+ | | - | | phy port/VF | | phy port/VF | | | - +-+--------------+--------------+--------------+-+ _| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.5 - -Physical port → vSwitch → VNF -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | ^ | | - | | | | Guest - | : | | - | +---------------+ | | - | | logical port 0| | | - +---+---------------+-------------------------------+ _| - ^ - | - : _ - +---+---------------+------------------------------+ | - | | logical port 0| | | - | +---------------+ | | - | ^ | | - | | | | Host - | : | | - | +--------------+ | | - | | phy port | vSwitch | | - +---+--------------+------------ -------------- ---+ _| - ^ - | - : - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.6 - -VNF → vSwitch → physical port -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +---------------------------------------------------+ | - | | | - | +-------------------------------------------+ | | - | | Application | | | - | +-------------------------------------------+ | | - | : | | - | | | | Guest - | v | | - | +---------------+ | | - | | logical port | | | - +-------------------------------+---------------+---+ _| - : - | - v _ - +------------------------------+---------------+---+ | - | | logical port | | | - | +---------------+ | | - | : | | - | | | | Host - | v | | - | +--------------+ | | - | vSwitch | phy port | | | - +-------------------------------+--------------+---+ _| - : - | - v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -.. 3.2.5.1.7 - -VNF → vSwitch → VNF → vSwitch -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +-------------------------+ +-------------------------+ | - | Guest 1 | | Guest 2 | | - | +-----------------+ | | +-----------------+ | | - | | Application | | | | Application | | | - | +-----------------+ | | +-----------------+ | | - | : | | ^ | | - | | | | | | | Guest - | v | | : | | - | +---------------+ | | +---------------+ | | - | | logical port 0| | | | logical port 0| | | - +-----+---------------+---+ +---+---------------+-----+ _| - : ^ - | | - v : _ - +----+---------------+------------+---------------+-----+ | - | | port 0 | | port 1 | | | - | +---------------+ +---------------+ | | - | : ^ | | - | | | | | Host - | +--------------------+ | | - | | | - | vswitch | | - +-------------------------------------------------------+ _| - -.. 3.2.5.1.8 - -HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port) -→ HOST 2(Physical port → virtual switch → VNF → virtual switch → Physical port) - -HOST 1 (PVP) → HOST 2 (PVP) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - _ - +----------------------+ +----------------------+ | - | Guest 1 | | Guest 2 | | - | +---------------+ | | +---------------+ | | - | | Application | | | | Application | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | - | | v | | | v | | Guests - | +---------------+ | | +---------------+ | | - | | logical ports | | | | logical ports | | | - | | 0 1 | | | | 0 1 | | | - +---+---------------+--+ +---+---------------+--+ _| - ^ : ^ : - | | | | - : v : v _ - +---+---------------+--+ +---+---------------+--+ | - | | 0 1 | | | | 3 4 | | | - | | logical ports | | | | logical ports | | | - | +---------------+ | | +---------------+ | | - | ^ | | | ^ | | | Hosts - | | v | | | v | | - | +--------------+ | | +--------------+ | | - | | phy ports | | | | phy ports | | | - +---+--------------+---+ +---+--------------+---+ _| - ^ : : : - | +-----------------+ | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - - - -**Note:** For tests where the traffic generator and/or measurement -receiver are implemented on VM and connected to the virtual switch -through vNIC, the issues of shared resources and interactions between -the measurement devices and the device under test must be considered. - -**Note:** Some RFC 2889 tests require a full-mesh sending and receiving -pattern involving more than two ports. This possibility is illustrated in the -Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port -diagram above (with 2 sending and 2 receiving ports, though all ports -could be used bi-directionally). - -**Note:** When Deployment Scenarios are used in RFC 2889 address learning -or cache capacity testing, an additional port from the vSwitch must be -connected to the test device. This port is used to listen for flooded -frames. - -.. 3.2.5.2 - -General Methodology: --------------------------- -To establish the baseline performance of the virtual switch, tests would -initially be run with a simple workload in the VNF (the recommended -simple workload VNF would be `DPDK <http://www.dpdk.org/>`__'s testpmd -application forwarding packets in a VM or vloop\_vnf a simple kernel -module that forwards traffic between two network interfaces inside the -virtualized environment while bypassing the networking stack). -Subsequently, the tests would also be executed with a real Telco -workload running in the VNF, which would exercise the virtual switch in -the context of higher level Telco NFV use cases, and prove that its -underlying characteristics and behaviour can be measured and validated. -Suitable real Telco workload VNFs are yet to be identified. - -.. 3.2.5.2.1 - -.. _default-test-parameters: - -Default Test Parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following list identifies the default parameters for suite of -tests: - -- Reference application: Simple forwarding or Open Source VNF. -- Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR - Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as - maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>. -- Reordering check: Tests should confirm that packets within a flow are - not reordered. -- Duplex: Unidirectional / Bidirectional. Default: Full duplex with - traffic transmitting in both directions, as network traffic generally - does not flow in a single direction. By default the data rate of - transmitted traffic should be the same in both directions, please - note that asymmetric traffic (e.g. downlink-heavy) tests will be - mentioned explicitly for the relevant test cases. -- Number of Flows: Default for non scalability tests is a single flow. - For scalability tests the goal is to test with maximum supported - flows but where possible will test up to 10 Million flows. Start with - a single flow and scale up. By default flows should be added - sequentially, tests that add flows simultaneously will explicitly - call out their flow addition behaviour. Packets are generated across - the flows uniformly with no burstiness. For multi-core tests should - consider the number of packet flows based on vSwitch/VNF multi-thread - implementation and behavior. - -- Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic. -- Deployment scenarios are: -- Physical → virtual switch → physical. -- Physical → virtual switch → VNF → virtual switch → physical. -- Physical → virtual switch → VNF → virtual switch → VNF → virtual - switch → physical. -- Physical → VNF → virtual switch → VNF → physical. -- Physical → virtual switch → VNF. -- VNF → virtual switch → Physical. -- VNF → virtual switch → VNF. - -Tests MUST have these parameters unless otherwise stated. **Test cases -with non default parameters will be stated explicitly**. - -**Note**: For throughput tests unless stated otherwise, test -configurations should ensure that traffic traverses the installed flows -through the virtual switch, i.e. flows are installed and have an appropriate -time out that doesn't expire before packet transmission starts. - -.. 3.2.5.2.2 - -Flow Classification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Virtual switches classify packets into flows by processing and matching -particular header fields in the packet/frame and/or the input port where -the packets/frames arrived. The vSwitch then carries out an action on -the group of packets that match the classification parameters. Thus a -flow is considered to be a sequence of packets that have a shared set of -header field values or have arrived on the same port and have the same -action applied to them. Performance results can vary based on the -parameters the vSwitch uses to match for a flow. The recommended flow -classification parameters for L3 vSwitch performance tests are: the -input port, the source IP address, the destination IP address and the -Ethernet protocol type field. It is essential to increase the flow -time-out time on a vSwitch before conducting any performance tests that -do not measure the flow set-up time. Normally the first packet of a -particular flow will install the flow in the vSwitch which adds an -additional latency, subsequent packets of the same flow are not subject -to this latency if the flow is already installed on the vSwitch. - -.. 3.2.5.2.3 - -Test Priority -~~~~~~~~~~~~~~~~~~~~~ - -Tests will be assigned a priority in order to determine which tests -should be implemented immediately and which tests implementations -can be deferred. - -Priority can be of following types: - Urgent: Must be implemented -immediately. - High: Must be implemented in the next release. - Medium: -May be implemented after the release. - Low: May or may not be -implemented at all. - -.. 3.2.5.2.4 - -SUT Setup -~~~~~~~~~~~~~~~~~~ - -The SUT should be configured to its "default" state. The -SUT's configuration or set-up must not change between tests in any way -other than what is required to do the test. All supported protocols must -be configured and enabled for each test set up. - -.. 3.2.5.2.5 - -Port Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The DUT should be configured with n ports where -n is a multiple of 2. Half of the ports on the DUT should be used as -ingress ports and the other half of the ports on the DUT should be used -as egress ports. Where a DUT has more than 2 ports, the ingress data -streams should be set-up so that they transmit packets to the egress -ports in sequence so that there is an even distribution of traffic -across ports. For example, if a DUT has 4 ports 0(ingress), 1(ingress), -2(egress) and 3(egress), the traffic stream directed at port 0 should -output a packet to port 2 followed by a packet to port 3. The traffic -stream directed at port 1 should also output a packet to port 2 followed -by a packet to port 3. - -.. 3.2.5.2.6 - -Frame Formats -~~~~~~~~~~~~~~~~~~~~~ - -**Frame formats Layer 2 (data link layer) protocols** - -- Ethernet II - -.. code-block:: console - - +---------------------------+-----------+ - | Ethernet Header | Payload | Check Sum | - +-----------------+---------+-----------+ - |_________________|_________|___________| - 14 Bytes 46 - 1500 4 Bytes - Bytes - - -**Layer 3 (network layer) protocols** - -- IPv4 - -.. code-block:: console - - +-----------------+-----------+---------+-----------+ - | Ethernet Header | IP Header | Payload | Checksum | - +-----------------+-----------+---------+-----------+ - |_________________|___________|_________|___________| - 14 Bytes 20 bytes 26 - 1480 4 Bytes - Bytes - -- IPv6 - -.. code-block:: console - - +-----------------+-----------+---------+-----------+ - | Ethernet Header | IP Header | Payload | Checksum | - +-----------------+-----------+---------+-----------+ - |_________________|___________|_________|___________| - 14 Bytes 40 bytes 26 - 1460 4 Bytes - Bytes - -**Layer 4 (transport layer) protocols** - - - TCP - - UDP - - SCTP - -.. code-block:: console - - +-----------------+-----------+-----------------+---------+-----------+ - | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum | - +-----------------+-----------+-----------------+---------+-----------+ - |_________________|___________|_________________|_________|___________| - 14 Bytes 40 bytes 20 Bytes 6 - 1460 4 Bytes - Bytes - - -**Layer 5 (application layer) protocols** - - - RTP - - GTP - -.. code-block:: console - - +-----------------+-----------+-----------------+---------+-----------+ - | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum | - +-----------------+-----------+-----------------+---------+-----------+ - |_________________|___________|_________________|_________|___________| - 14 Bytes 20 bytes 20 Bytes >= 6 Bytes 4 Bytes - -.. 3.2.5.2.7 - -Packet Throughput -~~~~~~~~~~~~~~~~~~~~~~~~~ -There is a difference between an Ethernet frame, -an IP packet, and a UDP datagram. In the seven-layer OSI model of -computer networking, packet refers to a data unit at layer 3 (network -layer). The correct term for a data unit at layer 2 (data link layer) is -a frame, and at layer 4 (transport layer) is a segment or datagram. - -Important concepts related to 10GbE performance are frame rate and -throughput. The MAC bit rate of 10GbE, defined in the IEEE standard 802 -.3ae, is 10 billion bits per second. Frame rate is based on the bit rate -and frame format definitions. Throughput, defined in IETF RFC 1242, is -the highest rate at which the system under test can forward the offered -load, without loss. - -The frame rate for 10GbE is determined by a formula that divides the 10 -billion bits per second by the preamble + frame length + inter-frame -gap. - -The maximum frame rate is calculated using the minimum values of the -following parameters, as described in the IEEE 802 .3ae standard: - -- Preamble: 8 bytes \* 8 = 64 bits -- Frame Length: 64 bytes (minimum) \* 8 = 512 bits -- Inter-frame Gap: 12 bytes (minimum) \* 8 = 96 bits - -Therefore, Maximum Frame Rate (64B Frames) -= MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame Gap) -= 10,000,000,000 / (64 + 512 + 96) -= 10,000,000,000 / 672 -= 14,880,952.38 frame per second (fps) - -.. 3.2.5.3 - -RFCs for testing virtual switch performance --------------------------------------------------- - -The starting point for defining the suite of tests for benchmarking the -performance of a virtual switch is to take existing RFCs and standards -that were designed to test their physical counterparts and adapting them -for testing virtual switches. The rationale behind this is to establish -a fair comparison between the performance of virtual and physical -switches. This section outlines the RFCs that are used by this -specification. - -.. 3.2.5.3.1 - -RFC 1242 Benchmarking Terminology for Network Interconnection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Devices RFC 1242 defines the terminology that is used in describing -performance benchmarking tests and their results. Definitions and -discussions covered include: Back-to-back, bridge, bridge/router, -constant load, data link frame size, frame loss rate, inter frame gap, -latency, and many more. - -.. 3.2.5.3.2 - -RFC 2544 Benchmarking Methodology for Network Interconnect Devices -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2544 outlines a benchmarking methodology for network Interconnect -Devices. The methodology results in performance metrics such as latency, -frame loss percentage, and maximum data throughput. - -In this document network “throughput” (measured in millions of frames -per second) is based on RFC 2544, unless otherwise noted. Frame size -refers to Ethernet frames ranging from smallest frames of 64 bytes to -largest frames of 9K bytes. - -Types of tests are: - -1. Throughput test defines the maximum number of frames per second - that can be transmitted without any error, or 0% loss ratio. - In some Throughput tests (and those tests with long duration), - evaluation of an additional frame loss ratio is suggested. The - current ratio (10^-7 %) is based on understanding the typical - user-to-user packet loss ratio needed for good application - performance and recognizing that a single transfer through a - vswitch must contribute a tiny fraction of user-to-user loss. - Further, the ratio 10^-7 % also recognizes practical limitations - when measuring loss ratio. - -2. Latency test measures the time required for a frame to travel from - the originating device through the network to the destination device. - Please note that RFC2544 Latency measurement will be superseded with - a measurement of average latency over all successfully transferred - packets or frames. - -3. Frame loss test measures the network’s - response in overload conditions - a critical indicator of the - network’s ability to support real-time applications in which a - large amount of frame loss will rapidly degrade service quality. - -4. Burst test assesses the buffering capability of a virtual switch. It - measures the maximum number of frames received at full line rate - before a frame is lost. In carrier Ethernet networks, this - measurement validates the excess information rate (EIR) as defined in - many SLAs. - -5. System recovery to characterize speed of recovery from an overload - condition. - -6. Reset to characterize speed of recovery from device or software - reset. This type of test has been updated by `RFC6201 - <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such, - the methodology defined by this specification will be that of RFC 6201. - -Although not included in the defined RFC 2544 standard, another crucial -measurement in Ethernet networking is packet delay variation. The -definition set out by this specification comes from -`RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. - -.. 3.2.5.3.3 - -RFC 2285 Benchmarking Terminology for LAN Switching Devices -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2285 defines the terminology that is used to describe the -terminology for benchmarking a LAN switching device. It extends RFC -1242 and defines: DUTs, SUTs, Traffic orientation and distribution, -bursts, loads, forwarding rates, etc. - -.. 3.2.5.3.4 - -RFC 2889 Benchmarking Methodology for LAN Switching -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 2889 outlines a benchmarking methodology for LAN switching, it -extends RFC 2544. The outlined methodology gathers performance -metrics for forwarding, congestion control, latency, address handling -and finally filtering. - -.. 3.2.5.3.5 - -RFC 3918 Methodology for IP Multicast Benchmarking -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 3918 outlines a methodology for IP Multicast benchmarking. - -.. 3.2.5.3.6 - -RFC 4737 Packet Reordering Metrics -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 4737 describes metrics for identifying and counting re-ordered -packets within a stream, and metrics to measure the extent each -packet has been re-ordered. - -.. 3.2.5.3.7 - -RFC 5481 Packet Delay Variation Applicability Statement -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 5481 defined two common, but different forms of delay variation -metrics, and compares the metrics over a range of networking -circumstances and tasks. The most suitable form for vSwitch -benchmarking is the "PDV" form. - -.. 3.2.5.3.8 - -RFC 6201 Device Reset Characterization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -RFC 6201 extends the methodology for characterizing the speed of -recovery of the DUT from device or software reset described in RFC -2544. - -.. 3.2.6: - -.. _PassFailCriteria: - -Item pass/fail criteria -========================= - -vswitchperf does not specify Pass/Fail criteria for the tests in terms of a -threshold, as benchmarks do not (and should not do this). The results/metrics -for a test are simply reported. If it had to be defined, a test is considered -to have passed if it succesfully completed and a relavent metric was -recorded/reported for the SUT. - -.. 3.2.7: - -.. _SuspensionResumptionReqs: - -Suspension criteria and resumption requirements -================================================ -In the case of a throughput test, a test should be suspended if a virtual -switch is failing to forward any traffic. A test should be restarted from a -clean state if the intention is to carry out the test again. - -.. 3.2.8: - -.. _TestDelierables: - -Test deliverables -================== -Each test should produce a test report that details SUT information as well as -the test results. There are a number of parameters related to the system, DUT -and tests that can affect the repeatability of a test results and should be -recorded. In order to minimise the variation in the results of a test, -it is recommended that the test report includes the following information: - -- Hardware details including: - - - Platform details. - - Processor details. - - Memory information (see below) - - Number of enabled cores. - - Number of cores used for the test. - - Number of physical NICs, as well as their details (manufacturer, - versions, type and the PCI slot they are plugged into). - - NIC interrupt configuration. - - BIOS version, release date and any configurations that were - modified. - -- Software details including: - - - OS version (for host and VNF) - - Kernel version (for host and VNF) - - GRUB boot parameters (for host and VNF). - - Hypervisor details (Type and version). - - Selected vSwitch, version number or commit id used. - - vSwitch launch command line if it has been parameterised. - - Memory allocation to the vSwitch – which NUMA node it is using, - and how many memory channels. - - Where the vswitch is built from source: compiler details including - versions and the flags that were used to compile the vSwitch. - - DPDK or any other SW dependency version number or commit id used. - - Memory allocation to a VM - if it's from Hugpages/elsewhere. - - VM storage type: snapshot/independent persistent/independent - non-persistent. - - Number of VMs. - - Number of Virtual NICs (vNICs), versions, type and driver. - - Number of virtual CPUs and their core affinity on the host. - - Number vNIC interrupt configuration. - - Thread affinitization for the applications (including the vSwitch - itself) on the host. - - Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - -- Memory Details - - - Total memory - - Type of memory - - Used memory - - Active memory - - Inactive memory - - Free memory - - Buffer memory - - Swap cache - - Total swap - - Used swap - - Free swap - -- Test duration. -- Number of flows. -- Traffic Information: - - - Traffic type - UDP, TCP, IMIX / Other. - - Packet Sizes. - -- Deployment Scenario. - -**Note**: Tests that require additional parameters to be recorded will -explicitly specify this. - - -.. 3.3: - -.. _TestManagement: - -Test management -================= -This section will detail the test activities that will be conducted by vsperf -as well as the infrastructure that will be used to complete the tests in OPNFV. - -.. 3.3.1: - -Planned activities and tasks; test progression -================================================= -A key consideration when conducting any sort of benchmark is trying to -ensure the consistency and repeatability of test results between runs. -When benchmarking the performance of a virtual switch there are many -factors that can affect the consistency of results. This section -describes these factors and the measures that can be taken to limit -their effects. In addition, this section will outline some system tests -to validate the platform and the VNF before conducting any vSwitch -benchmarking tests. - -**System Isolation:** - -When conducting a benchmarking test on any SUT, it is essential to limit -(and if reasonable, eliminate) any noise that may interfere with the -accuracy of the metrics collected by the test. This noise may be -introduced by other hardware or software (OS, other applications), and -can result in significantly varying performance metrics being collected -between consecutive runs of the same test. In the case of characterizing -the performance of a virtual switch, there are a number of configuration -parameters that can help increase the repeatability and stability of -test results, including: - -- OS/GRUB configuration: - - - maxcpus = n where n >= 0; limits the kernel to using 'n' - processors. Only use exactly what you need. - - isolcpus: Isolate CPUs from the general scheduler. Isolate all - CPUs bar one which will be used by the OS. - - use taskset to affinitize the forwarding application and the VNFs - onto isolated cores. VNFs and the vSwitch should be allocated - their own cores, i.e. must not share the same cores. vCPUs for the - VNF should be affinitized to individual cores also. - - Limit the amount of background applications that are running and - set OS to boot to runlevel 3. Make sure to kill any unnecessary - system processes/daemons. - - Only enable hardware that you need to use for your test – to - ensure there are no other interrupts on the system. - - Configure NIC interrupts to only use the cores that are not - allocated to any other process (VNF/vSwitch). - -- NUMA configuration: Any unused sockets in a multi-socket system - should be disabled. -- CPU pinning: The vSwitch and the VNF should each be affinitized to - separate logical cores using a combination of maxcpus, isolcpus and - taskset. -- BIOS configuration: BIOS should be configured for performance where - an explicit option exists, sleep states should be disabled, any - virtualization optimization technologies should be enabled, and - hyperthreading should also be enabled, turbo boost and overclocking - should be disabled. - -**System Validation:** - -System validation is broken down into two sub-categories: Platform -validation and VNF validation. The validation test itself involves -verifying the forwarding capability and stability for the sub-system -under test. The rationale behind system validation is two fold. Firstly -to give a tester confidence in the stability of the platform or VNF that -is being tested; and secondly to provide base performance comparison -points to understand the overhead introduced by the virtual switch. - -* Benchmark platform forwarding capability: This is an OPTIONAL test - used to verify the platform and measure the base performance (maximum - forwarding rate in fps and latency) that can be achieved by the - platform without a vSwitch or a VNF. The following diagram outlines - the set-up for benchmarking Platform forwarding capability: - - .. code-block:: console - - __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | l2fw or DPDK L2FWD app | | Host - | | | | | - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - -* Benchmark VNF forwarding capability: This test is used to verify - the VNF and measure the base performance (maximum forwarding rate in - fps and latency) that can be achieved by the VNF without a vSwitch. - The performance metrics collected by this test will serve as a key - comparison point for NIC passthrough technologies and vSwitches. VNF - in this context refers to the hypervisor and the VM. The following - diagram outlines the set-up for benchmarking VNF forwarding - capability: - - .. code-block:: console - - __ - +--------------------------------------------------+ | - | +------------------------------------------+ | | - | | | | | - | | VNF | | | - | | | | | - | +------------------------------------------+ | | - | | Passthrough/SR-IOV | | Host - | +------------------------------------------+ | | - | | NIC | | | - +---+------------------------------------------+---+ __| - ^ : - | | - : v - +--------------------------------------------------+ - | | - | traffic generator | - | | - +--------------------------------------------------+ - - -**Methodology to benchmark Platform/VNF forwarding capability** - - -The recommended methodology for the platform/VNF validation and -benchmark is: - Run `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ -Maximum Forwarding Rate test, this test will produce maximum -forwarding rate and latency results that will serve as the -expected values. These expected values can be used in -subsequent steps or compared with in subsequent validation tests. - -Transmit bidirectional traffic at line rate/max forwarding rate -(whichever is higher) for at least 72 hours, measure throughput (fps) -and latency. - Note: Traffic should be bidirectional. - Establish a -baseline forwarding rate for what the platform can achieve. - Additional -validation: After the test has completed for 72 hours run bidirectional -traffic at the maximum forwarding rate once more to see if the system is -still functional and measure throughput (fps) and latency. Compare the -measure the new obtained values with the expected values. - -**NOTE 1**: How the Platform is configured for its forwarding capability -test (BIOS settings, GRUB configuration, runlevel...) is how the -platform should be configured for every test after this - -**NOTE 2**: How the VNF is configured for its forwarding capability test -(# of vCPUs, vNICs, Memory, affinitization…) is how it should be -configured for every test that uses a VNF after this. - -**Methodology to benchmark the VNF to vSwitch to VNF deployment scenario** - -vsperf has identified the following concerns when benchmarking the VNF to -vSwitch to VNF deployment scenario: - -* The accuracy of the timing synchronization between VNFs/VMs. -* The clock accuracy of a VNF/VM if they were to be used as traffic generators. -* VNF traffic generator/receiver may be using resources of the system under - test, causing at least three forms of workload to increase as the traffic - load increases (generation, switching, receiving). - -The recommendation from vsperf is that tests for this sceanario must -include an external HW traffic generator to act as the tester/traffic transmitter -and receiver. The perscribed methodology to benchmark this deployment scanrio with -an external tester involves the following three steps: - -#. Determine the forwarding capability and latency through the virtual interface -connected to the VNF/VM. - -.. Figure:: vm2vm_virtual_interface_benchmark.png - - Virtual interfaces performance benchmark - -#. Determine the forwarding capability and latency through the VNF/hypervisor. - -.. Figure:: vm2vm_hypervisor_benchmark.png - - Hypervisor performance benchmark - -#. Determine the forwarding capability and latency for the VNF to vSwitch to VNF - taking the information from the previous two steps into account. - -.. Figure:: vm2vm_benchmark.png - - VNF to vSwitch to VNF performance benchmark - -vsperf also identified an alternative configuration for the final step: - -.. Figure:: vm2vm_alternative_benchmark.png - - VNF to vSwitch to VNF alternative performance benchmark - -.. 3.3.2: - -Environment/infrastructure -============================ -Intel is providing a hosted test-bed with nine bare-metal environments -allocated to different OPNFV projects. Currently a number of servers in -`Intel POD 3 <https://wiki.opnfv.org/display/pharos/Intel+Pod3>`__ are -allocated to vsperf: - - * pod3-wcp-node3 and pod3-wcp-node4 which are used for CI jobs. - * pod3-node6 which is used as a vsperf sandbox environment. - -vsperf CI ---------- -vsperf CI jobs are broken down into: - - * Daily job: - - * Runs everyday takes about 10 hours to complete. - * TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan - phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'. - * TESTPARAM_DAILY='--test-params TRAFFICGEN_PKT_SIZES=(64,128,512,1024,1518)'. - - * Merge job: - - * Runs whenever patches are merged to master. - * Runs a basic Sanity test. - - * Verify job: - - * Runs every time a patch is pushed to gerrit. - * Builds documentation. - -Scripts: --------- -There are 2 scripts that are part of VSPERFs CI: - - * build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is - used to run vsperf with the appropriate cli parameters. - * vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG - repository. - -More info on vsperf CI can be found here: -https://wiki.opnfv.org/display/vsperf/VSPERF+CI - -.. 3.3.3: - -Responsibilities and authority -=============================== -The group responsible for managing, designing, preparing and executing the -tests listed in the LTD are the vsperf committers and contributors. The vsperf -committers and contributors should work with the relavent OPNFV projects to -ensure that the infrastructure is in place for testing vswitches, and that the -results are published to common end point (a results database). - |