aboutsummaryrefslogtreecommitdiffstats
path: root/docs/test_spec
diff options
context:
space:
mode:
authorMaryam Tahhan <maryam.tahhan@intel.com>2015-12-02 13:42:50 +0000
committerMaryam Tahhan <maryam.tahhan@intel.com>2015-12-21 10:44:07 +0000
commitcc3a4bf85074989c28a09b7b141dea5e42701f5c (patch)
tree18740fd268ff0d631812b36eb45636489cf8b75d /docs/test_spec
parentde0aaffe31e1f787cefe21a5957a07924bb5956f (diff)
docs: updates and move traffic gens to separate doc
Move the traffic gen instructions to a separate user guide and add information on usage of the Dummy traffic generator. Update docs to fix PDF build failure and do general clean-up. Removed the numbering from the LTD and added the numbered directive to automate numbering for sections and headers. Add comment anchors that reflect the section numbers. Change-Id: I984ca38456a891c439697ebc1da041bc1d828a15 Signed-off-by: Maryam Tahhan <maryam.tahhan@intel.com>
Diffstat (limited to 'docs/test_spec')
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml964
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt1232
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml964
-rwxr-xr-xdocs/test_spec/index.rst17
-rw-r--r--docs/test_spec/vswitchperf_ltd.rst2050
5 files changed, 0 insertions, 5227 deletions
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml
deleted file mode 100755
index b5f7f833..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml
+++ /dev/null
@@ -1,964 +0,0 @@
-<?xml version="1.0" encoding="US-ASCII"?>
-<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
-<?rfc toc="yes"?>
-<?rfc tocompact="yes"?>
-<?rfc tocdepth="3"?>
-<?rfc tocindent="yes"?>
-<?rfc symrefs="yes"?>
-<?rfc sortrefs="yes"?>
-<?rfc comments="yes"?>
-<?rfc inline="yes"?>
-<?rfc compact="yes"?>
-<?rfc subcompact="no"?>
-<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-01"
- ipr="trust200902">
- <front>
- <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
- OPNFV</title>
-
- <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>maryam.tahhan@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>billy.o.mahony@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Al Morton" initials="A." surname="Morton">
- <organization>AT&amp;T Labs</organization>
-
- <address>
- <postal>
- <street>200 Laurel Avenue South</street>
-
- <city>Middletown,</city>
-
- <region>NJ</region>
-
- <code>07748</code>
-
- <country>USA</country>
- </postal>
-
- <phone>+1 732 420 1571</phone>
-
- <facsimile>+1 732 368 1192</facsimile>
-
- <email>acmorton@att.com</email>
-
- <uri>http://home.comcast.net/~acmacm/</uri>
- </address>
- </author>
-
- <date day="14" month="October" year="2015"/>
-
- <abstract>
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the Benchmarking
- Methodology Working Group in IETF, by referencing existing literature.
- The Benchmarking Methodology Working Group has traditionally conducted
- laboratory characterization of dedicated physical implementations of
- internetworking functions. Therefore, this memo begins to describe the
- additional considerations when virtual switches are implemented in
- general-purpose hardware. The expanded tests and benchmarks are also
- influenced by the OPNFV mission to support virtualization of the "telco"
- infrastructure.</t>
- </abstract>
-
- <note title="Requirements Language">
- <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in <xref
- target="RFC2119">RFC 2119</xref>.</t>
-
- <t/>
- </note>
- </front>
-
- <middle>
- <section title="Introduction">
- <t>Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box Benchmarks
- of Throughput, Latency, Forwarding Rates and others have served our
- industry for many years. Now, Network Function Virtualization (NFV) has
- the goal to transform how internetwork functions are implemented, and
- therefore has garnered much attention.</t>
-
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization, "VSWITCHPERF".
- This project intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing existing
- literature. For example, currently the most often referenced RFC is
- <xref target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and
- foundation of the benchmarking work in OPNFV is common and strong.</t>
-
- <t>See
- https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
- for more background, and the OPNFV website for general information:
- https://www.opnfv.org/</t>
-
- <t>The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on existing
- "telco" services as opposed to cloud-computing. There are many ways in
- which telco requirements have different emphasis on performance
- dimensions when compared to cloud computing: support for and transfer of
- isochronous media streams is one example.</t>
-
- <t>Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.</t>
- </section>
-
- <section title="Scope">
- <t>The primary purpose and scope of the memo is to inform BMWG of
- work-in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here is
- received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's chartered
- work to characterize the NFV Infrastructure (a virtual switch is an
- important aspect of that infrastructure).</t>
- </section>
-
- <section title="Benchmarking Considerations">
- <t>This section highlights some specific considerations (from <xref
- target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these areas,
- as they develop their specifications in the Level Test Design (LTD)
- document.</t>
-
- <section title="Comparison with Physical Network Functions">
- <t>To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this memo
- re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.</t>
-
- <t>It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.</t>
- </section>
-
- <section title="Continued Emphasis on Black-Box Benchmarks">
- <t>External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation will
- be provided in parallel to assist the development of operations
- procedures when the technology is deployed.</t>
- </section>
-
- <section title="New Configuration Parameters">
- <t>A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.</t>
-
- <t>Hardware details including:</t>
-
- <t><list style="symbols">
- <t>Platform details</t>
-
- <t>Processor details</t>
-
- <t>Memory information (type and size)</t>
-
- <t>Number of enabled cores</t>
-
- <t>Number of cores used for the test</t>
-
- <t>Number of physical NICs, as well as their details
- (manufacturer, versions, type and the PCI slot they are plugged
- into)</t>
-
- <t>NIC interrupt configuration</t>
-
- <t>BIOS version, release date and any configurations that were
- modified</t>
-
- <t>CPU microcode level</t>
-
- <t>Memory DIMM configurations (quad rank performance may not be
- the same as dual rank) in size, freq and slot locations</t>
-
- <t>PCI configuration parameters (payload size, early ack
- option...)</t>
-
- <t>Power management at all levels (ACPI sleep states, processor
- package, OS...)</t>
- </list>Software details including:</t>
-
- <t><list style="symbols">
- <t>OS parameters and behavior (text vs graphical no one typing at
- the console on one system)</t>
-
- <t>OS version (for host and VNF)</t>
-
- <t>Kernel version (for host and VNF)</t>
-
- <t>GRUB boot parameters (for host and VNF)</t>
-
- <t>Hypervisor details (Type and version)</t>
-
- <t>Selected vSwitch, version number or commit id used</t>
-
- <t>vSwitch launch command line if it has been parameterised</t>
-
- <t>Memory allocation to the vSwitch</t>
-
- <t>which NUMA node it is using, and how many memory channels</t>
-
- <t>DPDK or any other SW dependency version number or commit id
- used</t>
-
- <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t>
-
- <t>VM storage type: snapshot/independent persistent/independent
- non-persistent</t>
-
- <t>Number of VMs</t>
-
- <t>Number of Virtual NICs (vNICs), versions, type and driver</t>
-
- <t>Number of virtual CPUs and their core affinity on the host</t>
-
- <t>Number vNIC interrupt configuration</t>
-
- <t>Thread affinitization for the applications (including the
- vSwitch itself) on the host</t>
-
- <t>Details of Resource isolation, such as CPUs designated for
- Host/Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.</t>
- </list></t>
-
- <t>Test Traffic Information:<list style="symbols">
- <t>Traffic type - UDP, TCP, IMIX / Other</t>
-
- <t>Packet Sizes</t>
-
- <t>Deployment Scenario</t>
- </list></t>
-
- <t/>
- </section>
-
- <section title="Flow classification">
- <t>Virtual switches group packets into flows by processing and
- matching particular packet or frame header information, or by matching
- packets based on the input ports. Thus a flow can be thought of a
- sequence of packets that have the same set of header field values or
- have arrived on the same port. Performance results can vary based on
- the parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the flow
- timeout time on a vSwitch before conducting any performance tests that
- do not measure the flow setup time. Normally the first packet of a
- particular stream will install the flow in the virtual switch which
- adds an additional latency, subsequent packets of the same flow are
- not subject to this latency if the flow is already installed on the
- vSwitch.</t>
- </section>
-
- <section title="Benchmarks using Baselines with Resource Isolation">
- <t>This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.</t>
-
- <t><list style="numbers">
- <t>Baselines: <list style="symbols">
- <t>Optional: Benchmark platform forwarding capability without
- a vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency). <figure>
- <preamble>Benchmark platform forwarding
- capability</preamble>
-
- <artwork align="right"><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmark VNF forwarding capability with direct
- connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72
- hours (serves as a means of VNF validation and a means to
- obtain the base performance for the VNF in terms of its
- maximum forwarding rate and latency). The metrics gathered
- from this test will serve as a key comparison point for
- vSwitch bypass technologies performance and vSwitch
- performance. <figure align="right">
- <preamble>Benchmark VNF forwarding capability</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmarking with isolated resources alone, with other
- resources (both HW&amp;SW) disabled Example, vSw and VM are
- SUT</t>
-
- <t>Benchmarking with isolated resources alone, leaving some
- resources unused</t>
-
- <t>Benchmark with isolated resources and all resources
- occupied</t>
- </list></t>
-
- <t>Next Steps<list style="symbols">
- <t>Limited sharing</t>
-
- <t>Production scenarios</t>
-
- <t>Stressful scenarios</t>
- </list></t>
- </list></t>
- </section>
- </section>
-
- <section title="VSWITCHPERF Specification Summary">
- <t>The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the
- pre-existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2544"/> Benchmarking Methodology for Network
- Interconnect Devices</t>
-
- <t><xref target="RFC2889"/> Benchmarking Methodology for LAN
- Switching</t>
-
- <t><xref target="RFC6201"/> Device Reset Characterization</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t>Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test equipment
- developers. Fortunately, many members of the testing system community
- have engaged on the VSPERF project, including an open source test
- system.</t>
-
- <t>In addition to this, the LTD also re-uses the terminology defined
- by:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2285"/> Benchmarking Terminology for LAN
- Switching Devices</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t/>
-
- <t>Specifications to be included in future updates of the LTD
- include:<list style="symbols">
- <t><xref target="RFC3918"/> Methodology for IP Multicast
- Benchmarking</t>
-
- <t><xref target="RFC4737"/> Packet Reordering Metrics</t>
- </list></t>
-
- <t>As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.</t>
-
- <t>When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics. In
- this case, the project specifications have referenced metrics from the
- IETF IP Performance Metrics (IPPM) literature. This means that the <xref
- target="RFC2544"/> test of Latency is replaced by measurement of a
- metric derived from IPPM's <xref target="RFC2679"/>, where a set of
- statistical summaries will be provided (mean, max, min, etc.). Further
- metrics planned to be benchmarked include packet delay variation as
- defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
- availability, DUT capacity and packet loss in long term testing at
- Throughput level, where some low-level of background loss may be present
- and characterized.</t>
-
- <t>Tests have been (or will be) designed to collect the metrics
- below:</t>
-
- <t><list style="symbols">
- <t>Throughput Tests to measure the maximum forwarding rate (in
- frames per second or fps) and bit rate (in Mbps) for a constant load
- (as defined by RFC1242) without traffic loss.</t>
-
- <t>Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.</t>
-
- <t>Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.</t>
-
- <t>Scalability Tests to understand how the virtual switch performs
- as the number of flows, active ports, complexity of the forwarding
- logic&rsquo;s configuration&hellip; it has to deal with
- increases.</t>
-
- <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data through
- the switch.</t>
-
- <t>Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as the
- effect of this coupling on the performance of the DUT (example:
- delay of the initial packet of a flow).</t>
-
- <t>CPU and Memory Consumption Tests to understand the virtual
- switch&rsquo;s footprint on the system, usually conducted as
- auxiliary measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.</t>
- </list></t>
-
- <t>Future/planned test specs include:<list style="symbols">
- <t>Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.</t>
-
- <t>Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.</t>
-
- <t>Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements <xref target="IFA003"/> on characterization of
- acceleration technologies applied to vswitches.</t>
- </list>The flexibility of deployment of a virtual switch within a
- network means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:</t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to physical
- port</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure></t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|]]></artwork>
- </figure></t>
-
- <t>A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page <xref target="TestTopo"/>. </t>
- </section>
-
- <section title="3x3 Matrix Coverage">
- <t>This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>).
- Because the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all are
- occupied, also the matrix has grown to 3x4 to accommodate scale metrics
- when displaying the coverage of many metrics/benchmarks).</t>
-
- <t>The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.</t>
-
- <t>A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t>
-
- <section title="Speed of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressLearningRate</t>
-
- <t>PacketLatency.InitialPacketProcessingLatency</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Activation section">
- <t><list style="symbols">
- <t>CPDP.Coupling.Flow.Addition</t>
- </list></t>
- </section>
-
- <section title="Reliability of Activation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.SystemRecoveryTime</t>
-
- <t>Throughput.RFC2544.ResetTime</t>
- </list></t>
- </section>
-
- <section title="Scale of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressCachingCapacity</t>
- </list></t>
- </section>
-
- <section title="Speed of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.PacketLossRate</t>
-
- <t>CPU.RFC2544.0PacketLoss</t>
-
- <t>Throughput.RFC2544.PacketLossRateFrameModification</t>
-
- <t>Throughput.RFC2544.BackToBackFrames</t>
-
- <t>Throughput.RFC2889.MaxForwardingRate</t>
-
- <t>Throughput.RFC2889.ForwardPressure</t>
-
- <t>Throughput.RFC2889.BroadcastFrameForwarding</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.ErrorFramesFiltering</t>
-
- <t>Throughput.RFC2544.Profile</t>
- </list></t>
- </section>
-
- <section title="Reliability of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.Soak</t>
-
- <t>Throughput.RFC2889.SoakFrameModification</t>
-
- <t>PacketDelayVariation.RFC3393.Soak</t>
- </list></t>
- </section>
-
- <section title="Scalability of Operation">
- <t><list style="symbols">
- <t>Scalability.RFC2544.0PacketLoss</t>
-
- <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t>
- </list></t>
- </section>
-
- <section title="Summary">
- <t><figure>
- <artwork><![CDATA[|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|]]></artwork>
- </figure></t>
- </section>
- </section>
-
- <section title="Security Considerations">
- <t>Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.</t>
-
- <t>The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test traffic
- into a production network, or misroute traffic to the test management
- network.</t>
-
- <t>Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.</t>
-
- <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.</t>
- </section>
-
- <section anchor="IANA" title="IANA Considerations">
- <t>No IANA Action is requested at this time.</t>
- </section>
-
- <section title="Acknowledgements">
- <t>The authors acknowledge</t>
- </section>
- </middle>
-
- <back>
- <references title="Normative References">
- <?rfc ?>
-
- <?rfc include="reference.RFC.2119"?>
-
- <?rfc include="reference.RFC.2330"?>
-
- <?rfc include='reference.RFC.2544'?>
-
- <?rfc include="reference.RFC.2679"?>
-
- <?rfc include='reference.RFC.2680'?>
-
- <?rfc include='reference.RFC.3393'?>
-
- <?rfc include='reference.RFC.3432'?>
-
- <?rfc include='reference.RFC.2681'?>
-
- <?rfc include='reference.RFC.5905'?>
-
- <?rfc include='reference.RFC.4689'?>
-
- <?rfc include='reference.RFC.4737'?>
-
- <?rfc include='reference.RFC.5357'?>
-
- <?rfc include='reference.RFC.2889'?>
-
- <?rfc include='reference.RFC.3918'?>
-
- <?rfc include='reference.RFC.6201'?>
-
- <?rfc include='reference.RFC.2285'?>
-
- <reference anchor="NFV.PER001">
- <front>
- <title>Network Function Virtualization: Performance and Portability
- Best Practices</title>
-
- <author fullname="ETSI NFV" initials="" surname="">
- <organization/>
- </author>
-
- <date month="June" year="2014"/>
- </front>
-
- <seriesInfo name="Group Specification"
- value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
-
- <format type="PDF"/>
- </reference>
- </references>
-
- <references title="Informative References">
- <?rfc include='reference.RFC.1242'?>
-
- <?rfc include='reference.RFC.5481'?>
-
- <?rfc include='reference.RFC.6049'?>
-
- <?rfc include='reference.RFC.6248'?>
-
- <?rfc include='reference.RFC.6390'?>
-
- <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?>
-
- <reference anchor="TestTopo">
- <front>
- <title>Test Topologies
- https://wiki.opnfv.org/vsperf/test_methodology</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="LTDoverV">
- <front>
- <title>LTD Test Spec Overview
- https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review </title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="IFA003">
- <front>
- <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
- </references>
- </back>
-</rfc>
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt
deleted file mode 100755
index 81ae96c0..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt
+++ /dev/null
@@ -1,1232 +0,0 @@
-
-
-
-
-Network Working Group M. Tahhan
-Internet-Draft B. O'Mahony
-Intended status: Informational Intel
-Expires: April 16, 2016 A. Morton
- AT&T Labs
- October 14, 2015
-
-
- Benchmarking Virtual Switches in OPNFV
- draft-vsperf-bmwg-vswitch-opnfv-01
-
-Abstract
-
- This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing
- existing literature. The Benchmarking Methodology Working Group has
- traditionally conducted laboratory characterization of dedicated
- physical implementations of internetworking functions. Therefore,
- this memo begins to describe the additional considerations when
- virtual switches are implemented in general-purpose hardware. The
- expanded tests and benchmarks are also influenced by the OPNFV
- mission to support virtualization of the "telco" infrastructure.
-
-Requirements Language
-
- The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in RFC 2119 [RFC2119].
-
-Status of This Memo
-
- This Internet-Draft is submitted in full conformance with the
- provisions of BCP 78 and BCP 79.
-
- Internet-Drafts are working documents of the Internet Engineering
- Task Force (IETF). Note that other groups may also distribute
- working documents as Internet-Drafts. The list of current Internet-
- Drafts is at http://datatracker.ietf.org/drafts/current/.
-
- Internet-Drafts are draft documents valid for a maximum of six months
- and may be updated, replaced, or obsoleted by other documents at any
- time. It is inappropriate to use Internet-Drafts as reference
- material or to cite them other than as "work in progress."
-
- This Internet-Draft will expire on April 16, 2016.
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 1]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-Copyright Notice
-
- Copyright (c) 2015 IETF Trust and the persons identified as the
- document authors. All rights reserved.
-
- This document is subject to BCP 78 and the IETF Trust's Legal
- Provisions Relating to IETF Documents
- (http://trustee.ietf.org/license-info) in effect on the date of
- publication of this document. Please review these documents
- carefully, as they describe your rights and restrictions with respect
- to this document. Code Components extracted from this document must
- include Simplified BSD License text as described in Section 4.e of
- the Trust Legal Provisions and are provided without warranty as
- described in the Simplified BSD License.
-
-Table of Contents
-
- 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
- 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
- 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 4
- 3.1. Comparison with Physical Network Functions . . . . . . . 4
- 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 4
- 3.3. New Configuration Parameters . . . . . . . . . . . . . . 4
- 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 6
- 3.5. Benchmarks using Baselines with Resource Isolation . . . 7
- 4. VSWITCHPERF Specification Summary . . . . . . . . . . . . . . 8
- 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 16
- 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 17
- 5.2. Accuracy of Activation section . . . . . . . . . . . . . 17
- 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 17
- 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 17
- 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 17
- 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 17
- 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 17
- 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 18
- 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 18
- 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18
- 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
- 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19
- 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
- 9.1. Normative References . . . . . . . . . . . . . . . . . . 19
- 9.2. Informative References . . . . . . . . . . . . . . . . . 21
- Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 2]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-1. Introduction
-
- Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box
- Benchmarks of Throughput, Latency, Forwarding Rates and others have
- served our industry for many years. Now, Network Function
- Virtualization (NFV) has the goal to transform how internetwork
- functions are implemented, and therefore has garnered much attention.
-
- This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization,
- "VSWITCHPERF". This project intends to build on the current and
- completed work of the Benchmarking Methodology Working Group in IETF,
- by referencing existing literature. For example, currently the most
- often referenced RFC is [RFC2544] (which depends on [RFC1242]) and
- foundation of the benchmarking work in OPNFV is common and strong.
-
- See https://wiki.opnfv.org/
- characterize_vswitch_performance_for_telco_nfv_use_cases for more
- background, and the OPNFV website for general information:
- https://www.opnfv.org/
-
- The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on
- existing "telco" services as opposed to cloud-computing. There are
- many ways in which telco requirements have different emphasis on
- performance dimensions when compared to cloud computing: support for
- and transfer of isochronous media streams is one example.
-
- Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.
-
-2. Scope
-
- The primary purpose and scope of the memo is to inform BMWG of work-
- in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here
- is received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's
- chartered work to characterize the NFV Infrastructure (a virtual
- switch is an important aspect of that infrastructure).
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 3]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-3. Benchmarking Considerations
-
- This section highlights some specific considerations (from
- [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these
- areas, as they develop their specifications in the Level Test Design
- (LTD) document.
-
-3.1. Comparison with Physical Network Functions
-
- To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this
- memo re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.
-
- It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.
-
-3.2. Continued Emphasis on Black-Box Benchmarks
-
- External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation
- will be provided in parallel to assist the development of operations
- procedures when the technology is deployed.
-
-3.3. New Configuration Parameters
-
- A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.
-
- Hardware details including:
-
- o Platform details
-
- o Processor details
-
- o Memory information (type and size)
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 4]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o Number of enabled cores
-
- o Number of cores used for the test
-
- o Number of physical NICs, as well as their details (manufacturer,
- versions, type and the PCI slot they are plugged into)
-
- o NIC interrupt configuration
-
- o BIOS version, release date and any configurations that were
- modified
-
- o CPU microcode level
-
- o Memory DIMM configurations (quad rank performance may not be the
- same as dual rank) in size, freq and slot locations
-
- o PCI configuration parameters (payload size, early ack option...)
-
- o Power management at all levels (ACPI sleep states, processor
- package, OS...)
-
- Software details including:
-
- o OS parameters and behavior (text vs graphical no one typing at the
- console on one system)
-
- o OS version (for host and VNF)
-
- o Kernel version (for host and VNF)
-
- o GRUB boot parameters (for host and VNF)
-
- o Hypervisor details (Type and version)
-
- o Selected vSwitch, version number or commit id used
-
- o vSwitch launch command line if it has been parameterised
-
- o Memory allocation to the vSwitch
-
- o which NUMA node it is using, and how many memory channels
-
- o DPDK or any other SW dependency version number or commit id used
-
- o Memory allocation to a VM - if it's from Hugpages/elsewhere
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 5]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o VM storage type: snapshot/independent persistent/independent non-
- persistent
-
- o Number of VMs
-
- o Number of Virtual NICs (vNICs), versions, type and driver
-
- o Number of virtual CPUs and their core affinity on the host
-
- o Number vNIC interrupt configuration
-
- o Thread affinitization for the applications (including the vSwitch
- itself) on the host
-
- o Details of Resource isolation, such as CPUs designated for Host/
- Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.
-
- Test Traffic Information:
-
- o Traffic type - UDP, TCP, IMIX / Other
-
- o Packet Sizes
-
- o Deployment Scenario
-
-3.4. Flow classification
-
- Virtual switches group packets into flows by processing and matching
- particular packet or frame header information, or by matching packets
- based on the input ports. Thus a flow can be thought of a sequence
- of packets that have the same set of header field values or have
- arrived on the same port. Performance results can vary based on the
- parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the
- flow timeout time on a vSwitch before conducting any performance
- tests that do not measure the flow setup time. Normally the first
- packet of a particular stream will install the flow in the virtual
- switch which adds an additional latency, subsequent packets of the
- same flow are not subject to this latency if the flow is already
- installed on the vSwitch.
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 6]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-3.5. Benchmarks using Baselines with Resource Isolation
-
- This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.
-
- 1. Baselines:
-
- * Optional: Benchmark platform forwarding capability without a
- vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency).
-
- Benchmark platform forwarding capability
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
- * Benchmark VNF forwarding capability with direct connectivity
- (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves
- as a means of VNF validation and a means to obtain the base
- performance for the VNF in terms of its maximum forwarding
- rate and latency). The metrics gathered from this test will
- serve as a key comparison point for vSwitch bypass
- technologies performance and vSwitch performance.
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 7]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Benchmark VNF forwarding capability
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
- * Benchmarking with isolated resources alone, with other
- resources (both HW&SW) disabled Example, vSw and VM are SUT
-
- * Benchmarking with isolated resources alone, leaving some
- resources unused
-
- * Benchmark with isolated resources and all resources occupied
-
- 2. Next Steps
-
- * Limited sharing
-
- * Production scenarios
-
- * Stressful scenarios
-
-4. VSWITCHPERF Specification Summary
-
- The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the pre-
- existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:
-
- o [RFC2544] Benchmarking Methodology for Network Interconnect
- Devices
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 8]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o [RFC2889] Benchmarking Methodology for LAN Switching
-
- o [RFC6201] Device Reset Characterization
-
- o [RFC5481] Packet Delay Variation Applicability Statement
-
- Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test
- equipment developers. Fortunately, many members of the testing
- system community have engaged on the VSPERF project, including an
- open source test system.
-
- In addition to this, the LTD also re-uses the terminology defined by:
-
- o [RFC2285] Benchmarking Terminology for LAN Switching Devices
-
- o [RFC5481] Packet Delay Variation Applicability Statement
-
- Specifications to be included in future updates of the LTD include:
-
- o [RFC3918] Methodology for IP Multicast Benchmarking
-
- o [RFC4737] Packet Reordering Metrics
-
- As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.
-
- When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics.
- In this case, the project specifications have referenced metrics from
- the IETF IP Performance Metrics (IPPM) literature. This means that
- the [RFC2544] test of Latency is replaced by measurement of a metric
- derived from IPPM's [RFC2679], where a set of statistical summaries
- will be provided (mean, max, min, etc.). Further metrics planned to
- be benchmarked include packet delay variation as defined by [RFC5481]
- , reordering, burst behaviour, DUT availability, DUT capacity and
- packet loss in long term testing at Throughput level, where some low-
- level of background loss may be present and characterized.
-
- Tests have been (or will be) designed to collect the metrics below:
-
- o Throughput Tests to measure the maximum forwarding rate (in frames
- per second or fps) and bit rate (in Mbps) for a constant load (as
- defined by RFC1242) without traffic loss.
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 9]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.
-
- o Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.
-
- o Scalability Tests to understand how the virtual switch performs as
- the number of flows, active ports, complexity of the forwarding
- logic's configuration... it has to deal with increases.
-
- o Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data
- through the switch.
-
- o Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as
- the effect of this coupling on the performance of the DUT
- (example: delay of the initial packet of a flow).
-
- o CPU and Memory Consumption Tests to understand the virtual
- switch's footprint on the system, usually conducted as auxiliary
- measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.
-
- Future/planned test specs include:
-
- o Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.
-
- o Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.
-
- o Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements [IFA003] on characterization of acceleration
- technologies applied to vswitches.
-
- The flexibility of deployment of a virtual switch within a network
- means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 10]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to physical port
-
- __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 11]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF to virtual switch to physical
- port
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 12]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF to virtual switch to VNF to
- virtual switch to physical port
-
- __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 13]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 14]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- VNF to virtual switch to physical port
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 15]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- VNF to virtual switch to VNF
-
- __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|
-
- A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page [TestTopo].
-
-5. 3x3 Matrix Coverage
-
- This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]). Because
- the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all
- are occupied, also the matrix has grown to 3x4 to accommodate scale
- metrics when displaying the coverage of many metrics/benchmarks).
-
- The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.
-
- A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV].
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 16]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-5.1. Speed of Activation
-
- o Activation.RFC2889.AddressLearningRate
-
- o PacketLatency.InitialPacketProcessingLatency
-
-5.2. Accuracy of Activation section
-
- o CPDP.Coupling.Flow.Addition
-
-5.3. Reliability of Activation
-
- o Throughput.RFC2544.SystemRecoveryTime
-
- o Throughput.RFC2544.ResetTime
-
-5.4. Scale of Activation
-
- o Activation.RFC2889.AddressCachingCapacity
-
-5.5. Speed of Operation
-
- o Throughput.RFC2544.PacketLossRate
-
- o CPU.RFC2544.0PacketLoss
-
- o Throughput.RFC2544.PacketLossRateFrameModification
-
- o Throughput.RFC2544.BackToBackFrames
-
- o Throughput.RFC2889.MaxForwardingRate
-
- o Throughput.RFC2889.ForwardPressure
-
- o Throughput.RFC2889.BroadcastFrameForwarding
-
-5.6. Accuracy of Operation
-
- o Throughput.RFC2889.ErrorFramesFiltering
-
- o Throughput.RFC2544.Profile
-
-5.7. Reliability of Operation
-
- o Throughput.RFC2889.Soak
-
- o Throughput.RFC2889.SoakFrameModification
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 17]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o PacketDelayVariation.RFC3393.Soak
-
-5.8. Scalability of Operation
-
- o Scalability.RFC2544.0PacketLoss
-
- o MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-
-5.9. Summary
-
-|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|
-
-6. Security Considerations
-
- Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.
-
- The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test
- traffic into a production network, or misroute traffic to the test
- management network.
-
- Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.
-
- Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 18]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-7. IANA Considerations
-
- No IANA Action is requested at this time.
-
-8. Acknowledgements
-
- The authors acknowledge
-
-9. References
-
-9.1. Normative References
-
- [NFV.PER001]
- "Network Function Virtualization: Performance and
- Portability Best Practices", Group Specification ETSI GS
- NFV-PER 001 V1.1.1 (2014-06), June 2014.
-
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
- Requirement Levels", BCP 14, RFC 2119,
- DOI 10.17487/RFC2119, March 1997,
- <http://www.rfc-editor.org/info/rfc2119>.
-
- [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
- Switching Devices", RFC 2285, DOI 10.17487/RFC2285,
- February 1998, <http://www.rfc-editor.org/info/rfc2285>.
-
- [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
- "Framework for IP Performance Metrics", RFC 2330,
- DOI 10.17487/RFC2330, May 1998,
- <http://www.rfc-editor.org/info/rfc2330>.
-
- [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
- Network Interconnect Devices", RFC 2544,
- DOI 10.17487/RFC2544, March 1999,
- <http://www.rfc-editor.org/info/rfc2544>.
-
- [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
- Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679,
- September 1999, <http://www.rfc-editor.org/info/rfc2679>.
-
- [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
- Packet Loss Metric for IPPM", RFC 2680,
- DOI 10.17487/RFC2680, September 1999,
- <http://www.rfc-editor.org/info/rfc2680>.
-
- [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
- Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681,
- September 1999, <http://www.rfc-editor.org/info/rfc2681>.
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 19]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology
- for LAN Switching Devices", RFC 2889,
- DOI 10.17487/RFC2889, August 2000,
- <http://www.rfc-editor.org/info/rfc2889>.
-
- [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
- Metric for IP Performance Metrics (IPPM)", RFC 3393,
- DOI 10.17487/RFC3393, November 2002,
- <http://www.rfc-editor.org/info/rfc3393>.
-
- [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
- performance measurement with periodic streams", RFC 3432,
- DOI 10.17487/RFC3432, November 2002,
- <http://www.rfc-editor.org/info/rfc3432>.
-
- [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast
- Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October
- 2004, <http://www.rfc-editor.org/info/rfc3918>.
-
- [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
- "Terminology for Benchmarking Network-layer Traffic
- Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
- October 2006, <http://www.rfc-editor.org/info/rfc4689>.
-
- [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
- S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
- DOI 10.17487/RFC4737, November 2006,
- <http://www.rfc-editor.org/info/rfc4737>.
-
- [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
- Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
- RFC 5357, DOI 10.17487/RFC5357, October 2008,
- <http://www.rfc-editor.org/info/rfc5357>.
-
- [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
- "Network Time Protocol Version 4: Protocol and Algorithms
- Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
- <http://www.rfc-editor.org/info/rfc5905>.
-
- [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera,
- "Device Reset Characterization", RFC 6201,
- DOI 10.17487/RFC6201, March 2011,
- <http://www.rfc-editor.org/info/rfc6201>.
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 20]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-9.2. Informative References
-
- [I-D.ietf-bmwg-virtual-net]
- Morton, A., "Considerations for Benchmarking Virtual
- Network Functions and Their Infrastructure", draft-ietf-
- bmwg-virtual-net-01 (work in progress), September 2015.
-
- [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/
- IFA003_Acceleration_-_vSwitch_Spec/".
-
- [LTDoverV]
- "LTD Test Spec Overview https://wiki.opnfv.org/wiki/
- vswitchperf_test_spec_review".
-
- [RFC1242] Bradner, S., "Benchmarking Terminology for Network
- Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
- July 1991, <http://www.rfc-editor.org/info/rfc1242>.
-
- [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
- Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
- March 2009, <http://www.rfc-editor.org/info/rfc5481>.
-
- [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
- Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
- <http://www.rfc-editor.org/info/rfc6049>.
-
- [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics
- (IPPM) Registry of Metrics Are Obsolete", RFC 6248,
- DOI 10.17487/RFC6248, April 2011,
- <http://www.rfc-editor.org/info/rfc6248>.
-
- [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New
- Performance Metric Development", BCP 170, RFC 6390,
- DOI 10.17487/RFC6390, October 2011,
- <http://www.rfc-editor.org/info/rfc6390>.
-
- [TestTopo]
- "Test Topologies https://wiki.opnfv.org/vsperf/
- test_methodology".
-
-Authors' Addresses
-
- Maryam Tahhan
- Intel
-
- Email: maryam.tahhan@intel.com
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 21]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Billy O'Mahony
- Intel
-
- Email: billy.o.mahony@intel.com
-
-
- Al Morton
- AT&T Labs
- 200 Laurel Avenue South
- Middletown,, NJ 07748
- USA
-
- Phone: +1 732 420 1571
- Fax: +1 732 368 1192
- Email: acmorton@att.com
- URI: http://home.comcast.net/~acmacm/
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 22]
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml
deleted file mode 100755
index b5f7f833..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml
+++ /dev/null
@@ -1,964 +0,0 @@
-<?xml version="1.0" encoding="US-ASCII"?>
-<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
-<?rfc toc="yes"?>
-<?rfc tocompact="yes"?>
-<?rfc tocdepth="3"?>
-<?rfc tocindent="yes"?>
-<?rfc symrefs="yes"?>
-<?rfc sortrefs="yes"?>
-<?rfc comments="yes"?>
-<?rfc inline="yes"?>
-<?rfc compact="yes"?>
-<?rfc subcompact="no"?>
-<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-01"
- ipr="trust200902">
- <front>
- <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
- OPNFV</title>
-
- <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>maryam.tahhan@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>billy.o.mahony@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Al Morton" initials="A." surname="Morton">
- <organization>AT&amp;T Labs</organization>
-
- <address>
- <postal>
- <street>200 Laurel Avenue South</street>
-
- <city>Middletown,</city>
-
- <region>NJ</region>
-
- <code>07748</code>
-
- <country>USA</country>
- </postal>
-
- <phone>+1 732 420 1571</phone>
-
- <facsimile>+1 732 368 1192</facsimile>
-
- <email>acmorton@att.com</email>
-
- <uri>http://home.comcast.net/~acmacm/</uri>
- </address>
- </author>
-
- <date day="14" month="October" year="2015"/>
-
- <abstract>
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the Benchmarking
- Methodology Working Group in IETF, by referencing existing literature.
- The Benchmarking Methodology Working Group has traditionally conducted
- laboratory characterization of dedicated physical implementations of
- internetworking functions. Therefore, this memo begins to describe the
- additional considerations when virtual switches are implemented in
- general-purpose hardware. The expanded tests and benchmarks are also
- influenced by the OPNFV mission to support virtualization of the "telco"
- infrastructure.</t>
- </abstract>
-
- <note title="Requirements Language">
- <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in <xref
- target="RFC2119">RFC 2119</xref>.</t>
-
- <t/>
- </note>
- </front>
-
- <middle>
- <section title="Introduction">
- <t>Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box Benchmarks
- of Throughput, Latency, Forwarding Rates and others have served our
- industry for many years. Now, Network Function Virtualization (NFV) has
- the goal to transform how internetwork functions are implemented, and
- therefore has garnered much attention.</t>
-
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization, "VSWITCHPERF".
- This project intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing existing
- literature. For example, currently the most often referenced RFC is
- <xref target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and
- foundation of the benchmarking work in OPNFV is common and strong.</t>
-
- <t>See
- https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
- for more background, and the OPNFV website for general information:
- https://www.opnfv.org/</t>
-
- <t>The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on existing
- "telco" services as opposed to cloud-computing. There are many ways in
- which telco requirements have different emphasis on performance
- dimensions when compared to cloud computing: support for and transfer of
- isochronous media streams is one example.</t>
-
- <t>Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.</t>
- </section>
-
- <section title="Scope">
- <t>The primary purpose and scope of the memo is to inform BMWG of
- work-in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here is
- received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's chartered
- work to characterize the NFV Infrastructure (a virtual switch is an
- important aspect of that infrastructure).</t>
- </section>
-
- <section title="Benchmarking Considerations">
- <t>This section highlights some specific considerations (from <xref
- target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these areas,
- as they develop their specifications in the Level Test Design (LTD)
- document.</t>
-
- <section title="Comparison with Physical Network Functions">
- <t>To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this memo
- re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.</t>
-
- <t>It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.</t>
- </section>
-
- <section title="Continued Emphasis on Black-Box Benchmarks">
- <t>External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation will
- be provided in parallel to assist the development of operations
- procedures when the technology is deployed.</t>
- </section>
-
- <section title="New Configuration Parameters">
- <t>A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.</t>
-
- <t>Hardware details including:</t>
-
- <t><list style="symbols">
- <t>Platform details</t>
-
- <t>Processor details</t>
-
- <t>Memory information (type and size)</t>
-
- <t>Number of enabled cores</t>
-
- <t>Number of cores used for the test</t>
-
- <t>Number of physical NICs, as well as their details
- (manufacturer, versions, type and the PCI slot they are plugged
- into)</t>
-
- <t>NIC interrupt configuration</t>
-
- <t>BIOS version, release date and any configurations that were
- modified</t>
-
- <t>CPU microcode level</t>
-
- <t>Memory DIMM configurations (quad rank performance may not be
- the same as dual rank) in size, freq and slot locations</t>
-
- <t>PCI configuration parameters (payload size, early ack
- option...)</t>
-
- <t>Power management at all levels (ACPI sleep states, processor
- package, OS...)</t>
- </list>Software details including:</t>
-
- <t><list style="symbols">
- <t>OS parameters and behavior (text vs graphical no one typing at
- the console on one system)</t>
-
- <t>OS version (for host and VNF)</t>
-
- <t>Kernel version (for host and VNF)</t>
-
- <t>GRUB boot parameters (for host and VNF)</t>
-
- <t>Hypervisor details (Type and version)</t>
-
- <t>Selected vSwitch, version number or commit id used</t>
-
- <t>vSwitch launch command line if it has been parameterised</t>
-
- <t>Memory allocation to the vSwitch</t>
-
- <t>which NUMA node it is using, and how many memory channels</t>
-
- <t>DPDK or any other SW dependency version number or commit id
- used</t>
-
- <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t>
-
- <t>VM storage type: snapshot/independent persistent/independent
- non-persistent</t>
-
- <t>Number of VMs</t>
-
- <t>Number of Virtual NICs (vNICs), versions, type and driver</t>
-
- <t>Number of virtual CPUs and their core affinity on the host</t>
-
- <t>Number vNIC interrupt configuration</t>
-
- <t>Thread affinitization for the applications (including the
- vSwitch itself) on the host</t>
-
- <t>Details of Resource isolation, such as CPUs designated for
- Host/Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.</t>
- </list></t>
-
- <t>Test Traffic Information:<list style="symbols">
- <t>Traffic type - UDP, TCP, IMIX / Other</t>
-
- <t>Packet Sizes</t>
-
- <t>Deployment Scenario</t>
- </list></t>
-
- <t/>
- </section>
-
- <section title="Flow classification">
- <t>Virtual switches group packets into flows by processing and
- matching particular packet or frame header information, or by matching
- packets based on the input ports. Thus a flow can be thought of a
- sequence of packets that have the same set of header field values or
- have arrived on the same port. Performance results can vary based on
- the parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the flow
- timeout time on a vSwitch before conducting any performance tests that
- do not measure the flow setup time. Normally the first packet of a
- particular stream will install the flow in the virtual switch which
- adds an additional latency, subsequent packets of the same flow are
- not subject to this latency if the flow is already installed on the
- vSwitch.</t>
- </section>
-
- <section title="Benchmarks using Baselines with Resource Isolation">
- <t>This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.</t>
-
- <t><list style="numbers">
- <t>Baselines: <list style="symbols">
- <t>Optional: Benchmark platform forwarding capability without
- a vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency). <figure>
- <preamble>Benchmark platform forwarding
- capability</preamble>
-
- <artwork align="right"><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmark VNF forwarding capability with direct
- connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72
- hours (serves as a means of VNF validation and a means to
- obtain the base performance for the VNF in terms of its
- maximum forwarding rate and latency). The metrics gathered
- from this test will serve as a key comparison point for
- vSwitch bypass technologies performance and vSwitch
- performance. <figure align="right">
- <preamble>Benchmark VNF forwarding capability</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmarking with isolated resources alone, with other
- resources (both HW&amp;SW) disabled Example, vSw and VM are
- SUT</t>
-
- <t>Benchmarking with isolated resources alone, leaving some
- resources unused</t>
-
- <t>Benchmark with isolated resources and all resources
- occupied</t>
- </list></t>
-
- <t>Next Steps<list style="symbols">
- <t>Limited sharing</t>
-
- <t>Production scenarios</t>
-
- <t>Stressful scenarios</t>
- </list></t>
- </list></t>
- </section>
- </section>
-
- <section title="VSWITCHPERF Specification Summary">
- <t>The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the
- pre-existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2544"/> Benchmarking Methodology for Network
- Interconnect Devices</t>
-
- <t><xref target="RFC2889"/> Benchmarking Methodology for LAN
- Switching</t>
-
- <t><xref target="RFC6201"/> Device Reset Characterization</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t>Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test equipment
- developers. Fortunately, many members of the testing system community
- have engaged on the VSPERF project, including an open source test
- system.</t>
-
- <t>In addition to this, the LTD also re-uses the terminology defined
- by:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2285"/> Benchmarking Terminology for LAN
- Switching Devices</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t/>
-
- <t>Specifications to be included in future updates of the LTD
- include:<list style="symbols">
- <t><xref target="RFC3918"/> Methodology for IP Multicast
- Benchmarking</t>
-
- <t><xref target="RFC4737"/> Packet Reordering Metrics</t>
- </list></t>
-
- <t>As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.</t>
-
- <t>When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics. In
- this case, the project specifications have referenced metrics from the
- IETF IP Performance Metrics (IPPM) literature. This means that the <xref
- target="RFC2544"/> test of Latency is replaced by measurement of a
- metric derived from IPPM's <xref target="RFC2679"/>, where a set of
- statistical summaries will be provided (mean, max, min, etc.). Further
- metrics planned to be benchmarked include packet delay variation as
- defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
- availability, DUT capacity and packet loss in long term testing at
- Throughput level, where some low-level of background loss may be present
- and characterized.</t>
-
- <t>Tests have been (or will be) designed to collect the metrics
- below:</t>
-
- <t><list style="symbols">
- <t>Throughput Tests to measure the maximum forwarding rate (in
- frames per second or fps) and bit rate (in Mbps) for a constant load
- (as defined by RFC1242) without traffic loss.</t>
-
- <t>Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.</t>
-
- <t>Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.</t>
-
- <t>Scalability Tests to understand how the virtual switch performs
- as the number of flows, active ports, complexity of the forwarding
- logic&rsquo;s configuration&hellip; it has to deal with
- increases.</t>
-
- <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data through
- the switch.</t>
-
- <t>Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as the
- effect of this coupling on the performance of the DUT (example:
- delay of the initial packet of a flow).</t>
-
- <t>CPU and Memory Consumption Tests to understand the virtual
- switch&rsquo;s footprint on the system, usually conducted as
- auxiliary measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.</t>
- </list></t>
-
- <t>Future/planned test specs include:<list style="symbols">
- <t>Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.</t>
-
- <t>Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.</t>
-
- <t>Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements <xref target="IFA003"/> on characterization of
- acceleration technologies applied to vswitches.</t>
- </list>The flexibility of deployment of a virtual switch within a
- network means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:</t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to physical
- port</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure></t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|]]></artwork>
- </figure></t>
-
- <t>A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page <xref target="TestTopo"/>. </t>
- </section>
-
- <section title="3x3 Matrix Coverage">
- <t>This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>).
- Because the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all are
- occupied, also the matrix has grown to 3x4 to accommodate scale metrics
- when displaying the coverage of many metrics/benchmarks).</t>
-
- <t>The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.</t>
-
- <t>A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t>
-
- <section title="Speed of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressLearningRate</t>
-
- <t>PacketLatency.InitialPacketProcessingLatency</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Activation section">
- <t><list style="symbols">
- <t>CPDP.Coupling.Flow.Addition</t>
- </list></t>
- </section>
-
- <section title="Reliability of Activation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.SystemRecoveryTime</t>
-
- <t>Throughput.RFC2544.ResetTime</t>
- </list></t>
- </section>
-
- <section title="Scale of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressCachingCapacity</t>
- </list></t>
- </section>
-
- <section title="Speed of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.PacketLossRate</t>
-
- <t>CPU.RFC2544.0PacketLoss</t>
-
- <t>Throughput.RFC2544.PacketLossRateFrameModification</t>
-
- <t>Throughput.RFC2544.BackToBackFrames</t>
-
- <t>Throughput.RFC2889.MaxForwardingRate</t>
-
- <t>Throughput.RFC2889.ForwardPressure</t>
-
- <t>Throughput.RFC2889.BroadcastFrameForwarding</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.ErrorFramesFiltering</t>
-
- <t>Throughput.RFC2544.Profile</t>
- </list></t>
- </section>
-
- <section title="Reliability of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.Soak</t>
-
- <t>Throughput.RFC2889.SoakFrameModification</t>
-
- <t>PacketDelayVariation.RFC3393.Soak</t>
- </list></t>
- </section>
-
- <section title="Scalability of Operation">
- <t><list style="symbols">
- <t>Scalability.RFC2544.0PacketLoss</t>
-
- <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t>
- </list></t>
- </section>
-
- <section title="Summary">
- <t><figure>
- <artwork><![CDATA[|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|]]></artwork>
- </figure></t>
- </section>
- </section>
-
- <section title="Security Considerations">
- <t>Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.</t>
-
- <t>The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test traffic
- into a production network, or misroute traffic to the test management
- network.</t>
-
- <t>Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.</t>
-
- <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.</t>
- </section>
-
- <section anchor="IANA" title="IANA Considerations">
- <t>No IANA Action is requested at this time.</t>
- </section>
-
- <section title="Acknowledgements">
- <t>The authors acknowledge</t>
- </section>
- </middle>
-
- <back>
- <references title="Normative References">
- <?rfc ?>
-
- <?rfc include="reference.RFC.2119"?>
-
- <?rfc include="reference.RFC.2330"?>
-
- <?rfc include='reference.RFC.2544'?>
-
- <?rfc include="reference.RFC.2679"?>
-
- <?rfc include='reference.RFC.2680'?>
-
- <?rfc include='reference.RFC.3393'?>
-
- <?rfc include='reference.RFC.3432'?>
-
- <?rfc include='reference.RFC.2681'?>
-
- <?rfc include='reference.RFC.5905'?>
-
- <?rfc include='reference.RFC.4689'?>
-
- <?rfc include='reference.RFC.4737'?>
-
- <?rfc include='reference.RFC.5357'?>
-
- <?rfc include='reference.RFC.2889'?>
-
- <?rfc include='reference.RFC.3918'?>
-
- <?rfc include='reference.RFC.6201'?>
-
- <?rfc include='reference.RFC.2285'?>
-
- <reference anchor="NFV.PER001">
- <front>
- <title>Network Function Virtualization: Performance and Portability
- Best Practices</title>
-
- <author fullname="ETSI NFV" initials="" surname="">
- <organization/>
- </author>
-
- <date month="June" year="2014"/>
- </front>
-
- <seriesInfo name="Group Specification"
- value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
-
- <format type="PDF"/>
- </reference>
- </references>
-
- <references title="Informative References">
- <?rfc include='reference.RFC.1242'?>
-
- <?rfc include='reference.RFC.5481'?>
-
- <?rfc include='reference.RFC.6049'?>
-
- <?rfc include='reference.RFC.6248'?>
-
- <?rfc include='reference.RFC.6390'?>
-
- <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?>
-
- <reference anchor="TestTopo">
- <front>
- <title>Test Topologies
- https://wiki.opnfv.org/vsperf/test_methodology</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="LTDoverV">
- <front>
- <title>LTD Test Spec Overview
- https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review </title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="IFA003">
- <front>
- <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
- </references>
- </back>
-</rfc>
diff --git a/docs/test_spec/index.rst b/docs/test_spec/index.rst
deleted file mode 100755
index e36b7d85..00000000
--- a/docs/test_spec/index.rst
+++ /dev/null
@@ -1,17 +0,0 @@
-=========================
-VSPERF Test Specification
-=========================
-
-.. toctree::
- :numbered:
- :maxdepth: 4
-
- quickstart.rst
- installation.rst
- NEWS.rst
- vswitchperf_ltd.rst
- vswitchperf_design.rst
-
-Revision: _sha1_
-
-Build date: |today|
diff --git a/docs/test_spec/vswitchperf_ltd.rst b/docs/test_spec/vswitchperf_ltd.rst
deleted file mode 100644
index 7adc864f..00000000
--- a/docs/test_spec/vswitchperf_ltd.rst
+++ /dev/null
@@ -1,2050 +0,0 @@
-CHARACTERIZE VSWITCH PERFORMANCE FOR TELCO NFV USE CASES LEVEL TEST DESIGN
-==========================================================================
-
-.. contents:: Table of Contents
-
-1. Introduction
-===============
-
-The objective of the OPNFV project titled
-**“Characterize vSwitch Performance for Telco NFV Use Cases”**, is to
-evaluate a virtual switch to identify its suitability for a Telco
-Network Function Virtualization (NFV) environment. The intention of this
-Level Test Design (LTD) document is to specify the set of tests to carry
-out in order to objectively measure the current characteristics of a
-virtual switch in the Network Function Virtualization Infrastructure
-(NFVI) as well as the test pass criteria. The detailed test cases will
-be defined in `Section 2 <#DetailsOfTheLevelTestDesign>`__, preceded by
-the `Document identifier <#DocId>`__ and the `Scope <#Scope>`__.
-
-This document is currently in draft form.
-
-1.1. Document identifier
-------------------------
-
-The document id will be used to uniquely
-identify versions of the LTD. The format for the document id will be:
-OPNFV\_vswitchperf\_LTD\_ver\_NUM\_MONTH\_YEAR\_STATUS, where by the
-status is one of: draft, reviewed, corrected or final. The document id
-for this version of the LTD is:
-OPNFV\_vswitchperf\_LTD\_ver\_1.6\_Jan\_15\_DRAFT.
-
-1.2. Scope
-----------
-
-The main purpose of this project is to specify a suite of
-performance tests in order to objectively measure the current packet
-transfer characteristics of a virtual switch in the NFVI. The intent of
-the project is to facilitate testing of any virtual switch. Thus, a
-generic suite of tests shall be developed, with no hard dependencies to
-a single implementation. In addition, the test case suite shall be
-architecture independent.
-
-The test cases developed in this project shall not form part of a
-separate test framework, all of these tests may be inserted into the
-Continuous Integration Test Framework and/or the Platform Functionality
-Test Framework - if a vSwitch becomes a standard component of an OPNFV
-release.
-
-1.3. References
----------------
-
-* `RFC 1242 Benchmarking Terminology for Network Interconnection
- Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
-* `RFC 2544 Benchmarking Methodology for Network Interconnect
- Devices <http://www.ietf.org/rfc/rfc2544.txt>`__
-* `RFC 2285 Benchmarking Terminology for LAN Switching
- Devices <http://www.ietf.org/rfc/rfc2285.txt>`__
-* `RFC 2889 Benchmarking Methodology for LAN Switching
- Devices <http://www.ietf.org/rfc/rfc2889.txt>`__
-* `RFC 3918 Methodology for IP Multicast
- Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__
-* `RFC 4737 Packet Reordering
- Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__
-* `RFC 5481 Packet Delay Variation Applicability
- Statement <http://www.ietf.org/rfc/rfc5481.txt>`__
-* `RFC 6201 Device Reset
- Characterization <http://tools.ietf.org/html/rfc6201>`__
-
-2. Details of the Level Test Design
-===================================
-
-This section describes the features to be tested (`cf. 2.1
-<#FeaturesToBeTested>`__), the test approach (`cf. 2.2 <#Approach>`__);
-it also identifies the sets of test cases or scenarios (`cf. 2.3
-<#TestIdentification>`__) along with the pass/fail criteria (`cf. 2.4
-<#PassFail>`__) and the test deliverables (`cf. 2.5 <#TestDeliverables>`__).
-
-2.1. Features to be tested
---------------------------
-
-Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
-includes measuring the following performance metrics:
-
-- **Throughput** as defined by `RFC1242
- <https://www.rfc-editor.org/rfc/rfc1242.txt>`__: The maximum rate at which
- **none** of the offered frames are dropped by the DUT. The maximum frame
- rate and bit rate that can be transmitted by the DUT without any error
- should be recorded. Note there is an equivalent bit rate and a specific
- layer at which the payloads contribute to the bits. Errors and
- improperly formed frames or packets are dropped.
-- **Packet delay** introduced by the DUT and its cumulative effect on
- E2E networks. Frame delay can be measured equivalently.
-- **Packet delay variation**: measured from the perspective of the
- VNF/application. Packet delay variation is sometimes called "jitter".
- However, we will avoid the term "jitter" as the term holds different
- meaning to different groups of people. In this document we will
- simply use the term packet delay variation. The preferred form for this
- metric is the PDV form of delay variation defined in `RFC5481
- <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. The most relevant
- measurement of PDV considers the delay variation of a single user flow,
- as this will be relevant to the size of end-system buffers to compensate
- for delay variation. The measurement system's ability to store the
- delays of individual packets in the flow of interest is a key factor
- that determines the specific measurement method. At the outset, it is
- ideal to view the complete PDV distribution. Systems that can capture
- and store packets and their delays have the freedom to calculate the
- reference minimum delay and to determine various quantiles of the PDV
- distribution accurately (in post-measurement processing routines).
- Systems without storage must apply algorithms to calculate delay and
- statistical measurements on the fly. For example, a system may store
- temporary estimates of the mimimum delay and the set of (100) packets
- with the longest delays during measurement (to calculate a high quantile,
- and update these sets with new values periodically.
- In some cases, a limited number of delay histogram bins will be
- available, and the bin limits will need to be set using results from
- repeated experiments. See section 8 of `RFC5481
- <https://www.rfc-editor.org/rfc/rfc5481.txt>`__.
-- **Packet loss** (within a configured waiting time at the receiver): All
- packets sent to the DUT should be accounted for.
-- **Burst behaviour**: measures the ability of the DUT to buffer packets.
-- **Packet re-ordering**: measures the ability of the device under test to
- maintain sending order throughout transfer to the destination.
-- **Packet correctness**: packets or Frames must be well-formed, in that
- they include all required fields, conform to length requirements, pass
- integrity checks, etc.
-- **Availability and capacity** of the DUT i.e. when the DUT is fully “up”
- and connected:
-
- - Includes power consumption of the CPU (in various power states) and
- system.
- - Includes CPU utilization.
- - Includes the number of NIC interfaces supported.
- - Includes headroom of VM workload processing cores (i.e. available
- for applications).
-
-
-2.2. Approach
-==============
-
-In order to determine the packet transfer characteristics of a virtual
-switch, the tests will be broken down into the following categories:
-
-2.2.1 Test Categories
-----------------------
-- **Throughput Tests** to measure the maximum forwarding rate (in
- frames per second or fps) and bit rate (in Mbps) for a constant load
- (as defined by `RFC1242 <https://www.rfc-editor.org/rfc/rfc1242.txt>`__)
- without traffic loss.
-- **Packet and Frame Delay Tests** to measure average, min and max
- packet and frame delay for constant loads.
-- **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data through
- the virtual switch.
-- **Request/Response Performance** Tests (TCP, UDP) the measure the
- transaction rate through the virtual switch.
-- **Packet Delay Tests** to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.
-- **Scalability Tests** to understand how the virtual switch performs
- as the number of flows, active ports, complexity of the forwarding
- logic's configuration... it has to deal with increases.
-- **Control Path and Datapath Coupling** Tests, to understand how
- closely coupled the datapath and the control path are as well as the
- effect of this coupling on the performance of the DUT.
-- **CPU and Memory Consumption Tests** to understand the virtual
- switch’s footprint on the system, this includes:
-
- * CPU utilization
- * Cache utilization
- * Memory footprint
- * Time To Establish Flows Tests.
-
-- **Noisy Neighbour Tests**, to understand the effects of resource
- sharing on the performance of a virtual switch.
-
-**Note:** some of the tests above can be conducted simultaneously where
-the combined results would be insightful, for example Packet/Frame Delay
-and Scalability.
-
-2.2.2 Deployment Scenarios
---------------------------
-The following represents possible deployments which can help to
-determine the performance of both the virtual switch and the datapath
-into the VNF:
-
-Physical port → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
-
- _
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ _|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-Physical port → vSwitch → VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
-
- _
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ _|
- ^ :
- | |
- : v _
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ _|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+ _|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | L-----------------+ v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+ _|
- ^ ^ : :
- | | | |
- : : v v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-Physical port → VNF → vSwitch → VNF → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- |+-------------------+ | | +-------------------+| |
- || Application | | | | Application || |
- |+-------------------+ | | +-------------------+| |
- | ^ | | | ^ | | | Guests
- | | v | | | v | |
- |+-------------------+ | | +-------------------+| |
- || logical ports | | | | logical ports || |
- || 0 1 | | | | 0 1 || |
- ++--------------------++ ++--------------------++ _|
- ^ : ^ :
- (PCI passthrough) | | (PCI passthrough)
- | v : | _
- +--------++------------+-+------------++---------+ |
- | | || 0 | | 1 || | | |
- | | ||logical port| |logical port|| | | |
- | | |+------------+ +------------+| | | |
- | | | | ^ | | | |
- | | | L-----------------+ | | | |
- | | | | | | | Host
- | | | vSwitch | | | |
- | | +-----------------------------+ | | |
- | | | | |
- | | v | |
- | +--------------+ +--------------+ | |
- | | phy port/VF | | phy port/VF | | |
- +-+--------------+--------------+--------------+-+ _|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-Physical port → vSwitch → VNF
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ _|
- ^
- |
- : _
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ _|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ _|
- :
- |
- v _
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ _|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-VNF → vSwitch → VNF → vSwitch
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +-------------------------+ +-------------------------+ |
- | Guest 1 | | Guest 2 | |
- | +-----------------+ | | +-----------------+ | |
- | | Application | | | | Application | | |
- | +-----------------+ | | +-----------------+ | |
- | : | | ^ | |
- | | | | | | | Guest
- | v | | : | |
- | +---------------+ | | +---------------+ | |
- | | logical port 0| | | | logical port 0| | |
- +-----+---------------+---+ +---+---------------+-----+ _|
- : ^
- | |
- v : _
- +----+---------------+------------+---------------+-----+ |
- | | port 0 | | port 1 | | |
- | +---------------+ +---------------+ | |
- | : ^ | |
- | | | | | Host
- | +--------------------+ | |
- | | |
- | vswitch | |
- +-------------------------------------------------------+ _|
-
-HOST 1(Physical port → virtual switch → VNF → virtual switch →
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Physical port) → HOST 2(Physical port → virtual switch → VNF →
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-virtual switch → Physical port)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
- .. code-block:: console
-
- _
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+ _|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+--+ +---+---------------+--+ |
- | | 0 1 | | | | 3 4 | | |
- | | logical ports | | | | logical ports | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | | Hosts
- | | v | | | v | |
- | +--------------+ | | +--------------+ | |
- | | phy ports | | | | phy ports | | |
- +---+--------------+---+ +---+--------------+---+ _|
- ^ : : :
- | +-----------------+ |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-**Note:** For tests where the traffic generator and/or measurement
-receiver are implemented on VM and connected to the virtual switch
-through vNIC, the issues of shared resources and interactions between
-the measurement devices and the device under test must be considered.
-
-**Note:** Some RFC 2889 tests require a full-mesh sending and receiving
-pattern involving more than two ports. This possibility is illustrated in the
-Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
-diagram above (with 2 sending and 2 receiving ports, though all ports
-could be used bi-directionally).
-
-**Note:** When Deployment Scenarios are used in RFC 2889 address learning
-or cache capacity testing, an additional port from the vSwitch must be
-connected to the test device. This port is used to listen for flooded
-frames.
-
-2.2.3 General Methodology:
---------------------------
-To establish the baseline performance of the virtual switch, tests would
-initially be run with a simple workload in the VNF (the recommended
-simple workload VNF would be `DPDK <http://www.dpdk.org/>`__'s testpmd
-application forwarding packets in a VM or vloop\_vnf a simple kernel
-module that forwards traffic between two network interfaces inside the
-virtualized environment while bypassing the networking stack).
-Subsequently, the tests would also be executed with a real Telco
-workload running in the VNF, which would exercise the virtual switch in
-the context of higher level Telco NFV use cases, and prove that its
-underlying characteristics and behaviour can be measured and validated.
-Suitable real Telco workload VNFs are yet to be identified.
-
-2.2.3.1 Default Test Parameters
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following list identifies the default parameters for suite of
-tests:
-
-- Reference application: Simple forwarding or Open Source VNF.
-- Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR
- Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as
- maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>.
-- Reordering check: Tests should confirm that packets within a flow are
- not reordered.
-- Duplex: Unidirectional / Bidirectional. Default: Full duplex with
- traffic transmitting in both directions, as network traffic generally
- does not flow in a single direction. By default the data rate of
- transmitted traffic should be the same in both directions, please
- note that asymmetric traffic (e.g. downlink-heavy) tests will be
- mentioned explicitly for the relevant test cases.
-- Number of Flows: Default for non scalability tests is a single flow.
- For scalability tests the goal is to test with maximum supported
- flows but where possible will test up to 10 Million flows. Start with
- a single flow and scale up. By default flows should be added
- sequentially, tests that add flows simultaneously will explicitly
- call out their flow addition behaviour. Packets are generated across
- the flows uniformly with no burstiness. For multi-core tests should
- consider the number of packet flows based on vSwitch/VNF multi-thread
- implementation and behavior.
-
-- Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic.
-- Deployment scenarios are:
-- Physical → virtual switch → physical.
-- Physical → virtual switch → VNF → virtual switch → physical.
-- Physical → virtual switch → VNF → virtual switch → VNF → virtual
- switch → physical.
-- Physical → VNF → virtual switch → VNF → physical.
-- Physical → virtual switch → VNF.
-- VNF → virtual switch → Physical.
-- VNF → virtual switch → VNF.
-
-Tests MUST have these parameters unless otherwise stated. **Test cases
-with non default parameters will be stated explicitly**.
-
-**Note**: For throughput tests unless stated otherwise, test
-configurations should ensure that traffic traverses the installed flows
-through the virtual switch, i.e. flows are installed and have an appropriate
-time out that doesn't expire before packet transmission starts.
-
-2.2.3.2 Flow Classification
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Virtual switches classify packets into flows by processing and matching
-particular header fields in the packet/frame and/or the input port where
-the packets/frames arrived. The vSwitch then carries out an action on
-the group of packets that match the classification parameters. Thus a
-flow is considered to be a sequence of packets that have a shared set of
-header field values or have arrived on the same port and have the same
-action applied to them. Performance results can vary based on the
-parameters the vSwitch uses to match for a flow. The recommended flow
-classification parameters for L3 vSwitch performance tests are: the
-input port, the source IP address, the destination IP address and the
-Ethernet protocol type field. It is essential to increase the flow
-time-out time on a vSwitch before conducting any performance tests that
-do not measure the flow set-up time. Normally the first packet of a
-particular flow will install the flow in the vSwitch which adds an
-additional latency, subsequent packets of the same flow are not subject
-to this latency if the flow is already installed on the vSwitch.
-
-2.2.3.3 Test Priority
-~~~~~~~~~~~~~~~~~~~~~
-
-Tests will be assigned a priority in order to determine which tests
-should be implemented immediately and which tests implementations
-can be deferred.
-
-Priority can be of following types: - Urgent: Must be implemented
-immediately. - High: Must be implemented in the next release. - Medium:
-May be implemented after the release. - Low: May or may not be
-implemented at all.
-
-2.2.3.4 SUT Setup
-~~~~~~~~~~~~~~~~~
-
-The SUT should be configured to its "default" state. The
-SUT's configuration or set-up must not change between tests in any way
-other than what is required to do the test. All supported protocols must
-be configured and enabled for each test set up.
-
-2.2.3.4.1 Port Configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The DUT should be configured with n ports where
-n is a multiple of 2. Half of the ports on the DUT should be used as
-ingress ports and the other half of the ports on the DUT should be used
-as egress ports. Where a DUT has more than 2 ports, the ingress data
-streams should be set-up so that they transmit packets to the egress
-ports in sequence so that there is an even distribution of traffic
-across ports. For example, if a DUT has 4 ports 0(ingress), 1(ingress),
-2(egress) and 3(egress), the traffic stream directed at port 0 should
-output a packet to port 2 followed by a packet to port 3. The traffic
-stream directed at port 1 should also output a packet to port 2 followed
-by a packet to port 3.
-
-2.2.3.4.2 Frame Formats
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Frame formats Layer 2 (data link layer) protocols
-++++++++++++++++++++++++++++++++++++++++++++++++++
-- Ethernet II
-
- .. code-block:: console
-
- +---------------------+--------------------+-----------+
- | Ethernet Header | Payload | Check Sum |
- +---------------------+--------------------+-----------+
- |_____________________|____________________|___________|
- 14 Bytes 46 - 1500 Bytes 4 Bytes
-
-Layer 3 (network layer) protocols
-++++++++++++++++++++++++++++++++++
-
-- IPv4
-
- .. code-block:: console
-
- +---------------------+--------------------+--------------------+-----------+
- | Ethernet Header | IP Header | Payload | Check Sum |
- +---------------------+--------------------+--------------------+-----------+
- |_____________________|____________________|____________________|___________|
- 14 Bytes 20 bytes 26 - 1480 Bytes 4 Bytes
-
-- IPv6
-
- .. code-block:: console
-
- +---------------------+--------------------+--------------------+-----------+
- | Ethernet Header | IP Header | Payload | Check Sum |
- +---------------------+--------------------+--------------------+-----------+
- |_____________________|____________________|____________________|___________|
- 14 Bytes 40 bytes 26 - 1460 Bytes 4 Bytes
-
-Layer 4 (transport layer) protocols
-++++++++++++++++++++++++++++++++++++
- - TCP
- - UDP
- - SCTP
-
- .. code-block:: console
-
- +---------------------+--------------------+-----------------+--------------------+-----------+
- | Ethernet Header | IP Header | Layer 4 Header | Payload | Check Sum |
- +---------------------+--------------------+-----------------+--------------------+-----------+
- |_____________________|____________________|_________________|____________________|___________|
- 14 Bytes 40 bytes 20 Bytes 6 - 1460 Bytes 4 Bytes
-
-Layer 5 (application layer) protocols
-+++++++++++++++++++++++++++++++++++++
- - RTP
- - GTP
-
- .. code-block:: console
-
- +---------------------+--------------------+-----------------+--------------------+-----------+
- | Ethernet Header | IP Header | Layer 4 Header | Payload | Check Sum |
- +---------------------+--------------------+-----------------+--------------------+-----------+
- |_____________________|____________________|_________________|____________________|___________|
- 14 Bytes 20 bytes 20 Bytes Min 6 Bytes 4 Bytes
-
-
-2.2.3.4.3 Packet Throughput
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-There is a difference between an Ethernet frame,
-an IP packet, and a UDP datagram. In the seven-layer OSI model of
-computer networking, packet refers to a data unit at layer 3 (network
-layer). The correct term for a data unit at layer 2 (data link layer) is
-a frame, and at layer 4 (transport layer) is a segment or datagram.
-
-Important concepts related to 10GbE performance are frame rate and
-throughput. The MAC bit rate of 10GbE, defined in the IEEE standard 802
-.3ae, is 10 billion bits per second. Frame rate is based on the bit rate
-and frame format definitions. Throughput, defined in IETF RFC 1242, is
-the highest rate at which the system under test can forward the offered
-load, without loss.
-
-The frame rate for 10GbE is determined by a formula that divides the 10
-billion bits per second by the preamble + frame length + inter-frame
-gap.
-
-The maximum frame rate is calculated using the minimum values of the
-following parameters, as described in the IEEE 802 .3ae standard:
-
-- Preamble: 8 bytes \* 8 = 64 bits
-- Frame Length: 64 bytes (minimum) \* 8 = 512 bits
-- Inter-frame Gap: 12 bytes (minimum) \* 8 = 96 bits
-
-Therefore, Maximum Frame Rate (64B Frames)
-= MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame Gap)
-= 10,000,000,000 / (64 + 512 + 96)
-= 10,000,000,000 / 672
-= 14,880,952.38 frame per second (fps)
-
-2.2.3.4.4 System isolation and validation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A key consideration when conducting any sort of benchmark is trying to
-ensure the consistency and repeatability of test results between runs.
-When benchmarking the performance of a virtual switch there are many
-factors that can affect the consistency of results. This section
-describes these factors and the measures that can be taken to limit
-their effects. In addition, this section will outline some system tests
-to validate the platform and the VNF before conducting any vSwitch
-benchmarking tests.
-
-System Isolation
-++++++++++++++++
-When conducting a benchmarking test on any SUT, it is essential to limit
-(and if reasonable, eliminate) any noise that may interfere with the
-accuracy of the metrics collected by the test. This noise may be
-introduced by other hardware or software (OS, other applications), and
-can result in significantly varying performance metrics being collected
-between consecutive runs of the same test. In the case of characterizing
-the performance of a virtual switch, there are a number of configuration
-parameters that can help increase the repeatability and stability of
-test results, including:
-
-- OS/GRUB configuration:
-
- - maxcpus = n where n >= 0; limits the kernel to using 'n'
- processors. Only use exactly what you need.
- - isolcpus: Isolate CPUs from the general scheduler. Isolate all
- CPUs bar one which will be used by the OS.
- - use taskset to affinitize the forwarding application and the VNFs
- onto isolated cores. VNFs and the vSwitch should be allocated
- their own cores, i.e. must not share the same cores. vCPUs for the
- VNF should be affinitized to individual cores also.
- - Limit the amount of background applications that are running and
- set OS to boot to runlevel 3. Make sure to kill any unnecessary
- system processes/daemons.
- - Only enable hardware that you need to use for your test – to
- ensure there are no other interrupts on the system.
- - Configure NIC interrupts to only use the cores that are not
- allocated to any other process (VNF/vSwitch).
-
-- NUMA configuration: Any unused sockets in a multi-socket system
- should be disabled.
-- CPU pinning: The vSwitch and the VNF should each be affinitized to
- separate logical cores using a combination of maxcpus, isolcpus and
- taskset.
-- BIOS configuration: BIOS should be configured for performance where
- an explicit option exists, sleep states should be disabled, any
- virtualization optimization technologies should be enabled, and
- hyperthreading should also be enabled.
-
-System Validation
-+++++++++++++++++
-System validation is broken down into two sub-categories: Platform
-validation and VNF validation. The validation test itself involves
-verifying the forwarding capability and stability for the sub-system
-under test. The rationale behind system validation is two fold. Firstly
-to give a tester confidence in the stability of the platform or VNF that
-is being tested; and secondly to provide base performance comparison
-points to understand the overhead introduced by the virtual switch.
-
-* Benchmark platform forwarding capability: This is an OPTIONAL test
- used to verify the platform and measure the base performance (maximum
- forwarding rate in fps and latency) that can be achieved by the
- platform without a vSwitch or a VNF. The following diagram outlines
- the set-up for benchmarking Platform forwarding capability:
-
- .. code-block:: console
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | l2fw or DPDK L2FWD app | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-* Benchmark VNF forwarding capability: This test is used to verify
- the VNF and measure the base performance (maximum forwarding rate in
- fps and latency) that can be achieved by the VNF without a vSwitch.
- The performance metrics collected by this test will serve as a key
- comparison point for NIC passthrough technologies and vSwitches. VNF
- in this context refers to the hypervisor and the VM. The following
- diagram outlines the set-up for benchmarking VNF forwarding
- capability:
-
- .. code-block:: console
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-Methodology to benchmark Platform/VNF forwarding capability
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-The recommended methodology for the platform/VNF validation and
-benchmark is: - Run `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
-Maximum Forwarding Rate test, this test will produce maximum
-forwarding rate and latency results that will serve as the
-expected values. These expected values can be used in
-subsequent steps or compared with in subsequent validation tests. -
-Transmit bidirectional traffic at line rate/max forwarding rate
-(whichever is higher) for at least 72 hours, measure throughput (fps)
-and latency. - Note: Traffic should be bidirectional. - Establish a
-baseline forwarding rate for what the platform can achieve. - Additional
-validation: After the test has completed for 72 hours run bidirectional
-traffic at the maximum forwarding rate once more to see if the system is
-still functional and measure throughput (fps) and latency. Compare the
-measure the new obtained values with the expected values.
-
-**NOTE 1**: How the Platform is configured for its forwarding capability
-test (BIOS settings, GRUB configuration, runlevel...) is how the
-platform should be configured for every test after this
-
-**NOTE 2**: How the VNF is configured for its forwarding capability test
-(# of vCPUs, vNICs, Memory, affinitization…) is how it should be
-configured for every test that uses a VNF after this.
-
-2.2.4 RFCs for testing virtual switch performance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The starting point for defining the suite of tests for benchmarking the
-performance of a virtual switch is to take existing RFCs and standards
-that were designed to test their physical counterparts and adapting them
-for testing virtual switches. The rationale behind this is to establish
-a fair comparison between the performance of virtual and physical
-switches. This section outlines the RFCs that are used by this
-specification.
-
-RFC 1242 Benchmarking Terminology for Network Interconnection
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Devices RFC 1242 defines the terminology that is used in describing
-performance benchmarking tests and their results. Definitions and
-discussions covered include: Back-to-back, bridge, bridge/router,
-constant load, data link frame size, frame loss rate, inter frame gap,
-latency, and many more.
-
-RFC 2544 Benchmarking Methodology for Network Interconnect Devices
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 2544 outlines a benchmarking methodology for network Interconnect
-Devices. The methodology results in performance metrics such as latency,
-frame loss percentage, and maximum data throughput.
-
-In this document network “throughput” (measured in millions of frames
-per second) is based on RFC 2544, unless otherwise noted. Frame size
-refers to Ethernet frames ranging from smallest frames of 64 bytes to
-largest frames of 4K bytes.
-
-Types of tests are:
-
-1. Throughput test defines the maximum number of frames per second
- that can be transmitted without any error.
-
-2. Latency test measures the time required for a frame to travel from
- the originating device through the network to the destination device.
- Please note that RFC2544 Latency measurement will be superseded with
- a measurement of average latency over all successfully transferred
- packets or frames.
-
-3. Frame loss test measures the network’s
- response in overload conditions - a critical indicator of the
- network’s ability to support real-time applications in which a
- large amount of frame loss will rapidly degrade service quality.
-
-4. Burst test assesses the buffering capability of a virtual switch. It
- measures the maximum number of frames received at full line rate
- before a frame is lost. In carrier Ethernet networks, this
- measurement validates the excess information rate (EIR) as defined in
- many SLAs.
-
-5. System recovery to characterize speed of recovery from an overload
- condition.
-
-6. Reset to characterize speed of recovery from device or software
- reset. This type of test has been updated by `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such,
- the methodology defined by this specification will be that of RFC 6201.
-
-Although not included in the defined RFC 2544 standard, another crucial
-measurement in Ethernet networking is packet delay variation. The
-definition set out by this specification comes from
-`RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__.
-
-RFC 2285 Benchmarking Terminology for LAN Switching Devices
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 2285 defines the terminology that is used to describe the
-terminology for benchmarking a LAN switching device. It extends RFC
-1242 and defines: DUTs, SUTs, Traffic orientation and distribution,
-bursts, loads, forwarding rates, etc.
-
-RFC 2889 Benchmarking Methodology for LAN Switching
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 2889 outlines a benchmarking methodology for LAN switching, it
-extends RFC 2544. The outlined methodology gathers performance
-metrics for forwarding, congestion control, latency, address handling
-and finally filtering.
-
-RFC 3918 Methodology for IP Multicast Benchmarking
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 3918 outlines a methodology for IP Multicast benchmarking.
-
-RFC 4737 Packet Reordering Metrics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 4737 describes metrics for identifying and counting re-ordered
-packets within a stream, and metrics to measure the extent each
-packet has been re-ordered.
-
-RFC 5481 Packet Delay Variation Applicability Statement
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 5481 defined two common, but different forms of delay variation
-metrics, and compares the metrics over a range of networking
-circumstances and tasks. The most suitable form for vSwitch
-benchmarking is the "PDV" form.
-
-RFC 6201 Device Reset Characterization
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-RFC 6201 extends the methodology for characterizing the speed of
-recovery of the DUT from device or software reset described in RFC
-2544.
-
-2.2.5 Details of the Test Report
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There are a number of parameters related to the system, DUT and tests
-that can affect the repeatability of a test results and should be
-recorded. In order to minimise the variation in the results of a test,
-it is recommended that the test report includes the following information:
-
-- Hardware details including:
-
- - Platform details.
- - Processor details.
- - Memory information (see below)
- - Number of enabled cores.
- - Number of cores used for the test.
- - Number of physical NICs, as well as their details (manufacturer,
- versions, type and the PCI slot they are plugged into).
- - NIC interrupt configuration.
- - BIOS version, release date and any configurations that were
- modified.
-
-- Software details including:
-
- - OS version (for host and VNF)
- - Kernel version (for host and VNF)
- - GRUB boot parameters (for host and VNF).
- - Hypervisor details (Type and version).
- - Selected vSwitch, version number or commit id used.
- - vSwitch launch command line if it has been parameterised.
- - Memory allocation to the vSwitch – which NUMA node it is using,
- and how many memory channels.
- - Where the vswitch is built from source: compiler details including
- versions and the flags that were used to compile the vSwitch.
- - DPDK or any other SW dependency version number or commit id used.
- - Memory allocation to a VM - if it's from Hugpages/elsewhere.
- - VM storage type: snapshot/independent persistent/independent
- non-persistent.
- - Number of VMs.
- - Number of Virtual NICs (vNICs), versions, type and driver.
- - Number of virtual CPUs and their core affinity on the host.
- - Number vNIC interrupt configuration.
- - Thread affinitization for the applications (including the vSwitch
- itself) on the host.
- - Details of Resource isolation, such as CPUs designated for
- Host/Kernel (isolcpu) and CPUs designated for specific processes
- (taskset).
-
-- Memory Details
-
- - Total memory
- - Type of memory
- - Used memory
- - Active memory
- - Inactive memory
- - Free memory
- - Buffer memory
- - Swap cache
- - Total swap
- - Used swap
- - Free swap
-
-- Test duration.
-- Number of flows.
-- Traffic Information:
-
- - Traffic type - UDP, TCP, IMIX / Other.
- - Packet Sizes.
-
-- Deployment Scenario.
-
-**Note**: Tests that require additional parameters to be recorded will
-explicitly specify this.
-
-2.3. Test identification
-------------------------
-2.3.1 Throughput tests
-~~~~~~~~~~~~~~~~~~~~~~
-The following tests aim to determine the maximum forwarding rate that
-can be achieved with a virtual switch. The list is not exhaustive but
-should indicate the type of tests that should be required. It is
-expected that more will be added.
-
-Test ID: LTD.Throughput.RFC2544.PacketLossRatio
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- This test determines the DUT's maximum forwarding rate with X% traffic
- loss for a constant load (fixed length frames at a fixed interval time).
- The default loss percentages to be tested are: - X = 0% - X = 10^-7%
-
- Note: Other values can be tested if required by the user.
-
- The selected frame sizes are those previously defined under `Default
- Test Parameters <#DefaultParams>`__. The test can also be used to
- determine the average latency of the traffic.
-
- Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
- test methodology, the test duration will
- include a number of trials; each trial should run for a minimum period
- of 60 seconds. A binary search methodology must be applied for each
- trial to obtain the final result.
-
- **Expected Result**: At the end of each trial, the presence or absence
- of loss determines the modification of offered load for the next trial,
- converging on a maximum rate, or
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% loss.
- The Throughput load is re-used in related
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
- tests.
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
- the DUT for each frame size with X% packet loss.
- - The average latency of the traffic flow when passing through the DUT
- (if testing for latency, note that this average is different from the
- test specified in Section 26.3 of
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
-Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 X% packet loss Throughput and Latency Test with
- packet modification
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- This test determines the DUT's maximum forwarding rate with X% traffic
- loss for a constant load (fixed length frames at a fixed interval time).
- The default loss percentages to be tested are: - X = 0% - X = 10^-7%
-
- Note: Other values can be tested if required by the user.
-
- The selected frame sizes are those previously defined under `Default
- Test Parameters <#DefaultParams>`__. The test can also be used to
- determine the average latency of the traffic.
-
- Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
- test methodology, the test duration will
- include a number of trials; each trial should run for a minimum period
- of 60 seconds. A binary search methodology must be applied for each
- trial to obtain the final result.
-
- During this test, the DUT must perform the following operations on the
- traffic flow:
-
- - Perform packet parsing on the DUT's ingress port.
- - Perform any relevant address look-ups on the DUT's ingress ports.
- - Modify the packet header before forwarding the packet to the DUT's
- egress port. Packet modifications include:
-
- - Modifying the Ethernet source or destination MAC address.
- - Modifying/adding a VLAN tag. (**Recommended**).
- - Modifying/adding a MPLS tag.
- - Modifying the source or destination ip address.
- - Modifying the TOS/DSCP field.
- - Modifying the source or destination ports for UDP/TCP/SCTP.
- - Modifying the TTL.
-
- **Expected Result**: The Packet parsing/modifications require some
- additional degree of processing resource, therefore the
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
- Throughput is expected to be somewhat lower than the Throughput level
- measured without additional steps. The reduction is expected to be
- greatest on tests with the smallest packet sizes (greatest header
- processing rates).
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
- the DUT for each frame size with X% packet loss and packet
- modification operations being performed by the DUT.
- - The average latency of the traffic flow when passing through the DUT
- (if testing for latency, note that this average is different from the
- test specified in Section 26.3 of
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
- - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
- PDV form of delay variation on the traffic flow,
- using the 99th percentile.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
-Test ID: LTD.Throughput.RFC2544.Profile
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 Throughput and Latency Profile
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- This test reveals how throughput and latency degrades as the offered
- rate varies in the region of the DUT's maximum forwarding rate as
- determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss).
- For example it can be used to determine if the degradation of throughput
- and latency as the offered rate increases is slow and graceful or sudden
- and severe.
-
- The selected frame sizes are those previously defined under `Default
- Test Parameters <#DefaultParams>`__.
-
- The offered traffic rate is described as a percentage delta with respect
- to the DUT's RFC 2544 Throughput as determined by
- LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
- of 0% is equivalent to an offered traffic rate equal to the RFC 2544
- Throughput; A delta of +50% indicates an offered rate half-way
- between the Throughput and line-rate, whereas a delta of
- -50% indicates an offered rate of half the maximum rate. Therefore the
- range of the delta figure is natuarlly bounded at -100% (zero offered
- traffic) and +100% (traffic offered at line rate).
-
- The following deltas to the maximum forwarding rate should be applied:
-
- - -50%, -10%, 0%, +10% & +50%
-
- **Expected Result**: For each packet size a profile should be produced
- of how throughput and latency vary with offered rate.
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT
- for each delta to the maximum forwarding rate and for each frame
- size.
- - The average latency for each delta to the maximum forwarding rate and
- for each frame size.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
- - Any failures experienced (for example if the vSwitch crashes, stops
- processing packets, restarts or becomes unresponsive to commands)
- when the offered load is above Maximum Throughput MUST be recorded
- and reported with the results.
-
-Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 System Recovery Time Test
-
- **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to determine the length of time it takes the DUT
- to recover from an overload condition for a constant load (fixed length
- frames at a fixed interval time). The selected frame sizes are those
- previously defined under `Default Test Parameters <#DefaultParams>`__,
- traffic should be sent to the DUT under normal conditions. During the
- duration of the test and while the traffic flows are passing though the
- DUT, at least one situation leading to an overload condition for the DUT
- should occur. The time from the end of the overload condition to when
- the DUT returns to normal operations should be measured to determine
- recovery time. Prior to overloading the DUT, one should record the
- average latency for 10,000 packets forwarded through the DUT.
-
- The overload condition SHOULD be to transmit traffic at a very high
- frame rate to the DUT (150% of the maximum 0% packet loss rate as
- determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate
- whichever is lower), for at least 60 seconds, then reduce the frame rate
- to 75% of the maximum 0% packet loss rate. A number of time-stamps
- should be recorded: - Record the time-stamp at which the frame rate was
- reduced and record a second time-stamp at the time of the last frame
- lost. The recovery time is the difference between the two timestamps. -
- Record the average latency for 10,000 frames after the last frame loss
- and continue to record average latency measurements for every 10,000
- frames, when latency returns to within 10% of pre-overload levels record
- the time-stamp.
-
- **Expected Result**:
-
- **Metrics collected**
-
- The following are the metrics collected for this test:
-
- - The length of time it takes the DUT to recover from an overload
- condition.
- - The length of time it takes the DUT to recover the average latency to
- pre-overload conditions.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-Test ID: LTD.Throughput.RFC2544.BackToBackFrames
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2544 Back To Back Frames Test
-
- **Prerequisite Test**: N
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to characterize the ability of the DUT to
- process back-to-back frames. For each frame size previously defined
- under `Default Test Parameters <#DefaultParams>`__, a burst of traffic
- is sent to the DUT with the minimum inter-frame gap between each frame.
- If the number of received frames equals the number of frames that were
- transmitted, the burst size should be increased and traffic is sent to
- the DUT again. The value measured is the back-to-back value, that is the
- maximum burst size the DUT can handle without any frame loss. Please note
- a trial must run for a minimum of 2 seconds and should be repeated 50
- times (at a minimum).
-
- **Expected Result**:
-
- Tests of back-to-back frames with physical devices have produced
- unstable results in some cases. All tests should be repeated in multiple
- test sessions and results stability should be examined.
-
- **Metrics collected**
-
- The following are the metrics collected for this test:
-
- - The average back-to-back value across the trials, which is
- the number of frames in the longest burst that the DUT will
- handle without the loss of any frames.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
-
- **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand the Max Forwarding Rate stability
- over an extended test duration in order to uncover any outliers. To allow
- for an extended test duration, the test should ideally run for 24 hours
- or, if this is not possible, for at least 6 hours. For this test, each frame
- size must be sent at the highest Throughput rate with X% packet loss, as
- determined in the prerequisite test. The default loss percentages to be
- tested are: - X = 0% - X = 10^-7%
-
- Note: Other values can be tested if required by the user.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - Max Forwarding Rate stability of the DUT.
-
- - This means reporting the number of packets lost per time interval
- and reporting any time intervals with packet loss. The
- `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
- Forwarding Rate shall be measured in each interval.
- An interval of 60s is suggested.
-
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
- - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
- PDV form of delay variation on the traffic flow,
- using the 99th percentile.
-
-Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
-
- **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand the Max Forwarding Rate stability over an
- extended test duration in order to uncover any outliers. To allow for an
- extended test duration, the test should ideally run for 24 hours or, if
- this is not possible, for at least 6 hour. For this test, each frame
- size must be sent at the highest Throughput rate with 0% packet loss, as
- determined in the prerequisite test.
-
- During this test, the DUT must perform the following operations on the
- traffic flow:
-
- - Perform packet parsing on the DUT's ingress port.
- - Perform any relevant address look-ups on the DUT's ingress ports.
- - Modify the packet header before forwarding the packet to the DUT's
- egress port. Packet modifications include:
-
- - Modifying the Ethernet source or destination MAC address.
- - Modifying/adding a VLAN tag (**Recommended**).
- - Modifying/adding a MPLS tag.
- - Modifying the source or destination ip address.
- - Modifying the TOS/DSCP field.
- - Modifying the source or destination ports for UDP/TCP/SCTP.
- - Modifying the TTL.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - Max Forwarding Rate stability of the DUT.
-
- - This means reporting the number of packets lost per time interval
- and reporting any time intervals with packet loss. The
- `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
- Forwarding Rate shall be measured in each interval.
- An interval of 60s is suggested.
-
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
- - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ PDV form of delay variation on the traffic flow,
- using the 99th percentile.
-
-Test ID: LTD.Throughput.RFC6201.ResetTime
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 6201 Reset Time Test
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to determine the length of time it takes the DUT
- to recover from a reset.
-
- Two reset methods are defined - planned and unplanned. A planned reset
- requires stopping and restarting the virtual switch by the usual
- 'graceful' method defined by it's documentation. An unplanned reset
- requires simulating a fatal internal fault in the virtual switch - for
- example by using kill -SIGKILL on a Linux environment.
-
- Both reset methods SHOULD be exercised.
-
- For each frame size previously defined under `Default Test
- Parameters <#DefaultParams>`__, traffic should be sent to the DUT under
- normal conditions. During the duration of the test and while the traffic
- flows are passing through the DUT, the DUT should be reset and the Reset
- time measured. The Reset time is the total time that a device is
- determined to be out of operation and includes the time to perform the
- reset and the time to recover from it (cf. `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__).
-
- `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods to measure the Reset time:
- - Frame-Loss Method: which requires the monitoring of the number of
- lost frames and calculates the Reset time based on the number of
- frames lost and the offered rate according to the following
- formula:
-
- .. code-block:: console
-
- Frames_lost (packets)
- Reset_time = -------------------------------------
- Offered_rate (packets per second)
-
- - Timestamp Method: which measures the time from which the last frame
- is forwarded from the DUT to the time the first frame is forwarded
- after the reset. This involves time-stamping all transmitted frames
- and recording the timestamp of the last frame that was received prior
- to the reset and also measuring the timestamp of the first frame that
- is received after the reset. The Reset time is the difference between
- these two timestamps.
-
- According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the choice of method depends on the test
- tool's capability; the Frame-Loss method SHOULD be used if the test tool
- supports: - Counting the number of lost frames per stream. -
- Transmitting test frame despite the physical link status.
-
- whereas the Timestamp method SHOULD be used if the test tool supports: -
- Timestamping each frame. - Monitoring received frame's timestamp. -
- Transmitting frames only if the physical link status is up.
-
- **Expected Result**:
-
- **Metrics collected**
-
- The following are the metrics collected for this test: - Average Reset
- Time over the number of trials performed.
-
- Results of this test should include the following information: - The
- reset method used. - Throughput in Fps and Mbps. - Average Frame Loss
- over the number of trials performed. - Average Reset Time in
- milliseconds over the number of trials performed. - Number of trials
- performed. - Protocol: IPv4, IPv6, MPLS, etc. - Frame Size in Octets -
- Port Media: Ethernet, Gigabit Ethernet (GbE), etc. - Port Speed: 10
- Gbps, 40 Gbps etc. - Interface Encapsulation: Ethernet, Ethernet VLAN,
- etc.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Forwarding Rate Test
-
- **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
-
- **Priority**:
-
- **Description**:
-
- This test measures the DUT's Max Forwarding Rate when the Offered Load
- is varied between the throughput and the Maximum Offered Load for fixed
- length frames at a fixed time interval. The selected frame sizes are
- those previously defined under `Default Test
- Parameters <#DefaultParams>`__. The throughput is the maximum offered
- load with 0% frame loss (measured by the prerequisite test), and the
- Maximum Offered Load (as defined by
- `RFC2285 <https://www.rfc-editor.org/rfc/rfc2285.txt>`__) is *"the highest
- number of frames per second that an external source can transmit to a
- DUT/SUT for forwarding to a specified output interface or interfaces"*.
-
- Traffic should be sent to the DUT at a particular rate (TX rate)
- starting with TX rate equal to the throughput rate. The rate of
- successfully received frames at the destination counted (in FPS). If the
- RX rate is equal to the TX rate, the TX rate should be increased by a
- fixed step size and the RX rate measured again until the Max Forwarding
- Rate is found.
-
- The trial duration for each iteration should last for the period of time
- needed for the system to reach steady state for the frame size being
- tested. Under `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
- (Sec. 5.6.3.1) test methodology, the test
- duration should run for a minimum period of 30 seconds, regardless
- whether the system reaches steady state before the minimum duration
- ends.
-
- **Expected Result**: According to
- `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding Rate
- is the highest forwarding rate of a DUT taken from an iterative set of
- forwarding rate measurements. The iterative set of forwarding rate
- measurements are made by setting the intended load transmitted from an
- external source and measuring the offered load (i.e what the DUT is
- capable of forwarding). If the Throughput == the Maximum Offered Load,
- it follows that Max Forwarding Rate is equal to the Maximum Offered
- Load.
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The Max Forwarding Rate for the DUT for each packet size.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical. Note: Full mesh tests with
- multiple ingress and egress ports are a key aspect of RFC 2889
- benchmarks, and scenarios with both 2 and 4 ports should be tested.
- In any case, the number of ports used must be reported.
-
-
-Test ID: LTD.Throughput.RFC2889.ForwardPressure
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Forward Pressure Test
-
- **Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to determine if the DUT transmits frames with an
- inter-frame gap that is less than 12 bytes. This test overloads the DUT
- and measures the output for forward pressure. Traffic should be
- transmitted to the DUT with an inter-frame gap of 11 bytes, this will
- overload the DUT by 1 byte per frame. The forwarding rate of the DUT
- should be measured.
-
- **Expected Result**: The forwarding rate should not exceed the maximum
- forwarding rate of the DUT collected by
- LTD.Throughput.RFC2889.MaxForwardingRate.
-
- **Metrics collected**
-
- The following are the metrics collected for this test:
-
- - Forwarding rate of the DUT in FPS or Mbps.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-
-Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Error Frames Filtering Test
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to determine whether the DUT will propagate any
- erroneous frames it receives or whether it is capable of filtering out
- the erroneous frames. Traffic should be sent with erroneous frames
- included within the flow at random intervals. Illegal frames that must
- be tested include: - Oversize Frames. - Undersize Frames. - CRC Errored
- Frames. - Dribble Bit Errored Frames - Alignment Errored Frames
-
- The traffic flow exiting the DUT should be recorded and checked to
- determine if the erroneous frames where passed through the DUT.
-
- **Expected Result**: Broken frames are not passed!
-
- **Metrics collected**
-
- No Metrics are collected in this test, instead it determines:
-
- - Whether the DUT will propagate erroneous frames.
- - Or whether the DUT will correctly filter out any erroneous frames
- from traffic flow with out removing correct frames.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Broadcast Frame Forwarding Test
-
- **Prerequisite Test**: N
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to determine the maximum forwarding rate of the
- DUT when forwarding broadcast traffic. For each frame previously defined
- under `Default Test Parameters <#DefaultParams>`__, the traffic should
- be set up as broadcast traffic. The traffic throughput of the DUT should
- be measured.
-
- The test should be conducted with at least 4 physical ports on the DUT.
- The number of ports used MUST be recorded.
-
- As broadcast involves forwarding a single incoming packet to several
- destinations, the latency of a single packet is defined as the average
- of the latencies for each of the broadcast destinations.
-
- The incoming packet is transmitted on each of the other physical ports,
- it is not transmitted on the port on which it was received. The test MAY
- be conducted using different broadcasting ports to uncover any
- performance differences.
-
- **Expected Result**:
-
- **Metrics collected**:
-
- The following are the metrics collected for this test:
-
- - The forwarding rate of the DUT when forwarding broadcast traffic.
- - The minimum, average & maximum packets latencies observed.
-
- **Deployment scenario**:
-
- - Physical → virtual switch 3x physical. In the Broadcast rate testing,
- four test ports are required. One of the ports is connected to the test
- device, so it can send broadcast frames and listen for miss-routed frames.
-
-2.3.2 Packet Latency tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-These tests will measure the store and forward latency as well as the packet
-delay variation for various packet types through the virtual switch. The
-following list is not exhaustive but should indicate the type of tests
-that should be required. It is expected that more will be added.
-
-Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: Initial Packet Processing Latency
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- In some virtual switch architectures, the first packets of a flow will
- take the system longer to process than subsequent packets in the flow.
- This test determines the latency for these packets. The test will
- measure the latency of the packets as they are processed by the
- flow-setup-path of the DUT. There are two methods for this test, a
- recommended method and a nalternative method that can be used if it is
- possible to disable the fastpath of the virtual switch.
-
- Recommended method: This test will send 64,000 packets to the DUT, each
- belonging to a different flow. Average packet latency will be determined
- over the 64,000 packets.
-
- Alternative method: This test will send a single packet to the DUT after
- a fixed interval of time. The time interval will be equivalent to the
- amount of time it takes for a flow to time out in the virtual switch
- plus 10%. Average packet latency will be determined over 1,000,000
- packets.
-
- This test is intended only for non-learning virtual switches; For learning
- virtual switches use RFC2889.
-
- For this test, only unidirectional traffic is required.
-
- **Expected Result**: The average latency for the initial packet of all
- flows should be greater than the latency of subsequent traffic.
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - Average latency of the initial packets of all flows that are
- processed by the DUT.
-
- **Deployment scenario**:
-
- - Physical → Virtual Switch → Physical.
-
-Test ID: LTD.PacketDelayVariation.RFC3393.Soak
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: Packet Delay Variation Soak Test
-
- **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand the distribution of packet delay
- variation for different frame sizes over an extended test duration and
- to determine if there are any outliers. To allow for an extended test
- duration, the test should ideally run for 24 hours or, if this is not
- possible, for at least 6 hour. For this test, each frame size must be
- sent at the highest possible throughput with 0% packet loss, as
- determined in the prerequisite test.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The packet delay variation value for traffic passing through the DUT.
- - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
- PDV form of delay variation on the traffic flow,
- using the 99th percentile, for each 60s interval during the test.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
-2.3.3 Scalability tests
-~~~~~~~~~~~~~~~~~~~~~~~~
-The general aim of these tests is to understand the impact of large flow
-table size and flow lookups on throughput. The following list is not
-exhaustive but should indicate the type of tests that should be required.
-It is expected that more will be added.
-
-Test ID: LTD.Scalability.RFC2544.0PacketLoss
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 0% loss Scalability throughput test
-
- **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
- delta Throughput between the single-flow RFC2544 test and this test with
- a variable number of flows is desired.
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to measure how throughput changes as the number
- of flows in the DUT increases. The test will measure the throughput
- through the fastpath, as such the flows need to be installed on the DUT
- before passing traffic.
-
- For each frame size previously defined under `Default Test
- Parameters <#DefaultParams>`__ and for each of the following number of
- flows:
-
- - 1,000
- - 2,000
- - 4,000
- - 8,000
- - 16,000
- - 32,000
- - 64,000
- - Max supported number of flows.
-
- This test will be conducted under two conditions following the
- establishment of all flows as required by RFC 2544, regarding the flow
- expiration time-out:
-
- 1) The time-out never expires during each trial.
-
- 2) The time-out expires for all flows periodically. This would require a
- short time-out compared with flow re-appearance for a small number of
- flows, and may not be possible for all flow conditions.
-
- The maximum 0% packet loss Throughput should be determined in a manner
- identical to LTD.Throughput.RFC2544.PacketLossRatio.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The maximum number of frames per second that can be forwarded at the
- specified number of flows and the specified frame size, with zero
- packet loss.
-
-Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 0% loss Memory Bandwidth Scalability test
-
- **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
- delta Throughput between an undisturbed RFC2544 test and this test with
- the Throughput affected by cache and memory bandwidth contention is desired.
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand how the DUT's performance is
- affected by cache sharing and memory bandwidth between processes.
-
- During the test all cores not used by the vSwitch should be running a
- memory intensive application. This application should read and write
- random data to random addresses in unused physical memory. The random
- nature of the data and addresses is intended to consume cache, exercise
- main memory access (as opposed to cache) and exercise all memory buses
- equally. Furthermore:
-
- - the ratio of reads to writes should be recorded. A ratio of 1:1
- SHOULD be used.
- - the reads and writes MUST be of cache-line size and be cache-line aligned.
- - in NUMA architectures memory access SHOULD be local to the core's node.
- Whether only local memory or a mix of local and remote memory is used
- MUST be recorded.
- - the memory bandwidth (reads plus writes) used per-core MUST be recorded;
- the test MUST be run with a per-core memory bandwidth equal to half the
- maximum system memory bandwidth divided by the number of cores. The test
- MAY be run with other values for the per-core memory bandwidth.
- - the test MAY also be run with the memory intensive application running
- on all cores.
-
- Under these conditions the DUT's 0% packet loss throughput is determined
- as per LTD.Throughput.RFC2544.PacketLossRatio.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The DUT's 0% packet loss throughput in the presence of cache sharing and memory bandwidth between processes.
-
-2.3.4 Activation tests
-~~~~~~~~~~~~~~~~~~~~~~~~
-The general aim of these tests is to understand the capacity of the
-and speed with which the vswitch can accomodate new flows.
-
-Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Address Caching Capacity Test
-
- **Prerequisite Test**: N/A
-
- **Priority**:
-
- **Description**:
-
- Please note this test is only applicable to virtual switches that are capable of
- MAC learning. The aim of this test is to determine the address caching
- capacity of the DUT for a constant load (fixed length frames at a fixed
- interval time). The selected frame sizes are those previously defined
- under `Default Test Parameters <#DefaultParams>`__.
-
- In order to run this test the aging time, that is the maximum time the
- DUT will keep a learned address in its flow table, and a set of initial
- addresses, whose value should be >= 1 and <= the max number supported by
- the implementation must be known. Please note that if the aging time is
- configurable it must be longer than the time necessary to produce frames
- from the external source at the specified rate. If the aging time is
- fixed the frame rate must be brought down to a value that the external
- source can produce in a time that is less than the aging time.
-
- Learning Frames should be sent from an external source to the DUT to
- install a number of flows. The Learning Frames must have a fixed
- destination address and must vary the source address of the frames. The
- DUT should install flows in its flow table based on the varying source
- addresses. Frames should then be transmitted from an external source at
- a suitable frame rate to see if the DUT has properly learned all of the
- addresses. If there is no frame loss and no flooding, the number of
- addresses sent to the DUT should be increased and the test is repeated
- until the max number of cached addresses supported by the DUT
- determined.
-
- **Expected Result**:
-
- **Metrics collected**:
-
- The following are the metrics collected for this test:
-
- - Number of cached addresses supported by the DUT.
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → 2 x physical (one receiving, one listening).
-
-Test ID: LTD.Activation.RFC2889.AddressLearningRate
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC2889 Address Learning Rate Test
-
- **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
-
- **Priority**:
-
- **Description**:
-
- Please note this test is only applicable to virtual switches that are capable of
- MAC learning. The aim of this test is to determine the rate of address
- learning of the DUT for a constant load (fixed length frames at a fixed
- interval time). The selected frame sizes are those previously defined
- under `Default Test Parameters <#DefaultParams>`__, traffic should be
- sent with each IPv4/IPv6 address incremented by one. The rate at which
- the DUT learns a new address should be measured. The maximum caching
- capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken
- into consideration as the maximum number of addresses for which the
- learning rate can be obtained.
-
- **Expected Result**: It may be worthwhile to report the behaviour when
- operating beyond address capacity - some DUTs may be more friendly to
- new addresses than others.
-
- **Metrics collected**:
-
- The following are the metrics collected for this test:
-
- - The address learning rate of the DUT.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → 2 x physical (one receiving, one listening).
-
-
-2.3.5 Coupling between control path and datapath Tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The following tests aim to determine how tightly coupled the datapath
-and the control path are within a virtual switch. The following list
-is not exhaustive but should indicate the type of tests that should be
-required. It is expected that more will be added.
-
-Test ID: LTD.CPDPCouplingFlowAddition
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: Control Path and Datapath Coupling
-
- **Prerequisite Test**:
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand how exercising the DUT's control
- path affects datapath performance.
-
- Initially a certain number of flow table entries are installed in the
- vSwitch. Then over the duration of an RFC2544 throughput test
- flow-entries are added and removed at the rates specified below. No
- traffic is 'hitting' these flow-entries, they are simply added and
- removed.
-
- The test MUST be repeated with the following initial number of
- flow-entries installed: - < 10 - 1000 - 100,000 - 10,000,000 (or the
- maximum supported number of flow-entries)
-
- The test MUST be repeated with the following rates of flow-entry
- addition and deletion per second: - 0 - 1 (i.e. 1 addition plus 1
- deletion) - 100 - 10,000
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
- the DUT.
- - The average latency of the traffic flow when passing through the DUT
- (if testing for latency, note that this average is different from the
- test specified in Section 26.3 of
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
- - CPU and memory utilization may also be collected as part of this
- test, to determine the vSwitch's performance footprint on the system.
-
- **Deployment scenario**:
-
- - Physical → virtual switch → physical.
-
-2.3.6 CPU and memory consumption
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The following tests will profile a virtual switch's CPU and memory
-utilization under various loads and circumstances. The following
-list is not exhaustive but should indicate the type of tests that
-should be required. It is expected that more will be added.
-
-Test ID: LTD.CPU.RFC2544.0PacketLoss
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- **Title**: RFC 2544 0% Loss Compute Test
-
- **Prerequisite Test**:
-
- **Priority**:
-
- **Description**:
-
- The aim of this test is to understand the overall performance of the
- system when a CPU intensive application is run on the same DUT as the
- Virtual Switch. For each frame size, an
- LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be
- performed. Throughout the entire test a CPU intensive application should
- be run on all cores on the system not in use by the Virtual Switch. For
- NUMA system only cores on the same NUMA node are loaded.
-
- It is recommended that stress-ng be used for loading the non-Virtual
- Switch cores but any stress tool MAY be used.
-
- **Expected Result**:
-
- **Metrics Collected**:
-
- The following are the metrics collected for this test:
-
- - CPU utilization of the cores running the Virtual Switch.
- - The number of identity of the cores allocated to the Virtual Switch.
- - The configuration of the stress tool (for example the command line
- parameters used to start it.)
-
-2.3.7 Summary List of Tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-1. Throughput tests
-
- - Test ID: LTD.Throughput.RFC2544.PacketLossRatio
- - Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
- - Test ID: LTD.Throughput.RFC2544.Profile
- - Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
- - Test ID: LTD.Throughput.RFC2544.BackToBackFrames
- - Test ID: LTD.Throughput.RFC2889.Soak
- - Test ID: LTD.Throughput.RFC2889.SoakFrameModification
- - Test ID: LTD.Throughput.RFC6201.ResetTime
- - Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
- - Test ID: LTD.Throughput.RFC2889.ForwardPressure
- - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
- - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
-
-2. Packet Latency tests
-
- - Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
- - Test ID: LTD.PacketDelayVariation.RFC3393.Soak
-
-3. Scalability tests
-
- - Test ID: LTD.Scalability.RFC2544.0PacketLoss
- - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-
-4. Acivation tests
-
- - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
- - Test ID: LTD.Activation.RFC2889.AddressLearningRate
-
-5. Coupling between control path and datapath Tests
-
- - Test ID: LTD.CPDPCouplingFlowAddition
-
-6. CPU and memory consumption
-
- - Test ID: LTD.CPU.RFC2544.0PacketLoss