From ffc0fc7902e2f46cb5982f55aacd262073f08e1c Mon Sep 17 00:00:00 2001 From: Trevor Cooper Date: Tue, 22 Aug 2017 15:29:08 -0700 Subject: Moved devguide for consitency with docs dir structure for all testing projects Updated RFC description based on IETF approval of Internet Draft Change-Id: Ifd37da946866f350b8968bbbe8c5a3f5ce762cfa Signed-off-by: Trevor Cooper --- .../devguide/requirements/ietf_draft/LICENSE | 12 + .../draft-ietf-bmwg-vswitch-opnfv-00.xml | 1016 +++++++++++++++++++ .../draft-ietf-bmwg-vswitch-opnfv-01.xml | 1027 ++++++++++++++++++++ .../draft-vsperf-bmwg-vswitch-opnfv-00.xml | 964 ++++++++++++++++++ .../draft-vsperf-bmwg-vswitch-opnfv-01.xml | 1016 +++++++++++++++++++ .../draft-vsperf-bmwg-vswitch-opnfv-02.xml | 1016 +++++++++++++++++++ 6 files changed, 5051 insertions(+) create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/LICENSE create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml create mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml (limited to 'docs/testing/developer/devguide/requirements/ietf_draft') diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/LICENSE b/docs/testing/developer/devguide/requirements/ietf_draft/LICENSE new file mode 100644 index 00000000..7fc9ae14 --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/LICENSE @@ -0,0 +1,12 @@ +Copyright (c) 2016 IETF Trust and the persons identified as the +document authors. All rights reserved. + +This document is subject to BCP 78 and the IETF Trust's Legal +Provisions Relating to IETF Documents +(http://trustee.ietf.org/license-info) in effect on the date of +publication of this document. Please review these documents +carefully, as they describe your rights and restrictions with respect +to this document. Code Components extracted from this document must +include Simplified BSD License text as described in Section 4.e of +the Trust Legal Provisions and are provided without warranty as +described in the Simplified BSD License. diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml new file mode 100644 index 00000000..2259b23c --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml @@ -0,0 +1,1016 @@ + + + + + + + + + + + + + + + Benchmarking Virtual Switches in + OPNFV + + + Intel + +
+ + + + + + + + + + + + + + + + + maryam.tahhan@intel.com + + +
+ + + + Intel + +
+ + + + + + + + + + + + + + + + + billy.o.mahony@intel.com + + +
+ + + + AT&T Labs + +
+ + 200 Laurel Avenue South + + Middletown, + + NJ + + 07748 + + USA + + + +1 732 420 1571 + + +1 732 368 1192 + + acmorton@att.com + + http://home.comcast.net/~acmacm/ +
+ + + + + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure. + + + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119. + + + + + + +
+ Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention. + + This memo summarizes the progress of the Open Platform for NFV + (OPNFV) project on virtual switch performance characterization, + "VSWITCHPERF", through the Brahmaputra (second) release . This project intends to build on the current and + completed work of the Benchmarking Methodology Working Group in IETF, by + referencing existing literature. For example, currently the most often + referenced RFC is (which depends on ) and foundation of the benchmarking work in OPNFV is + common and strong. + + See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/ + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example. + + Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry. The authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is one part of the efforts. We + acknowledge the early work in , and useful + discussion with the authors. +
+ +
+ The primary purpose and scope of the memo is to inform the industry + of work-in-progress that builds on the body of extensive BMWG literature + and experience, and describe the extensions needed for benchmarking + virtual switches. Inital feedback indicates that many of these + extensions may be applicable beyond the current scope (to hardware + switches in the NFV Infrastructure and to virtual routers, for example). + Additionally, this memo serves as a vehicle to include more detail and + commentary from BMWG and other Open Source communities, under BMWG's + chartered work to characterize the NFV Infrastructure (a virtual switch + is an important aspect of that infrastructure). +
+ +
+ This section highlights some specific considerations (from )related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document. + +
+ To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions. +
+ +
+ External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed. +
+ +
+ A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability. + + Hardware details including: + + + Platform details + + Processor details + + Memory information (type and size) + + Number of enabled cores + + Number of cores used for the test + + Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into) + + NIC interrupt configuration + + BIOS version, release date and any configurations that were + modified + + CPU microcode level + + Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations + + PCI configuration parameters (payload size, early ack + option...) + + Power management at all levels (ACPI sleep states, processor + package, OS...) + Software details including: + + + OS parameters and behavior (text vs graphical no one typing at + the console on one system) + + OS version (for host and VNF) + + Kernel version (for host and VNF) + + GRUB boot parameters (for host and VNF) + + Hypervisor details (Type and version) + + Selected vSwitch, version number or commit id used + + vSwitch launch command line if it has been parameterised + + Memory allocation to the vSwitch + + which NUMA node it is using, and how many memory channels + + DPDK or any other SW dependency version number or commit id + used + + Memory allocation to a VM - if it's from Hugpages/elsewhere + + VM storage type: snapshot/independent persistent/independent + non-persistent + + Number of VMs + + Number of Virtual NICs (vNICs), versions, type and driver + + Number of virtual CPUs and their core affinity on the host + + Number vNIC interrupt configuration + + Thread affinitization for the applications (including the + vSwitch itself) on the host + + Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows. + + + Test Traffic Information: + Traffic type - UDP, TCP, IMIX / Other + + Packet Sizes + + Deployment Scenario + + + +
+ +
+ Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values + (5-tuple) or have arrived on the same port. Performance results can + vary based on the parameters the vSwitch uses to match for a flow. The + recommended flow classification parameters for any vSwitch performance + tests are: the input port, the source IP address, the destination IP + address and the Ethernet protocol type field. It is essential to + increase the flow timeout time on a vSwitch before conducting any + performance tests that do not measure the flow setup time. Normally + the first packet of a particular stream will install the flow in the + virtual switch which adds an additional latency, subsequent packets of + the same flow are not subject to this latency if the flow is already + installed on the vSwitch. +
+ +
+ This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time. + + + Baselines: + Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency).
+ Benchmark platform forwarding + capability + + + + +
+ + Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance.
+ Benchmark VNF forwarding capability + + + + +
+ + Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT + + Benchmarking with isolated resources alone, leaving some + resources unused + + Benchmark with isolated resources and all resources + occupied + + + Next Steps + Limited sharing + + Production scenarios + + Stressful scenarios + + +
+
+ +
+ The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include: + + + Benchmarking Methodology for Network + Interconnect Devices + + Benchmarking Methodology for LAN + Switching + + Device Reset Characterization + + Packet Delay Variation Applicability + Statement + + + Some of the above/newer RFCs are being applied in benchmarking for + the first time, and represent a development challenge for test equipment + developers. Fortunately, many members of the testing system community + have engaged on the VSPERF project, including an open source test + system. + + In addition to this, the LTD also re-uses the terminology defined + by: + + + Benchmarking Terminology for LAN + Switching Devices + + Packet Delay Variation Applicability + Statement + + + + + Specifications to be included in future updates of the LTD + include: + Methodology for IP Multicast + Benchmarking + + Packet Reordering Metrics + + + As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a + metric derived from IPPM's , where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized. + + Tests have been (or will be) designed to collect the metrics + below: + + + Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by ) without traffic loss. + + Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads. + + Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers. + + Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases. + + Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch. + + Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow). + + CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint. + + The so-called "Soak" tests, where the selected test is conducted + over a long period of time (with an ideal duration of 24 hours, and + at least 6 hours). The purpose of soak tests is to capture transient + changes in performance which may occur due to infrequent processes + or the low probability coincidence of two or more processes. The + performance must be evaluated periodically during continuous + testing, and this results in use of Frame + Rate metrics instead of Throughput (which + requires stopping traffic to allow time for all traffic to exit + internal queues). + + + Future/planned test specs include: + Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch. + + Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch. + + Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements on characterization of + acceleration technologies applied to vswitches. + The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include: + +
+ Physical port to virtual switch to physical + port + + +
+ +
+ Physical port to virtual switch to VNF to virtual switch + to physical port + + +
+ Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port + + +
+ Physical port to virtual switch to VNF + + +
+ VNF to virtual switch to physical port + + +
+ VNF to virtual switch to VNF + + +
+ + A set of Deployment Scenario figures is available on the VSPERF Test + Methodology Wiki page . +
+ +
+ This section organizes the many existing test specifications into the + "3x3" matrix (introduced in ). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale metrics + when displaying the coverage of many metrics/benchmarks). The current + version of the LTD specification is available . + + The tests listed below assess the activation of paths in the data + plane, rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" Wiki page . + +
+ + Activation.RFC2889.AddressLearningRate + + PacketLatency.InitialPacketProcessingLatency + +
+ +
+ + CPDP.Coupling.Flow.Addition + +
+ +
+ + Throughput.RFC2544.SystemRecoveryTime + + Throughput.RFC2544.ResetTime + +
+ +
+ + Activation.RFC2889.AddressCachingCapacity + +
+ +
+ + Throughput.RFC2544.PacketLossRate + + CPU.RFC2544.0PacketLoss + + Throughput.RFC2544.PacketLossRateFrameModification + + Throughput.RFC2544.BackToBackFrames + + Throughput.RFC2889.MaxForwardingRate + + Throughput.RFC2889.ForwardPressure + + Throughput.RFC2889.BroadcastFrameForwarding + +
+ +
+ + Throughput.RFC2889.ErrorFramesFiltering + + Throughput.RFC2544.Profile + +
+ +
+ + Throughput.RFC2889.Soak + + Throughput.RFC2889.SoakFrameModification + + PacketDelayVariation.RFC3393.Soak + +
+ +
+ + Scalability.RFC2544.0PacketLoss + + MemoryBandwidth.RFC2544.0PacketLoss.Scalability + +
+ +
+
+ +
+
+
+ +
+ Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. +
+ +
+ No IANA Action is requested at this time. +
+ +
+ The authors appreciate and acknowledge comments from Scott Bradner, + Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, + Christian Trautman, and others for their reviews. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network Function Virtualization: Performance and Portability + Best Practices + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Test Topologies + https://wiki.opnfv.org/vsperf/test_methodology + + + + + + + + + + + + LTD Test Spec Overview + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review + + + + + + + + + + + + LTD Test Specification + http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html + + + + + + + + + + + + Brahmaputra, Second OPNFV Release + https://www.opnfv.org/brahmaputra + + + + + + + + + + + + https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ + + + + + + + + + + + diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml new file mode 100644 index 00000000..c8a3d99b --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-01.xml @@ -0,0 +1,1027 @@ + + + + + + + + + + + + + + + Benchmarking Virtual Switches in + OPNFV + + + Intel + +
+ + + + + + + + + + + + + + + + + maryam.tahhan@intel.com + + +
+ + + + Intel + +
+ + + + + + + + + + + + + + + + + billy.o.mahony@intel.com + + +
+ + + + AT&T Labs + +
+ + 200 Laurel Avenue South + + Middletown, + + NJ + + 07748 + + USA + + + +1 732 420 1571 + + +1 732 368 1192 + + acmorton@att.com + + http://home.comcast.net/~acmacm/ +
+ + + + + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure. + + + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119. + + + + + + +
+ Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention. + + This memo summarizes the progress of the Open Platform for NFV + (OPNFV) project on virtual switch performance characterization, + "VSWITCHPERF", through the Brahmaputra (second) release . This project intends to build on the current and + completed work of the Benchmarking Methodology Working Group in IETF, by + referencing existing literature. For example, currently the most often + referenced RFC is (which depends on ) and foundation of the benchmarking work in OPNFV is + common and strong. + + See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/ + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example. + + Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry. The authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is one part of the efforts. We + acknowledge the early work in , and useful + discussion with the authors. +
+ +
+ The primary purpose and scope of the memo is to inform the industry + of work-in-progress that builds on the body of extensive BMWG literature + and experience, and describe the extensions needed for benchmarking + virtual switches. Inital feedback indicates that many of these + extensions may be applicable beyond the current scope (to hardware + switches in the NFV Infrastructure and to virtual routers, for example). + Additionally, this memo serves as a vehicle to include more detail and + commentary from BMWG and other Open Source communities, under BMWG's + chartered work to characterize the NFV Infrastructure (a virtual switch + is an important aspect of that infrastructure). + + The benchmarking covered in this memo should be applicable to many + types of vswitches, and remain vswitch-agnostic to great degree. There + has been no attempt to track and test all features of any specific + vswitch implementation. +
+ +
+ This section highlights some specific considerations (from )related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document. + +
+ To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions. +
+ +
+ External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed. +
+ +
+ A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability. + + Hardware details including: + + + Platform details + + Processor details + + Memory information (type and size) + + Number of enabled cores + + Number of cores used for the test + + Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into) + + NIC interrupt configuration + + BIOS version, release date and any configurations that were + modified + + CPU microcode level + + Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations + + PCI configuration parameters (payload size, early ack + option...) + + Power management at all levels (ACPI sleep states, processor + package, OS...) + Software details including: + + + OS parameters and behavior (text vs graphical no one typing at + the console on one system) + + OS version (for host and VNF) + + Kernel version (for host and VNF) + + GRUB boot parameters (for host and VNF) + + Hypervisor details (Type and version) + + Selected vSwitch, version number or commit id used + + vSwitch launch command line if it has been parameterised + + Memory allocation to the vSwitch + + which NUMA node it is using, and how many memory channels + + DPDK or any other SW dependency version number or commit id + used + + Memory allocation to a VM - if it's from Hugpages/elsewhere + + VM storage type: snapshot/independent persistent/independent + non-persistent + + Number of VMs + + Number of Virtual NICs (vNICs), versions, type and driver + + Number of virtual CPUs and their core affinity on the host + + Number vNIC interrupt configuration + + Thread affinitization for the applications (including the + vSwitch itself) on the host + + Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows. + + + Test Traffic Information: + Traffic type - UDP, TCP, IMIX / Other + + Packet Sizes + + Deployment Scenario + + + +
+ +
+ Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values + (5-tuple) or have arrived on the same port. Performance results can + vary based on the parameters the vSwitch uses to match for a flow. The + recommended flow classification parameters for any vSwitch performance + tests are: the input port, the source IP address, the destination IP + address and the Ethernet protocol type field. It is essential to + increase the flow timeout time on a vSwitch before conducting any + performance tests that do not measure the flow setup time. Normally + the first packet of a particular stream will install the flow in the + virtual switch which adds an additional latency, subsequent packets of + the same flow are not subject to this latency if the flow is already + installed on the vSwitch. +
+ +
+ This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time. + + + Baselines: + Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency).
+ Benchmark platform forwarding + capability + + + + +
+ + Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance.
+ Benchmark VNF forwarding capability + + + + +
+ + Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT + + Benchmarking with isolated resources alone, leaving some + resources unused + + Benchmark with isolated resources and all resources + occupied + + + Next Steps + Limited sharing + + Production scenarios + + Stressful scenarios + + +
+
+ +
+ The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include: + + + Benchmarking Methodology for Network + Interconnect Devices + + Benchmarking Methodology for LAN + Switching + + Device Reset Characterization + + Packet Delay Variation Applicability + Statement + + + Some of the above/newer RFCs are being applied in benchmarking for + the first time, and represent a development challenge for test equipment + developers. Fortunately, many members of the testing system community + have engaged on the VSPERF project, including an open source test + system. + + In addition to this, the LTD also re-uses the terminology defined + by: + + + Benchmarking Terminology for LAN + Switching Devices + + Packet Delay Variation Applicability + Statement + + + + + Specifications to be included in future updates of the LTD + include: + Methodology for IP Multicast + Benchmarking + + Packet Reordering Metrics + + + As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a + metric derived from IPPM's , where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized. + + Tests have been (or will be) designed to collect the metrics + below: + + + Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by ) without traffic loss. + + Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads. + + Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers. + + Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases. + + Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch. + + Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow). + + CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint. + + The so-called "Soak" tests, where the selected test is conducted + over a long period of time (with an ideal duration of 24 hours, but + only long enough to determine that stability issues exist when + found; there is no requirement to continue a test when a DUT + exhibits instability over time). The key performance characteristics + and benchmarks for a DUT are determined (using short duration tests) + prior to conducting soak tests. The purpose of soak tests is to + capture transient changes in performance which may occur due to + infrequent processes, memory leaks, or the low probability + coincidence of two or more processes. The stability of the DUT is + the paramount consideration, so performance must be evaluated + periodically during continuous testing, and this results in use of + Frame Rate metrics instead of Throughput (which requires stopping traffic to + allow time for all traffic to exit internal queues), for + example. + + + Future/planned test specs include: + Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch. + + Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch. + + Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements on characterization of + acceleration technologies applied to vswitches. + The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include: + +
+ Physical port to virtual switch to physical + port + + +
+ +
+ Physical port to virtual switch to VNF to virtual switch + to physical port + + +
+ Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port + + +
+ Physical port to virtual switch to VNF + + +
+ VNF to virtual switch to physical port + + +
+ VNF to virtual switch to VNF + + +
+ + A set of Deployment Scenario figures is available on the VSPERF Test + Methodology Wiki page . +
+ +
+ This section organizes the many existing test specifications into the + "3x3" matrix (introduced in ). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale metrics + when displaying the coverage of many metrics/benchmarks). The current + version of the LTD specification is available . + + The tests listed below assess the activation of paths in the data + plane, rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" Wiki page . + +
+ + Activation.RFC2889.AddressLearningRate + + PacketLatency.InitialPacketProcessingLatency + +
+ +
+ + CPDP.Coupling.Flow.Addition + +
+ +
+ + Throughput.RFC2544.SystemRecoveryTime + + Throughput.RFC2544.ResetTime + +
+ +
+ + Activation.RFC2889.AddressCachingCapacity + +
+ +
+ + Throughput.RFC2544.PacketLossRate + + CPU.RFC2544.0PacketLoss + + Throughput.RFC2544.PacketLossRateFrameModification + + Throughput.RFC2544.BackToBackFrames + + Throughput.RFC2889.MaxForwardingRate + + Throughput.RFC2889.ForwardPressure + + Throughput.RFC2889.BroadcastFrameForwarding + +
+ +
+ + Throughput.RFC2889.ErrorFramesFiltering + + Throughput.RFC2544.Profile + +
+ +
+ + Throughput.RFC2889.Soak + + Throughput.RFC2889.SoakFrameModification + + PacketDelayVariation.RFC3393.Soak + +
+ +
+ + Scalability.RFC2544.0PacketLoss + + MemoryBandwidth.RFC2544.0PacketLoss.Scalability + +
+ +
+
+ +
+
+
+ +
+ Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. +
+ +
+ No IANA Action is requested at this time. +
+ +
+ The authors appreciate and acknowledge comments from Scott Bradner, + Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, + Christian Trautman, and others for their reviews. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network Function Virtualization: Performance and Portability + Best Practices + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Test Topologies + https://wiki.opnfv.org/vsperf/test_methodology + + + + + + + + + + + + LTD Test Spec Overview + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review + + + + + + + + + + + + LTD Test Specification + http://artifacts.opnfv.org/vswitchperf/brahmaputra/docs/requirements/index.html + + + + + + + + + + + + Brahmaputra, Second OPNFV Release + https://www.opnfv.org/brahmaputra + + + + + + + + + + + + https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ + + + + + + + + + + + diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml new file mode 100644 index 00000000..b5f7f833 --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml @@ -0,0 +1,964 @@ + + + + + + + + + + + + + + + Benchmarking Virtual Switches in + OPNFV + + + Intel + +
+ + + + + + + + + + + + + + + + + maryam.tahhan@intel.com + + +
+ + + + Intel + +
+ + + + + + + + + + + + + + + + + billy.o.mahony@intel.com + + +
+ + + + AT&T Labs + +
+ + 200 Laurel Avenue South + + Middletown, + + NJ + + 07748 + + USA + + + +1 732 420 1571 + + +1 732 368 1192 + + acmorton@att.com + + http://home.comcast.net/~acmacm/ +
+ + + + + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure. + + + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119. + + + + + + +
+ Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention. + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance characterization, "VSWITCHPERF". + This project intends to build on the current and completed work of the + Benchmarking Methodology Working Group in IETF, by referencing existing + literature. For example, currently the most often referenced RFC is + (which depends on ) and + foundation of the benchmarking work in OPNFV is common and strong. + + See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/ + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example. + + Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry, and the authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is evidence of the efforts. +
+ +
+ The primary purpose and scope of the memo is to inform BMWG of + work-in-progress that builds on the body of extensive literature and + experience. Additionally, once the initial information conveyed here is + received, this memo may be expanded to include more detail and + commentary from both BMWG and OPNFV communities, under BMWG's chartered + work to characterize the NFV Infrastructure (a virtual switch is an + important aspect of that infrastructure). +
+ +
+ This section highlights some specific considerations (from )related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document. + +
+ To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions. +
+ +
+ External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed. +
+ +
+ A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability. + + Hardware details including: + + + Platform details + + Processor details + + Memory information (type and size) + + Number of enabled cores + + Number of cores used for the test + + Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into) + + NIC interrupt configuration + + BIOS version, release date and any configurations that were + modified + + CPU microcode level + + Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations + + PCI configuration parameters (payload size, early ack + option...) + + Power management at all levels (ACPI sleep states, processor + package, OS...) + Software details including: + + + OS parameters and behavior (text vs graphical no one typing at + the console on one system) + + OS version (for host and VNF) + + Kernel version (for host and VNF) + + GRUB boot parameters (for host and VNF) + + Hypervisor details (Type and version) + + Selected vSwitch, version number or commit id used + + vSwitch launch command line if it has been parameterised + + Memory allocation to the vSwitch + + which NUMA node it is using, and how many memory channels + + DPDK or any other SW dependency version number or commit id + used + + Memory allocation to a VM - if it's from Hugpages/elsewhere + + VM storage type: snapshot/independent persistent/independent + non-persistent + + Number of VMs + + Number of Virtual NICs (vNICs), versions, type and driver + + Number of virtual CPUs and their core affinity on the host + + Number vNIC interrupt configuration + + Thread affinitization for the applications (including the + vSwitch itself) on the host + + Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows. + + + Test Traffic Information: + Traffic type - UDP, TCP, IMIX / Other + + Packet Sizes + + Deployment Scenario + + + +
+ +
+ Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values or + have arrived on the same port. Performance results can vary based on + the parameters the vSwitch uses to match for a flow. The recommended + flow classification parameters for any vSwitch performance tests are: + the input port, the source IP address, the destination IP address and + the Ethernet protocol type field. It is essential to increase the flow + timeout time on a vSwitch before conducting any performance tests that + do not measure the flow setup time. Normally the first packet of a + particular stream will install the flow in the virtual switch which + adds an additional latency, subsequent packets of the same flow are + not subject to this latency if the flow is already installed on the + vSwitch. +
+ +
+ This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time. + + + Baselines: + Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency).
+ Benchmark platform forwarding + capability + + + + +
+ + Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance.
+ Benchmark VNF forwarding capability + + + + +
+ + Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT + + Benchmarking with isolated resources alone, leaving some + resources unused + + Benchmark with isolated resources and all resources + occupied + + + Next Steps + Limited sharing + + Production scenarios + + Stressful scenarios + + +
+
+ +
+ The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include: + + + Benchmarking Methodology for Network + Interconnect Devices + + Benchmarking Methodology for LAN + Switching + + Device Reset Characterization + + Packet Delay Variation Applicability + Statement + + + Some of the above/newer RFCs are being applied in benchmarking for + the first time, and represent a development challenge for test equipment + developers. Fortunately, many members of the testing system community + have engaged on the VSPERF project, including an open source test + system. + + In addition to this, the LTD also re-uses the terminology defined + by: + + + Benchmarking Terminology for LAN + Switching Devices + + Packet Delay Variation Applicability + Statement + + + + + Specifications to be included in future updates of the LTD + include: + Methodology for IP Multicast + Benchmarking + + Packet Reordering Metrics + + + As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a + metric derived from IPPM's , where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized. + + Tests have been (or will be) designed to collect the metrics + below: + + + Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by RFC1242) without traffic loss. + + Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads. + + Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers. + + Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases. + + Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch. + + Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow). + + CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint. + + + Future/planned test specs include: + Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch. + + Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch. + + Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements on characterization of + acceleration technologies applied to vswitches. + The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include: + +
+ Physical port to virtual switch to physical + port + + +
+ +
+ Physical port to virtual switch to VNF to virtual switch + to physical port + + +
+ Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port + + +
+ Physical port to virtual switch to VNF + + +
+ VNF to virtual switch to physical port + + +
+ VNF to virtual switch to VNF + + +
+ + A set of Deployment Scenario figures is available on the VSPERF Test + Methodology Wiki page . +
+ +
+ This section organizes the many existing test specifications into the + "3x3" matrix (introduced in ). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale metrics + when displaying the coverage of many metrics/benchmarks). + + The tests listed below assess the activation of paths in the data + plane, rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" Wiki page . + +
+ + Activation.RFC2889.AddressLearningRate + + PacketLatency.InitialPacketProcessingLatency + +
+ +
+ + CPDP.Coupling.Flow.Addition + +
+ +
+ + Throughput.RFC2544.SystemRecoveryTime + + Throughput.RFC2544.ResetTime + +
+ +
+ + Activation.RFC2889.AddressCachingCapacity + +
+ +
+ + Throughput.RFC2544.PacketLossRate + + CPU.RFC2544.0PacketLoss + + Throughput.RFC2544.PacketLossRateFrameModification + + Throughput.RFC2544.BackToBackFrames + + Throughput.RFC2889.MaxForwardingRate + + Throughput.RFC2889.ForwardPressure + + Throughput.RFC2889.BroadcastFrameForwarding + +
+ +
+ + Throughput.RFC2889.ErrorFramesFiltering + + Throughput.RFC2544.Profile + +
+ +
+ + Throughput.RFC2889.Soak + + Throughput.RFC2889.SoakFrameModification + + PacketDelayVariation.RFC3393.Soak + +
+ +
+ + Scalability.RFC2544.0PacketLoss + + MemoryBandwidth.RFC2544.0PacketLoss.Scalability + +
+ +
+
+ +
+
+
+ +
+ Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. +
+ +
+ No IANA Action is requested at this time. +
+ +
+ The authors acknowledge +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network Function Virtualization: Performance and Portability + Best Practices + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Test Topologies + https://wiki.opnfv.org/vsperf/test_methodology + + + + + + + + + + + + LTD Test Spec Overview + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review + + + + + + + + + + + + https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ + + + + + + + + + + + diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml new file mode 100644 index 00000000..a9405a77 --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml @@ -0,0 +1,1016 @@ + + + + + + + + + + + + + + + Benchmarking Virtual Switches in + OPNFV + + + Intel + +
+ + + + + + + + + + + + + + + + + maryam.tahhan@intel.com + + +
+ + + + Intel + +
+ + + + + + + + + + + + + + + + + billy.o.mahony@intel.com + + +
+ + + + AT&T Labs + +
+ + 200 Laurel Avenue South + + Middletown, + + NJ + + 07748 + + USA + + + +1 732 420 1571 + + +1 732 368 1192 + + acmorton@att.com + + http://home.comcast.net/~acmacm/ +
+ + + + + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure. + + + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119. + + + + + + +
+ Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention. + + This memo summarizes the progress of the Open Platform for NFV + (OPNFV) project on virtual switch performance characterization, + "VSWITCHPERF", through the Brahmaputra (second) release . This project intends to build on the current and + completed work of the Benchmarking Methodology Working Group in IETF, by + referencing existing literature. For example, currently the most often + referenced RFC is (which depends on ) and foundation of the benchmarking work in OPNFV is + common and strong. + + See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/ + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example. + + Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry. The authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is one part of the efforts. We + acknowledge the early work in , and useful + discussion with the authors. +
+ +
+ The primary purpose and scope of the memo is to inform the industry + of work-in-progress that builds on the body of extensive BMWG literature + and experience, and describe the extensions needed for benchmarking + virtual switches. Inital feedback indicates that many of these + extensions may be applicable beyond the current scope (to hardware + switches in the NFV Infrastructure and to virtual routers, for example). + Additionally, this memo serves as a vehicle to include more detail and + commentary from BMWG and other Open Source communities, under BMWG's + chartered work to characterize the NFV Infrastructure (a virtual switch + is an important aspect of that infrastructure). +
+ +
+ This section highlights some specific considerations (from )related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document. + +
+ To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions. +
+ +
+ External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed. +
+ +
+ A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability. + + Hardware details including: + + + Platform details + + Processor details + + Memory information (type and size) + + Number of enabled cores + + Number of cores used for the test + + Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into) + + NIC interrupt configuration + + BIOS version, release date and any configurations that were + modified + + CPU microcode level + + Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations + + PCI configuration parameters (payload size, early ack + option...) + + Power management at all levels (ACPI sleep states, processor + package, OS...) + Software details including: + + + OS parameters and behavior (text vs graphical no one typing at + the console on one system) + + OS version (for host and VNF) + + Kernel version (for host and VNF) + + GRUB boot parameters (for host and VNF) + + Hypervisor details (Type and version) + + Selected vSwitch, version number or commit id used + + vSwitch launch command line if it has been parameterised + + Memory allocation to the vSwitch + + which NUMA node it is using, and how many memory channels + + DPDK or any other SW dependency version number or commit id + used + + Memory allocation to a VM - if it's from Hugpages/elsewhere + + VM storage type: snapshot/independent persistent/independent + non-persistent + + Number of VMs + + Number of Virtual NICs (vNICs), versions, type and driver + + Number of virtual CPUs and their core affinity on the host + + Number vNIC interrupt configuration + + Thread affinitization for the applications (including the + vSwitch itself) on the host + + Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows. + + + Test Traffic Information: + Traffic type - UDP, TCP, IMIX / Other + + Packet Sizes + + Deployment Scenario + + + +
+ +
+ Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values or + have arrived on the same port. Performance results can vary based on + the parameters the vSwitch uses to match for a flow. The recommended + flow classification parameters for any vSwitch performance tests are: + the input port, the source IP address, the destination IP address and + the Ethernet protocol type field. It is essential to increase the flow + timeout time on a vSwitch before conducting any performance tests that + do not measure the flow setup time. Normally the first packet of a + particular stream will install the flow in the virtual switch which + adds an additional latency, subsequent packets of the same flow are + not subject to this latency if the flow is already installed on the + vSwitch. +
+ +
+ This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time. + + + Baselines: + Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency).
+ Benchmark platform forwarding + capability + + + + +
+ + Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance.
+ Benchmark VNF forwarding capability + + + + +
+ + Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT + + Benchmarking with isolated resources alone, leaving some + resources unused + + Benchmark with isolated resources and all resources + occupied + + + Next Steps + Limited sharing + + Production scenarios + + Stressful scenarios + + +
+
+ +
+ The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include: + + + Benchmarking Methodology for Network + Interconnect Devices + + Benchmarking Methodology for LAN + Switching + + Device Reset Characterization + + Packet Delay Variation Applicability + Statement + + + Some of the above/newer RFCs are being applied in benchmarking for + the first time, and represent a development challenge for test equipment + developers. Fortunately, many members of the testing system community + have engaged on the VSPERF project, including an open source test + system. + + In addition to this, the LTD also re-uses the terminology defined + by: + + + Benchmarking Terminology for LAN + Switching Devices + + Packet Delay Variation Applicability + Statement + + + + + Specifications to be included in future updates of the LTD + include: + Methodology for IP Multicast + Benchmarking + + Packet Reordering Metrics + + + As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a + metric derived from IPPM's , where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized. + + Tests have been (or will be) designed to collect the metrics + below: + + + Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by ) without traffic loss. + + Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads. + + Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers. + + Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases. + + Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch. + + Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow). + + CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint. + + The so-called "Soak" tests, where the selected test is conducted + over a long period of time (with an ideal duration of 24 hours, and + at least 6 hours). The purpose of soak tests is to capture transient + changes in performance which may occur due to infrequent processes + or the low probability coincidence of two or more processes. The + performance must be evaluated periodically during continuous + testing, and this results in use of Frame + Rate metrics instead of Throughput (which + requires stopping traffic to allow time for all traffic to exit + internal queues). + + + Future/planned test specs include: + Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch. + + Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch. + + Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements on characterization of + acceleration technologies applied to vswitches. + The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include: + +
+ Physical port to virtual switch to physical + port + + +
+ +
+ Physical port to virtual switch to VNF to virtual switch + to physical port + + +
+ Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port + + +
+ Physical port to virtual switch to VNF + + +
+ VNF to virtual switch to physical port + + +
+ VNF to virtual switch to VNF + + +
+ + A set of Deployment Scenario figures is available on the VSPERF Test + Methodology Wiki page . +
+ +
+ This section organizes the many existing test specifications into the + "3x3" matrix (introduced in ). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale metrics + when displaying the coverage of many metrics/benchmarks). The current + version of the LTD specification is available . + + The tests listed below assess the activation of paths in the data + plane, rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" Wiki page . + +
+ + Activation.RFC2889.AddressLearningRate + + PacketLatency.InitialPacketProcessingLatency + +
+ +
+ + CPDP.Coupling.Flow.Addition + +
+ +
+ + Throughput.RFC2544.SystemRecoveryTime + + Throughput.RFC2544.ResetTime + +
+ +
+ + Activation.RFC2889.AddressCachingCapacity + +
+ +
+ + Throughput.RFC2544.PacketLossRate + + CPU.RFC2544.0PacketLoss + + Throughput.RFC2544.PacketLossRateFrameModification + + Throughput.RFC2544.BackToBackFrames + + Throughput.RFC2889.MaxForwardingRate + + Throughput.RFC2889.ForwardPressure + + Throughput.RFC2889.BroadcastFrameForwarding + +
+ +
+ + Throughput.RFC2889.ErrorFramesFiltering + + Throughput.RFC2544.Profile + +
+ +
+ + Throughput.RFC2889.Soak + + Throughput.RFC2889.SoakFrameModification + + PacketDelayVariation.RFC3393.Soak + +
+ +
+ + Scalability.RFC2544.0PacketLoss + + MemoryBandwidth.RFC2544.0PacketLoss.Scalability + +
+ +
+
+ +
+
+
+ +
+ Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. +
+ +
+ No IANA Action is requested at this time. +
+ +
+ The authors appreciate and acknowledge comments from Scott Bradner, + Marius Georgescu, Ramki Krishnan, and Doug Montgomery, and others for + their reviews. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network Function Virtualization: Performance and Portability + Best Practices + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Test Topologies + https://wiki.opnfv.org/vsperf/test_methodology + + + + + + + + + + + + LTD Test Spec Overview + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review + + + + + + + + + + + + LTD Test Specification + http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html + + + + + + + + + + + + Brahmaputra, Second OPNFV Release + https://www.opnfv.org/brahmaputra + + + + + + + + + + + + https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ + + + + + + + + + + + diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml new file mode 100644 index 00000000..9157763e --- /dev/null +++ b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-02.xml @@ -0,0 +1,1016 @@ + + + + + + + + + + + + + + + Benchmarking Virtual Switches in + OPNFV + + + Intel + +
+ + + + + + + + + + + + + + + + + maryam.tahhan@intel.com + + +
+ + + + Intel + +
+ + + + + + + + + + + + + + + + + billy.o.mahony@intel.com + + +
+ + + + AT&T Labs + +
+ + 200 Laurel Avenue South + + Middletown, + + NJ + + 07748 + + USA + + + +1 732 420 1571 + + +1 732 368 1192 + + acmorton@att.com + + http://home.comcast.net/~acmacm/ +
+ + + + + + This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure. + + + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in RFC 2119. + + + + + + +
+ Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention. + + This memo summarizes the progress of the Open Platform for NFV + (OPNFV) project on virtual switch performance characterization, + "VSWITCHPERF", through the Brahmaputra (second) release . This project intends to build on the current and + completed work of the Benchmarking Methodology Working Group in IETF, by + referencing existing literature. For example, currently the most often + referenced RFC is (which depends on ) and foundation of the benchmarking work in OPNFV is + common and strong. + + See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/ + + The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example. + + Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry. The authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is one part of the efforts. We + acknowledge the early work in , and useful + discussion with the authors. +
+ +
+ The primary purpose and scope of the memo is to inform the industry + of work-in-progress that builds on the body of extensive BMWG literature + and experience, and describe the extensions needed for benchmarking + virtual switches. Inital feedback indicates that many of these + extensions may be applicable beyond the current scope (to hardware + switches in the NFV Infrastructure and to virtual routers, for example). + Additionally, this memo serves as a vehicle to include more detail and + commentary from BMWG and other Open Source communities, under BMWG's + chartered work to characterize the NFV Infrastructure (a virtual switch + is an important aspect of that infrastructure). +
+ +
+ This section highlights some specific considerations (from )related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document. + +
+ To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level. + + It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions. +
+ +
+ External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed. +
+ +
+ A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability. + + Hardware details including: + + + Platform details + + Processor details + + Memory information (type and size) + + Number of enabled cores + + Number of cores used for the test + + Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into) + + NIC interrupt configuration + + BIOS version, release date and any configurations that were + modified + + CPU microcode level + + Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations + + PCI configuration parameters (payload size, early ack + option...) + + Power management at all levels (ACPI sleep states, processor + package, OS...) + Software details including: + + + OS parameters and behavior (text vs graphical no one typing at + the console on one system) + + OS version (for host and VNF) + + Kernel version (for host and VNF) + + GRUB boot parameters (for host and VNF) + + Hypervisor details (Type and version) + + Selected vSwitch, version number or commit id used + + vSwitch launch command line if it has been parameterised + + Memory allocation to the vSwitch + + which NUMA node it is using, and how many memory channels + + DPDK or any other SW dependency version number or commit id + used + + Memory allocation to a VM - if it's from Hugpages/elsewhere + + VM storage type: snapshot/independent persistent/independent + non-persistent + + Number of VMs + + Number of Virtual NICs (vNICs), versions, type and driver + + Number of virtual CPUs and their core affinity on the host + + Number vNIC interrupt configuration + + Thread affinitization for the applications (including the + vSwitch itself) on the host + + Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows. + + + Test Traffic Information: + Traffic type - UDP, TCP, IMIX / Other + + Packet Sizes + + Deployment Scenario + + + +
+ +
+ Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values or + have arrived on the same port. Performance results can vary based on + the parameters the vSwitch uses to match for a flow. The recommended + flow classification parameters for any vSwitch performance tests are: + the input port, the source IP address, the destination IP address and + the Ethernet protocol type field. It is essential to increase the flow + timeout time on a vSwitch before conducting any performance tests that + do not measure the flow setup time. Normally the first packet of a + particular stream will install the flow in the virtual switch which + adds an additional latency, subsequent packets of the same flow are + not subject to this latency if the flow is already installed on the + vSwitch. +
+ +
+ This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time. + + + Baselines: + Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency).
+ Benchmark platform forwarding + capability + + + + +
+ + Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance.
+ Benchmark VNF forwarding capability + + + + +
+ + Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT + + Benchmarking with isolated resources alone, leaving some + resources unused + + Benchmark with isolated resources and all resources + occupied + + + Next Steps + Limited sharing + + Production scenarios + + Stressful scenarios + + +
+
+ +
+ The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include: + + + Benchmarking Methodology for Network + Interconnect Devices + + Benchmarking Methodology for LAN + Switching + + Device Reset Characterization + + Packet Delay Variation Applicability + Statement + + + Some of the above/newer RFCs are being applied in benchmarking for + the first time, and represent a development challenge for test equipment + developers. Fortunately, many members of the testing system community + have engaged on the VSPERF project, including an open source test + system. + + In addition to this, the LTD also re-uses the terminology defined + by: + + + Benchmarking Terminology for LAN + Switching Devices + + Packet Delay Variation Applicability + Statement + + + + + Specifications to be included in future updates of the LTD + include: + Methodology for IP Multicast + Benchmarking + + Packet Reordering Metrics + + + As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification. + + When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a + metric derived from IPPM's , where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized. + + Tests have been (or will be) designed to collect the metrics + below: + + + Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by ) without traffic loss. + + Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads. + + Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers. + + Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases. + + Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch. + + Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow). + + CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint. + + The so-called "Soak" tests, where the selected test is conducted + over a long period of time (with an ideal duration of 24 hours, and + at least 6 hours). The purpose of soak tests is to capture transient + changes in performance which may occur due to infrequent processes + or the low probability coincidence of two or more processes. The + performance must be evaluated periodically during continuous + testing, and this results in use of Frame + Rate metrics instead of Throughput (which + requires stopping traffic to allow time for all traffic to exit + internal queues). + + + Future/planned test specs include: + Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch. + + Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch. + + Tests derived from examination of ETSI NFV Draft GS IFA003 + requirements on characterization of + acceleration technologies applied to vswitches. + The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include: + +
+ Physical port to virtual switch to physical + port + + +
+ +
+ Physical port to virtual switch to VNF to virtual switch + to physical port + + +
+ Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port + + +
+ Physical port to virtual switch to VNF + + +
+ VNF to virtual switch to physical port + + +
+ VNF to virtual switch to VNF + + +
+ + A set of Deployment Scenario figures is available on the VSPERF Test + Methodology Wiki page . +
+ +
+ This section organizes the many existing test specifications into the + "3x3" matrix (introduced in ). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale metrics + when displaying the coverage of many metrics/benchmarks). The current + version of the LTD specification is available . + + The tests listed below assess the activation of paths in the data + plane, rather than the control plane. + + A complete list of tests with short summaries is available on the + VSPERF "LTD Test Spec Overview" Wiki page . + +
+ + Activation.RFC2889.AddressLearningRate + + PacketLatency.InitialPacketProcessingLatency + +
+ +
+ + CPDP.Coupling.Flow.Addition + +
+ +
+ + Throughput.RFC2544.SystemRecoveryTime + + Throughput.RFC2544.ResetTime + +
+ +
+ + Activation.RFC2889.AddressCachingCapacity + +
+ +
+ + Throughput.RFC2544.PacketLossRate + + CPU.RFC2544.0PacketLoss + + Throughput.RFC2544.PacketLossRateFrameModification + + Throughput.RFC2544.BackToBackFrames + + Throughput.RFC2889.MaxForwardingRate + + Throughput.RFC2889.ForwardPressure + + Throughput.RFC2889.BroadcastFrameForwarding + +
+ +
+ + Throughput.RFC2889.ErrorFramesFiltering + + Throughput.RFC2544.Profile + +
+ +
+ + Throughput.RFC2889.Soak + + Throughput.RFC2889.SoakFrameModification + + PacketDelayVariation.RFC3393.Soak + +
+ +
+ + Scalability.RFC2544.0PacketLoss + + MemoryBandwidth.RFC2544.0PacketLoss.Scalability + +
+ +
+
+ +
+
+
+ +
+ Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network. + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. +
+ +
+ No IANA Action is requested at this time. +
+ +
+ The authors appreciate and acknowledge comments from Scott Bradner, + Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik, + Christian Trautman, and others for their reviews. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network Function Virtualization: Performance and Portability + Best Practices + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Test Topologies + https://wiki.opnfv.org/vsperf/test_methodology + + + + + + + + + + + + LTD Test Spec Overview + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review + + + + + + + + + + + + LTD Test Specification + http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html + + + + + + + + + + + + Brahmaputra, Second OPNFV Release + https://www.opnfv.org/brahmaputra + + + + + + + + + + + + https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ + + + + + + + + + + + -- cgit 1.2.3-korg