From c17fe61eba519ca1c49aee2214438f1a248fe9f8 Mon Sep 17 00:00:00 2001 From: Trevor Cooper Date: Sun, 1 Oct 2017 21:51:47 -0700 Subject: Removed drafts of test specificaiton now published as RFC8204. Also removed IETF license notice and added a file with link to RFC8204 Change-Id: I44b4cb75afeff74a58f402a7cb8816ddfbe52b87 Signed-off-by: Trevor Cooper --- .../draft-vsperf-bmwg-vswitch-opnfv-00.xml | 964 --------------------- 1 file changed, 964 deletions(-) delete mode 100644 docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml (limited to 'docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml') diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml deleted file mode 100644 index b5f7f833..00000000 --- a/docs/testing/developer/devguide/requirements/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml +++ /dev/null @@ -1,964 +0,0 @@ - - - - - - - - - - - - - - - Benchmarking Virtual Switches in - OPNFV - - - Intel - -
- - - - - - - - - - - - - - - - - maryam.tahhan@intel.com - - -
- - - - Intel - -
- - - - - - - - - - - - - - - - - billy.o.mahony@intel.com - - -
- - - - AT&T Labs - -
- - 200 Laurel Avenue South - - Middletown, - - NJ - - 07748 - - USA - - - +1 732 420 1571 - - +1 732 368 1192 - - acmorton@att.com - - http://home.comcast.net/~acmacm/ -
- - - - - - This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance "VSWITCHPERF". This project - intends to build on the current and completed work of the Benchmarking - Methodology Working Group in IETF, by referencing existing literature. - The Benchmarking Methodology Working Group has traditionally conducted - laboratory characterization of dedicated physical implementations of - internetworking functions. Therefore, this memo begins to describe the - additional considerations when virtual switches are implemented in - general-purpose hardware. The expanded tests and benchmarks are also - influenced by the OPNFV mission to support virtualization of the "telco" - infrastructure. - - - - The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", - "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this - document are to be interpreted as described in RFC 2119. - - - - - - -
- Benchmarking Methodology Working Group (BMWG) has traditionally - conducted laboratory characterization of dedicated physical - implementations of internetworking functions. The Black-box Benchmarks - of Throughput, Latency, Forwarding Rates and others have served our - industry for many years. Now, Network Function Virtualization (NFV) has - the goal to transform how internetwork functions are implemented, and - therefore has garnered much attention. - - This memo describes the progress of the Open Platform for NFV (OPNFV) - project on virtual switch performance characterization, "VSWITCHPERF". - This project intends to build on the current and completed work of the - Benchmarking Methodology Working Group in IETF, by referencing existing - literature. For example, currently the most often referenced RFC is - (which depends on ) and - foundation of the benchmarking work in OPNFV is common and strong. - - See - https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases - for more background, and the OPNFV website for general information: - https://www.opnfv.org/ - - The authors note that OPNFV distinguishes itself from other open - source compute and networking projects through its emphasis on existing - "telco" services as opposed to cloud-computing. There are many ways in - which telco requirements have different emphasis on performance - dimensions when compared to cloud computing: support for and transfer of - isochronous media streams is one example. - - Note also that the move to NFV Infrastructure has resulted in many - new benchmarking initiatives across the industry, and the authors are - currently doing their best to maintain alignment with many other - projects, and this Internet Draft is evidence of the efforts. -
- -
- The primary purpose and scope of the memo is to inform BMWG of - work-in-progress that builds on the body of extensive literature and - experience. Additionally, once the initial information conveyed here is - received, this memo may be expanded to include more detail and - commentary from both BMWG and OPNFV communities, under BMWG's chartered - work to characterize the NFV Infrastructure (a virtual switch is an - important aspect of that infrastructure). -
- -
- This section highlights some specific considerations (from )related to Benchmarks for virtual - switches. The OPNFV project is sharing its present view on these areas, - as they develop their specifications in the Level Test Design (LTD) - document. - -
- To compare the performance of virtual designs and implementations - with their physical counterparts, identical benchmarks are needed. - BMWG has developed specifications for many network functions this memo - re-uses existing benchmarks through references, and expands them - during development of new methods. A key configuration aspect is the - number of parallel cores required to achieve comparable performance - with a given physical device, or whether some limit of scale was - reached before the cores could achieve the comparable level. - - It's unlikely that the virtual switch will be the only application - running on the SUT, so CPU utilization, Cache utilization, and Memory - footprint should also be recorded for the virtual implementations of - internetworking functions. -
- -
- External observations remain essential as the basis for Benchmarks. - Internal observations with fixed specification and interpretation will - be provided in parallel to assist the development of operations - procedures when the technology is deployed. -
- -
- A key consideration when conducting any sort of benchmark is trying - to ensure the consistency and repeatability of test results. When - benchmarking the performance of a vSwitch there are many factors that - can affect the consistency of results, one key factor is matching the - various hardware and software details of the SUT. This section lists - some of the many new parameters which this project believes are - critical to report in order to achieve repeatability. - - Hardware details including: - - - Platform details - - Processor details - - Memory information (type and size) - - Number of enabled cores - - Number of cores used for the test - - Number of physical NICs, as well as their details - (manufacturer, versions, type and the PCI slot they are plugged - into) - - NIC interrupt configuration - - BIOS version, release date and any configurations that were - modified - - CPU microcode level - - Memory DIMM configurations (quad rank performance may not be - the same as dual rank) in size, freq and slot locations - - PCI configuration parameters (payload size, early ack - option...) - - Power management at all levels (ACPI sleep states, processor - package, OS...) - Software details including: - - - OS parameters and behavior (text vs graphical no one typing at - the console on one system) - - OS version (for host and VNF) - - Kernel version (for host and VNF) - - GRUB boot parameters (for host and VNF) - - Hypervisor details (Type and version) - - Selected vSwitch, version number or commit id used - - vSwitch launch command line if it has been parameterised - - Memory allocation to the vSwitch - - which NUMA node it is using, and how many memory channels - - DPDK or any other SW dependency version number or commit id - used - - Memory allocation to a VM - if it's from Hugpages/elsewhere - - VM storage type: snapshot/independent persistent/independent - non-persistent - - Number of VMs - - Number of Virtual NICs (vNICs), versions, type and driver - - Number of virtual CPUs and their core affinity on the host - - Number vNIC interrupt configuration - - Thread affinitization for the applications (including the - vSwitch itself) on the host - - Details of Resource isolation, such as CPUs designated for - Host/Kernel (isolcpu) and CPUs designated for specific processes - (taskset). - Test duration. - Number of flows. - - - Test Traffic Information: - Traffic type - UDP, TCP, IMIX / Other - - Packet Sizes - - Deployment Scenario - - - -
- -
- Virtual switches group packets into flows by processing and - matching particular packet or frame header information, or by matching - packets based on the input ports. Thus a flow can be thought of a - sequence of packets that have the same set of header field values or - have arrived on the same port. Performance results can vary based on - the parameters the vSwitch uses to match for a flow. The recommended - flow classification parameters for any vSwitch performance tests are: - the input port, the source IP address, the destination IP address and - the Ethernet protocol type field. It is essential to increase the flow - timeout time on a vSwitch before conducting any performance tests that - do not measure the flow setup time. Normally the first packet of a - particular stream will install the flow in the virtual switch which - adds an additional latency, subsequent packets of the same flow are - not subject to this latency if the flow is already installed on the - vSwitch. -
- -
- This outline describes measurement of baseline with isolated - resources at a high level, which is the intended approach at this - time. - - - Baselines: - Optional: Benchmark platform forwarding capability without - a vswitch or VNF for at least 72 hours (serves as a means of - platform validation and a means to obtain the base performance - for the platform in terms of its maximum forwarding rate and - latency).
- Benchmark platform forwarding - capability - - - - -
- - Benchmark VNF forwarding capability with direct - connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 - hours (serves as a means of VNF validation and a means to - obtain the base performance for the VNF in terms of its - maximum forwarding rate and latency). The metrics gathered - from this test will serve as a key comparison point for - vSwitch bypass technologies performance and vSwitch - performance.
- Benchmark VNF forwarding capability - - - - -
- - Benchmarking with isolated resources alone, with other - resources (both HW&SW) disabled Example, vSw and VM are - SUT - - Benchmarking with isolated resources alone, leaving some - resources unused - - Benchmark with isolated resources and all resources - occupied - - - Next Steps - Limited sharing - - Production scenarios - - Stressful scenarios - - -
-
- -
- The overall specification in preparation is referred to as a Level - Test Design (LTD) document, which will contain a suite of performance - tests. The base performance tests in the LTD are based on the - pre-existing specifications developed by BMWG to test the performance of - physical switches. These specifications include: - - - Benchmarking Methodology for Network - Interconnect Devices - - Benchmarking Methodology for LAN - Switching - - Device Reset Characterization - - Packet Delay Variation Applicability - Statement - - - Some of the above/newer RFCs are being applied in benchmarking for - the first time, and represent a development challenge for test equipment - developers. Fortunately, many members of the testing system community - have engaged on the VSPERF project, including an open source test - system. - - In addition to this, the LTD also re-uses the terminology defined - by: - - - Benchmarking Terminology for LAN - Switching Devices - - Packet Delay Variation Applicability - Statement - - - - - Specifications to be included in future updates of the LTD - include: - Methodology for IP Multicast - Benchmarking - - Packet Reordering Metrics - - - As one might expect, the most fundamental internetworking - characteristics of Throughput and Latency remain important when the - switch is virtualized, and these benchmarks figure prominently in the - specification. - - When considering characteristics important to "telco" network - functions, we must begin to consider additional performance metrics. In - this case, the project specifications have referenced metrics from the - IETF IP Performance Metrics (IPPM) literature. This means that the test of Latency is replaced by measurement of a - metric derived from IPPM's , where a set of - statistical summaries will be provided (mean, max, min, etc.). Further - metrics planned to be benchmarked include packet delay variation as - defined by , reordering, burst behaviour, DUT - availability, DUT capacity and packet loss in long term testing at - Throughput level, where some low-level of background loss may be present - and characterized. - - Tests have been (or will be) designed to collect the metrics - below: - - - Throughput Tests to measure the maximum forwarding rate (in - frames per second or fps) and bit rate (in Mbps) for a constant load - (as defined by RFC1242) without traffic loss. - - Packet and Frame Delay Distribution Tests to measure average, min - and max packet and frame delay for constant loads. - - Packet Delay Tests to understand latency distribution for - different packet sizes and over an extended test run to uncover - outliers. - - Scalability Tests to understand how the virtual switch performs - as the number of flows, active ports, complexity of the forwarding - logic’s configuration… it has to deal with - increases. - - Stream Performance Tests (TCP, UDP) to measure bulk data transfer - performance, i.e. how fast systems can send and receive data through - the switch. - - Control Path and Datapath Coupling Tests, to understand how - closely coupled the datapath and the control path are as well as the - effect of this coupling on the performance of the DUT (example: - delay of the initial packet of a flow). - - CPU and Memory Consumption Tests to understand the virtual - switch’s footprint on the system, usually conducted as - auxiliary measurements with benchmarks above. They include: CPU - utilization, Cache utilization and Memory footprint. - - - Future/planned test specs include: - Request/Response Performance Tests (TCP, UDP) which measure the - transaction rate through the switch. - - Noisy Neighbour Tests, to understand the effects of resource - sharing on the performance of a virtual switch. - - Tests derived from examination of ETSI NFV Draft GS IFA003 - requirements on characterization of - acceleration technologies applied to vswitches. - The flexibility of deployment of a virtual switch within a - network means that the BMWG IETF existing literature needs to be used to - characterize the performance of a switch in various deployment - scenarios. The deployment scenarios under consideration include: - -
- Physical port to virtual switch to physical - port - - -
- -
- Physical port to virtual switch to VNF to virtual switch - to physical port - - -
- Physical port to virtual switch to VNF to virtual switch - to VNF to virtual switch to physical port - - -
- Physical port to virtual switch to VNF - - -
- VNF to virtual switch to physical port - - -
- VNF to virtual switch to VNF - - -
- - A set of Deployment Scenario figures is available on the VSPERF Test - Methodology Wiki page . -
- -
- This section organizes the many existing test specifications into the - "3x3" matrix (introduced in ). - Because the LTD specification ID names are quite long, this section is - organized into lists for each occupied cell of the matrix (not all are - occupied, also the matrix has grown to 3x4 to accommodate scale metrics - when displaying the coverage of many metrics/benchmarks). - - The tests listed below assess the activation of paths in the data - plane, rather than the control plane. - - A complete list of tests with short summaries is available on the - VSPERF "LTD Test Spec Overview" Wiki page . - -
- - Activation.RFC2889.AddressLearningRate - - PacketLatency.InitialPacketProcessingLatency - -
- -
- - CPDP.Coupling.Flow.Addition - -
- -
- - Throughput.RFC2544.SystemRecoveryTime - - Throughput.RFC2544.ResetTime - -
- -
- - Activation.RFC2889.AddressCachingCapacity - -
- -
- - Throughput.RFC2544.PacketLossRate - - CPU.RFC2544.0PacketLoss - - Throughput.RFC2544.PacketLossRateFrameModification - - Throughput.RFC2544.BackToBackFrames - - Throughput.RFC2889.MaxForwardingRate - - Throughput.RFC2889.ForwardPressure - - Throughput.RFC2889.BroadcastFrameForwarding - -
- -
- - Throughput.RFC2889.ErrorFramesFiltering - - Throughput.RFC2544.Profile - -
- -
- - Throughput.RFC2889.Soak - - Throughput.RFC2889.SoakFrameModification - - PacketDelayVariation.RFC3393.Soak - -
- -
- - Scalability.RFC2544.0PacketLoss - - MemoryBandwidth.RFC2544.0PacketLoss.Scalability - -
- -
-
- -
-
-
- -
- Benchmarking activities as described in this memo are limited to - technology characterization of a Device Under Test/System Under Test - (DUT/SUT) using controlled stimuli in a laboratory environment, with - dedicated address space and the constraints specified in the sections - above. - - The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test traffic - into a production network, or misroute traffic to the test management - network. - - Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT. - - Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks. -
- -
- No IANA Action is requested at this time. -
- -
- The authors acknowledge -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Network Function Virtualization: Performance and Portability - Best Practices - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Test Topologies - https://wiki.opnfv.org/vsperf/test_methodology - - - - - - - - - - - - LTD Test Spec Overview - https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review - - - - - - - - - - - - https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/ - - - - - - - - - - - -- cgit 1.2.3-korg