summaryrefslogtreecommitdiffstats
path: root/docs/test_spec/ietf_draft
diff options
context:
space:
mode:
authorMaryam Tahhan <maryam.tahhan@intel.com>2015-12-02 13:42:50 +0000
committerMaryam Tahhan <maryam.tahhan@intel.com>2015-12-21 10:44:07 +0000
commitcc3a4bf85074989c28a09b7b141dea5e42701f5c (patch)
tree18740fd268ff0d631812b36eb45636489cf8b75d /docs/test_spec/ietf_draft
parentde0aaffe31e1f787cefe21a5957a07924bb5956f (diff)
docs: updates and move traffic gens to separate doc
Move the traffic gen instructions to a separate user guide and add information on usage of the Dummy traffic generator. Update docs to fix PDF build failure and do general clean-up. Removed the numbering from the LTD and added the numbered directive to automate numbering for sections and headers. Add comment anchors that reflect the section numbers. Change-Id: I984ca38456a891c439697ebc1da041bc1d828a15 Signed-off-by: Maryam Tahhan <maryam.tahhan@intel.com>
Diffstat (limited to 'docs/test_spec/ietf_draft')
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml964
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt1232
-rwxr-xr-xdocs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml964
3 files changed, 0 insertions, 3160 deletions
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml
deleted file mode 100755
index b5f7f833..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml
+++ /dev/null
@@ -1,964 +0,0 @@
-<?xml version="1.0" encoding="US-ASCII"?>
-<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
-<?rfc toc="yes"?>
-<?rfc tocompact="yes"?>
-<?rfc tocdepth="3"?>
-<?rfc tocindent="yes"?>
-<?rfc symrefs="yes"?>
-<?rfc sortrefs="yes"?>
-<?rfc comments="yes"?>
-<?rfc inline="yes"?>
-<?rfc compact="yes"?>
-<?rfc subcompact="no"?>
-<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-01"
- ipr="trust200902">
- <front>
- <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
- OPNFV</title>
-
- <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>maryam.tahhan@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>billy.o.mahony@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Al Morton" initials="A." surname="Morton">
- <organization>AT&amp;T Labs</organization>
-
- <address>
- <postal>
- <street>200 Laurel Avenue South</street>
-
- <city>Middletown,</city>
-
- <region>NJ</region>
-
- <code>07748</code>
-
- <country>USA</country>
- </postal>
-
- <phone>+1 732 420 1571</phone>
-
- <facsimile>+1 732 368 1192</facsimile>
-
- <email>acmorton@att.com</email>
-
- <uri>http://home.comcast.net/~acmacm/</uri>
- </address>
- </author>
-
- <date day="14" month="October" year="2015"/>
-
- <abstract>
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the Benchmarking
- Methodology Working Group in IETF, by referencing existing literature.
- The Benchmarking Methodology Working Group has traditionally conducted
- laboratory characterization of dedicated physical implementations of
- internetworking functions. Therefore, this memo begins to describe the
- additional considerations when virtual switches are implemented in
- general-purpose hardware. The expanded tests and benchmarks are also
- influenced by the OPNFV mission to support virtualization of the "telco"
- infrastructure.</t>
- </abstract>
-
- <note title="Requirements Language">
- <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in <xref
- target="RFC2119">RFC 2119</xref>.</t>
-
- <t/>
- </note>
- </front>
-
- <middle>
- <section title="Introduction">
- <t>Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box Benchmarks
- of Throughput, Latency, Forwarding Rates and others have served our
- industry for many years. Now, Network Function Virtualization (NFV) has
- the goal to transform how internetwork functions are implemented, and
- therefore has garnered much attention.</t>
-
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization, "VSWITCHPERF".
- This project intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing existing
- literature. For example, currently the most often referenced RFC is
- <xref target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and
- foundation of the benchmarking work in OPNFV is common and strong.</t>
-
- <t>See
- https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
- for more background, and the OPNFV website for general information:
- https://www.opnfv.org/</t>
-
- <t>The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on existing
- "telco" services as opposed to cloud-computing. There are many ways in
- which telco requirements have different emphasis on performance
- dimensions when compared to cloud computing: support for and transfer of
- isochronous media streams is one example.</t>
-
- <t>Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.</t>
- </section>
-
- <section title="Scope">
- <t>The primary purpose and scope of the memo is to inform BMWG of
- work-in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here is
- received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's chartered
- work to characterize the NFV Infrastructure (a virtual switch is an
- important aspect of that infrastructure).</t>
- </section>
-
- <section title="Benchmarking Considerations">
- <t>This section highlights some specific considerations (from <xref
- target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these areas,
- as they develop their specifications in the Level Test Design (LTD)
- document.</t>
-
- <section title="Comparison with Physical Network Functions">
- <t>To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this memo
- re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.</t>
-
- <t>It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.</t>
- </section>
-
- <section title="Continued Emphasis on Black-Box Benchmarks">
- <t>External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation will
- be provided in parallel to assist the development of operations
- procedures when the technology is deployed.</t>
- </section>
-
- <section title="New Configuration Parameters">
- <t>A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.</t>
-
- <t>Hardware details including:</t>
-
- <t><list style="symbols">
- <t>Platform details</t>
-
- <t>Processor details</t>
-
- <t>Memory information (type and size)</t>
-
- <t>Number of enabled cores</t>
-
- <t>Number of cores used for the test</t>
-
- <t>Number of physical NICs, as well as their details
- (manufacturer, versions, type and the PCI slot they are plugged
- into)</t>
-
- <t>NIC interrupt configuration</t>
-
- <t>BIOS version, release date and any configurations that were
- modified</t>
-
- <t>CPU microcode level</t>
-
- <t>Memory DIMM configurations (quad rank performance may not be
- the same as dual rank) in size, freq and slot locations</t>
-
- <t>PCI configuration parameters (payload size, early ack
- option...)</t>
-
- <t>Power management at all levels (ACPI sleep states, processor
- package, OS...)</t>
- </list>Software details including:</t>
-
- <t><list style="symbols">
- <t>OS parameters and behavior (text vs graphical no one typing at
- the console on one system)</t>
-
- <t>OS version (for host and VNF)</t>
-
- <t>Kernel version (for host and VNF)</t>
-
- <t>GRUB boot parameters (for host and VNF)</t>
-
- <t>Hypervisor details (Type and version)</t>
-
- <t>Selected vSwitch, version number or commit id used</t>
-
- <t>vSwitch launch command line if it has been parameterised</t>
-
- <t>Memory allocation to the vSwitch</t>
-
- <t>which NUMA node it is using, and how many memory channels</t>
-
- <t>DPDK or any other SW dependency version number or commit id
- used</t>
-
- <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t>
-
- <t>VM storage type: snapshot/independent persistent/independent
- non-persistent</t>
-
- <t>Number of VMs</t>
-
- <t>Number of Virtual NICs (vNICs), versions, type and driver</t>
-
- <t>Number of virtual CPUs and their core affinity on the host</t>
-
- <t>Number vNIC interrupt configuration</t>
-
- <t>Thread affinitization for the applications (including the
- vSwitch itself) on the host</t>
-
- <t>Details of Resource isolation, such as CPUs designated for
- Host/Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.</t>
- </list></t>
-
- <t>Test Traffic Information:<list style="symbols">
- <t>Traffic type - UDP, TCP, IMIX / Other</t>
-
- <t>Packet Sizes</t>
-
- <t>Deployment Scenario</t>
- </list></t>
-
- <t/>
- </section>
-
- <section title="Flow classification">
- <t>Virtual switches group packets into flows by processing and
- matching particular packet or frame header information, or by matching
- packets based on the input ports. Thus a flow can be thought of a
- sequence of packets that have the same set of header field values or
- have arrived on the same port. Performance results can vary based on
- the parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the flow
- timeout time on a vSwitch before conducting any performance tests that
- do not measure the flow setup time. Normally the first packet of a
- particular stream will install the flow in the virtual switch which
- adds an additional latency, subsequent packets of the same flow are
- not subject to this latency if the flow is already installed on the
- vSwitch.</t>
- </section>
-
- <section title="Benchmarks using Baselines with Resource Isolation">
- <t>This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.</t>
-
- <t><list style="numbers">
- <t>Baselines: <list style="symbols">
- <t>Optional: Benchmark platform forwarding capability without
- a vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency). <figure>
- <preamble>Benchmark platform forwarding
- capability</preamble>
-
- <artwork align="right"><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmark VNF forwarding capability with direct
- connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72
- hours (serves as a means of VNF validation and a means to
- obtain the base performance for the VNF in terms of its
- maximum forwarding rate and latency). The metrics gathered
- from this test will serve as a key comparison point for
- vSwitch bypass technologies performance and vSwitch
- performance. <figure align="right">
- <preamble>Benchmark VNF forwarding capability</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmarking with isolated resources alone, with other
- resources (both HW&amp;SW) disabled Example, vSw and VM are
- SUT</t>
-
- <t>Benchmarking with isolated resources alone, leaving some
- resources unused</t>
-
- <t>Benchmark with isolated resources and all resources
- occupied</t>
- </list></t>
-
- <t>Next Steps<list style="symbols">
- <t>Limited sharing</t>
-
- <t>Production scenarios</t>
-
- <t>Stressful scenarios</t>
- </list></t>
- </list></t>
- </section>
- </section>
-
- <section title="VSWITCHPERF Specification Summary">
- <t>The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the
- pre-existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2544"/> Benchmarking Methodology for Network
- Interconnect Devices</t>
-
- <t><xref target="RFC2889"/> Benchmarking Methodology for LAN
- Switching</t>
-
- <t><xref target="RFC6201"/> Device Reset Characterization</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t>Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test equipment
- developers. Fortunately, many members of the testing system community
- have engaged on the VSPERF project, including an open source test
- system.</t>
-
- <t>In addition to this, the LTD also re-uses the terminology defined
- by:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2285"/> Benchmarking Terminology for LAN
- Switching Devices</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t/>
-
- <t>Specifications to be included in future updates of the LTD
- include:<list style="symbols">
- <t><xref target="RFC3918"/> Methodology for IP Multicast
- Benchmarking</t>
-
- <t><xref target="RFC4737"/> Packet Reordering Metrics</t>
- </list></t>
-
- <t>As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.</t>
-
- <t>When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics. In
- this case, the project specifications have referenced metrics from the
- IETF IP Performance Metrics (IPPM) literature. This means that the <xref
- target="RFC2544"/> test of Latency is replaced by measurement of a
- metric derived from IPPM's <xref target="RFC2679"/>, where a set of
- statistical summaries will be provided (mean, max, min, etc.). Further
- metrics planned to be benchmarked include packet delay variation as
- defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
- availability, DUT capacity and packet loss in long term testing at
- Throughput level, where some low-level of background loss may be present
- and characterized.</t>
-
- <t>Tests have been (or will be) designed to collect the metrics
- below:</t>
-
- <t><list style="symbols">
- <t>Throughput Tests to measure the maximum forwarding rate (in
- frames per second or fps) and bit rate (in Mbps) for a constant load
- (as defined by RFC1242) without traffic loss.</t>
-
- <t>Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.</t>
-
- <t>Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.</t>
-
- <t>Scalability Tests to understand how the virtual switch performs
- as the number of flows, active ports, complexity of the forwarding
- logic&rsquo;s configuration&hellip; it has to deal with
- increases.</t>
-
- <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data through
- the switch.</t>
-
- <t>Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as the
- effect of this coupling on the performance of the DUT (example:
- delay of the initial packet of a flow).</t>
-
- <t>CPU and Memory Consumption Tests to understand the virtual
- switch&rsquo;s footprint on the system, usually conducted as
- auxiliary measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.</t>
- </list></t>
-
- <t>Future/planned test specs include:<list style="symbols">
- <t>Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.</t>
-
- <t>Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.</t>
-
- <t>Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements <xref target="IFA003"/> on characterization of
- acceleration technologies applied to vswitches.</t>
- </list>The flexibility of deployment of a virtual switch within a
- network means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:</t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to physical
- port</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure></t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|]]></artwork>
- </figure></t>
-
- <t>A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page <xref target="TestTopo"/>. </t>
- </section>
-
- <section title="3x3 Matrix Coverage">
- <t>This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>).
- Because the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all are
- occupied, also the matrix has grown to 3x4 to accommodate scale metrics
- when displaying the coverage of many metrics/benchmarks).</t>
-
- <t>The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.</t>
-
- <t>A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t>
-
- <section title="Speed of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressLearningRate</t>
-
- <t>PacketLatency.InitialPacketProcessingLatency</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Activation section">
- <t><list style="symbols">
- <t>CPDP.Coupling.Flow.Addition</t>
- </list></t>
- </section>
-
- <section title="Reliability of Activation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.SystemRecoveryTime</t>
-
- <t>Throughput.RFC2544.ResetTime</t>
- </list></t>
- </section>
-
- <section title="Scale of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressCachingCapacity</t>
- </list></t>
- </section>
-
- <section title="Speed of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.PacketLossRate</t>
-
- <t>CPU.RFC2544.0PacketLoss</t>
-
- <t>Throughput.RFC2544.PacketLossRateFrameModification</t>
-
- <t>Throughput.RFC2544.BackToBackFrames</t>
-
- <t>Throughput.RFC2889.MaxForwardingRate</t>
-
- <t>Throughput.RFC2889.ForwardPressure</t>
-
- <t>Throughput.RFC2889.BroadcastFrameForwarding</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.ErrorFramesFiltering</t>
-
- <t>Throughput.RFC2544.Profile</t>
- </list></t>
- </section>
-
- <section title="Reliability of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.Soak</t>
-
- <t>Throughput.RFC2889.SoakFrameModification</t>
-
- <t>PacketDelayVariation.RFC3393.Soak</t>
- </list></t>
- </section>
-
- <section title="Scalability of Operation">
- <t><list style="symbols">
- <t>Scalability.RFC2544.0PacketLoss</t>
-
- <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t>
- </list></t>
- </section>
-
- <section title="Summary">
- <t><figure>
- <artwork><![CDATA[|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|]]></artwork>
- </figure></t>
- </section>
- </section>
-
- <section title="Security Considerations">
- <t>Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.</t>
-
- <t>The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test traffic
- into a production network, or misroute traffic to the test management
- network.</t>
-
- <t>Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.</t>
-
- <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.</t>
- </section>
-
- <section anchor="IANA" title="IANA Considerations">
- <t>No IANA Action is requested at this time.</t>
- </section>
-
- <section title="Acknowledgements">
- <t>The authors acknowledge</t>
- </section>
- </middle>
-
- <back>
- <references title="Normative References">
- <?rfc ?>
-
- <?rfc include="reference.RFC.2119"?>
-
- <?rfc include="reference.RFC.2330"?>
-
- <?rfc include='reference.RFC.2544'?>
-
- <?rfc include="reference.RFC.2679"?>
-
- <?rfc include='reference.RFC.2680'?>
-
- <?rfc include='reference.RFC.3393'?>
-
- <?rfc include='reference.RFC.3432'?>
-
- <?rfc include='reference.RFC.2681'?>
-
- <?rfc include='reference.RFC.5905'?>
-
- <?rfc include='reference.RFC.4689'?>
-
- <?rfc include='reference.RFC.4737'?>
-
- <?rfc include='reference.RFC.5357'?>
-
- <?rfc include='reference.RFC.2889'?>
-
- <?rfc include='reference.RFC.3918'?>
-
- <?rfc include='reference.RFC.6201'?>
-
- <?rfc include='reference.RFC.2285'?>
-
- <reference anchor="NFV.PER001">
- <front>
- <title>Network Function Virtualization: Performance and Portability
- Best Practices</title>
-
- <author fullname="ETSI NFV" initials="" surname="">
- <organization/>
- </author>
-
- <date month="June" year="2014"/>
- </front>
-
- <seriesInfo name="Group Specification"
- value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
-
- <format type="PDF"/>
- </reference>
- </references>
-
- <references title="Informative References">
- <?rfc include='reference.RFC.1242'?>
-
- <?rfc include='reference.RFC.5481'?>
-
- <?rfc include='reference.RFC.6049'?>
-
- <?rfc include='reference.RFC.6248'?>
-
- <?rfc include='reference.RFC.6390'?>
-
- <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?>
-
- <reference anchor="TestTopo">
- <front>
- <title>Test Topologies
- https://wiki.opnfv.org/vsperf/test_methodology</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="LTDoverV">
- <front>
- <title>LTD Test Spec Overview
- https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review </title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="IFA003">
- <front>
- <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
- </references>
- </back>
-</rfc>
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt
deleted file mode 100755
index 81ae96c0..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.txt
+++ /dev/null
@@ -1,1232 +0,0 @@
-
-
-
-
-Network Working Group M. Tahhan
-Internet-Draft B. O'Mahony
-Intended status: Informational Intel
-Expires: April 16, 2016 A. Morton
- AT&T Labs
- October 14, 2015
-
-
- Benchmarking Virtual Switches in OPNFV
- draft-vsperf-bmwg-vswitch-opnfv-01
-
-Abstract
-
- This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing
- existing literature. The Benchmarking Methodology Working Group has
- traditionally conducted laboratory characterization of dedicated
- physical implementations of internetworking functions. Therefore,
- this memo begins to describe the additional considerations when
- virtual switches are implemented in general-purpose hardware. The
- expanded tests and benchmarks are also influenced by the OPNFV
- mission to support virtualization of the "telco" infrastructure.
-
-Requirements Language
-
- The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in RFC 2119 [RFC2119].
-
-Status of This Memo
-
- This Internet-Draft is submitted in full conformance with the
- provisions of BCP 78 and BCP 79.
-
- Internet-Drafts are working documents of the Internet Engineering
- Task Force (IETF). Note that other groups may also distribute
- working documents as Internet-Drafts. The list of current Internet-
- Drafts is at http://datatracker.ietf.org/drafts/current/.
-
- Internet-Drafts are draft documents valid for a maximum of six months
- and may be updated, replaced, or obsoleted by other documents at any
- time. It is inappropriate to use Internet-Drafts as reference
- material or to cite them other than as "work in progress."
-
- This Internet-Draft will expire on April 16, 2016.
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 1]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-Copyright Notice
-
- Copyright (c) 2015 IETF Trust and the persons identified as the
- document authors. All rights reserved.
-
- This document is subject to BCP 78 and the IETF Trust's Legal
- Provisions Relating to IETF Documents
- (http://trustee.ietf.org/license-info) in effect on the date of
- publication of this document. Please review these documents
- carefully, as they describe your rights and restrictions with respect
- to this document. Code Components extracted from this document must
- include Simplified BSD License text as described in Section 4.e of
- the Trust Legal Provisions and are provided without warranty as
- described in the Simplified BSD License.
-
-Table of Contents
-
- 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
- 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
- 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 4
- 3.1. Comparison with Physical Network Functions . . . . . . . 4
- 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 4
- 3.3. New Configuration Parameters . . . . . . . . . . . . . . 4
- 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 6
- 3.5. Benchmarks using Baselines with Resource Isolation . . . 7
- 4. VSWITCHPERF Specification Summary . . . . . . . . . . . . . . 8
- 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 16
- 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 17
- 5.2. Accuracy of Activation section . . . . . . . . . . . . . 17
- 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 17
- 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 17
- 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 17
- 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 17
- 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 17
- 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 18
- 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 18
- 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18
- 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
- 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19
- 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
- 9.1. Normative References . . . . . . . . . . . . . . . . . . 19
- 9.2. Informative References . . . . . . . . . . . . . . . . . 21
- Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 2]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-1. Introduction
-
- Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box
- Benchmarks of Throughput, Latency, Forwarding Rates and others have
- served our industry for many years. Now, Network Function
- Virtualization (NFV) has the goal to transform how internetwork
- functions are implemented, and therefore has garnered much attention.
-
- This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization,
- "VSWITCHPERF". This project intends to build on the current and
- completed work of the Benchmarking Methodology Working Group in IETF,
- by referencing existing literature. For example, currently the most
- often referenced RFC is [RFC2544] (which depends on [RFC1242]) and
- foundation of the benchmarking work in OPNFV is common and strong.
-
- See https://wiki.opnfv.org/
- characterize_vswitch_performance_for_telco_nfv_use_cases for more
- background, and the OPNFV website for general information:
- https://www.opnfv.org/
-
- The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on
- existing "telco" services as opposed to cloud-computing. There are
- many ways in which telco requirements have different emphasis on
- performance dimensions when compared to cloud computing: support for
- and transfer of isochronous media streams is one example.
-
- Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.
-
-2. Scope
-
- The primary purpose and scope of the memo is to inform BMWG of work-
- in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here
- is received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's
- chartered work to characterize the NFV Infrastructure (a virtual
- switch is an important aspect of that infrastructure).
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 3]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-3. Benchmarking Considerations
-
- This section highlights some specific considerations (from
- [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these
- areas, as they develop their specifications in the Level Test Design
- (LTD) document.
-
-3.1. Comparison with Physical Network Functions
-
- To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this
- memo re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.
-
- It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.
-
-3.2. Continued Emphasis on Black-Box Benchmarks
-
- External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation
- will be provided in parallel to assist the development of operations
- procedures when the technology is deployed.
-
-3.3. New Configuration Parameters
-
- A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.
-
- Hardware details including:
-
- o Platform details
-
- o Processor details
-
- o Memory information (type and size)
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 4]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o Number of enabled cores
-
- o Number of cores used for the test
-
- o Number of physical NICs, as well as their details (manufacturer,
- versions, type and the PCI slot they are plugged into)
-
- o NIC interrupt configuration
-
- o BIOS version, release date and any configurations that were
- modified
-
- o CPU microcode level
-
- o Memory DIMM configurations (quad rank performance may not be the
- same as dual rank) in size, freq and slot locations
-
- o PCI configuration parameters (payload size, early ack option...)
-
- o Power management at all levels (ACPI sleep states, processor
- package, OS...)
-
- Software details including:
-
- o OS parameters and behavior (text vs graphical no one typing at the
- console on one system)
-
- o OS version (for host and VNF)
-
- o Kernel version (for host and VNF)
-
- o GRUB boot parameters (for host and VNF)
-
- o Hypervisor details (Type and version)
-
- o Selected vSwitch, version number or commit id used
-
- o vSwitch launch command line if it has been parameterised
-
- o Memory allocation to the vSwitch
-
- o which NUMA node it is using, and how many memory channels
-
- o DPDK or any other SW dependency version number or commit id used
-
- o Memory allocation to a VM - if it's from Hugpages/elsewhere
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 5]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o VM storage type: snapshot/independent persistent/independent non-
- persistent
-
- o Number of VMs
-
- o Number of Virtual NICs (vNICs), versions, type and driver
-
- o Number of virtual CPUs and their core affinity on the host
-
- o Number vNIC interrupt configuration
-
- o Thread affinitization for the applications (including the vSwitch
- itself) on the host
-
- o Details of Resource isolation, such as CPUs designated for Host/
- Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.
-
- Test Traffic Information:
-
- o Traffic type - UDP, TCP, IMIX / Other
-
- o Packet Sizes
-
- o Deployment Scenario
-
-3.4. Flow classification
-
- Virtual switches group packets into flows by processing and matching
- particular packet or frame header information, or by matching packets
- based on the input ports. Thus a flow can be thought of a sequence
- of packets that have the same set of header field values or have
- arrived on the same port. Performance results can vary based on the
- parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the
- flow timeout time on a vSwitch before conducting any performance
- tests that do not measure the flow setup time. Normally the first
- packet of a particular stream will install the flow in the virtual
- switch which adds an additional latency, subsequent packets of the
- same flow are not subject to this latency if the flow is already
- installed on the vSwitch.
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 6]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-3.5. Benchmarks using Baselines with Resource Isolation
-
- This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.
-
- 1. Baselines:
-
- * Optional: Benchmark platform forwarding capability without a
- vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency).
-
- Benchmark platform forwarding capability
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
- * Benchmark VNF forwarding capability with direct connectivity
- (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves
- as a means of VNF validation and a means to obtain the base
- performance for the VNF in terms of its maximum forwarding
- rate and latency). The metrics gathered from this test will
- serve as a key comparison point for vSwitch bypass
- technologies performance and vSwitch performance.
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 7]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Benchmark VNF forwarding capability
-
- __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
- * Benchmarking with isolated resources alone, with other
- resources (both HW&SW) disabled Example, vSw and VM are SUT
-
- * Benchmarking with isolated resources alone, leaving some
- resources unused
-
- * Benchmark with isolated resources and all resources occupied
-
- 2. Next Steps
-
- * Limited sharing
-
- * Production scenarios
-
- * Stressful scenarios
-
-4. VSWITCHPERF Specification Summary
-
- The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the pre-
- existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:
-
- o [RFC2544] Benchmarking Methodology for Network Interconnect
- Devices
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 8]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o [RFC2889] Benchmarking Methodology for LAN Switching
-
- o [RFC6201] Device Reset Characterization
-
- o [RFC5481] Packet Delay Variation Applicability Statement
-
- Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test
- equipment developers. Fortunately, many members of the testing
- system community have engaged on the VSPERF project, including an
- open source test system.
-
- In addition to this, the LTD also re-uses the terminology defined by:
-
- o [RFC2285] Benchmarking Terminology for LAN Switching Devices
-
- o [RFC5481] Packet Delay Variation Applicability Statement
-
- Specifications to be included in future updates of the LTD include:
-
- o [RFC3918] Methodology for IP Multicast Benchmarking
-
- o [RFC4737] Packet Reordering Metrics
-
- As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.
-
- When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics.
- In this case, the project specifications have referenced metrics from
- the IETF IP Performance Metrics (IPPM) literature. This means that
- the [RFC2544] test of Latency is replaced by measurement of a metric
- derived from IPPM's [RFC2679], where a set of statistical summaries
- will be provided (mean, max, min, etc.). Further metrics planned to
- be benchmarked include packet delay variation as defined by [RFC5481]
- , reordering, burst behaviour, DUT availability, DUT capacity and
- packet loss in long term testing at Throughput level, where some low-
- level of background loss may be present and characterized.
-
- Tests have been (or will be) designed to collect the metrics below:
-
- o Throughput Tests to measure the maximum forwarding rate (in frames
- per second or fps) and bit rate (in Mbps) for a constant load (as
- defined by RFC1242) without traffic loss.
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 9]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.
-
- o Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.
-
- o Scalability Tests to understand how the virtual switch performs as
- the number of flows, active ports, complexity of the forwarding
- logic's configuration... it has to deal with increases.
-
- o Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data
- through the switch.
-
- o Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as
- the effect of this coupling on the performance of the DUT
- (example: delay of the initial packet of a flow).
-
- o CPU and Memory Consumption Tests to understand the virtual
- switch's footprint on the system, usually conducted as auxiliary
- measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.
-
- Future/planned test specs include:
-
- o Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.
-
- o Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.
-
- o Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements [IFA003] on characterization of acceleration
- technologies applied to vswitches.
-
- The flexibility of deployment of a virtual switch within a network
- means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 10]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to physical port
-
- __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 11]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF to virtual switch to physical
- port
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 12]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF to virtual switch to VNF to
- virtual switch to physical port
-
- __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 13]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Physical port to virtual switch to VNF
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 14]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- VNF to virtual switch to physical port
-
- __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 15]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- VNF to virtual switch to VNF
-
- __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|
-
- A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page [TestTopo].
-
-5. 3x3 Matrix Coverage
-
- This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]). Because
- the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all
- are occupied, also the matrix has grown to 3x4 to accommodate scale
- metrics when displaying the coverage of many metrics/benchmarks).
-
- The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.
-
- A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV].
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 16]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-5.1. Speed of Activation
-
- o Activation.RFC2889.AddressLearningRate
-
- o PacketLatency.InitialPacketProcessingLatency
-
-5.2. Accuracy of Activation section
-
- o CPDP.Coupling.Flow.Addition
-
-5.3. Reliability of Activation
-
- o Throughput.RFC2544.SystemRecoveryTime
-
- o Throughput.RFC2544.ResetTime
-
-5.4. Scale of Activation
-
- o Activation.RFC2889.AddressCachingCapacity
-
-5.5. Speed of Operation
-
- o Throughput.RFC2544.PacketLossRate
-
- o CPU.RFC2544.0PacketLoss
-
- o Throughput.RFC2544.PacketLossRateFrameModification
-
- o Throughput.RFC2544.BackToBackFrames
-
- o Throughput.RFC2889.MaxForwardingRate
-
- o Throughput.RFC2889.ForwardPressure
-
- o Throughput.RFC2889.BroadcastFrameForwarding
-
-5.6. Accuracy of Operation
-
- o Throughput.RFC2889.ErrorFramesFiltering
-
- o Throughput.RFC2544.Profile
-
-5.7. Reliability of Operation
-
- o Throughput.RFC2889.Soak
-
- o Throughput.RFC2889.SoakFrameModification
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 17]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- o PacketDelayVariation.RFC3393.Soak
-
-5.8. Scalability of Operation
-
- o Scalability.RFC2544.0PacketLoss
-
- o MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-
-5.9. Summary
-
-|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|
-
-6. Security Considerations
-
- Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.
-
- The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test
- traffic into a production network, or misroute traffic to the test
- management network.
-
- Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.
-
- Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 18]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-7. IANA Considerations
-
- No IANA Action is requested at this time.
-
-8. Acknowledgements
-
- The authors acknowledge
-
-9. References
-
-9.1. Normative References
-
- [NFV.PER001]
- "Network Function Virtualization: Performance and
- Portability Best Practices", Group Specification ETSI GS
- NFV-PER 001 V1.1.1 (2014-06), June 2014.
-
- [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
- Requirement Levels", BCP 14, RFC 2119,
- DOI 10.17487/RFC2119, March 1997,
- <http://www.rfc-editor.org/info/rfc2119>.
-
- [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
- Switching Devices", RFC 2285, DOI 10.17487/RFC2285,
- February 1998, <http://www.rfc-editor.org/info/rfc2285>.
-
- [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
- "Framework for IP Performance Metrics", RFC 2330,
- DOI 10.17487/RFC2330, May 1998,
- <http://www.rfc-editor.org/info/rfc2330>.
-
- [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
- Network Interconnect Devices", RFC 2544,
- DOI 10.17487/RFC2544, March 1999,
- <http://www.rfc-editor.org/info/rfc2544>.
-
- [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
- Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679,
- September 1999, <http://www.rfc-editor.org/info/rfc2679>.
-
- [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
- Packet Loss Metric for IPPM", RFC 2680,
- DOI 10.17487/RFC2680, September 1999,
- <http://www.rfc-editor.org/info/rfc2680>.
-
- [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
- Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681,
- September 1999, <http://www.rfc-editor.org/info/rfc2681>.
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 19]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology
- for LAN Switching Devices", RFC 2889,
- DOI 10.17487/RFC2889, August 2000,
- <http://www.rfc-editor.org/info/rfc2889>.
-
- [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
- Metric for IP Performance Metrics (IPPM)", RFC 3393,
- DOI 10.17487/RFC3393, November 2002,
- <http://www.rfc-editor.org/info/rfc3393>.
-
- [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
- performance measurement with periodic streams", RFC 3432,
- DOI 10.17487/RFC3432, November 2002,
- <http://www.rfc-editor.org/info/rfc3432>.
-
- [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast
- Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October
- 2004, <http://www.rfc-editor.org/info/rfc3918>.
-
- [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
- "Terminology for Benchmarking Network-layer Traffic
- Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
- October 2006, <http://www.rfc-editor.org/info/rfc4689>.
-
- [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
- S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
- DOI 10.17487/RFC4737, November 2006,
- <http://www.rfc-editor.org/info/rfc4737>.
-
- [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
- Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
- RFC 5357, DOI 10.17487/RFC5357, October 2008,
- <http://www.rfc-editor.org/info/rfc5357>.
-
- [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
- "Network Time Protocol Version 4: Protocol and Algorithms
- Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
- <http://www.rfc-editor.org/info/rfc5905>.
-
- [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera,
- "Device Reset Characterization", RFC 6201,
- DOI 10.17487/RFC6201, March 2011,
- <http://www.rfc-editor.org/info/rfc6201>.
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 20]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
-9.2. Informative References
-
- [I-D.ietf-bmwg-virtual-net]
- Morton, A., "Considerations for Benchmarking Virtual
- Network Functions and Their Infrastructure", draft-ietf-
- bmwg-virtual-net-01 (work in progress), September 2015.
-
- [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/
- IFA003_Acceleration_-_vSwitch_Spec/".
-
- [LTDoverV]
- "LTD Test Spec Overview https://wiki.opnfv.org/wiki/
- vswitchperf_test_spec_review".
-
- [RFC1242] Bradner, S., "Benchmarking Terminology for Network
- Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
- July 1991, <http://www.rfc-editor.org/info/rfc1242>.
-
- [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
- Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
- March 2009, <http://www.rfc-editor.org/info/rfc5481>.
-
- [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
- Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
- <http://www.rfc-editor.org/info/rfc6049>.
-
- [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics
- (IPPM) Registry of Metrics Are Obsolete", RFC 6248,
- DOI 10.17487/RFC6248, April 2011,
- <http://www.rfc-editor.org/info/rfc6248>.
-
- [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New
- Performance Metric Development", BCP 170, RFC 6390,
- DOI 10.17487/RFC6390, October 2011,
- <http://www.rfc-editor.org/info/rfc6390>.
-
- [TestTopo]
- "Test Topologies https://wiki.opnfv.org/vsperf/
- test_methodology".
-
-Authors' Addresses
-
- Maryam Tahhan
- Intel
-
- Email: maryam.tahhan@intel.com
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 21]
-
-Internet-Draft Benchmarking vSwitches October 2015
-
-
- Billy O'Mahony
- Intel
-
- Email: billy.o.mahony@intel.com
-
-
- Al Morton
- AT&T Labs
- 200 Laurel Avenue South
- Middletown,, NJ 07748
- USA
-
- Phone: +1 732 420 1571
- Fax: +1 732 368 1192
- Email: acmorton@att.com
- URI: http://home.comcast.net/~acmacm/
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Tahhan, et al. Expires April 16, 2016 [Page 22]
diff --git a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml b/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml
deleted file mode 100755
index b5f7f833..00000000
--- a/docs/test_spec/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-01.xml
+++ /dev/null
@@ -1,964 +0,0 @@
-<?xml version="1.0" encoding="US-ASCII"?>
-<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
-<?rfc toc="yes"?>
-<?rfc tocompact="yes"?>
-<?rfc tocdepth="3"?>
-<?rfc tocindent="yes"?>
-<?rfc symrefs="yes"?>
-<?rfc sortrefs="yes"?>
-<?rfc comments="yes"?>
-<?rfc inline="yes"?>
-<?rfc compact="yes"?>
-<?rfc subcompact="no"?>
-<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-01"
- ipr="trust200902">
- <front>
- <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
- OPNFV</title>
-
- <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>maryam.tahhan@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony">
- <organization>Intel</organization>
-
- <address>
- <postal>
- <street/>
-
- <city/>
-
- <region/>
-
- <code/>
-
- <country/>
- </postal>
-
- <phone/>
-
- <facsimile/>
-
- <email>billy.o.mahony@intel.com</email>
-
- <uri/>
- </address>
- </author>
-
- <author fullname="Al Morton" initials="A." surname="Morton">
- <organization>AT&amp;T Labs</organization>
-
- <address>
- <postal>
- <street>200 Laurel Avenue South</street>
-
- <city>Middletown,</city>
-
- <region>NJ</region>
-
- <code>07748</code>
-
- <country>USA</country>
- </postal>
-
- <phone>+1 732 420 1571</phone>
-
- <facsimile>+1 732 368 1192</facsimile>
-
- <email>acmorton@att.com</email>
-
- <uri>http://home.comcast.net/~acmacm/</uri>
- </address>
- </author>
-
- <date day="14" month="October" year="2015"/>
-
- <abstract>
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance "VSWITCHPERF". This project
- intends to build on the current and completed work of the Benchmarking
- Methodology Working Group in IETF, by referencing existing literature.
- The Benchmarking Methodology Working Group has traditionally conducted
- laboratory characterization of dedicated physical implementations of
- internetworking functions. Therefore, this memo begins to describe the
- additional considerations when virtual switches are implemented in
- general-purpose hardware. The expanded tests and benchmarks are also
- influenced by the OPNFV mission to support virtualization of the "telco"
- infrastructure.</t>
- </abstract>
-
- <note title="Requirements Language">
- <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
- "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
- document are to be interpreted as described in <xref
- target="RFC2119">RFC 2119</xref>.</t>
-
- <t/>
- </note>
- </front>
-
- <middle>
- <section title="Introduction">
- <t>Benchmarking Methodology Working Group (BMWG) has traditionally
- conducted laboratory characterization of dedicated physical
- implementations of internetworking functions. The Black-box Benchmarks
- of Throughput, Latency, Forwarding Rates and others have served our
- industry for many years. Now, Network Function Virtualization (NFV) has
- the goal to transform how internetwork functions are implemented, and
- therefore has garnered much attention.</t>
-
- <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
- project on virtual switch performance characterization, "VSWITCHPERF".
- This project intends to build on the current and completed work of the
- Benchmarking Methodology Working Group in IETF, by referencing existing
- literature. For example, currently the most often referenced RFC is
- <xref target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and
- foundation of the benchmarking work in OPNFV is common and strong.</t>
-
- <t>See
- https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
- for more background, and the OPNFV website for general information:
- https://www.opnfv.org/</t>
-
- <t>The authors note that OPNFV distinguishes itself from other open
- source compute and networking projects through its emphasis on existing
- "telco" services as opposed to cloud-computing. There are many ways in
- which telco requirements have different emphasis on performance
- dimensions when compared to cloud computing: support for and transfer of
- isochronous media streams is one example.</t>
-
- <t>Note also that the move to NFV Infrastructure has resulted in many
- new benchmarking initiatives across the industry, and the authors are
- currently doing their best to maintain alignment with many other
- projects, and this Internet Draft is evidence of the efforts.</t>
- </section>
-
- <section title="Scope">
- <t>The primary purpose and scope of the memo is to inform BMWG of
- work-in-progress that builds on the body of extensive literature and
- experience. Additionally, once the initial information conveyed here is
- received, this memo may be expanded to include more detail and
- commentary from both BMWG and OPNFV communities, under BMWG's chartered
- work to characterize the NFV Infrastructure (a virtual switch is an
- important aspect of that infrastructure).</t>
- </section>
-
- <section title="Benchmarking Considerations">
- <t>This section highlights some specific considerations (from <xref
- target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
- switches. The OPNFV project is sharing its present view on these areas,
- as they develop their specifications in the Level Test Design (LTD)
- document.</t>
-
- <section title="Comparison with Physical Network Functions">
- <t>To compare the performance of virtual designs and implementations
- with their physical counterparts, identical benchmarks are needed.
- BMWG has developed specifications for many network functions this memo
- re-uses existing benchmarks through references, and expands them
- during development of new methods. A key configuration aspect is the
- number of parallel cores required to achieve comparable performance
- with a given physical device, or whether some limit of scale was
- reached before the cores could achieve the comparable level.</t>
-
- <t>It's unlikely that the virtual switch will be the only application
- running on the SUT, so CPU utilization, Cache utilization, and Memory
- footprint should also be recorded for the virtual implementations of
- internetworking functions.</t>
- </section>
-
- <section title="Continued Emphasis on Black-Box Benchmarks">
- <t>External observations remain essential as the basis for Benchmarks.
- Internal observations with fixed specification and interpretation will
- be provided in parallel to assist the development of operations
- procedures when the technology is deployed.</t>
- </section>
-
- <section title="New Configuration Parameters">
- <t>A key consideration when conducting any sort of benchmark is trying
- to ensure the consistency and repeatability of test results. When
- benchmarking the performance of a vSwitch there are many factors that
- can affect the consistency of results, one key factor is matching the
- various hardware and software details of the SUT. This section lists
- some of the many new parameters which this project believes are
- critical to report in order to achieve repeatability.</t>
-
- <t>Hardware details including:</t>
-
- <t><list style="symbols">
- <t>Platform details</t>
-
- <t>Processor details</t>
-
- <t>Memory information (type and size)</t>
-
- <t>Number of enabled cores</t>
-
- <t>Number of cores used for the test</t>
-
- <t>Number of physical NICs, as well as their details
- (manufacturer, versions, type and the PCI slot they are plugged
- into)</t>
-
- <t>NIC interrupt configuration</t>
-
- <t>BIOS version, release date and any configurations that were
- modified</t>
-
- <t>CPU microcode level</t>
-
- <t>Memory DIMM configurations (quad rank performance may not be
- the same as dual rank) in size, freq and slot locations</t>
-
- <t>PCI configuration parameters (payload size, early ack
- option...)</t>
-
- <t>Power management at all levels (ACPI sleep states, processor
- package, OS...)</t>
- </list>Software details including:</t>
-
- <t><list style="symbols">
- <t>OS parameters and behavior (text vs graphical no one typing at
- the console on one system)</t>
-
- <t>OS version (for host and VNF)</t>
-
- <t>Kernel version (for host and VNF)</t>
-
- <t>GRUB boot parameters (for host and VNF)</t>
-
- <t>Hypervisor details (Type and version)</t>
-
- <t>Selected vSwitch, version number or commit id used</t>
-
- <t>vSwitch launch command line if it has been parameterised</t>
-
- <t>Memory allocation to the vSwitch</t>
-
- <t>which NUMA node it is using, and how many memory channels</t>
-
- <t>DPDK or any other SW dependency version number or commit id
- used</t>
-
- <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t>
-
- <t>VM storage type: snapshot/independent persistent/independent
- non-persistent</t>
-
- <t>Number of VMs</t>
-
- <t>Number of Virtual NICs (vNICs), versions, type and driver</t>
-
- <t>Number of virtual CPUs and their core affinity on the host</t>
-
- <t>Number vNIC interrupt configuration</t>
-
- <t>Thread affinitization for the applications (including the
- vSwitch itself) on the host</t>
-
- <t>Details of Resource isolation, such as CPUs designated for
- Host/Kernel (isolcpu) and CPUs designated for specific processes
- (taskset). - Test duration. - Number of flows.</t>
- </list></t>
-
- <t>Test Traffic Information:<list style="symbols">
- <t>Traffic type - UDP, TCP, IMIX / Other</t>
-
- <t>Packet Sizes</t>
-
- <t>Deployment Scenario</t>
- </list></t>
-
- <t/>
- </section>
-
- <section title="Flow classification">
- <t>Virtual switches group packets into flows by processing and
- matching particular packet or frame header information, or by matching
- packets based on the input ports. Thus a flow can be thought of a
- sequence of packets that have the same set of header field values or
- have arrived on the same port. Performance results can vary based on
- the parameters the vSwitch uses to match for a flow. The recommended
- flow classification parameters for any vSwitch performance tests are:
- the input port, the source IP address, the destination IP address and
- the Ethernet protocol type field. It is essential to increase the flow
- timeout time on a vSwitch before conducting any performance tests that
- do not measure the flow setup time. Normally the first packet of a
- particular stream will install the flow in the virtual switch which
- adds an additional latency, subsequent packets of the same flow are
- not subject to this latency if the flow is already installed on the
- vSwitch.</t>
- </section>
-
- <section title="Benchmarks using Baselines with Resource Isolation">
- <t>This outline describes measurement of baseline with isolated
- resources at a high level, which is the intended approach at this
- time.</t>
-
- <t><list style="numbers">
- <t>Baselines: <list style="symbols">
- <t>Optional: Benchmark platform forwarding capability without
- a vswitch or VNF for at least 72 hours (serves as a means of
- platform validation and a means to obtain the base performance
- for the platform in terms of its maximum forwarding rate and
- latency). <figure>
- <preamble>Benchmark platform forwarding
- capability</preamble>
-
- <artwork align="right"><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | Simple Forwarding App | | Host
- | | | | |
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmark VNF forwarding capability with direct
- connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72
- hours (serves as a means of VNF validation and a means to
- obtain the base performance for the VNF in terms of its
- maximum forwarding rate and latency). The metrics gathered
- from this test will serve as a key comparison point for
- vSwitch bypass technologies performance and vSwitch
- performance. <figure align="right">
- <preamble>Benchmark VNF forwarding capability</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +------------------------------------------+ | |
- | | | | |
- | | VNF | | |
- | | | | |
- | +------------------------------------------+ | |
- | | Passthrough/SR-IOV | | Host
- | +------------------------------------------+ | |
- | | NIC | | |
- +---+------------------------------------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
-
- <postamble/>
- </figure></t>
-
- <t>Benchmarking with isolated resources alone, with other
- resources (both HW&amp;SW) disabled Example, vSw and VM are
- SUT</t>
-
- <t>Benchmarking with isolated resources alone, leaving some
- resources unused</t>
-
- <t>Benchmark with isolated resources and all resources
- occupied</t>
- </list></t>
-
- <t>Next Steps<list style="symbols">
- <t>Limited sharing</t>
-
- <t>Production scenarios</t>
-
- <t>Stressful scenarios</t>
- </list></t>
- </list></t>
- </section>
- </section>
-
- <section title="VSWITCHPERF Specification Summary">
- <t>The overall specification in preparation is referred to as a Level
- Test Design (LTD) document, which will contain a suite of performance
- tests. The base performance tests in the LTD are based on the
- pre-existing specifications developed by BMWG to test the performance of
- physical switches. These specifications include:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2544"/> Benchmarking Methodology for Network
- Interconnect Devices</t>
-
- <t><xref target="RFC2889"/> Benchmarking Methodology for LAN
- Switching</t>
-
- <t><xref target="RFC6201"/> Device Reset Characterization</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t>Some of the above/newer RFCs are being applied in benchmarking for
- the first time, and represent a development challenge for test equipment
- developers. Fortunately, many members of the testing system community
- have engaged on the VSPERF project, including an open source test
- system.</t>
-
- <t>In addition to this, the LTD also re-uses the terminology defined
- by:</t>
-
- <t><list style="symbols">
- <t><xref target="RFC2285"/> Benchmarking Terminology for LAN
- Switching Devices</t>
-
- <t><xref target="RFC5481"/> Packet Delay Variation Applicability
- Statement</t>
- </list></t>
-
- <t/>
-
- <t>Specifications to be included in future updates of the LTD
- include:<list style="symbols">
- <t><xref target="RFC3918"/> Methodology for IP Multicast
- Benchmarking</t>
-
- <t><xref target="RFC4737"/> Packet Reordering Metrics</t>
- </list></t>
-
- <t>As one might expect, the most fundamental internetworking
- characteristics of Throughput and Latency remain important when the
- switch is virtualized, and these benchmarks figure prominently in the
- specification.</t>
-
- <t>When considering characteristics important to "telco" network
- functions, we must begin to consider additional performance metrics. In
- this case, the project specifications have referenced metrics from the
- IETF IP Performance Metrics (IPPM) literature. This means that the <xref
- target="RFC2544"/> test of Latency is replaced by measurement of a
- metric derived from IPPM's <xref target="RFC2679"/>, where a set of
- statistical summaries will be provided (mean, max, min, etc.). Further
- metrics planned to be benchmarked include packet delay variation as
- defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
- availability, DUT capacity and packet loss in long term testing at
- Throughput level, where some low-level of background loss may be present
- and characterized.</t>
-
- <t>Tests have been (or will be) designed to collect the metrics
- below:</t>
-
- <t><list style="symbols">
- <t>Throughput Tests to measure the maximum forwarding rate (in
- frames per second or fps) and bit rate (in Mbps) for a constant load
- (as defined by RFC1242) without traffic loss.</t>
-
- <t>Packet and Frame Delay Distribution Tests to measure average, min
- and max packet and frame delay for constant loads.</t>
-
- <t>Packet Delay Tests to understand latency distribution for
- different packet sizes and over an extended test run to uncover
- outliers.</t>
-
- <t>Scalability Tests to understand how the virtual switch performs
- as the number of flows, active ports, complexity of the forwarding
- logic&rsquo;s configuration&hellip; it has to deal with
- increases.</t>
-
- <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
- performance, i.e. how fast systems can send and receive data through
- the switch.</t>
-
- <t>Control Path and Datapath Coupling Tests, to understand how
- closely coupled the datapath and the control path are as well as the
- effect of this coupling on the performance of the DUT (example:
- delay of the initial packet of a flow).</t>
-
- <t>CPU and Memory Consumption Tests to understand the virtual
- switch&rsquo;s footprint on the system, usually conducted as
- auxiliary measurements with benchmarks above. They include: CPU
- utilization, Cache utilization and Memory footprint.</t>
- </list></t>
-
- <t>Future/planned test specs include:<list style="symbols">
- <t>Request/Response Performance Tests (TCP, UDP) which measure the
- transaction rate through the switch.</t>
-
- <t>Noisy Neighbour Tests, to understand the effects of resource
- sharing on the performance of a virtual switch.</t>
-
- <t>Tests derived from examination of ETSI NFV Draft GS IFA003
- requirements <xref target="IFA003"/> on characterization of
- acceleration technologies applied to vswitches.</t>
- </list>The flexibility of deployment of a virtual switch within a
- network means that the BMWG IETF existing literature needs to be used to
- characterize the performance of a switch in various deployment
- scenarios. The deployment scenarios under consideration include:</t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to physical
- port</preamble>
-
- <artwork><![CDATA[ __
- +--------------------------------------------------+ |
- | +--------------------+ | |
- | | | | |
- | | v | | Host
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure></t>
-
- <t><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ : | |
- | | | | | Guest
- | : v | |
- | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ __|
- ^ :
- | |
- : v __
- +---+---------------+----------+---------------+---+ |
- | | logical port 0| | logical port 1| | |
- | +---------------+ +---------------+ | |
- | ^ : | |
- | | | | | Host
- | : v | |
- | +--------------+ +--------------+ | |
- | | phy port | vSwitch | phy port | | |
- +---+--------------+------------+--------------+---+ __|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF to virtual switch
- to VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | ^ | | | ^ | | |
- | | v | | | v | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 1 | | | | 0 1 | | |
- +---+---------------+--+ +---+---------------+--+__|
- ^ : ^ :
- | | | |
- : v : v _
- +---+---------------+---------+---------------+--+ |
- | | 0 1 | | 3 4 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | ^ | ^ | | | Host
- | | |-----------------| v | |
- | +--------------+ +--------------+ | |
- | | phy ports | vSwitch | phy ports | | |
- +---+--------------+----------+--------------+---+_|
- ^ :
- | |
- : v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>Physical port to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | ^ | |
- | | | | Guest
- | : | |
- | +---------------+ | |
- | | logical port 0| | |
- +---+---------------+-------------------------------+ __|
- ^
- |
- : __
- +---+---------------+------------------------------+ |
- | | logical port 0| | |
- | +---------------+ | |
- | ^ | |
- | | | | Host
- | : | |
- | +--------------+ | |
- | | phy port | vSwitch | |
- +---+--------------+------------ -------------- ---+ __|
- ^
- |
- :
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to physical port</preamble>
-
- <artwork><![CDATA[ __
- +---------------------------------------------------+ |
- | | |
- | +-------------------------------------------+ | |
- | | Application | | |
- | +-------------------------------------------+ | |
- | : | |
- | | | | Guest
- | v | |
- | +---------------+ | |
- | | logical port | | |
- +-------------------------------+---------------+---+ __|
- :
- |
- v __
- +------------------------------+---------------+---+ |
- | | logical port | | |
- | +---------------+ | |
- | : | |
- | | | | Host
- | v | |
- | +--------------+ | |
- | vSwitch | phy port | | |
- +-------------------------------+--------------+---+ __|
- :
- |
- v
- +--------------------------------------------------+
- | |
- | traffic generator |
- | |
- +--------------------------------------------------+]]></artwork>
- </figure><figure>
- <preamble>VNF to virtual switch to VNF</preamble>
-
- <artwork><![CDATA[ __
- +----------------------+ +----------------------+ |
- | Guest 1 | | Guest 2 | |
- | +---------------+ | | +---------------+ | |
- | | Application | | | | Application | | |
- | +---------------+ | | +---------------+ | |
- | | | | ^ | |
- | v | | | | | Guests
- | +---------------+ | | +---------------+ | |
- | | logical ports | | | | logical ports | | |
- | | 0 | | | | 0 | | |
- +---+---------------+--+ +---+---------------+--+__|
- : ^
- | |
- v : _
- +---+---------------+---------+---------------+--+ |
- | | 1 | | 1 | | |
- | | logical ports | | logical ports | | |
- | +---------------+ +---------------+ | |
- | | ^ | | Host
- | L-----------------+ | |
- | | |
- | vSwitch | |
- +------------------------------------------------+_|]]></artwork>
- </figure></t>
-
- <t>A set of Deployment Scenario figures is available on the VSPERF Test
- Methodology Wiki page <xref target="TestTopo"/>. </t>
- </section>
-
- <section title="3x3 Matrix Coverage">
- <t>This section organizes the many existing test specifications into the
- "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>).
- Because the LTD specification ID names are quite long, this section is
- organized into lists for each occupied cell of the matrix (not all are
- occupied, also the matrix has grown to 3x4 to accommodate scale metrics
- when displaying the coverage of many metrics/benchmarks).</t>
-
- <t>The tests listed below assess the activation of paths in the data
- plane, rather than the control plane.</t>
-
- <t>A complete list of tests with short summaries is available on the
- VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t>
-
- <section title="Speed of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressLearningRate</t>
-
- <t>PacketLatency.InitialPacketProcessingLatency</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Activation section">
- <t><list style="symbols">
- <t>CPDP.Coupling.Flow.Addition</t>
- </list></t>
- </section>
-
- <section title="Reliability of Activation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.SystemRecoveryTime</t>
-
- <t>Throughput.RFC2544.ResetTime</t>
- </list></t>
- </section>
-
- <section title="Scale of Activation">
- <t><list style="symbols">
- <t>Activation.RFC2889.AddressCachingCapacity</t>
- </list></t>
- </section>
-
- <section title="Speed of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2544.PacketLossRate</t>
-
- <t>CPU.RFC2544.0PacketLoss</t>
-
- <t>Throughput.RFC2544.PacketLossRateFrameModification</t>
-
- <t>Throughput.RFC2544.BackToBackFrames</t>
-
- <t>Throughput.RFC2889.MaxForwardingRate</t>
-
- <t>Throughput.RFC2889.ForwardPressure</t>
-
- <t>Throughput.RFC2889.BroadcastFrameForwarding</t>
- </list></t>
- </section>
-
- <section title="Accuracy of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.ErrorFramesFiltering</t>
-
- <t>Throughput.RFC2544.Profile</t>
- </list></t>
- </section>
-
- <section title="Reliability of Operation">
- <t><list style="symbols">
- <t>Throughput.RFC2889.Soak</t>
-
- <t>Throughput.RFC2889.SoakFrameModification</t>
-
- <t>PacketDelayVariation.RFC3393.Soak</t>
- </list></t>
- </section>
-
- <section title="Scalability of Operation">
- <t><list style="symbols">
- <t>Scalability.RFC2544.0PacketLoss</t>
-
- <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t>
- </list></t>
- </section>
-
- <section title="Summary">
- <t><figure>
- <artwork><![CDATA[|------------------------------------------------------------------------|
-| | | | | |
-| | SPEED | ACCURACY | RELIABILITY | SCALE |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Activation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| Operation | X | X | X | X |
-| | | | | |
-|------------------------------------------------------------------------|
-| | | | | |
-| De-activation | | | | |
-| | | | | |
-|------------------------------------------------------------------------|]]></artwork>
- </figure></t>
- </section>
- </section>
-
- <section title="Security Considerations">
- <t>Benchmarking activities as described in this memo are limited to
- technology characterization of a Device Under Test/System Under Test
- (DUT/SUT) using controlled stimuli in a laboratory environment, with
- dedicated address space and the constraints specified in the sections
- above.</t>
-
- <t>The benchmarking network topology will be an independent test setup
- and MUST NOT be connected to devices that may forward the test traffic
- into a production network, or misroute traffic to the test management
- network.</t>
-
- <t>Further, benchmarking is performed on a "black-box" basis, relying
- solely on measurements observable external to the DUT/SUT.</t>
-
- <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
- benchmarking purposes. Any implications for network security arising
- from the DUT/SUT SHOULD be identical in the lab and in production
- networks.</t>
- </section>
-
- <section anchor="IANA" title="IANA Considerations">
- <t>No IANA Action is requested at this time.</t>
- </section>
-
- <section title="Acknowledgements">
- <t>The authors acknowledge</t>
- </section>
- </middle>
-
- <back>
- <references title="Normative References">
- <?rfc ?>
-
- <?rfc include="reference.RFC.2119"?>
-
- <?rfc include="reference.RFC.2330"?>
-
- <?rfc include='reference.RFC.2544'?>
-
- <?rfc include="reference.RFC.2679"?>
-
- <?rfc include='reference.RFC.2680'?>
-
- <?rfc include='reference.RFC.3393'?>
-
- <?rfc include='reference.RFC.3432'?>
-
- <?rfc include='reference.RFC.2681'?>
-
- <?rfc include='reference.RFC.5905'?>
-
- <?rfc include='reference.RFC.4689'?>
-
- <?rfc include='reference.RFC.4737'?>
-
- <?rfc include='reference.RFC.5357'?>
-
- <?rfc include='reference.RFC.2889'?>
-
- <?rfc include='reference.RFC.3918'?>
-
- <?rfc include='reference.RFC.6201'?>
-
- <?rfc include='reference.RFC.2285'?>
-
- <reference anchor="NFV.PER001">
- <front>
- <title>Network Function Virtualization: Performance and Portability
- Best Practices</title>
-
- <author fullname="ETSI NFV" initials="" surname="">
- <organization/>
- </author>
-
- <date month="June" year="2014"/>
- </front>
-
- <seriesInfo name="Group Specification"
- value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
-
- <format type="PDF"/>
- </reference>
- </references>
-
- <references title="Informative References">
- <?rfc include='reference.RFC.1242'?>
-
- <?rfc include='reference.RFC.5481'?>
-
- <?rfc include='reference.RFC.6049'?>
-
- <?rfc include='reference.RFC.6248'?>
-
- <?rfc include='reference.RFC.6390'?>
-
- <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?>
-
- <reference anchor="TestTopo">
- <front>
- <title>Test Topologies
- https://wiki.opnfv.org/vsperf/test_methodology</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="LTDoverV">
- <front>
- <title>LTD Test Spec Overview
- https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review </title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
-
- <reference anchor="IFA003">
- <front>
- <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title>
-
- <author>
- <organization/>
- </author>
-
- <date/>
- </front>
- </reference>
- </references>
- </back>
-</rfc>