Merge "New IETF Draft WG version"

2 files changed, 2248 insertions, 0 deletions

diff --git a/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.txt b/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.txt
new file mode 100644
index 00000000..7375b618
--- /dev/null
+++ b/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.txt
@@ -0,0 +1,1232 @@
+
+
+
+
+Network Working Group                                          M. Tahhan
+Internet-Draft                                               B. O'Mahony
+Intended status: Informational                                     Intel
+Expires: January 9, 2017                                       A. Morton
+                                                               AT&T Labs
+                                                            July 8, 2016
+
+
+                 Benchmarking Virtual Switches in OPNFV
+                    draft-ietf-bmwg-vswitch-opnfv-00
+
+Abstract
+
+   This memo describes the progress of the Open Platform for NFV (OPNFV)
+   project on virtual switch performance "VSWITCHPERF".  This project
+   intends to build on the current and completed work of the
+   Benchmarking Methodology Working Group in IETF, by referencing
+   existing literature.  The Benchmarking Methodology Working Group has
+   traditionally conducted laboratory characterization of dedicated
+   physical implementations of internetworking functions.  Therefore,
+   this memo begins to describe the additional considerations when
+   virtual switches are implemented in general-purpose hardware.  The
+   expanded tests and benchmarks are also influenced by the OPNFV
+   mission to support virtualization of the "telco" infrastructure.
+
+Requirements Language
+
+   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+   document are to be interpreted as described in RFC 2119 [RFC2119].
+
+Status of This Memo
+
+   This Internet-Draft is submitted in full conformance with the
+   provisions of BCP 78 and BCP 79.
+
+   Internet-Drafts are working documents of the Internet Engineering
+   Task Force (IETF).  Note that other groups may also distribute
+   working documents as Internet-Drafts.  The list of current Internet-
+   Drafts is at http://datatracker.ietf.org/drafts/current/.
+
+   Internet-Drafts are draft documents valid for a maximum of six months
+   and may be updated, replaced, or obsoleted by other documents at any
+   time.  It is inappropriate to use Internet-Drafts as reference
+   material or to cite them other than as "work in progress."
+
+   This Internet-Draft will expire on January 9, 2017.
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 1]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+Copyright Notice
+
+   Copyright (c) 2016 IETF Trust and the persons identified as the
+   document authors.  All rights reserved.
+
+   This document is subject to BCP 78 and the IETF Trust's Legal
+   Provisions Relating to IETF Documents
+   (http://trustee.ietf.org/license-info) in effect on the date of
+   publication of this document.  Please review these documents
+   carefully, as they describe your rights and restrictions with respect
+   to this document.  Code Components extracted from this document must
+   include Simplified BSD License text as described in Section 4.e of
+   the Trust Legal Provisions and are provided without warranty as
+   described in the Simplified BSD License.
+
+Table of Contents
+
+   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
+   2.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
+   3.  Benchmarking Considerations . . . . . . . . . . . . . . . . .   4
+     3.1.  Comparison with Physical Network Functions  . . . . . . .   4
+     3.2.  Continued Emphasis on Black-Box Benchmarks  . . . . . . .   4
+     3.3.  New Configuration Parameters  . . . . . . . . . . . . . .   4
+     3.4.  Flow classification . . . . . . . . . . . . . . . . . . .   6
+     3.5.  Benchmarks using Baselines with Resource Isolation  . . .   7
+   4.  VSWITCHPERF Specification Summary . . . . . . . . . . . . . .   8
+   5.  3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . .  16
+     5.1.  Speed of Activation . . . . . . . . . . . . . . . . . . .  17
+     5.2.  Accuracy of Activation section  . . . . . . . . . . . . .  17
+     5.3.  Reliability of Activation . . . . . . . . . . . . . . . .  17
+     5.4.  Scale of Activation . . . . . . . . . . . . . . . . . . .  17
+     5.5.  Speed of Operation  . . . . . . . . . . . . . . . . . . .  17
+     5.6.  Accuracy of Operation . . . . . . . . . . . . . . . . . .  17
+     5.7.  Reliability of Operation  . . . . . . . . . . . . . . . .  17
+     5.8.  Scalability of Operation  . . . . . . . . . . . . . . . .  18
+     5.9.  Summary . . . . . . . . . . . . . . . . . . . . . . . . .  18
+   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  18
+   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  19
+   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  19
+   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  19
+     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  19
+     9.2.  Informative References  . . . . . . . . . . . . . . . . .  21
+   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  22
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 2]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+1.  Introduction
+
+   Benchmarking Methodology Working Group (BMWG) has traditionally
+   conducted laboratory characterization of dedicated physical
+   implementations of internetworking functions.  The Black-box
+   Benchmarks of Throughput, Latency, Forwarding Rates and others have
+   served our industry for many years.  Now, Network Function
+   Virtualization (NFV) has the goal to transform how internetwork
+   functions are implemented, and therefore has garnered much attention.
+
+   This memo summarizes the progress of the Open Platform for NFV
+   (OPNFV) project on virtual switch performance characterization,
+   "VSWITCHPERF", through the Brahmaputra (second) release [BrahRel].
+   This project intends to build on the current and completed work of
+   the Benchmarking Methodology Working Group in IETF, by referencing
+   existing literature.  For example, currently the most often
+   referenced RFC is [RFC2544] (which depends on [RFC1242]) and
+   foundation of the benchmarking work in OPNFV is common and strong.
+
+   See https://wiki.opnfv.org/
+   characterize_vswitch_performance_for_telco_nfv_use_cases for more
+   background, and the OPNFV website for general information:
+   https://www.opnfv.org/
+
+   The authors note that OPNFV distinguishes itself from other open
+   source compute and networking projects through its emphasis on
+   existing "telco" services as opposed to cloud-computing.  There are
+   many ways in which telco requirements have different emphasis on
+   performance dimensions when compared to cloud computing: support for
+   and transfer of isochronous media streams is one example.
+
+   Note also that the move to NFV Infrastructure has resulted in many
+   new benchmarking initiatives across the industry.  The authors are
+   currently doing their best to maintain alignment with many other
+   projects, and this Internet Draft is one part of the efforts.  We
+   acknowledge the early work in
+   [I-D.huang-bmwg-virtual-network-performance], and useful discussion
+   with the authors.
+
+2.  Scope
+
+   The primary purpose and scope of the memo is to inform the industry
+   of work-in-progress that builds on the body of extensive BMWG
+   literature and experience, and describe the extensions needed for
+   benchmarking virtual switches.  Inital feedback indicates that many
+   of these extensions may be applicable beyond the current scope (to
+   hardware switches in the NFV Infrastructure and to virtual routers,
+   for example).  Additionally, this memo serves as a vehicle to include
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 3]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   more detail and commentary from BMWG and other Open Source
+   communities, under BMWG's chartered work to characterize the NFV
+   Infrastructure (a virtual switch is an important aspect of that
+   infrastructure).
+
+3.  Benchmarking Considerations
+
+   This section highlights some specific considerations (from
+   [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
+   switches.  The OPNFV project is sharing its present view on these
+   areas, as they develop their specifications in the Level Test Design
+   (LTD) document.
+
+3.1.  Comparison with Physical Network Functions
+
+   To compare the performance of virtual designs and implementations
+   with their physical counterparts, identical benchmarks are needed.
+   BMWG has developed specifications for many network functions this
+   memo re-uses existing benchmarks through references, and expands them
+   during development of new methods.  A key configuration aspect is the
+   number of parallel cores required to achieve comparable performance
+   with a given physical device, or whether some limit of scale was
+   reached before the cores could achieve the comparable level.
+
+   It's unlikely that the virtual switch will be the only application
+   running on the SUT, so CPU utilization, Cache utilization, and Memory
+   footprint should also be recorded for the virtual implementations of
+   internetworking functions.
+
+3.2.  Continued Emphasis on Black-Box Benchmarks
+
+   External observations remain essential as the basis for Benchmarks.
+   Internal observations with fixed specification and interpretation
+   will be provided in parallel to assist the development of operations
+   procedures when the technology is deployed.
+
+3.3.  New Configuration Parameters
+
+   A key consideration when conducting any sort of benchmark is trying
+   to ensure the consistency and repeatability of test results.  When
+   benchmarking the performance of a vSwitch there are many factors that
+   can affect the consistency of results, one key factor is matching the
+   various hardware and software details of the SUT.  This section lists
+   some of the many new parameters which this project believes are
+   critical to report in order to achieve repeatability.
+
+   Hardware details including:
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 4]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   o  Platform details
+
+   o  Processor details
+
+   o  Memory information (type and size)
+
+   o  Number of enabled cores
+
+   o  Number of cores used for the test
+
+   o  Number of physical NICs, as well as their details (manufacturer,
+      versions, type and the PCI slot they are plugged into)
+
+   o  NIC interrupt configuration
+
+   o  BIOS version, release date and any configurations that were
+      modified
+
+   o  CPU microcode level
+
+   o  Memory DIMM configurations (quad rank performance may not be the
+      same as dual rank) in size, freq and slot locations
+
+   o  PCI configuration parameters (payload size, early ack option...)
+
+   o  Power management at all levels (ACPI sleep states, processor
+      package, OS...)
+
+   Software details including:
+
+   o  OS parameters and behavior (text vs graphical no one typing at the
+      console on one system)
+
+   o  OS version (for host and VNF)
+
+   o  Kernel version (for host and VNF)
+
+   o  GRUB boot parameters (for host and VNF)
+
+   o  Hypervisor details (Type and version)
+
+   o  Selected vSwitch, version number or commit id used
+
+   o  vSwitch launch command line if it has been parameterised
+
+   o  Memory allocation to the vSwitch
+
+   o  which NUMA node it is using, and how many memory channels
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 5]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   o  DPDK or any other SW dependency version number or commit id used
+
+   o  Memory allocation to a VM - if it's from Hugpages/elsewhere
+
+   o  VM storage type: snapshot/independent persistent/independent non-
+      persistent
+
+   o  Number of VMs
+
+   o  Number of Virtual NICs (vNICs), versions, type and driver
+
+   o  Number of virtual CPUs and their core affinity on the host
+
+   o  Number vNIC interrupt configuration
+
+   o  Thread affinitization for the applications (including the vSwitch
+      itself) on the host
+
+   o  Details of Resource isolation, such as CPUs designated for Host/
+      Kernel (isolcpu) and CPUs designated for specific processes
+      (taskset). - Test duration. - Number of flows.
+
+   Test Traffic Information:
+
+   o  Traffic type - UDP, TCP, IMIX / Other
+
+   o  Packet Sizes
+
+   o  Deployment Scenario
+
+3.4.  Flow classification
+
+   Virtual switches group packets into flows by processing and matching
+   particular packet or frame header information, or by matching packets
+   based on the input ports.  Thus a flow can be thought of a sequence
+   of packets that have the same set of header field values (5-tuple) or
+   have arrived on the same port.  Performance results can vary based on
+   the parameters the vSwitch uses to match for a flow.  The recommended
+   flow classification parameters for any vSwitch performance tests are:
+   the input port, the source IP address, the destination IP address and
+   the Ethernet protocol type field.  It is essential to increase the
+   flow timeout time on a vSwitch before conducting any performance
+   tests that do not measure the flow setup time.  Normally the first
+   packet of a particular stream will install the flow in the virtual
+   switch which adds an additional latency, subsequent packets of the
+   same flow are not subject to this latency if the flow is already
+   installed on the vSwitch.
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 6]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+3.5.  Benchmarks using Baselines with Resource Isolation
+
+   This outline describes measurement of baseline with isolated
+   resources at a high level, which is the intended approach at this
+   time.
+
+   1.  Baselines:
+
+       *  Optional: Benchmark platform forwarding capability without a
+          vswitch or VNF for at least 72 hours (serves as a means of
+          platform validation and a means to obtain the base performance
+          for the platform in terms of its maximum forwarding rate and
+          latency).
+
+   Benchmark platform forwarding capability
+
+                                                                   __
+              +--------------------------------------------------+   |
+              |   +------------------------------------------+   |   |
+              |   |                                          |   |   |
+              |   |          Simple Forwarding App           |   |  Host
+              |   |                                          |   |   |
+              |   +------------------------------------------+   |   |
+              |   |                 NIC                      |   |   |
+              +---+------------------------------------------+---+ __|
+                         ^                           :
+                         |                           |
+                         :                           v
+              +--------------------------------------------------+
+              |                                                  |
+              |                traffic generator                 |
+              |                                                  |
+              +--------------------------------------------------+
+
+       *  Benchmark VNF forwarding capability with direct connectivity
+          (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves
+          as a means of VNF validation and a means to obtain the base
+          performance for the VNF in terms of its maximum forwarding
+          rate and latency).  The metrics gathered from this test will
+          serve as a key comparison point for vSwitch bypass
+          technologies performance and vSwitch performance.
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 7]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+                                     Benchmark VNF forwarding capability
+
+                                                         __
+    +--------------------------------------------------+   |
+    |   +------------------------------------------+   |   |
+    |   |                                          |   |   |
+    |   |                 VNF                      |   |   |
+    |   |                                          |   |   |
+    |   +------------------------------------------+   |   |
+    |   |          Passthrough/SR-IOV              |   |  Host
+    |   +------------------------------------------+   |   |
+    |   |                 NIC                      |   |   |
+    +---+------------------------------------------+---+ __|
+               ^                           :
+               |                           |
+               :                           v
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+       *  Benchmarking with isolated resources alone, with other
+          resources (both HW&SW) disabled Example, vSw and VM are SUT
+
+       *  Benchmarking with isolated resources alone, leaving some
+          resources unused
+
+       *  Benchmark with isolated resources and all resources occupied
+
+   2.  Next Steps
+
+       *  Limited sharing
+
+       *  Production scenarios
+
+       *  Stressful scenarios
+
+4.  VSWITCHPERF Specification Summary
+
+   The overall specification in preparation is referred to as a Level
+   Test Design (LTD) document, which will contain a suite of performance
+   tests.  The base performance tests in the LTD are based on the pre-
+   existing specifications developed by BMWG to test the performance of
+   physical switches.  These specifications include:
+
+   o  [RFC2544] Benchmarking Methodology for Network Interconnect
+      Devices
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 8]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   o  [RFC2889] Benchmarking Methodology for LAN Switching
+
+   o  [RFC6201] Device Reset Characterization
+
+   o  [RFC5481] Packet Delay Variation Applicability Statement
+
+   Some of the above/newer RFCs are being applied in benchmarking for
+   the first time, and represent a development challenge for test
+   equipment developers.  Fortunately, many members of the testing
+   system community have engaged on the VSPERF project, including an
+   open source test system.
+
+   In addition to this, the LTD also re-uses the terminology defined by:
+
+   o  [RFC2285] Benchmarking Terminology for LAN Switching Devices
+
+   o  [RFC5481] Packet Delay Variation Applicability Statement
+
+   Specifications to be included in future updates of the LTD include:
+
+   o  [RFC3918] Methodology for IP Multicast Benchmarking
+
+   o  [RFC4737] Packet Reordering Metrics
+
+   As one might expect, the most fundamental internetworking
+   characteristics of Throughput and Latency remain important when the
+   switch is virtualized, and these benchmarks figure prominently in the
+   specification.
+
+   When considering characteristics important to "telco" network
+   functions, we must begin to consider additional performance metrics.
+   In this case, the project specifications have referenced metrics from
+   the IETF IP Performance Metrics (IPPM) literature.  This means that
+   the [RFC2544] test of Latency is replaced by measurement of a metric
+   derived from IPPM's [RFC2679], where a set of statistical summaries
+   will be provided (mean, max, min, etc.).  Further metrics planned to
+   be benchmarked include packet delay variation as defined by [RFC5481]
+   , reordering, burst behaviour, DUT availability, DUT capacity and
+   packet loss in long term testing at Throughput level, where some low-
+   level of background loss may be present and characterized.
+
+   Tests have been (or will be) designed to collect the metrics below:
+
+   o  Throughput Tests to measure the maximum forwarding rate (in frames
+      per second or fps) and bit rate (in Mbps) for a constant load (as
+      defined by [RFC1242]) without traffic loss.
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017                [Page 9]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   o  Packet and Frame Delay Distribution Tests to measure average, min
+      and max packet and frame delay for constant loads.
+
+   o  Packet Delay Tests to understand latency distribution for
+      different packet sizes and over an extended test run to uncover
+      outliers.
+
+   o  Scalability Tests to understand how the virtual switch performs as
+      the number of flows, active ports, complexity of the forwarding
+      logic's configuration... it has to deal with increases.
+
+   o  Stream Performance Tests (TCP, UDP) to measure bulk data transfer
+      performance, i.e. how fast systems can send and receive data
+      through the switch.
+
+   o  Control Path and Datapath Coupling Tests, to understand how
+      closely coupled the datapath and the control path are as well as
+      the effect of this coupling on the performance of the DUT
+      (example: delay of the initial packet of a flow).
+
+   o  CPU and Memory Consumption Tests to understand the virtual
+      switch's footprint on the system, usually conducted as auxiliary
+      measurements with benchmarks above.  They include: CPU
+      utilization, Cache utilization and Memory footprint.
+
+   o  The so-called "Soak" tests, where the selected test is conducted
+      over a long period of time (with an ideal duration of 24 hours,
+      and at least 6 hours).  The purpose of soak tests is to capture
+      transient changes in performance which may occur due to infrequent
+      processes or the low probability coincidence of two or more
+      processes.  The performance must be evaluated periodically during
+      continuous testing, and this results in use of [RFC2889] Frame
+      Rate metrics instead of [RFC2544] Throughput (which requires
+      stopping traffic to allow time for all traffic to exit internal
+      queues).
+
+   Future/planned test specs include:
+
+   o  Request/Response Performance Tests (TCP, UDP) which measure the
+      transaction rate through the switch.
+
+   o  Noisy Neighbour Tests, to understand the effects of resource
+      sharing on the performance of a virtual switch.
+
+   o  Tests derived from examination of ETSI NFV Draft GS IFA003
+      requirements [IFA003] on characterization of acceleration
+      technologies applied to vswitches.
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 10]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   The flexibility of deployment of a virtual switch within a network
+   means that the BMWG IETF existing literature needs to be used to
+   characterize the performance of a switch in various deployment
+   scenarios.  The deployment scenarios under consideration include:
+
+   Physical port to virtual switch to physical port
+
+                                                         __
+    +--------------------------------------------------+   |
+    |              +--------------------+              |   |
+    |              |                    |              |   |
+    |              |                    v              |   |  Host
+    |   +--------------+            +--------------+   |   |
+    |   |   phy port   |  vSwitch   |   phy port   |   |   |
+    +---+--------------+------------+--------------+---+ __|
+               ^                           :
+               |                           |
+               :                           v
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 11]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   Physical port to virtual switch to VNF to virtual switch to physical
+   port
+
+                                                         __
+    +---------------------------------------------------+   |
+    |                                                   |   |
+    |   +-------------------------------------------+   |   |
+    |   |                 Application               |   |   |
+    |   +-------------------------------------------+   |   |
+    |       ^                                  :        |   |
+    |       |                                  |        |   |  Guest
+    |       :                                  v        |   |
+    |   +---------------+           +---------------+   |   |
+    |   | logical port 0|           | logical port 1|   |   |
+    +---+---------------+-----------+---------------+---+ __|
+            ^                                  :
+            |                                  |
+            :                                  v         __
+    +---+---------------+----------+---------------+---+   |
+    |   | logical port 0|          | logical port 1|   |   |
+    |   +---------------+          +---------------+   |   |
+    |       ^                                  :       |   |
+    |       |                                  |       |   |  Host
+    |       :                                  v       |   |
+    |   +--------------+            +--------------+   |   |
+    |   |   phy port   |  vSwitch   |   phy port   |   |   |
+    +---+--------------+------------+--------------+---+ __|
+               ^                           :
+               |                           |
+               :                           v
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 12]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   Physical port to virtual switch to VNF to virtual switch to VNF to
+   virtual switch to physical port
+
+                                                      __
+    +----------------------+  +----------------------+  |
+    |   Guest 1            |  |   Guest 2            |  |
+    |   +---------------+  |  |   +---------------+  |  |
+    |   |  Application  |  |  |   |  Application  |  |  |
+    |   +---------------+  |  |   +---------------+  |  |
+    |       ^       |      |  |       ^       |      |  |
+    |       |       v      |  |       |       v      |  |  Guests
+    |   +---------------+  |  |   +---------------+  |  |
+    |   | logical ports |  |  |   | logical ports |  |  |
+    |   |   0       1   |  |  |   |   0       1   |  |  |
+    +---+---------------+--+  +---+---------------+--+__|
+            ^       :                 ^       :
+            |       |                 |       |
+            :       v                 :       v       _
+    +---+---------------+---------+---------------+--+ |
+    |   |   0       1   |         |   3       4   |  | |
+    |   | logical ports |         | logical ports |  | |
+    |   +---------------+         +---------------+  | |
+    |       ^       |                 ^       |      | |  Host
+    |       |       |-----------------|       v      | |
+    |   +--------------+          +--------------+   | |
+    |   |   phy ports  | vSwitch  |   phy ports  |   | |
+    +---+--------------+----------+--------------+---+_|
+            ^                                 :
+            |                                 |
+            :                                 v
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 13]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   Physical port to virtual switch to VNF
+
+                                                          __
+    +---------------------------------------------------+   |
+    |                                                   |   |
+    |   +-------------------------------------------+   |   |
+    |   |                 Application               |   |   |
+    |   +-------------------------------------------+   |   |
+    |       ^                                           |   |
+    |       |                                           |   |  Guest
+    |       :                                           |   |
+    |   +---------------+                               |   |
+    |   | logical port 0|                               |   |
+    +---+---------------+-------------------------------+ __|
+            ^
+            |
+            :                                            __
+    +---+---------------+------------------------------+   |
+    |   | logical port 0|                              |   |
+    |   +---------------+                              |   |
+    |       ^                                          |   |
+    |       |                                          |   |  Host
+    |       :                                          |   |
+    |   +--------------+                               |   |
+    |   |   phy port   |  vSwitch                      |   |
+    +---+--------------+------------ -------------- ---+ __|
+               ^
+               |
+               :
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 14]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   VNF to virtual switch to physical port
+
+                                                          __
+    +---------------------------------------------------+   |
+    |                                                   |   |
+    |   +-------------------------------------------+   |   |
+    |   |                 Application               |   |   |
+    |   +-------------------------------------------+   |   |
+    |                                          :        |   |
+    |                                          |        |   |  Guest
+    |                                          v        |   |
+    |                               +---------------+   |   |
+    |                               | logical port  |   |   |
+    +-------------------------------+---------------+---+ __|
+                                               :
+                                               |
+                                               v         __
+    +------------------------------+---------------+---+   |
+    |                              | logical port  |   |   |
+    |                              +---------------+   |   |
+    |                                          :       |   |
+    |                                          |       |   |  Host
+    |                                          v       |   |
+    |                               +--------------+   |   |
+    |                     vSwitch   |   phy port   |   |   |
+    +-------------------------------+--------------+---+ __|
+                                           :
+                                           |
+                                           v
+    +--------------------------------------------------+
+    |                                                  |
+    |                traffic generator                 |
+    |                                                  |
+    +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 15]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   VNF to virtual switch to VNF
+
+                                                      __
+    +----------------------+  +----------------------+  |
+    |   Guest 1            |  |   Guest 2            |  |
+    |   +---------------+  |  |   +---------------+  |  |
+    |   |  Application  |  |  |   |  Application  |  |  |
+    |   +---------------+  |  |   +---------------+  |  |
+    |              |       |  |       ^              |  |
+    |              v       |  |       |              |  |  Guests
+    |   +---------------+  |  |   +---------------+  |  |
+    |   | logical ports |  |  |   | logical ports |  |  |
+    |   |           0   |  |  |   |   0           |  |  |
+    +---+---------------+--+  +---+---------------+--+__|
+                    :                 ^
+                    |                 |
+                    v                 :               _
+    +---+---------------+---------+---------------+--+ |
+    |   |           1   |         |   1           |  | |
+    |   | logical ports |         | logical ports |  | |
+    |   +---------------+         +---------------+  | |
+    |               |                 ^              | |  Host
+    |               L-----------------+              | |
+    |                                                | |
+    |                    vSwitch                     | |
+    +------------------------------------------------+_|
+
+   A set of Deployment Scenario figures is available on the VSPERF Test
+   Methodology Wiki page [TestTopo].
+
+5.  3x3 Matrix Coverage
+
+   This section organizes the many existing test specifications into the
+   "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]).  Because
+   the LTD specification ID names are quite long, this section is
+   organized into lists for each occupied cell of the matrix (not all
+   are occupied, also the matrix has grown to 3x4 to accommodate scale
+   metrics when displaying the coverage of many metrics/benchmarks).
+   The current version of the LTD specification is available [LTD].
+
+   The tests listed below assess the activation of paths in the data
+   plane, rather than the control plane.
+
+   A complete list of tests with short summaries is available on the
+   VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV].
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 16]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+5.1.  Speed of Activation
+
+   o  Activation.RFC2889.AddressLearningRate
+
+   o  PacketLatency.InitialPacketProcessingLatency
+
+5.2.  Accuracy of Activation section
+
+   o  CPDP.Coupling.Flow.Addition
+
+5.3.  Reliability of Activation
+
+   o  Throughput.RFC2544.SystemRecoveryTime
+
+   o  Throughput.RFC2544.ResetTime
+
+5.4.  Scale of Activation
+
+   o  Activation.RFC2889.AddressCachingCapacity
+
+5.5.  Speed of Operation
+
+   o  Throughput.RFC2544.PacketLossRate
+
+   o  CPU.RFC2544.0PacketLoss
+
+   o  Throughput.RFC2544.PacketLossRateFrameModification
+
+   o  Throughput.RFC2544.BackToBackFrames
+
+   o  Throughput.RFC2889.MaxForwardingRate
+
+   o  Throughput.RFC2889.ForwardPressure
+
+   o  Throughput.RFC2889.BroadcastFrameForwarding
+
+5.6.  Accuracy of Operation
+
+   o  Throughput.RFC2889.ErrorFramesFiltering
+
+   o  Throughput.RFC2544.Profile
+
+5.7.  Reliability of Operation
+
+   o  Throughput.RFC2889.Soak
+
+   o  Throughput.RFC2889.SoakFrameModification
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 17]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   o  PacketDelayVariation.RFC3393.Soak
+
+5.8.  Scalability of Operation
+
+   o  Scalability.RFC2544.0PacketLoss
+
+   o  MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+
+5.9.  Summary
+
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|               |   SPEED     |  ACCURACY  |  RELIABILITY  |    SCALE    |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|  Activation   |      X      |     X      |       X       |      X      |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|  Operation    |      X      |      X     |       X       |      X      |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+| De-activation |             |            |               |             |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+
+6.  Security Considerations
+
+   Benchmarking activities as described in this memo are limited to
+   technology characterization of a Device Under Test/System Under Test
+   (DUT/SUT) using controlled stimuli in a laboratory environment, with
+   dedicated address space and the constraints specified in the sections
+   above.
+
+   The benchmarking network topology will be an independent test setup
+   and MUST NOT be connected to devices that may forward the test
+   traffic into a production network, or misroute traffic to the test
+   management network.
+
+   Further, benchmarking is performed on a "black-box" basis, relying
+   solely on measurements observable external to the DUT/SUT.
+
+   Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+   benchmarking purposes.  Any implications for network security arising
+   from the DUT/SUT SHOULD be identical in the lab and in production
+   networks.
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 18]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+7.  IANA Considerations
+
+   No IANA Action is requested at this time.
+
+8.  Acknowledgements
+
+   The authors appreciate and acknowledge comments from Scott Bradner,
+   Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik,
+   Christian Trautman, and others for their reviews.
+
+9.  References
+
+9.1.  Normative References
+
+   [NFV.PER001]
+              "Network Function Virtualization: Performance and
+              Portability Best Practices", Group Specification ETSI GS
+              NFV-PER 001 V1.1.1 (2014-06), June 2014.
+
+   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
+              Requirement Levels", BCP 14, RFC 2119,
+              DOI 10.17487/RFC2119, March 1997,
+              <http://www.rfc-editor.org/info/rfc2119>.
+
+   [RFC2285]  Mandeville, R., "Benchmarking Terminology for LAN
+              Switching Devices", RFC 2285, DOI 10.17487/RFC2285,
+              February 1998, <http://www.rfc-editor.org/info/rfc2285>.
+
+   [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
+              "Framework for IP Performance Metrics", RFC 2330,
+              DOI 10.17487/RFC2330, May 1998,
+              <http://www.rfc-editor.org/info/rfc2330>.
+
+   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
+              Network Interconnect Devices", RFC 2544,
+              DOI 10.17487/RFC2544, March 1999,
+              <http://www.rfc-editor.org/info/rfc2544>.
+
+   [RFC2679]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
+              Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679,
+              September 1999, <http://www.rfc-editor.org/info/rfc2679>.
+
+   [RFC2680]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
+              Packet Loss Metric for IPPM", RFC 2680,
+              DOI 10.17487/RFC2680, September 1999,
+              <http://www.rfc-editor.org/info/rfc2680>.
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 19]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   [RFC2681]  Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
+              Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681,
+              September 1999, <http://www.rfc-editor.org/info/rfc2681>.
+
+   [RFC2889]  Mandeville, R. and J. Perser, "Benchmarking Methodology
+              for LAN Switching Devices", RFC 2889,
+              DOI 10.17487/RFC2889, August 2000,
+              <http://www.rfc-editor.org/info/rfc2889>.
+
+   [RFC3393]  Demichelis, C. and P. Chimento, "IP Packet Delay Variation
+              Metric for IP Performance Metrics (IPPM)", RFC 3393,
+              DOI 10.17487/RFC3393, November 2002,
+              <http://www.rfc-editor.org/info/rfc3393>.
+
+   [RFC3432]  Raisanen, V., Grotefeld, G., and A. Morton, "Network
+              performance measurement with periodic streams", RFC 3432,
+              DOI 10.17487/RFC3432, November 2002,
+              <http://www.rfc-editor.org/info/rfc3432>.
+
+   [RFC3918]  Stopp, D. and B. Hickman, "Methodology for IP Multicast
+              Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October
+              2004, <http://www.rfc-editor.org/info/rfc3918>.
+
+   [RFC4689]  Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
+              "Terminology for Benchmarking Network-layer Traffic
+              Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
+              October 2006, <http://www.rfc-editor.org/info/rfc4689>.
+
+   [RFC4737]  Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
+              S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
+              DOI 10.17487/RFC4737, November 2006,
+              <http://www.rfc-editor.org/info/rfc4737>.
+
+   [RFC5357]  Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
+              Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
+              RFC 5357, DOI 10.17487/RFC5357, October 2008,
+              <http://www.rfc-editor.org/info/rfc5357>.
+
+   [RFC5905]  Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
+              "Network Time Protocol Version 4: Protocol and Algorithms
+              Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
+              <http://www.rfc-editor.org/info/rfc5905>.
+
+   [RFC6201]  Asati, R., Pignataro, C., Calabria, F., and C. Olvera,
+              "Device Reset Characterization", RFC 6201,
+              DOI 10.17487/RFC6201, March 2011,
+              <http://www.rfc-editor.org/info/rfc6201>.
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 20]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+9.2.  Informative References
+
+   [BrahRel]  "Brahmaputra, Second OPNFV Release https://www.opnfv.org/
+              brahmaputra".
+
+   [I-D.huang-bmwg-virtual-network-performance]
+              Huang, L., Rong, G., Mandeville, B., and B. Hickman,
+              "Benchmarking Methodology for Virtualization Network
+              Performance", draft-huang-bmwg-virtual-network-
+              performance-01 (work in progress), April 2015.
+
+   [I-D.ietf-bmwg-virtual-net]
+              Morton, A., "Considerations for Benchmarking Virtual
+              Network Functions and Their Infrastructure", draft-ietf-
+              bmwg-virtual-net-03 (work in progress), June 2016.
+
+   [IFA003]   "https://docbox.etsi.org/ISG/NFV/Open/Drafts/
+              IFA003_Acceleration_-_vSwitch_Spec/".
+
+   [LTD]      "LTD Test Specification
+              http://artifacts.opnfv.org/vswitchperf/docs/requirements/
+              index.html".
+
+   [LTDoverV]
+              "LTD Test Spec Overview https://wiki.opnfv.org/wiki/
+              vswitchperf_test_spec_review".
+
+   [RFC1242]  Bradner, S., "Benchmarking Terminology for Network
+              Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
+              July 1991, <http://www.rfc-editor.org/info/rfc1242>.
+
+   [RFC5481]  Morton, A. and B. Claise, "Packet Delay Variation
+              Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
+              March 2009, <http://www.rfc-editor.org/info/rfc5481>.
+
+   [RFC6049]  Morton, A. and E. Stephan, "Spatial Composition of
+              Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
+              <http://www.rfc-editor.org/info/rfc6049>.
+
+   [RFC6248]  Morton, A., "RFC 4148 and the IP Performance Metrics
+              (IPPM) Registry of Metrics Are Obsolete", RFC 6248,
+              DOI 10.17487/RFC6248, April 2011,
+              <http://www.rfc-editor.org/info/rfc6248>.
+
+   [RFC6390]  Clark, A. and B. Claise, "Guidelines for Considering New
+              Performance Metric Development", BCP 170, RFC 6390,
+              DOI 10.17487/RFC6390, October 2011,
+              <http://www.rfc-editor.org/info/rfc6390>.
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 21]
+
+Internet-Draft           Benchmarking vSwitches                July 2016
+
+
+   [TestTopo]
+              "Test Topologies https://wiki.opnfv.org/vsperf/
+              test_methodology".
+
+Authors' Addresses
+
+   Maryam Tahhan
+   Intel
+
+   Email: maryam.tahhan@intel.com
+
+
+   Billy O'Mahony
+   Intel
+
+   Email: billy.o.mahony@intel.com
+
+
+   Al Morton
+   AT&T Labs
+   200 Laurel Avenue South
+   Middletown,, NJ  07748
+   USA
+
+   Phone: +1 732 420 1571
+   Fax:   +1 732 368 1192
+   Email: acmorton@att.com
+   URI:   http://home.comcast.net/~acmacm/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al.           Expires January 9, 2017               [Page 22]
diff --git a/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml b/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml
new file mode 100644
index 00000000..2259b23c
--- /dev/null
+++ b/docs/requirements/ietf_draft/draft-ietf-bmwg-vswitch-opnfv-00.xml
@@ -0,0 +1,1016 @@
+<?xml version="1.0" encoding="US-ASCII"?>
+<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
+<?rfc toc="yes"?>
+<?rfc tocompact="yes"?>
+<?rfc tocdepth="3"?>
+<?rfc tocindent="yes"?>
+<?rfc symrefs="yes"?>
+<?rfc sortrefs="yes"?>
+<?rfc comments="yes"?>
+<?rfc inline="yes"?>
+<?rfc compact="yes"?>
+<?rfc subcompact="no"?>
+<rfc category="info" docName="draft-ietf-bmwg-vswitch-opnfv-00"
+     ipr="trust200902">
+  <front>
+    <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
+    OPNFV</title>
+
+    <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
+      <organization>Intel</organization>
+
+      <address>
+        <postal>
+          <street/>
+
+          <city/>
+
+          <region/>
+
+          <code/>
+
+          <country/>
+        </postal>
+
+        <phone/>
+
+        <facsimile/>
+
+        <email>maryam.tahhan@intel.com</email>
+
+        <uri/>
+      </address>
+    </author>
+
+    <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony">
+      <organization>Intel</organization>
+
+      <address>
+        <postal>
+          <street/>
+
+          <city/>
+
+          <region/>
+
+          <code/>
+
+          <country/>
+        </postal>
+
+        <phone/>
+
+        <facsimile/>
+
+        <email>billy.o.mahony@intel.com</email>
+
+        <uri/>
+      </address>
+    </author>
+
+    <author fullname="Al Morton" initials="A." surname="Morton">
+      <organization>AT&amp;T Labs</organization>
+
+      <address>
+        <postal>
+          <street>200 Laurel Avenue South</street>
+
+          <city>Middletown,</city>
+
+          <region>NJ</region>
+
+          <code>07748</code>
+
+          <country>USA</country>
+        </postal>
+
+        <phone>+1 732 420 1571</phone>
+
+        <facsimile>+1 732 368 1192</facsimile>
+
+        <email>acmorton@att.com</email>
+
+        <uri>http://home.comcast.net/~acmacm/</uri>
+      </address>
+    </author>
+
+    <date day="8" month="July" year="2016"/>
+
+    <abstract>
+      <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
+      project on virtual switch performance "VSWITCHPERF". This project
+      intends to build on the current and completed work of the Benchmarking
+      Methodology Working Group in IETF, by referencing existing literature.
+      The Benchmarking Methodology Working Group has traditionally conducted
+      laboratory characterization of dedicated physical implementations of
+      internetworking functions. Therefore, this memo begins to describe the
+      additional considerations when virtual switches are implemented in
+      general-purpose hardware. The expanded tests and benchmarks are also
+      influenced by the OPNFV mission to support virtualization of the "telco"
+      infrastructure.</t>
+    </abstract>
+
+    <note title="Requirements Language">
+      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+      "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+      document are to be interpreted as described in <xref
+      target="RFC2119">RFC 2119</xref>.</t>
+
+      <t/>
+    </note>
+  </front>
+
+  <middle>
+    <section title="Introduction">
+      <t>Benchmarking Methodology Working Group (BMWG) has traditionally
+      conducted laboratory characterization of dedicated physical
+      implementations of internetworking functions. The Black-box Benchmarks
+      of Throughput, Latency, Forwarding Rates and others have served our
+      industry for many years. Now, Network Function Virtualization (NFV) has
+      the goal to transform how internetwork functions are implemented, and
+      therefore has garnered much attention.</t>
+
+      <t>This memo summarizes the progress of the Open Platform for NFV
+      (OPNFV) project on virtual switch performance characterization,
+      "VSWITCHPERF", through the Brahmaputra (second) release <xref
+      target="BrahRel"/>. This project intends to build on the current and
+      completed work of the Benchmarking Methodology Working Group in IETF, by
+      referencing existing literature. For example, currently the most often
+      referenced RFC is <xref target="RFC2544"/> (which depends on <xref
+      target="RFC1242"/>) and foundation of the benchmarking work in OPNFV is
+      common and strong.</t>
+
+      <t>See
+      https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
+      for more background, and the OPNFV website for general information:
+      https://www.opnfv.org/</t>
+
+      <t>The authors note that OPNFV distinguishes itself from other open
+      source compute and networking projects through its emphasis on existing
+      "telco" services as opposed to cloud-computing. There are many ways in
+      which telco requirements have different emphasis on performance
+      dimensions when compared to cloud computing: support for and transfer of
+      isochronous media streams is one example.</t>
+
+      <t>Note also that the move to NFV Infrastructure has resulted in many
+      new benchmarking initiatives across the industry. The authors are
+      currently doing their best to maintain alignment with many other
+      projects, and this Internet Draft is one part of the efforts. We
+      acknowledge the early work in <xref
+      target="I-D.huang-bmwg-virtual-network-performance"/>, and useful
+      discussion with the authors.</t>
+    </section>
+
+    <section title="Scope">
+      <t>The primary purpose and scope of the memo is to inform the industry
+      of work-in-progress that builds on the body of extensive BMWG literature
+      and experience, and describe the extensions needed for benchmarking
+      virtual switches. Inital feedback indicates that many of these
+      extensions may be applicable beyond the current scope (to hardware
+      switches in the NFV Infrastructure and to virtual routers, for example).
+      Additionally, this memo serves as a vehicle to include more detail and
+      commentary from BMWG and other Open Source communities, under BMWG's
+      chartered work to characterize the NFV Infrastructure (a virtual switch
+      is an important aspect of that infrastructure).</t>
+    </section>
+
+    <section title="Benchmarking Considerations">
+      <t>This section highlights some specific considerations (from <xref
+      target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
+      switches. The OPNFV project is sharing its present view on these areas,
+      as they develop their specifications in the Level Test Design (LTD)
+      document.</t>
+
+      <section title="Comparison with Physical Network Functions">
+        <t>To compare the performance of virtual designs and implementations
+        with their physical counterparts, identical benchmarks are needed.
+        BMWG has developed specifications for many network functions this memo
+        re-uses existing benchmarks through references, and expands them
+        during development of new methods. A key configuration aspect is the
+        number of parallel cores required to achieve comparable performance
+        with a given physical device, or whether some limit of scale was
+        reached before the cores could achieve the comparable level.</t>
+
+        <t>It's unlikely that the virtual switch will be the only application
+        running on the SUT, so CPU utilization, Cache utilization, and Memory
+        footprint should also be recorded for the virtual implementations of
+        internetworking functions.</t>
+      </section>
+
+      <section title="Continued Emphasis on Black-Box Benchmarks">
+        <t>External observations remain essential as the basis for Benchmarks.
+        Internal observations with fixed specification and interpretation will
+        be provided in parallel to assist the development of operations
+        procedures when the technology is deployed.</t>
+      </section>
+
+      <section title="New Configuration Parameters">
+        <t>A key consideration when conducting any sort of benchmark is trying
+        to ensure the consistency and repeatability of test results. When
+        benchmarking the performance of a vSwitch there are many factors that
+        can affect the consistency of results, one key factor is matching the
+        various hardware and software details of the SUT. This section lists
+        some of the many new parameters which this project believes are
+        critical to report in order to achieve repeatability.</t>
+
+        <t>Hardware details including:</t>
+
+        <t><list style="symbols">
+            <t>Platform details</t>
+
+            <t>Processor details</t>
+
+            <t>Memory information (type and size)</t>
+
+            <t>Number of enabled cores</t>
+
+            <t>Number of cores used for the test</t>
+
+            <t>Number of physical NICs, as well as their details
+            (manufacturer, versions, type and the PCI slot they are plugged
+            into)</t>
+
+            <t>NIC interrupt configuration</t>
+
+            <t>BIOS version, release date and any configurations that were
+            modified</t>
+
+            <t>CPU microcode level</t>
+
+            <t>Memory DIMM configurations (quad rank performance may not be
+            the same as dual rank) in size, freq and slot locations</t>
+
+            <t>PCI configuration parameters (payload size, early ack
+            option...)</t>
+
+            <t>Power management at all levels (ACPI sleep states, processor
+            package, OS...)</t>
+          </list>Software details including:</t>
+
+        <t><list style="symbols">
+            <t>OS parameters and behavior (text vs graphical no one typing at
+            the console on one system)</t>
+
+            <t>OS version (for host and VNF)</t>
+
+            <t>Kernel version (for host and VNF)</t>
+
+            <t>GRUB boot parameters (for host and VNF)</t>
+
+            <t>Hypervisor details (Type and version)</t>
+
+            <t>Selected vSwitch, version number or commit id used</t>
+
+            <t>vSwitch launch command line if it has been parameterised</t>
+
+            <t>Memory allocation to the vSwitch</t>
+
+            <t>which NUMA node it is using, and how many memory channels</t>
+
+            <t>DPDK or any other SW dependency version number or commit id
+            used</t>
+
+            <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t>
+
+            <t>VM storage type: snapshot/independent persistent/independent
+            non-persistent</t>
+
+            <t>Number of VMs</t>
+
+            <t>Number of Virtual NICs (vNICs), versions, type and driver</t>
+
+            <t>Number of virtual CPUs and their core affinity on the host</t>
+
+            <t>Number vNIC interrupt configuration</t>
+
+            <t>Thread affinitization for the applications (including the
+            vSwitch itself) on the host</t>
+
+            <t>Details of Resource isolation, such as CPUs designated for
+            Host/Kernel (isolcpu) and CPUs designated for specific processes
+            (taskset). - Test duration. - Number of flows.</t>
+          </list></t>
+
+        <t>Test Traffic Information:<list style="symbols">
+            <t>Traffic type - UDP, TCP, IMIX / Other</t>
+
+            <t>Packet Sizes</t>
+
+            <t>Deployment Scenario</t>
+          </list></t>
+
+        <t/>
+      </section>
+
+      <section title="Flow classification">
+        <t>Virtual switches group packets into flows by processing and
+        matching particular packet or frame header information, or by matching
+        packets based on the input ports. Thus a flow can be thought of a
+        sequence of packets that have the same set of header field values
+        (5-tuple) or have arrived on the same port. Performance results can
+        vary based on the parameters the vSwitch uses to match for a flow. The
+        recommended flow classification parameters for any vSwitch performance
+        tests are: the input port, the source IP address, the destination IP
+        address and the Ethernet protocol type field. It is essential to
+        increase the flow timeout time on a vSwitch before conducting any
+        performance tests that do not measure the flow setup time. Normally
+        the first packet of a particular stream will install the flow in the
+        virtual switch which adds an additional latency, subsequent packets of
+        the same flow are not subject to this latency if the flow is already
+        installed on the vSwitch.</t>
+      </section>
+
+      <section title="Benchmarks using Baselines with Resource Isolation">
+        <t>This outline describes measurement of baseline with isolated
+        resources at a high level, which is the intended approach at this
+        time.</t>
+
+        <t><list style="numbers">
+            <t>Baselines: <list style="symbols">
+                <t>Optional: Benchmark platform forwarding capability without
+                a vswitch or VNF for at least 72 hours (serves as a means of
+                platform validation and a means to obtain the base performance
+                for the platform in terms of its maximum forwarding rate and
+                latency). <figure>
+                    <preamble>Benchmark platform forwarding
+                    capability</preamble>
+
+                    <artwork align="right"><![CDATA[                                                      __
+ +--------------------------------------------------+   |
+ |   +------------------------------------------+   |   |
+ |   |                                          |   |   |
+ |   |          Simple Forwarding App           |   |  Host
+ |   |                                          |   |   |
+ |   +------------------------------------------+   |   |
+ |   |                 NIC                      |   |   |
+ +---+------------------------------------------+---+ __|
+            ^                           :
+            |                           |
+            :                           v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+
+                    <postamble/>
+                  </figure></t>
+
+                <t>Benchmark VNF forwarding capability with direct
+                connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72
+                hours (serves as a means of VNF validation and a means to
+                obtain the base performance for the VNF in terms of its
+                maximum forwarding rate and latency). The metrics gathered
+                from this test will serve as a key comparison point for
+                vSwitch bypass technologies performance and vSwitch
+                performance. <figure align="right">
+                    <preamble>Benchmark VNF forwarding capability</preamble>
+
+                    <artwork><![CDATA[                                                      __
+ +--------------------------------------------------+   |
+ |   +------------------------------------------+   |   |
+ |   |                                          |   |   |
+ |   |                 VNF                      |   |   |
+ |   |                                          |   |   |
+ |   +------------------------------------------+   |   |
+ |   |          Passthrough/SR-IOV              |   |  Host
+ |   +------------------------------------------+   |   |
+ |   |                 NIC                      |   |   |
+ +---+------------------------------------------+---+ __|
+            ^                           :
+            |                           |
+            :                           v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+
+                    <postamble/>
+                  </figure></t>
+
+                <t>Benchmarking with isolated resources alone, with other
+                resources (both HW&amp;SW) disabled Example, vSw and VM are
+                SUT</t>
+
+                <t>Benchmarking with isolated resources alone, leaving some
+                resources unused</t>
+
+                <t>Benchmark with isolated resources and all resources
+                occupied</t>
+              </list></t>
+
+            <t>Next Steps<list style="symbols">
+                <t>Limited sharing</t>
+
+                <t>Production scenarios</t>
+
+                <t>Stressful scenarios</t>
+              </list></t>
+          </list></t>
+      </section>
+    </section>
+
+    <section title="VSWITCHPERF Specification Summary">
+      <t>The overall specification in preparation is referred to as a Level
+      Test Design (LTD) document, which will contain a suite of performance
+      tests. The base performance tests in the LTD are based on the
+      pre-existing specifications developed by BMWG to test the performance of
+      physical switches. These specifications include:</t>
+
+      <t><list style="symbols">
+          <t><xref target="RFC2544"/> Benchmarking Methodology for Network
+          Interconnect Devices</t>
+
+          <t><xref target="RFC2889"/> Benchmarking Methodology for LAN
+          Switching</t>
+
+          <t><xref target="RFC6201"/> Device Reset Characterization</t>
+
+          <t><xref target="RFC5481"/> Packet Delay Variation Applicability
+          Statement</t>
+        </list></t>
+
+      <t>Some of the above/newer RFCs are being applied in benchmarking for
+      the first time, and represent a development challenge for test equipment
+      developers. Fortunately, many members of the testing system community
+      have engaged on the VSPERF project, including an open source test
+      system.</t>
+
+      <t>In addition to this, the LTD also re-uses the terminology defined
+      by:</t>
+
+      <t><list style="symbols">
+          <t><xref target="RFC2285"/> Benchmarking Terminology for LAN
+          Switching Devices</t>
+
+          <t><xref target="RFC5481"/> Packet Delay Variation Applicability
+          Statement</t>
+        </list></t>
+
+      <t/>
+
+      <t>Specifications to be included in future updates of the LTD
+      include:<list style="symbols">
+          <t><xref target="RFC3918"/> Methodology for IP Multicast
+          Benchmarking</t>
+
+          <t><xref target="RFC4737"/> Packet Reordering Metrics</t>
+        </list></t>
+
+      <t>As one might expect, the most fundamental internetworking
+      characteristics of Throughput and Latency remain important when the
+      switch is virtualized, and these benchmarks figure prominently in the
+      specification.</t>
+
+      <t>When considering characteristics important to "telco" network
+      functions, we must begin to consider additional performance metrics. In
+      this case, the project specifications have referenced metrics from the
+      IETF IP Performance Metrics (IPPM) literature. This means that the <xref
+      target="RFC2544"/> test of Latency is replaced by measurement of a
+      metric derived from IPPM's <xref target="RFC2679"/>, where a set of
+      statistical summaries will be provided (mean, max, min, etc.). Further
+      metrics planned to be benchmarked include packet delay variation as
+      defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
+      availability, DUT capacity and packet loss in long term testing at
+      Throughput level, where some low-level of background loss may be present
+      and characterized.</t>
+
+      <t>Tests have been (or will be) designed to collect the metrics
+      below:</t>
+
+      <t><list style="symbols">
+          <t>Throughput Tests to measure the maximum forwarding rate (in
+          frames per second or fps) and bit rate (in Mbps) for a constant load
+          (as defined by <xref target="RFC1242"/>) without traffic loss.</t>
+
+          <t>Packet and Frame Delay Distribution Tests to measure average, min
+          and max packet and frame delay for constant loads.</t>
+
+          <t>Packet Delay Tests to understand latency distribution for
+          different packet sizes and over an extended test run to uncover
+          outliers.</t>
+
+          <t>Scalability Tests to understand how the virtual switch performs
+          as the number of flows, active ports, complexity of the forwarding
+          logic&rsquo;s configuration&hellip; it has to deal with
+          increases.</t>
+
+          <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
+          performance, i.e. how fast systems can send and receive data through
+          the switch.</t>
+
+          <t>Control Path and Datapath Coupling Tests, to understand how
+          closely coupled the datapath and the control path are as well as the
+          effect of this coupling on the performance of the DUT (example:
+          delay of the initial packet of a flow).</t>
+
+          <t>CPU and Memory Consumption Tests to understand the virtual
+          switch&rsquo;s footprint on the system, usually conducted as
+          auxiliary measurements with benchmarks above. They include: CPU
+          utilization, Cache utilization and Memory footprint.</t>
+
+          <t>The so-called "Soak" tests, where the selected test is conducted
+          over a long period of time (with an ideal duration of 24 hours, and
+          at least 6 hours). The purpose of soak tests is to capture transient
+          changes in performance which may occur due to infrequent processes
+          or the low probability coincidence of two or more processes. The
+          performance must be evaluated periodically during continuous
+          testing, and this results in use of <xref target="RFC2889"/> Frame
+          Rate metrics instead of <xref target="RFC2544"/> Throughput (which
+          requires stopping traffic to allow time for all traffic to exit
+          internal queues).</t>
+        </list></t>
+
+      <t>Future/planned test specs include:<list style="symbols">
+          <t>Request/Response Performance Tests (TCP, UDP) which measure the
+          transaction rate through the switch.</t>
+
+          <t>Noisy Neighbour Tests, to understand the effects of resource
+          sharing on the performance of a virtual switch.</t>
+
+          <t>Tests derived from examination of ETSI NFV Draft GS IFA003
+          requirements <xref target="IFA003"/> on characterization of
+          acceleration technologies applied to vswitches.</t>
+        </list>The flexibility of deployment of a virtual switch within a
+      network means that the BMWG IETF existing literature needs to be used to
+      characterize the performance of a switch in various deployment
+      scenarios. The deployment scenarios under consideration include:</t>
+
+      <t><figure>
+          <preamble>Physical port to virtual switch to physical
+          port</preamble>
+
+          <artwork><![CDATA[                                                      __
+ +--------------------------------------------------+   |
+ |              +--------------------+              |   |
+ |              |                    |              |   |
+ |              |                    v              |   |  Host
+ |   +--------------+            +--------------+   |   |
+ |   |   phy port   |  vSwitch   |   phy port   |   |   |
+ +---+--------------+------------+--------------+---+ __|
+            ^                           :
+            |                           |
+            :                           v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+        </figure></t>
+
+      <t><figure>
+          <preamble>Physical port to virtual switch to VNF to virtual switch
+          to physical port</preamble>
+
+          <artwork><![CDATA[                                                      __
+ +---------------------------------------------------+   |
+ |                                                   |   |
+ |   +-------------------------------------------+   |   |
+ |   |                 Application               |   |   |
+ |   +-------------------------------------------+   |   |
+ |       ^                                  :        |   |
+ |       |                                  |        |   |  Guest
+ |       :                                  v        |   |
+ |   +---------------+           +---------------+   |   |
+ |   | logical port 0|           | logical port 1|   |   |
+ +---+---------------+-----------+---------------+---+ __|
+         ^                                  :
+         |                                  |
+         :                                  v         __
+ +---+---------------+----------+---------------+---+   |
+ |   | logical port 0|          | logical port 1|   |   |
+ |   +---------------+          +---------------+   |   |
+ |       ^                                  :       |   |
+ |       |                                  |       |   |  Host
+ |       :                                  v       |   |
+ |   +--------------+            +--------------+   |   |
+ |   |   phy port   |  vSwitch   |   phy port   |   |   |
+ +---+--------------+------------+--------------+---+ __|
+            ^                           :
+            |                           |
+            :                           v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+        </figure><figure>
+          <preamble>Physical port to virtual switch to VNF to virtual switch
+          to VNF to virtual switch to physical port</preamble>
+
+          <artwork><![CDATA[                                                   __
+ +----------------------+  +----------------------+  |
+ |   Guest 1            |  |   Guest 2            |  |
+ |   +---------------+  |  |   +---------------+  |  |
+ |   |  Application  |  |  |   |  Application  |  |  |
+ |   +---------------+  |  |   +---------------+  |  |
+ |       ^       |      |  |       ^       |      |  |
+ |       |       v      |  |       |       v      |  |  Guests
+ |   +---------------+  |  |   +---------------+  |  |
+ |   | logical ports |  |  |   | logical ports |  |  |
+ |   |   0       1   |  |  |   |   0       1   |  |  |
+ +---+---------------+--+  +---+---------------+--+__|
+         ^       :                 ^       :
+         |       |                 |       |
+         :       v                 :       v       _
+ +---+---------------+---------+---------------+--+ |
+ |   |   0       1   |         |   3       4   |  | |
+ |   | logical ports |         | logical ports |  | |
+ |   +---------------+         +---------------+  | |
+ |       ^       |                 ^       |      | |  Host
+ |       |       |-----------------|       v      | |
+ |   +--------------+          +--------------+   | |
+ |   |   phy ports  | vSwitch  |   phy ports  |   | |
+ +---+--------------+----------+--------------+---+_|
+         ^                                 :
+         |                                 |
+         :                                 v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+        </figure><figure>
+          <preamble>Physical port to virtual switch to VNF</preamble>
+
+          <artwork><![CDATA[                                                       __
+ +---------------------------------------------------+   |
+ |                                                   |   |
+ |   +-------------------------------------------+   |   |
+ |   |                 Application               |   |   |
+ |   +-------------------------------------------+   |   |
+ |       ^                                           |   |
+ |       |                                           |   |  Guest
+ |       :                                           |   |
+ |   +---------------+                               |   |
+ |   | logical port 0|                               |   |
+ +---+---------------+-------------------------------+ __|
+         ^
+         |
+         :                                            __
+ +---+---------------+------------------------------+   |
+ |   | logical port 0|                              |   |
+ |   +---------------+                              |   |
+ |       ^                                          |   |
+ |       |                                          |   |  Host
+ |       :                                          |   |
+ |   +--------------+                               |   |
+ |   |   phy port   |  vSwitch                      |   |
+ +---+--------------+------------ -------------- ---+ __|
+            ^
+            |
+            :
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+        </figure><figure>
+          <preamble>VNF to virtual switch to physical port</preamble>
+
+          <artwork><![CDATA[                                                       __
+ +---------------------------------------------------+   |
+ |                                                   |   |
+ |   +-------------------------------------------+   |   |
+ |   |                 Application               |   |   |
+ |   +-------------------------------------------+   |   |
+ |                                          :        |   |
+ |                                          |        |   |  Guest
+ |                                          v        |   |
+ |                               +---------------+   |   |
+ |                               | logical port  |   |   |
+ +-------------------------------+---------------+---+ __|
+                                            :
+                                            |
+                                            v         __
+ +------------------------------+---------------+---+   |
+ |                              | logical port  |   |   |
+ |                              +---------------+   |   |
+ |                                          :       |   |
+ |                                          |       |   |  Host
+ |                                          v       |   |
+ |                               +--------------+   |   |
+ |                     vSwitch   |   phy port   |   |   |
+ +-------------------------------+--------------+---+ __|
+                                        :
+                                        |
+                                        v
+ +--------------------------------------------------+
+ |                                                  |
+ |                traffic generator                 |
+ |                                                  |
+ +--------------------------------------------------+]]></artwork>
+        </figure><figure>
+          <preamble>VNF to virtual switch to VNF</preamble>
+
+          <artwork><![CDATA[                                                   __
+ +----------------------+  +----------------------+  |
+ |   Guest 1            |  |   Guest 2            |  |
+ |   +---------------+  |  |   +---------------+  |  |
+ |   |  Application  |  |  |   |  Application  |  |  |
+ |   +---------------+  |  |   +---------------+  |  |
+ |              |       |  |       ^              |  |
+ |              v       |  |       |              |  |  Guests
+ |   +---------------+  |  |   +---------------+  |  |
+ |   | logical ports |  |  |   | logical ports |  |  |
+ |   |           0   |  |  |   |   0           |  |  |
+ +---+---------------+--+  +---+---------------+--+__|
+                 :                 ^
+                 |                 |
+                 v                 :               _
+ +---+---------------+---------+---------------+--+ |
+ |   |           1   |         |   1           |  | |
+ |   | logical ports |         | logical ports |  | |
+ |   +---------------+         +---------------+  | |
+ |               |                 ^              | |  Host
+ |               L-----------------+              | |
+ |                                                | |
+ |                    vSwitch                     | |
+ +------------------------------------------------+_|]]></artwork>
+        </figure></t>
+
+      <t>A set of Deployment Scenario figures is available on the VSPERF Test
+      Methodology Wiki page <xref target="TestTopo"/>.</t>
+    </section>
+
+    <section title="3x3 Matrix Coverage">
+      <t>This section organizes the many existing test specifications into the
+      "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>).
+      Because the LTD specification ID names are quite long, this section is
+      organized into lists for each occupied cell of the matrix (not all are
+      occupied, also the matrix has grown to 3x4 to accommodate scale metrics
+      when displaying the coverage of many metrics/benchmarks). The current
+      version of the LTD specification is available <xref target="LTD"/>.</t>
+
+      <t>The tests listed below assess the activation of paths in the data
+      plane, rather than the control plane.</t>
+
+      <t>A complete list of tests with short summaries is available on the
+      VSPERF "LTD Test Spec Overview" Wiki page <xref target="LTDoverV"/>.</t>
+
+      <section title="Speed of Activation">
+        <t><list style="symbols">
+            <t>Activation.RFC2889.AddressLearningRate</t>
+
+            <t>PacketLatency.InitialPacketProcessingLatency</t>
+          </list></t>
+      </section>
+
+      <section title="Accuracy of Activation section">
+        <t><list style="symbols">
+            <t>CPDP.Coupling.Flow.Addition</t>
+          </list></t>
+      </section>
+
+      <section title="Reliability of Activation">
+        <t><list style="symbols">
+            <t>Throughput.RFC2544.SystemRecoveryTime</t>
+
+            <t>Throughput.RFC2544.ResetTime</t>
+          </list></t>
+      </section>
+
+      <section title="Scale of Activation">
+        <t><list style="symbols">
+            <t>Activation.RFC2889.AddressCachingCapacity</t>
+          </list></t>
+      </section>
+
+      <section title="Speed of Operation">
+        <t><list style="symbols">
+            <t>Throughput.RFC2544.PacketLossRate</t>
+
+            <t>CPU.RFC2544.0PacketLoss</t>
+
+            <t>Throughput.RFC2544.PacketLossRateFrameModification</t>
+
+            <t>Throughput.RFC2544.BackToBackFrames</t>
+
+            <t>Throughput.RFC2889.MaxForwardingRate</t>
+
+            <t>Throughput.RFC2889.ForwardPressure</t>
+
+            <t>Throughput.RFC2889.BroadcastFrameForwarding</t>
+          </list></t>
+      </section>
+
+      <section title="Accuracy of Operation">
+        <t><list style="symbols">
+            <t>Throughput.RFC2889.ErrorFramesFiltering</t>
+
+            <t>Throughput.RFC2544.Profile</t>
+          </list></t>
+      </section>
+
+      <section title="Reliability of Operation">
+        <t><list style="symbols">
+            <t>Throughput.RFC2889.Soak</t>
+
+            <t>Throughput.RFC2889.SoakFrameModification</t>
+
+            <t>PacketDelayVariation.RFC3393.Soak</t>
+          </list></t>
+      </section>
+
+      <section title="Scalability of Operation">
+        <t><list style="symbols">
+            <t>Scalability.RFC2544.0PacketLoss</t>
+
+            <t>MemoryBandwidth.RFC2544.0PacketLoss.Scalability</t>
+          </list></t>
+      </section>
+
+      <section title="Summary">
+        <t><figure>
+            <artwork><![CDATA[|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|               |   SPEED     |  ACCURACY  |  RELIABILITY  |    SCALE    |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|  Activation   |      X      |     X      |       X       |      X      |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+|  Operation    |      X      |      X     |       X       |      X      |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|
+|               |             |            |               |             |
+| De-activation |             |            |               |             |
+|               |             |            |               |             |
+|------------------------------------------------------------------------|]]></artwork>
+          </figure></t>
+      </section>
+    </section>
+
+    <section title="Security Considerations">
+      <t>Benchmarking activities as described in this memo are limited to
+      technology characterization of a Device Under Test/System Under Test
+      (DUT/SUT) using controlled stimuli in a laboratory environment, with
+      dedicated address space and the constraints specified in the sections
+      above.</t>
+
+      <t>The benchmarking network topology will be an independent test setup
+      and MUST NOT be connected to devices that may forward the test traffic
+      into a production network, or misroute traffic to the test management
+      network.</t>
+
+      <t>Further, benchmarking is performed on a "black-box" basis, relying
+      solely on measurements observable external to the DUT/SUT.</t>
+
+      <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+      benchmarking purposes. Any implications for network security arising
+      from the DUT/SUT SHOULD be identical in the lab and in production
+      networks.</t>
+    </section>
+
+    <section anchor="IANA" title="IANA Considerations">
+      <t>No IANA Action is requested at this time.</t>
+    </section>
+
+    <section title="Acknowledgements">
+      <t>The authors appreciate and acknowledge comments from Scott Bradner,
+      Marius Georgescu, Ramki Krishnan, Doug Montgomery, Martin Klozik,
+      Christian Trautman, and others for their reviews.</t>
+    </section>
+  </middle>
+
+  <back>
+    <references title="Normative References">
+      <?rfc ?>
+
+      <?rfc include="reference.RFC.2119"?>
+
+      <?rfc ?>
+
+      <?rfc include="reference.RFC.2330"?>
+
+      <?rfc include='reference.RFC.2544'?>
+
+      <?rfc include="reference.RFC.2679"?>
+
+      <?rfc include='reference.RFC.2680'?>
+
+      <?rfc include='reference.RFC.3393'?>
+
+      <?rfc include='reference.RFC.3432'?>
+
+      <?rfc include='reference.RFC.2681'?>
+
+      <?rfc include='reference.RFC.5905'?>
+
+      <?rfc include='reference.RFC.4689'?>
+
+      <?rfc include='reference.RFC.4737'?>
+
+      <?rfc include='reference.RFC.5357'?>
+
+      <?rfc include='reference.RFC.2889'?>
+
+      <?rfc include='reference.RFC.3918'?>
+
+      <?rfc include='reference.RFC.6201'?>
+
+      <?rfc include='reference.RFC.2285'?>
+
+      <reference anchor="NFV.PER001">
+        <front>
+          <title>Network Function Virtualization: Performance and Portability
+          Best Practices</title>
+
+          <author fullname="ETSI NFV" initials="" surname="">
+            <organization/>
+          </author>
+
+          <date month="June" year="2014"/>
+        </front>
+
+        <seriesInfo name="Group Specification"
+                    value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
+
+        <format type="PDF"/>
+      </reference>
+    </references>
+
+    <references title="Informative References">
+      <?rfc include='reference.RFC.1242'?>
+
+      <?rfc include='reference.RFC.5481'?>
+
+      <?rfc include='reference.RFC.6049'?>
+
+      <?rfc include='reference.RFC.6248'?>
+
+      <?rfc include='reference.RFC.6390'?>
+
+      <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?>
+
+      <?rfc include='reference.I-D.huang-bmwg-virtual-network-performance'?>
+
+      <reference anchor="TestTopo">
+        <front>
+          <title>Test Topologies
+          https://wiki.opnfv.org/vsperf/test_methodology</title>
+
+          <author>
+            <organization/>
+          </author>
+
+          <date/>
+        </front>
+      </reference>
+
+      <reference anchor="LTDoverV">
+        <front>
+          <title>LTD Test Spec Overview
+          https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review</title>
+
+          <author>
+            <organization/>
+          </author>
+
+          <date/>
+        </front>
+      </reference>
+
+      <reference anchor="LTD">
+        <front>
+          <title>LTD Test Specification
+          http://artifacts.opnfv.org/vswitchperf/docs/requirements/index.html</title>
+
+          <author>
+            <organization/>
+          </author>
+
+          <date/>
+        </front>
+      </reference>
+
+      <reference anchor="BrahRel">
+        <front>
+          <title>Brahmaputra, Second OPNFV Release
+          https://www.opnfv.org/brahmaputra</title>
+
+          <author>
+            <organization/>
+          </author>
+
+          <date/>
+        </front>
+      </reference>
+
+      <reference anchor="IFA003">
+        <front>
+          <title>https://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA003_Acceleration_-_vSwitch_Spec/</title>
+
+          <author>
+            <organization/>
+          </author>
+
+          <date/>
+        </front>
+      </reference>
+    </references>
+  </back>
+</rfc>