diff options
Diffstat (limited to 'docs/ietf_draft')
| -rw-r--r-- | docs/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml | 907 |
1 files changed, 907 insertions, 0 deletions
diff --git a/docs/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml b/docs/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml new file mode 100644 index 00000000..d8351957 --- /dev/null +++ b/docs/ietf_draft/draft-vsperf-bmwg-vswitch-opnfv-00.xml @@ -0,0 +1,907 @@ +<?xml version="1.0" encoding="US-ASCII"?> +<!DOCTYPE rfc SYSTEM "rfc2629.dtd"> +<?rfc toc="yes"?> +<?rfc tocompact="yes"?> +<?rfc tocdepth="3"?> +<?rfc tocindent="yes"?> +<?rfc symrefs="yes"?> +<?rfc sortrefs="yes"?> +<?rfc comments="yes"?> +<?rfc inline="yes"?> +<?rfc compact="yes"?> +<?rfc subcompact="no"?> +<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-00" + ipr="trust200902"> + <front> + <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in + OPNFV</title> + + <author fullname="Maryam Tahhan" initials="M." surname="Tahhan"> + <organization>Intel</organization> + + <address> + <postal> + <street/> + + <city/> + + <region/> + + <code/> + + <country/> + </postal> + + <phone/> + + <facsimile/> + + <email>maryam.tahhan@intel.com</email> + + <uri/> + </address> + </author> + + <author fullname="Billy O'Mahony" initials="B." surname="O'Mahony"> + <organization>Intel</organization> + + <address> + <postal> + <street/> + + <city/> + + <region/> + + <code/> + + <country/> + </postal> + + <phone/> + + <facsimile/> + + <email>billy.o.mahony@intel.com</email> + + <uri/> + </address> + </author> + + <author fullname="Al Morton" initials="A." surname="Morton"> + <organization>AT&T Labs</organization> + + <address> + <postal> + <street>200 Laurel Avenue South</street> + + <city>Middletown,</city> + + <region>NJ</region> + + <code>07748</code> + + <country>USA</country> + </postal> + + <phone>+1 732 420 1571</phone> + + <facsimile>+1 732 368 1192</facsimile> + + <email>acmorton@att.com</email> + + <uri>http://home.comcast.net/~acmacm/</uri> + </address> + </author> + + <date day="3" month="July" year="2015"/> + + <abstract> + <t>This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance "VSWITCHPERF". This project + intends to build on the current and completed work of the Benchmarking + Methodology Working Group in IETF, by referencing existing literature. + The Benchmarking Methodology Working Group has traditionally conducted + laboratory characterization of dedicated physical implementations of + internetworking functions. Therefore, this memo begins to describe the + additional considerations when virtual switches are implemented in + general-purpose hardware. The expanded tests and benchmarks are also + influenced by the OPNFV mission to support virtualization of the "telco" + infrastructure.</t> + </abstract> + + <note title="Requirements Language"> + <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in <xref + target="RFC2119">RFC 2119</xref>.</t> + + <t/> + </note> + </front> + + <middle> + <section title="Introduction"> + <t>Benchmarking Methodology Working Group (BMWG) has traditionally + conducted laboratory characterization of dedicated physical + implementations of internetworking functions. The Black-box Benchmarks + of Throughput, Latency, Forwarding Rates and others have served our + industry for many years. Now, Network Function Virtualization (NFV) has + the goal to transform how internetwork functions are implemented, and + therefore has garnered much attention.</t> + + <t>This memo describes the progress of the Open Platform for NFV (OPNFV) + project on virtual switch performance characterization, "VSWITCHPERF". + This project intends to build on the current and completed work of the + Benchmarking Methodology Working Group in IETF, by referencing existing + literature. For example, currently the most referenced RFC is <xref + target="RFC2544"/> (which depends on <xref target="RFC1242"/>) and + foundation of the benchmarking work in OPNFV is common and strong.</t> + + <t>See + https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases + for more background, and the OPNFV website for general information: + https://www.opnfv.org/</t> + + <t>The authors note that OPNFV distinguishes itself from other open + source compute and networking projects through its emphasis on existing + "telco" services as opposed to cloud-computing. There are many ways in + which telco requirements have different emphasis on performance + dimensions when compared to cloud computing: support for and transfer of + isochronous media streams is one example.</t> + + <t>Note also that the move to NFV Infrastructure has resulted in many + new benchmarking initiatives across the industry, and the authors are + currently doing their best to maintain alignment with many other + projects, and this Internet Draft is evidence of the efforts.</t> + </section> + + <section title="Scope"> + <t>The primary purpose and scope of the memo is to inform BMWG of + work-in-progress that builds on the body of extensive literature and + experience. Additionally, once the initial information conveyed here is + received, this memo may be expanded to include more detail and + commentary from both BMWG and OPNFV communities, under BMWG's chartered + work to characterize the NFV Infrastructure (a virtual switch is an + important aspect of that infrastructure).</t> + </section> + + <section title="Benchmarking Considerations"> + <t>This section highlights some specific considerations (from <xref + target="I-D.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual + switches. The OPNFV project is sharing its present view on these areas, + as they develop their specifications in the Level Test Design (LTD) + document.</t> + + <section title="Comparison with Physical Network Functions"> + <t>To compare the performance of virtual designs and implementations + with their physical counterparts, identical benchmarks are needed. + BMWG has developed specifications for many network functions this memo + re-uses existing benchmarks through references, and expands them + during development of new methods. A key configuration aspect is the + number of parallel cores required to achieve comparable performance + with a given physical device, or whether some limit of scale was + reached before the cores could achieve the comparable level.</t> + + <t>It's unlikely that the virtual switch will be the only application + running on the SUT, so CPU utilization, Cache utilization, and Memory + footprint should also be recorded for the virtual implementations of + internetworking functions.</t> + </section> + + <section title="Continued Emphasis on Black-Box Benchmarks"> + <t>External observations remain essential as the basis for Benchmarks. + Internal observations with fixed specification and interpretation will + be provided in parallel to assist the development of operations + procedures when the technology is deployed.</t> + </section> + + <section title="New Configuration Parameters"> + <t>A key consideration when conducting any sort of benchmark is trying + to ensure the consistency and repeatability of test results. When + benchmarking the performance of a vSwitch there are many factors that + can affect the consistency of results, one key factor is matching the + various hardware and software details of the SUT. This section lists + some of the many new parameters which this project believes are + critical to report in order to achieve repeatability.</t> + + <t>Hardware details including:</t> + + <t><list style="symbols"> + <t>Platform details</t> + + <t>Processor details</t> + + <t>Memory information (type and size)</t> + + <t>Number of enabled cores</t> + + <t>Number of cores used for the test</t> + + <t>Number of physical NICs, as well as their details + (manufacturer, versions, type and the PCI slot they are plugged + into)</t> + + <t>NIC interrupt configuration</t> + + <t>BIOS version, release date and any configurations that were + modified</t> + + <t>CPU microcode level</t> + + <t>Memory DIMM configurations (quad rank performance may not be + the same as dual rank) in size, freq and slot locations</t> + + <t>PCI configuration parameters (payload size, early ack + option...)</t> + + <t>Power management at all levels (ACPI sleep states, processor + package, OS...)</t> + </list>Software details including:</t> + + <t><list style="symbols"> + <t>OS parameters and behavior (text vs graphical no one typing at + the console on one system)</t> + + <t>OS version (for host and VNF)</t> + + <t>Kernel version (for host and VNF)</t> + + <t>GRUB boot parameters (for host and VNF)</t> + + <t>Hypervisor details (Type and version)</t> + + <t>Selected vSwitch, version number or commit id used</t> + + <t>vSwitch launch command line if it has been parameterised</t> + + <t>Memory allocation to the vSwitch</t> + + <t>which NUMA node it is using, and how many memory channels</t> + + <t>DPDK or any other SW dependency version number or commit id + used</t> + + <t>Memory allocation to a VM - if it's from Hugpages/elsewhere</t> + + <t>VM storage type: snapshot/independent persistent/independent + non-persistent</t> + + <t>Number of VMs</t> + + <t>Number of Virtual NICs (vNICs), versions, type and driver</t> + + <t>Number of virtual CPUs and their core affinity on the host</t> + + <t>Number vNIC interrupt configuration</t> + + <t>Thread affinitization for the applications (including the + vSwitch itself) on the host</t> + + <t>Details of Resource isolation, such as CPUs designated for + Host/Kernel (isolcpu) and CPUs designated for specific processes + (taskset). - Test duration. - Number of flows.</t> + </list></t> + + <t>Test Traffic Information:<list style="symbols"> + <t>Traffic type - UDP, TCP, IMIX / Other</t> + + <t>Packet Sizes</t> + + <t>Deployment Scenario</t> + </list></t> + + <t/> + </section> + + <section title="Flow classification"> + <t>Virtual switches group packets into flows by processing and + matching particular packet or frame header information, or by matching + packets based on the input ports. Thus a flow can be thought of a + sequence of packets that have the same set of header field values or + have arrived on the same port. Performance results can vary based on + the parameters the vSwitch uses to match for a flow. The recommended + flow classification parameters for any vSwitch performance tests are: + the input port, the source IP address, the destination IP address and + the ethernet protocol type field. It is essential to increase the flow + timeout time on a vSwitch before conducting any performance tests that + do not measure the flow setup time. Normally the first packet of a + particular stream will install the flow in the virtual switch which + adds an additional latency, subsequent packets of the same flow are + not subject to this latency if the flow is already installed on the + vSwitch.</t> + </section> + + <section title="Benchmarks using Baselines with Resource Isolation"> + <t>This outline describes measurement of baseline with isolated + resources at a high level, which is the intended approach at this + time.</t> + + <t><list style="numbers"> + <t>Baselines: <list style="symbols"> + <t>Optional: Benchmark platform forwarding capability without + a vswitch or VNF for at least 72 hours (serves as a means of + platform validation and a means to obtain the base performance + for the platform in terms of its maximum forwarding rate and + latency). <figure> + <preamble>Benchmark platform forwarding + capability</preamble> + + <artwork align="right"><![CDATA[ __ + +--------------------------------------------------+ | + | +------------------------------------------+ | | + | | | | | + | | Simple Forwarding App | | Host + | | | | | + | +------------------------------------------+ | | + | | NIC | | | + +---+------------------------------------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + + <postamble/> + </figure></t> + + <t>Benchmark VNF forwarding capability with direct + connectivity (vSwitch bypass, e.g., SR/IOV) for at least 72 + hours (serves as a means of VNF validation and a means to + obtain the base performance for the VNF in terms of its + maximum forwarding rate and latency). The metrics gathered + from this test will serve as a key comparison point for + vSwitch bypass technologies performance and vSwitch + performance. <figure align="right"> + <preamble>Benchmark VNF forwarding capability</preamble> + + <artwork><![CDATA[ __ + +--------------------------------------------------+ | + | +------------------------------------------+ | | + | | | | | + | | VNF | | | + | | | | | + | +------------------------------------------+ | | + | | Passthrough/SR-IOV | | Host + | +------------------------------------------+ | | + | | NIC | | | + +---+------------------------------------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + + <postamble/> + </figure></t> + + <t>Benchmarking with isolated resources alone, with other + resources (both HW&SW) disabled Example, vSw and VM are + SUT</t> + + <t>Benchmarking with isolated resources alone, leaving some + resources unused</t> + + <t>Benchmark with isolated resources and all resources + occupied</t> + </list></t> + + <t>Next Steps<list style="symbols"> + <t>Limited sharing</t> + + <t>Production scenarios</t> + + <t>Stressful scenarios</t> + </list></t> + </list></t> + </section> + </section> + + <section title="VSWITCHPERF Specification Summary"> + <t>The overall specification in preparation is referred to as a Level + Test Design (LTD) document, which will contain a suite of performance + tests. The base performance tests in the LTD are based on the + pre-existing specifications developed by BMWG to test the performance of + physical switches. These specifications include:</t> + + <t><list style="symbols"> + <t><xref target="RFC2544"/> Benchmarking Methodology for Network + Interconnect Devices</t> + + <t><xref target="RFC2889"/> Benchmarking Methodology for LAN + Switching</t> + + <t><xref target="RFC6201"/> Device Reset Characterization</t> + + <t><xref target="RFC5481"/> Packet Delay Variation Applicability + Statement</t> + </list></t> + + <t/> + + <t>In addition to this, the LTD also re-uses the terminology defined + by:</t> + + <t><list style="symbols"> + <t><xref target="RFC2285"/> Benchmarking Terminology for LAN + Switching Devices</t> + + <t><xref target="RFC5481"/> Packet Delay Variation Applicability + Statement</t> + </list></t> + + <t/> + + <t>Specifications to be included in future updates of the LTD + include:<list style="symbols"> + <t><xref target="RFC3918"/> Methodology for IP Multicast + Benchmarking</t> + + <t><xref target="RFC4737"/> Packet Reordering Metrics</t> + </list></t> + + <t>As one might expect, the most fundamental internetworking + characteristics of Throughput and Latency remain important when the + switch is virtualized, and these benchmarks figure prominently in the + specification.</t> + + <t>When considering characteristics important to "telco" network + functions, we must begin to consider additional performance metrics. In + this case, the project specifications have referenced metrics from the + IETF IP Performance Metrics (IPPM) literature. This means that the <xref + target="RFC2544"/> test of Latency is replaced by measurement of a + metric derived from IPPM's <xref target="RFC2679"/>, where a set of + statistical summaries will be provided (mean, max, min, etc.). Further + metrics planned to be benchmarked include packet delay variation as + defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT + availability, DUT capacity and packet loss in long term testing at + Throughput level, where some low-level of background loss may be present + and characterized.</t> + + <t>Tests have been (or will be) designed to collect the metrics + below:</t> + + <t><list style="symbols"> + <t>Throughput Tests to measure the maximum forwarding rate (in + frames per second or fps) and bit rate (in Mbps) for a constant load + (as defined by RFC1242) without traffic loss.</t> + + <t>Packet and Frame Delay Distribution Tests to measure average, min + and max packet and frame delay for constant loads.</t> + + <t>Packet Delay Tests to understand latency distribution for + different packet sizes and over an extended test run to uncover + outliers.</t> + + <t>Scalability Tests to understand how the virtual switch performs + as the number of flows, active ports, complexity of the forwarding + logic’s configuration… it has to deal with + increases.</t> + + <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer + performance, i.e. how fast systems can send and receive data through + the switch.</t> + + <t>Control Path and Datapath Coupling Tests, to understand how + closely coupled the datapath and the control path are as well as the + effect of this coupling on the performance of the DUT (example: + delay of the initial packet of a flow).</t> + + <t>CPU and Memory Consumption Tests to understand the virtual + switch’s footprint on the system, usually conducted as + auxiliary measurements with benchmarks above. They include: CPU + utilization, Cache utilization and Memory footprint.</t> + </list></t> + + <t>Future/planned test specs include:<list style="symbols"> + <t>Request/Response Performance Tests (TCP, UDP) which measure the + transaction rate through the switch.</t> + + <t>Noisy Neighbour Tests, to understand the effects of resource + sharing on the performance of a virtual switch.</t> + </list>The flexibility of deployment of a virtual switch within a + network means that the BMWG IETF existing literature needs to be used to + characterize the performance of a switch in various deployment + scenarios. The deployment scenarios under consideration include:</t> + + <t><figure> + <preamble>Physical port to virtual switch to physical + port</preamble> + + <artwork><![CDATA[ __ + +--------------------------------------------------+ | + | +--------------------+ | | + | | | | | + | | v | | Host + | +--------------+ +--------------+ | | + | | phy port | vSwitch | phy port | | | + +---+--------------+------------+--------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + </figure></t> + + <t><figure> + <preamble>Physical port to virtual switch to VNF to virtual switch + to physical port</preamble> + + <artwork><![CDATA[ __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | ^ : | | + | | | | | Guest + | : v | | + | +---------------+ +---------------+ | | + | | logical port 0| | logical port 1| | | + +---+---------------+-----------+---------------+---+ __| + ^ : + | | + : v __ + +---+---------------+----------+---------------+---+ | + | | logical port 0| | logical port 1| | | + | +---------------+ +---------------+ | | + | ^ : | | + | | | | | Host + | : v | | + | +--------------+ +--------------+ | | + | | phy port | vSwitch | phy port | | | + +---+--------------+------------+--------------+---+ __| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + </figure><figure> + <preamble>Physical port to virtual switch to VNF to virtual switch + to VNF to virtual switch to physical port</preamble> + + <artwork><![CDATA[ __ + +----------------------+ +----------------------+ | + | Guest 1 | | Guest 2 | | + | +---------------+ | | +---------------+ | | + | | Application | | | | Application | | | + | +---------------+ | | +---------------+ | | + | ^ | | | ^ | | | + | | v | | | v | | Guests + | +---------------+ | | +---------------+ | | + | | logical ports | | | | logical ports | | | + | | 0 1 | | | | 0 1 | | | + +---+---------------+--+ +---+---------------+--+__| + ^ : ^ : + | | | | + : v : v _ + +---+---------------+---------+---------------+--+ | + | | 0 1 | | 3 4 | | | + | | logical ports | | logical ports | | | + | +---------------+ +---------------+ | | + | ^ | ^ | | | Host + | | |-----------------| v | | + | +--------------+ +--------------+ | | + | | phy ports | vSwitch | phy ports | | | + +---+--------------+----------+--------------+---+_| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + </figure><figure> + <preamble>Physical port to virtual switch to VNF</preamble> + + <artwork><![CDATA[ __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | ^ | | + | | | | Guest + | : | | + | +---------------+ | | + | | logical port 0| | | + +---+---------------+-------------------------------+ __| + ^ + | + : __ + +---+---------------+------------------------------+ | + | | logical port 0| | | + | +---------------+ | | + | ^ | | + | | | | Host + | : | | + | +--------------+ | | + | | phy port | vSwitch | | + +---+--------------+------------ -------------- ---+ __| + ^ + | + : + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + </figure><figure> + <preamble>VNF to virtual switch to physical port</preamble> + + <artwork><![CDATA[ __ + +---------------------------------------------------+ | + | | | + | +-------------------------------------------+ | | + | | Application | | | + | +-------------------------------------------+ | | + | : | | + | | | | Guest + | v | | + | +---------------+ | | + | | logical port | | | + +-------------------------------+---------------+---+ __| + : + | + v __ + +------------------------------+---------------+---+ | + | | logical port | | | + | +---------------+ | | + | : | | + | | | | Host + | v | | + | +--------------+ | | + | vSwitch | phy port | | | + +-------------------------------+--------------+---+ __| + : + | + v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+]]></artwork> + </figure><figure> + <preamble>VNF to virtual switch to VNF</preamble> + + <artwork><![CDATA[ __ + +----------------------+ +----------------------+ | + | Guest 1 | | Guest 2 | | + | +---------------+ | | +---------------+ | | + | | Application | | | | Application | | | + | +---------------+ | | +---------------+ | | + | | | | ^ | | + | v | | | | | Guests + | +---------------+ | | +---------------+ | | + | | logical ports | | | | logical ports | | | + | | 0 | | | | 0 | | | + +---+---------------+--+ +---+---------------+--+__| + : ^ + | | + v : _ + +---+---------------+---------+---------------+--+ | + | | 1 | | 1 | | | + | | logical ports | | logical ports | | | + | +---------------+ +---------------+ | | + | | ^ | | Host + | L-----------------+ | | + | | | + | vSwitch | | + +------------------------------------------------+_|]]></artwork> + </figure></t> + </section> + + <section title="3x3 Matrix Coverage"> + <t>This section organizes the many existing test specifications into the + "3x3" matrix (introduced in <xref target="I-D.ietf-bmwg-virtual-net"/>). + Because the LTD specification ID names are quite long, this section is + organized into lists for each occupied cell of the matrix (not all are + occupied, also the matrix has grown to 3x4 to accommodate scale + metrics).</t> + + <t>The tests listed below assess the activation of paths in the data + plane, rather than the control plane.</t> + + <t>(Editor's Note: a complete list of tests is available here: + https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review )</t> + + <section title="Speed of Activation"> + <t><list style="symbols"> + <t>Throughput.RFC2889.AddressLearningRate</t> + + <t>Throughput.RFC2889.AddressCachingCapacity</t> + + <t>PacketLatency.InitialPacketProcessingLatency</t> + + <t/> + </list></t> + </section> + + <section title="Reliability of Activation"> + <t><list style="symbols"> + <t>Throughput.RFC2544.SystemRecoveryTime</t> + + <t>Throughput.RFC2544.ResetTime</t> + </list></t> + </section> + + <section title="Scale of Activation"> + <t><list style="symbols"> + <t>Throughput.RFC2889.AddressCachingCapacity</t> + + <t/> + </list></t> + </section> + + <section title="Speed of Operation"> + <t><list style="symbols"> + <t>Throughput.RFC2544.PacketLossRate</t> + + <t>Throughput.RFC2544.PacketLossRateFrameModification</t> + + <t>Throughput.RFC2544.BackToBackFrames</t> + + <t>Throughput.RFC2889.ForwardingRate</t> + + <t>Throughput.RFC2889.ForwardPressure</t> + + <t>Throughput.RFC2889.BroadcastFrameForwarding</t> + + <t>RFC2889 Broadcast Frame Latency test</t> + </list></t> + </section> + + <section title="Accuracy of Operation"> + <t><list style="symbols"> + <t>Throughput.RFC2889.ErrorFramesFiltering</t> + + <t/> + </list></t> + </section> + + <section title="Reliability of Operation"> + <t><list style="symbols"> + <t>Throughput.RFC2544.Soak</t> + + <t>Throughput.RFC2544.SoakFrameModification</t> + + <t/> + </list></t> + </section> + + <section title="Summary"> + <t><figure> + <artwork><![CDATA[|------------------------------------------------------------------------| +| | | | | | +| | SPEED | ACCURACY | RELIABILITY | SCALE | +| | | | | | +|------------------------------------------------------------------------| +| | | | | | +| Activation | X | | X | X | +| | | | | | +|------------------------------------------------------------------------| +| | | | | | +| Operation | X | X | X | | +| | | | | | +|------------------------------------------------------------------------| +| | | | | | +| De-activation | | | | | +| | | | | | +|------------------------------------------------------------------------|]]></artwork> + </figure></t> + </section> + </section> + + <section title="Security Considerations"> + <t>Benchmarking activities as described in this memo are limited to + technology characterization of a Device Under Test/System Under Test + (DUT/SUT) using controlled stimuli in a laboratory environment, with + dedicated address space and the constraints specified in the sections + above.</t> + + <t>The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test traffic + into a production network, or misroute traffic to the test management + network.</t> + + <t>Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT.</t> + + <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks.</t> + </section> + + <section anchor="IANA" title="IANA Considerations"> + <t>No IANA Action is requested at this time.</t> + </section> + + <section title="Acknowledgements"> + <t>The authors acknowledge</t> + </section> + </middle> + + <back> + <references title="Normative References"> + <?rfc ?> + + <?rfc include="reference.RFC.2119"?> + + <?rfc include="reference.RFC.2330"?> + + <?rfc include='reference.RFC.2544'?> + + <?rfc include="reference.RFC.2679"?> + + <?rfc include='reference.RFC.2680'?> + + <?rfc include='reference.RFC.3393'?> + + <?rfc include='reference.RFC.3432'?> + + <?rfc include='reference.RFC.2681'?> + + <?rfc include='reference.RFC.5905'?> + + <?rfc include='reference.RFC.4689'?> + + <?rfc include='reference.RFC.4737'?> + + <?rfc include='reference.RFC.5357'?> + + <?rfc include='reference.RFC.2889'?> + + <?rfc include='reference.RFC.3918'?> + + <?rfc include='reference.RFC.6201'?> + + <?rfc include='reference.RFC.2285'?> + + <reference anchor="NFV.PER001"> + <front> + <title>Network Function Virtualization: Performance and Portability + Best Practices</title> + + <author fullname="ETSI NFV" initials="" surname=""> + <organization/> + </author> + + <date month="June" year="2014"/> + </front> + + <seriesInfo name="Group Specification" + value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/> + + <format type="PDF"/> + </reference> + </references> + + <references title="Informative References"> + <?rfc include='reference.RFC.1242'?> + + <?rfc include='reference.RFC.5481'?> + + <?rfc include='reference.RFC.6049'?> + + <?rfc include='reference.RFC.6248'?> + + <?rfc include='reference.RFC.6390'?> + + <?rfc include='reference.I-D.ietf-bmwg-virtual-net'?> + </references> + </back> +</rfc> |