summaryrefslogtreecommitdiffstats
path: root/docs/requirements/vswitchperf_ltd.rst
diff options
context:
space:
mode:
authorTrevor Cooper <trevor.cooper@intel.com>2017-03-22 00:49:18 +0000
committerGerrit Code Review <gerrit@opnfv.org>2017-03-22 00:49:18 +0000
commitf56bcee58ec3710b02a0f7639f13d7a8ed903ebf (patch)
treee90758d0f0ad0df6698a144c3052b9f8f0308375 /docs/requirements/vswitchperf_ltd.rst
parenta224f56b6750062078b881606092003eaa9e81eb (diff)
parentf4a955b25a59af2984b0910e5f2cb10a0d1150e5 (diff)
Merge "Revert "Moved doc files to testing document structure"
Diffstat (limited to 'docs/requirements/vswitchperf_ltd.rst')
-rw-r--r--docs/requirements/vswitchperf_ltd.rst1712
1 files changed, 1712 insertions, 0 deletions
diff --git a/docs/requirements/vswitchperf_ltd.rst b/docs/requirements/vswitchperf_ltd.rst
new file mode 100644
index 00000000..e1372520
--- /dev/null
+++ b/docs/requirements/vswitchperf_ltd.rst
@@ -0,0 +1,1712 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T and others.
+
+******************************
+VSPERF LEVEL TEST DESIGN (LTD)
+******************************
+
+.. 3.1
+
+============
+Introduction
+============
+
+The intention of this Level Test Design (LTD) document is to specify the set of
+tests to carry out in order to objectively measure the current characteristics
+of a virtual switch in the Network Function Virtualization Infrastructure
+(NFVI) as well as the test pass criteria. The detailed test cases will be
+defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_.
+
+This document is currently in draft form.
+
+.. 3.1.1
+
+
+.. _doc-id-of-LTD:
+
+Document identifier
+===================
+
+The document id will be used to uniquely
+identify versions of the LTD. The format for the document id will be:
+OPNFV\_vswitchperf\_LTD\_REL\_STATUS, where by the
+status is one of: draft, reviewed, corrected or final. The document id
+for this version of the LTD is:
+OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED.
+
+.. 3.1.2
+
+.. _scope-of-LTD:
+
+Scope
+=====
+
+The main purpose of this project is to specify a suite of
+performance tests in order to objectively measure the current packet
+transfer characteristics of a virtual switch in the NFVI. The intent of
+the project is to facilitate testing of any virtual switch. Thus, a
+generic suite of tests shall be developed, with no hard dependencies to
+a single implementation. In addition, the test case suite shall be
+architecture independent.
+
+The test cases developed in this project shall not form part of a
+separate test framework, all of these tests may be inserted into the
+Continuous Integration Test Framework and/or the Platform Functionality
+Test Framework - if a vSwitch becomes a standard component of an OPNFV
+release.
+
+.. 3.1.3
+
+References
+==========
+
+* `RFC 1242 Benchmarking Terminology for Network Interconnection
+ Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
+* `RFC 2544 Benchmarking Methodology for Network Interconnect
+ Devices <http://www.ietf.org/rfc/rfc2544.txt>`__
+* `RFC 2285 Benchmarking Terminology for LAN Switching
+ Devices <http://www.ietf.org/rfc/rfc2285.txt>`__
+* `RFC 2889 Benchmarking Methodology for LAN Switching
+ Devices <http://www.ietf.org/rfc/rfc2889.txt>`__
+* `RFC 3918 Methodology for IP Multicast
+ Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__
+* `RFC 4737 Packet Reordering
+ Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__
+* `RFC 5481 Packet Delay Variation Applicability
+ Statement <http://www.ietf.org/rfc/rfc5481.txt>`__
+* `RFC 6201 Device Reset
+ Characterization <http://tools.ietf.org/html/rfc6201>`__
+
+.. 3.2
+
+.. _details-of-LTD:
+
+================================
+Details of the Level Test Design
+================================
+
+This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and
+identifies the sets of test cases or scenarios (TestIdentification-of-LTD_).
+
+.. 3.2.1
+
+.. _FeaturesToBeTested-of-LTD:
+
+Features to be tested
+=====================
+
+Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
+includes measuring the following performance metrics:
+
+- Throughput
+- Packet delay
+- Packet delay variation
+- Packet loss
+- Burst behaviour
+- Packet re-ordering
+- Packet correctness
+- Availability and capacity of the DUT
+
+.. 3.2.2
+
+.. _TestIdentification-of-LTD:
+
+Test identification
+===================
+
+.. 3.2.2.1
+
+Throughput tests
+----------------
+
+The following tests aim to determine the maximum forwarding rate that
+can be achieved with a virtual switch. The list is not exhaustive but
+should indicate the type of tests that should be required. It is
+expected that more will be added.
+
+.. 3.2.2.1.1
+
+.. _PacketLossRatio:
+
+Test ID: LTD.Throughput.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's maximum forwarding rate with X% traffic
+ loss for a constant load (fixed length frames at a fixed interval time).
+ The default loss percentages to be tested are: - X = 0% - X = 10^-7%
+
+ Note: Other values can be tested if required by the user.
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.2.1.2
+
+.. _PacketLossRatioFrameModification:
+
+Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 X% packet loss Throughput and Latency Test with
+ packet modification
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's maximum forwarding rate with X% traffic
+ loss for a constant load (fixed length frames at a fixed interval time).
+ The default loss percentages to be tested are: - X = 0% - X = 10^-7%
+
+ Note: Other values can be tested if required by the user.
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result.
+
+ During this test, the DUT must perform the following operations on the
+ traffic flow:
+
+ - Perform packet parsing on the DUT's ingress port.
+ - Perform any relevant address look-ups on the DUT's ingress ports.
+ - Modify the packet header before forwarding the packet to the DUT's
+ egress port. Packet modifications include:
+
+ - Modifying the Ethernet source or destination MAC address.
+ - Modifying/adding a VLAN tag. (**Recommended**).
+ - Modifying/adding a MPLS tag.
+ - Modifying the source or destination ip address.
+ - Modifying the TOS/DSCP field.
+ - Modifying the source or destination ports for UDP/TCP/SCTP.
+ - Modifying the TTL.
+
+ **Expected Result**: The Packet parsing/modifications require some
+ additional degree of processing resource, therefore the
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ Throughput is expected to be somewhat lower than the Throughput level
+ measured without additional steps. The reduction is expected to be
+ greatest on tests with the smallest packet sizes (greatest header
+ processing rates).
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss and packet
+ modification operations being performed by the DUT.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
+ PDV form of delay variation on the traffic flow,
+ using the 99th percentile.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.2.1.3
+
+Test ID: LTD.Throughput.RFC2544.Profile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 Throughput and Latency Profile
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test reveals how throughput and latency degrades as the offered
+ rate varies in the region of the DUT's maximum forwarding rate as
+ determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss).
+ For example it can be used to determine if the degradation of throughput
+ and latency as the offered rate increases is slow and graceful or sudden
+ and severe.
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+
+ The offered traffic rate is described as a percentage delta with respect
+ to the DUT's RFC 2544 Throughput as determined by
+ LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
+ of 0% is equivalent to an offered traffic rate equal to the RFC 2544
+ Maximum Throughput; A delta of +50% indicates an offered rate half-way
+ between the Maximum RFC2544 Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the RFC 2544 Maximum Throughput.
+ Therefore the range of the delta figure is natuarlly bounded at -100%
+ (zero offered traffic) and +100% (traffic offered at line rate).
+
+ The following deltas to the maximum forwarding rate should be applied:
+
+ - -50%, -10%, 0%, +10% & +50%
+
+ **Expected Result**: For each packet size a profile should be produced
+ of how throughput and latency vary with offered rate.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT
+ for each delta to the maximum forwarding rate and for each frame
+ size.
+ - The average latency for each delta to the maximum forwarding rate and
+ for each frame size.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - Any failures experienced (for example if the vSwitch crashes, stops
+ processing packets, restarts or becomes unresponsive to commands)
+ when the offered load is above Maximum Throughput MUST be recorded
+ and reported with the results.
+
+.. 3.2.2.1.4
+
+Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 System Recovery Time Test
+
+ **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the length of time it takes the DUT
+ to recover from an overload condition for a constant load (fixed length
+ frames at a fixed interval time). The selected frame sizes are those
+ previously defined under :ref:`default-test-parameters`,
+ traffic should be sent to the DUT under normal conditions. During the
+ duration of the test and while the traffic flows are passing though the
+ DUT, at least one situation leading to an overload condition for the DUT
+ should occur. The time from the end of the overload condition to when
+ the DUT returns to normal operations should be measured to determine
+ recovery time. Prior to overloading the DUT, one should record the
+ average latency for 10,000 packets forwarded through the DUT.
+
+ The overload condition SHOULD be to transmit traffic at a very high
+ frame rate to the DUT (150% of the maximum 0% packet loss rate as
+ determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate
+ whichever is lower), for at least 60 seconds, then reduce the frame rate
+ to 75% of the maximum 0% packet loss rate. A number of time-stamps
+ should be recorded: - Record the time-stamp at which the frame rate was
+ reduced and record a second time-stamp at the time of the last frame
+ lost. The recovery time is the difference between the two timestamps. -
+ Record the average latency for 10,000 frames after the last frame loss
+ and continue to record average latency measurements for every 10,000
+ frames, when latency returns to within 10% of pre-overload levels record
+ the time-stamp.
+
+ **Expected Result**:
+
+ **Metrics collected**
+
+ The following are the metrics collected for this test:
+
+ - The length of time it takes the DUT to recover from an overload
+ condition.
+ - The length of time it takes the DUT to recover the average latency to
+ pre-overload conditions.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+.. 3.2.2.1.5
+
+.. _BackToBackFrames:
+
+Test ID: LTD.Throughput.RFC2544.BackToBackFrames
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2544 Back To Back Frames Test
+
+ **Prerequisite Test**: N
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to characterize the ability of the DUT to
+ process back-to-back frames. For each frame size previously defined
+ under :ref:`default-test-parameters`, a burst of traffic
+ is sent to the DUT with the minimum inter-frame gap between each frame.
+ If the number of received frames equals the number of frames that were
+ transmitted, the burst size should be increased and traffic is sent to
+ the DUT again. The value measured is the back-to-back value, that is the
+ maximum burst size the DUT can handle without any frame loss. Please note
+ a trial must run for a minimum of 2 seconds and should be repeated 50
+ times (at a minimum).
+
+ **Expected Result**:
+
+ Tests of back-to-back frames with physical devices have produced
+ unstable results in some cases. All tests should be repeated in multiple
+ test sessions and results stability should be examined.
+
+ **Metrics collected**
+
+ The following are the metrics collected for this test:
+
+ - The average back-to-back value across the trials, which is
+ the number of frames in the longest burst that the DUT will
+ handle without the loss of any frames.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+.. 3.2.2.1.6
+
+Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
+
+ **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand the Max Forwarding Rate stability
+ over an extended test duration in order to uncover any outliers. To allow
+ for an extended test duration, the test should ideally run for 24 hours
+ or, if this is not possible, for at least 6 hours. For this test, each frame
+ size must be sent at the highest Throughput rate with X% packet loss, as
+ determined in the prerequisite test. The default loss percentages to be
+ tested are: - X = 0% - X = 10^-7%
+
+ Note: Other values can be tested if required by the user.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - Max Forwarding Rate stability of the DUT.
+
+ - This means reporting the number of packets lost per time interval
+ and reporting any time intervals with packet loss. The
+ `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
+ Forwarding Rate shall be measured in each interval.
+ An interval of 60s is suggested.
+
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
+ PDV form of delay variation on the traffic flow,
+ using the 99th percentile.
+
+.. 3.2.2.1.7
+
+Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
+
+ **Prerequisite Test**:
+ LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand the Max Forwarding Rate stability over an
+ extended test duration in order to uncover any outliers. To allow for an
+ extended test duration, the test should ideally run for 24 hours or, if
+ this is not possible, for at least 6 hour. For this test, each frame
+ size must be sent at the highest Throughput rate with 0% packet loss, as
+ determined in the prerequisite test.
+
+ During this test, the DUT must perform the following operations on the
+ traffic flow:
+
+ - Perform packet parsing on the DUT's ingress port.
+ - Perform any relevant address look-ups on the DUT's ingress ports.
+ - Modify the packet header before forwarding the packet to the DUT's
+ egress port. Packet modifications include:
+
+ - Modifying the Ethernet source or destination MAC address.
+ - Modifying/adding a VLAN tag (**Recommended**).
+ - Modifying/adding a MPLS tag.
+ - Modifying the source or destination ip address.
+ - Modifying the TOS/DSCP field.
+ - Modifying the source or destination ports for UDP/TCP/SCTP.
+ - Modifying the TTL.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - Max Forwarding Rate stability of the DUT.
+
+ - This means reporting the number of packets lost per time interval
+ and reporting any time intervals with packet loss. The
+ `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
+ Forwarding Rate shall be measured in each interval.
+ An interval of 60s is suggested.
+
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
+ PDV form of delay variation on the traffic flow, using the 99th
+ percentile.
+
+.. 3.2.2.1.8
+
+Test ID: LTD.Throughput.RFC6201.ResetTime
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 6201 Reset Time Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the length of time it takes the DUT
+ to recover from a reset.
+
+ Two reset methods are defined - planned and unplanned. A planned reset
+ requires stopping and restarting the virtual switch by the usual
+ 'graceful' method defined by it's documentation. An unplanned reset
+ requires simulating a fatal internal fault in the virtual switch - for
+ example by using kill -SIGKILL on a Linux environment.
+
+ Both reset methods SHOULD be exercised.
+
+ For each frame size previously defined under :ref:`default-test-parameters`,
+ traffic should be sent to the DUT under
+ normal conditions. During the duration of the test and while the traffic
+ flows are passing through the DUT, the DUT should be reset and the Reset
+ time measured. The Reset time is the total time that a device is
+ determined to be out of operation and includes the time to perform the
+ reset and the time to recover from it (cf. `RFC6201
+ <https://www.rfc-editor.org/rfc/rfc6201.txt>`__).
+
+ `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods
+ to measure the Reset time:
+
+ - Frame-Loss Method: which requires the monitoring of the number of
+ lost frames and calculates the Reset time based on the number of
+ frames lost and the offered rate according to the following
+ formula:
+
+ .. code-block:: console
+
+ Frames_lost (packets)
+ Reset_time = -------------------------------------
+ Offered_rate (packets per second)
+
+ - Timestamp Method: which measures the time from which the last frame
+ is forwarded from the DUT to the time the first frame is forwarded
+ after the reset. This involves time-stamping all transmitted frames
+ and recording the timestamp of the last frame that was received prior
+ to the reset and also measuring the timestamp of the first frame that
+ is received after the reset. The Reset time is the difference between
+ these two timestamps.
+
+ According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the
+ choice of method depends on the test tool's capability; the Frame-Loss
+ method SHOULD be used if the test tool supports:
+
+ * Counting the number of lost frames per stream.
+ * Transmitting test frame despite the physical link status.
+
+ whereas the Timestamp method SHOULD be used if the test tool supports:
+
+ * Timestamping each frame.
+ * Monitoring received frame's timestamp.
+ * Transmitting frames only if the physical link status is up.
+
+ **Expected Result**:
+
+ **Metrics collected**
+
+ The following are the metrics collected for this test:
+
+ * Average Reset Time over the number of trials performed.
+
+ Results of this test should include the following information:
+
+ * The reset method used.
+ * Throughput in Fps and Mbps.
+ * Average Frame Loss over the number of trials performed.
+ * Average Reset Time in milliseconds over the number of trials performed.
+ * Number of trials performed.
+ * Protocol: IPv4, IPv6, MPLS, etc.
+ * Frame Size in Octets
+ * Port Media: Ethernet, Gigabit Ethernet (GbE), etc.
+ * Port Speed: 10 Gbps, 40 Gbps etc.
+ * Interface Encapsulation: Ethernet, Ethernet VLAN, etc.
+
+ **Deployment scenario**:
+
+ * Physical → virtual switch → physical.
+
+.. 3.2.2.1.9
+
+Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Forwarding Rate Test
+
+ **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
+
+ **Priority**:
+
+ **Description**:
+
+ This test measures the DUT's Max Forwarding Rate when the Offered Load
+ is varied between the throughput and the Maximum Offered Load for fixed
+ length frames at a fixed time interval. The selected frame sizes are
+ those previously defined under :ref:`default-test-parameters`.
+ The throughput is the maximum offered
+ load with 0% frame loss (measured by the prerequisite test), and the
+ Maximum Offered Load (as defined by
+ `RFC2285 <https://www.rfc-editor.org/rfc/rfc2285.txt>`__) is *"the highest
+ number of frames per second that an external source can transmit to a
+ DUT/SUT for forwarding to a specified output interface or interfaces"*.
+
+ Traffic should be sent to the DUT at a particular rate (TX rate)
+ starting with TX rate equal to the throughput rate. The rate of
+ successfully received frames at the destination counted (in FPS). If the
+ RX rate is equal to the TX rate, the TX rate should be increased by a
+ fixed step size and the RX rate measured again until the Max Forwarding
+ Rate is found.
+
+ The trial duration for each iteration should last for the period of time
+ needed for the system to reach steady state for the frame size being
+ tested. Under `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
+ (Sec. 5.6.3.1) test methodology, the test
+ duration should run for a minimum period of 30 seconds, regardless
+ whether the system reaches steady state before the minimum duration
+ ends.
+
+ **Expected Result**: According to
+ `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding
+ Rate is the highest forwarding rate of a DUT taken from an iterative set of
+ forwarding rate measurements. The iterative set of forwarding rate measurements
+ are made by setting the intended load transmitted from an external source and
+ measuring the offered load (i.e what the DUT is capable of forwarding). If the
+ Throughput == the Maximum Offered Load, it follows that Max Forwarding Rate is
+ equal to the Maximum Offered Load.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The Max Forwarding Rate for the DUT for each packet size.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical. Note: Full mesh tests with
+ multiple ingress and egress ports are a key aspect of RFC 2889
+ benchmarks, and scenarios with both 2 and 4 ports should be tested.
+ In any case, the number of ports used must be reported.
+
+.. 3.2.2.1.10
+
+Test ID: LTD.Throughput.RFC2889.ForwardPressure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Forward Pressure Test
+
+ **Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine if the DUT transmits frames with an
+ inter-frame gap that is less than 12 bytes. This test overloads the DUT
+ and measures the output for forward pressure. Traffic should be
+ transmitted to the DUT with an inter-frame gap of 11 bytes, this will
+ overload the DUT by 1 byte per frame. The forwarding rate of the DUT
+ should be measured.
+
+ **Expected Result**: The forwarding rate should not exceed the maximum
+ forwarding rate of the DUT collected by
+ LTD.Throughput.RFC2889.MaxForwardingRate.
+
+ **Metrics collected**
+
+ The following are the metrics collected for this test:
+
+ - Forwarding rate of the DUT in FPS or Mbps.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+.. 3.2.2.1.11
+
+Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Error Frames Filtering Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine whether the DUT will propagate any
+ erroneous frames it receives or whether it is capable of filtering out
+ the erroneous frames. Traffic should be sent with erroneous frames
+ included within the flow at random intervals. Illegal frames that must
+ be tested include: - Oversize Frames. - Undersize Frames. - CRC Errored
+ Frames. - Dribble Bit Errored Frames - Alignment Errored Frames
+
+ The traffic flow exiting the DUT should be recorded and checked to
+ determine if the erroneous frames where passed through the DUT.
+
+ **Expected Result**: Broken frames are not passed!
+
+ **Metrics collected**
+
+ No Metrics are collected in this test, instead it determines:
+
+ - Whether the DUT will propagate erroneous frames.
+ - Or whether the DUT will correctly filter out any erroneous frames
+ from traffic flow with out removing correct frames.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+.. 3.2.2.1.12
+
+Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Broadcast Frame Forwarding Test
+
+ **Prerequisite Test**: N
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the maximum forwarding rate of the
+ DUT when forwarding broadcast traffic. For each frame previously defined
+ under :ref:`default-test-parameters`, the traffic should
+ be set up as broadcast traffic. The traffic throughput of the DUT should
+ be measured.
+
+ The test should be conducted with at least 4 physical ports on the DUT.
+ The number of ports used MUST be recorded.
+
+ As broadcast involves forwarding a single incoming packet to several
+ destinations, the latency of a single packet is defined as the average
+ of the latencies for each of the broadcast destinations.
+
+ The incoming packet is transmitted on each of the other physical ports,
+ it is not transmitted on the port on which it was received. The test MAY
+ be conducted using different broadcasting ports to uncover any
+ performance differences.
+
+ **Expected Result**:
+
+ **Metrics collected**:
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate of the DUT when forwarding broadcast traffic.
+ - The minimum, average & maximum packets latencies observed.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch 3x physical. In the Broadcast rate testing,
+ four test ports are required. One of the ports is connected to the test
+ device, so it can send broadcast frames and listen for miss-routed frames.
+
+.. 3.2.2.1.13
+
+Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's maximum forwarding rate with X% traffic
+ loss for a constant load (fixed length frames at a fixed interval time).
+ The default loss percentages to be tested are: X = 0%, X = 10^-7%
+
+ Modified RFC 2544 throughput benchmarking methodology aims to quantify
+ the throughput measurement variations observed during standard RFC 2544
+ benchmarking measurements of virtual switches and VNFs. The RFC2544
+ binary search algorithm is modified to use more samples per test trial
+ to drive the binary search and yield statistically more meaningful
+ results. This keeps the heart of the RFC2544 methodology, still relying
+ on the binary search of throughput at specified loss tolerance, while
+ providing more useful information about the range of results seen in
+ testing. Instead of using a single traffic trial per iteration step,
+ each traffic trial is repeated N times and the success/failure of the
+ iteration step is based on these N traffic trials. Two types of revised
+ tests are defined - *Worst-of-N* and *Best-of-N*.
+
+ **Worst-of-N**
+
+ *Worst-of-N* indicates the lowest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same test run N times at a set packet rate, record each
+ result.
+ 2. Take the WORST result (highest packet loss) out of N result samples,
+ called the Worst-of-N sample.
+ 3. If Worst-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Worst-of-N sample has loss greater than the set loss tolerance
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ **Best-of-N**
+
+ *Best-of-N* indicates the highest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same traffic run N times at a set packet rate, record
+ each result.
+ 2. Take the BEST result (least packet loss) out of N result samples,
+ called the Best-of-N sample.
+ 3. If Best-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Best-of-N sample has loss greater than the set loss tolerance,
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ Performing both Worst-of-N and Best-of-N benchmark tests yields lower
+ and upper bounds of expected maximum throughput under the operating
+ conditions, giving a very good indication to the user of the
+ deterministic performance range for the tested setup.
+
+ **Expected Result**: At the end of each trial series, the presence or
+ absence of loss determines the modification of offered load for the
+ next trial series, converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput
+ with X% loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - Following may also be collected as part of this test, to determine
+ the vSwitch's performance footprint on the system:
+
+ - CPU core utilization.
+ - CPU cache utilization.
+ - Memory footprint.
+ - System bus (QPI, PCI, ...) utilization.
+ - CPU cycles consumed per packet.
+
+.. 3.2.2.1.14
+
+Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+
+ NOTE: Throughout this test, four interchangeable overlay technologies are covered by the
+ same test description. They are: VXLAN, GRE, NVGRE and GENEVE.
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+ This test evaluates standard switch performance benchmarks for the scenario where an
+ Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered
+ (replacing <tech> in the test name) include:
+
+ - VXLAN
+ - GRE
+ - NVGRE
+ - GENEVE
+
+ Performance will be assessed for each of the following overlay network functions:
+
+ - Encapsulation only
+ - De-encapsulation only
+ - Both Encapsulation and De-encapsulation
+
+ For each native packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct overlay net (tunnel) assignment
+ - Encapsulate the packet
+ - Switch the packet to the correct port
+
+ For each encapsulated packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct native network assignment
+ - De-encapsulate the packet, if required
+ - Switch the packet to the correct port
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+
+ Thus, each test comprises an overlay technology, a network function,
+ and a packet size *with* overlay network overhead included
+ (but see also the discussion at
+ https://etherpad.opnfv.org/p/vSwitchTestsDrafts ).
+
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss (where the value of X is typically equal to zero).
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.3.1.15
+
+Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test
+
+ **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the cost of carrying out match
+ action(s) on the DUT’s RFC2544 Throughput with X% traffic loss for
+ a constant load (fixed length frames at a fixed interval time).
+
+ Each test case requires:
+
+ * selection of a specific match action(s),
+ * specifying a percentage of total traffic that is elligible
+ for the match action,
+ * determination of the specific test configuration (number
+ of flows, number of test ports, presence of an external
+ controller, etc.), and
+ * measurement of the RFC 2544 Throughput level with X% packet
+ loss: Traffic shall be bi-directional and symmetric.
+
+ Note: It would be ideal to verify that all match action-elligible
+ traffic was forwarded to the correct port, and if forwarded to
+ an unintended port it should be considered lost.
+
+ A match action is an action that is typically carried on a frame
+ or packet that matches a set of flow classification parameters
+ (typically frame/packet header fields). A match action may or may
+ not modify a packet/frame. Match actions include [1]:
+
+ * output : outputs a packet to a particular port.
+ * normal: Subjects the packet to traditional L2/L3 processing
+ (MAC learning).
+ * flood: Outputs the packet on all switch physical ports
+ other than the port on which it was received and any ports
+ on which flooding is disabled.
+ * all: Outputs the packet on all switch physical ports other
+ than the port on which it was received.
+ * local: Outputs the packet on the ``local port``, which
+ corresponds to the network device that has the same name as
+ the bridge.
+ * in_port: Outputs the packet on the port from which it was
+ received.
+ * Controller: Sends the packet and its metadata to the
+ OpenFlow controller as a ``packet in`` message.
+ * enqueue: Enqueues the packet on the specified queue
+ within port.
+ * drop: discard the packet.
+
+ Modifications include [1]:
+
+ * mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification
+ * mod_dl_src: Sets the source Ethernet address.
+ * mod_dl_dst: Sets the destination Ethernet address.
+ * mod_nw_src: Sets the IPv4 source address.
+ * mod_nw_dst: Sets the IPv4 destination address.
+ * mod_tp_src: Sets the TCP or UDP or SCTP source port.
+ * mod_tp_dst: Sets the TCP or UDP or SCTP destination port.
+ * mod_nw_tos: Sets the DSCP bits in the IPv4 ToS/DSCP or
+ IPv6 traffic class field.
+ * mod_nw_ecn: Sets the ECN bits in the appropriate IPv4 or
+ IPv6 field.
+ * mod_nw_ttl: Sets the IPv4 TTL or IPv6 hop limit field.
+
+ Note: This comprehensive list requires extensive traffic generator
+ capabilities.
+
+ The match action(s) that were applied as part of the test should be
+ reported in the final test report.
+
+ During this test, the DUT must perform the following operations on
+ the traffic flow:
+
+ * Perform packet parsing on the DUT’s ingress port.
+ * Perform any relevant address look-ups on the DUT’s ingress
+ ports.
+ * Carry out one or more of the match actions specified above.
+
+ The default loss percentages to be tested are: - X = 0% - X = 10^-7%
+ Other values can be tested if required by the user. The selected
+ frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+
+ The test can also be used to determine the average latency of the
+ traffic when a match action is applied to packets in a flow. Under
+ the RFC2544 test methodology, the test duration will include a
+ number of trials; each trial should run for a minimum period of 60
+ seconds. A binary search methodology must be applied for each
+ trial to obtain the final result.
+
+ **Expected Result:**
+
+ At the end of each trial, the presence or absence of loss
+ determines the modification of offered load for the next trial,
+ converging on a maximum rate, or RFC2544Throughput with X% loss.
+ The Throughput load is re-used in related RFC2544 tests and other
+ tests.
+
+ **Metrics Collected:**
+
+ The following are the metrics collected for this test:
+
+ * The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps
+ of the DUT for each frame size with X% packet loss.
+ * The average latency of the traffic flow when passing through
+ the DUT (if testing for latency, note that this average is
+ different from the test specified in Section 26.3 ofRFC2544).
+ * CPU and memory utilization may also be collected as part of
+ this test, to determine the vSwitch’s performance footprint
+ on the system.
+
+ The metrics collected can be compared to that of the prerequisite
+ test to determine the cost of the match action(s) in the pipeline.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical (and others are possible)
+
+ [1] ovs-ofctl - administer OpenFlow switches
+ [http://openvswitch.org/support/dist-docs/ovs-ofctl.8.txt ]
+
+
+.. 3.2.2.2
+
+Packet Latency tests
+--------------------
+
+These tests will measure the store and forward latency as well as the packet
+delay variation for various packet types through the virtual switch. The
+following list is not exhaustive but should indicate the type of tests
+that should be required. It is expected that more will be added.
+
+.. 3.2.2.2.1
+
+Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: Initial Packet Processing Latency
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ In some virtual switch architectures, the first packets of a flow will
+ take the system longer to process than subsequent packets in the flow.
+ This test determines the latency for these packets. The test will
+ measure the latency of the packets as they are processed by the
+ flow-setup-path of the DUT. There are two methods for this test, a
+ recommended method and a nalternative method that can be used if it is
+ possible to disable the fastpath of the virtual switch.
+
+ Recommended method: This test will send 64,000 packets to the DUT, each
+ belonging to a different flow. Average packet latency will be determined
+ over the 64,000 packets.
+
+ Alternative method: This test will send a single packet to the DUT after
+ a fixed interval of time. The time interval will be equivalent to the
+ amount of time it takes for a flow to time out in the virtual switch
+ plus 10%. Average packet latency will be determined over 1,000,000
+ packets.
+
+ This test is intended only for non-learning virtual switches; For learning
+ virtual switches use RFC2889.
+
+ For this test, only unidirectional traffic is required.
+
+ **Expected Result**: The average latency for the initial packet of all
+ flows should be greater than the latency of subsequent traffic.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - Average latency of the initial packets of all flows that are
+ processed by the DUT.
+
+ **Deployment scenario**:
+
+ - Physical → Virtual Switch → Physical.
+
+.. 3.2.2.2.2
+
+Test ID: LTD.PacketDelayVariation.RFC3393.Soak
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: Packet Delay Variation Soak Test
+
+ **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand the distribution of packet delay
+ variation for different frame sizes over an extended test duration and
+ to determine if there are any outliers. To allow for an extended test
+ duration, the test should ideally run for 24 hours or, if this is not
+ possible, for at least 6 hour. For this test, each frame size must be
+ sent at the highest possible throughput with 0% packet loss, as
+ determined in the prerequisite test.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The packet delay variation value for traffic passing through the DUT.
+ - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
+ PDV form of delay variation on the traffic flow,
+ using the 99th percentile, for each 60s interval during the test.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.2.3
+
+Scalability tests
+-----------------
+
+The general aim of these tests is to understand the impact of large flow
+table size and flow lookups on throughput. The following list is not
+exhaustive but should indicate the type of tests that should be required.
+It is expected that more will be added.
+
+.. 3.2.2.3.1
+
+.. _Scalability0PacketLoss:
+
+Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 0% loss Flow Scalability throughput test
+
+ **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
+ delta Throughput between the single-flow RFC2544 test and this test with
+ a variable number of flows is desired.
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to measure how throughput changes as the number
+ of flows in the DUT increases. The test will measure the throughput
+ through the fastpath, as such the flows need to be installed on the DUT
+ before passing traffic.
+
+ For each frame size previously defined under :ref:`default-test-parameters`
+ and for each of the following number of flows:
+
+ - 1,000
+ - 2,000
+ - 4,000
+ - 8,000
+ - 16,000
+ - 32,000
+ - 64,000
+ - Max supported number of flows.
+
+ This test will be conducted under two conditions following the
+ establishment of all flows as required by RFC 2544, regarding the flow
+ expiration time-out:
+
+ 1) The time-out never expires during each trial.
+
+ 2) The time-out expires for all flows periodically. This would require a
+ short time-out compared with flow re-appearance for a small number of
+ flows, and may not be possible for all flow conditions.
+
+ The maximum 0% packet loss Throughput should be determined in a manner
+ identical to LTD.Throughput.RFC2544.PacketLossRatio.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum number of frames per second that can be forwarded at the
+ specified number of flows and the specified frame size, with zero
+ packet loss.
+
+.. 3.2.2.3.2
+
+Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 0% loss Memory Bandwidth Scalability test
+
+ **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
+ delta Throughput between an undisturbed RFC2544 test and this test with
+ the Throughput affected by cache and memory bandwidth contention is desired.
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand how the DUT's performance is
+ affected by cache sharing and memory bandwidth between processes.
+
+ During the test all cores not used by the vSwitch should be running a
+ memory intensive application. This application should read and write
+ random data to random addresses in unused physical memory. The random
+ nature of the data and addresses is intended to consume cache, exercise
+ main memory access (as opposed to cache) and exercise all memory buses
+ equally. Furthermore:
+
+ - the ratio of reads to writes should be recorded. A ratio of 1:1
+ SHOULD be used.
+ - the reads and writes MUST be of cache-line size and be cache-line aligned.
+ - in NUMA architectures memory access SHOULD be local to the core's node.
+ Whether only local memory or a mix of local and remote memory is used
+ MUST be recorded.
+ - the memory bandwidth (reads plus writes) used per-core MUST be recorded;
+ the test MUST be run with a per-core memory bandwidth equal to half the
+ maximum system memory bandwidth divided by the number of cores. The test
+ MAY be run with other values for the per-core memory bandwidth.
+ - the test MAY also be run with the memory intensive application running
+ on all cores.
+
+ Under these conditions the DUT's 0% packet loss throughput is determined
+ as per LTD.Throughput.RFC2544.PacketLossRatio.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The DUT's 0% packet loss throughput in the presence of cache sharing and
+ memory bandwidth between processes.
+
+.. 3.2.2.3.3
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
+ Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's throughput rate with X% traffic loss for
+ a constant load (fixed length frames at a fixed interval time) when the
+ number of VNFs on the DUT increases. The default loss percentages
+ to be tested are: - X = 0% - X = 10^-7% . The minimum number of
+ VNFs to be tested are 3.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The vSwitch is responsible for forwarding frames between the VNFs
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: The traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT.
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ If the test VNFs are rather light-weight in terms of processing, the test
+ provides a view of multiple passes through the vswitch on logical
+ interfaces. In other words, the test produces an optimistic count of
+ daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is
+ "real" (assuming that the vSwitch has some dedicated resources, and the
+ effects on shared resources is understood).
+
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.2.3.4
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test reveals how throughput and latency degrades as the number
+ of VNFs increases and offered rate varies in the region of the DUT's
+ maximum forwarding rate as determined by
+ LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss).
+ For example it can be used to determine if the degradation of throughput
+ and latency as the number of VNFs and offered rate increases is slow
+ and graceful, or sudden and severe. The minimum number of VNFs to
+ be tested is 3.
+
+ The selected frame sizes are those previously defined under
+ :ref:`default-test-parameters`.
+
+ The offered traffic rate is described as a percentage delta with respect
+ to the DUT's RFC 2544 Throughput as determined by
+ LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
+ of 0% is equivalent to an offered traffic rate equal to the RFC 2544
+ Throughput; A delta of +50% indicates an offered rate half-way
+ between the Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the maximum rate. Therefore the
+ range of the delta figure is natuarlly bounded at -100% (zero offered
+ traffic) and +100% (traffic offered at line rate).
+
+ The following deltas to the maximum forwarding rate should be applied:
+
+ - -50%, -10%, 0%, +10% & +50%
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: the traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT
+
+ **Expected Result**: For each packet size a profile should be produced
+ of how throughput and latency vary with offered rate.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT
+ for each delta to the maximum forwarding rate and for each frame
+ size.
+ - The average latency for each delta to the maximum forwarding rate and
+ for each frame size.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - Any failures experienced (for example if the vSwitch crashes, stops
+ processing packets, restarts or becomes unresponsive to commands)
+ when the offered load is above Maximum Throughput MUST be recorded
+ and reported with the results.
+
+.. 3.2.2.4
+
+Activation tests
+----------------
+
+The general aim of these tests is to understand the capacity of the
+and speed with which the vswitch can accommodate new flows.
+
+.. 3.2.2.4.1
+
+Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Address Caching Capacity Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ Please note this test is only applicable to virtual switches that are capable of
+ MAC learning. The aim of this test is to determine the address caching
+ capacity of the DUT for a constant load (fixed length frames at a fixed
+ interval time). The selected frame sizes are those previously defined
+ under :ref:`default-test-parameters`.
+
+ In order to run this test the aging time, that is the maximum time the
+ DUT will keep a learned address in its flow table, and a set of initial
+ addresses, whose value should be >= 1 and <= the max number supported by
+ the implementation must be known. Please note that if the aging time is
+ configurable it must be longer than the time necessary to produce frames
+ from the external source at the specified rate. If the aging time is
+ fixed the frame rate must be brought down to a value that the external
+ source can produce in a time that is less than the aging time.
+
+ Learning Frames should be sent from an external source to the DUT to
+ install a number of flows. The Learning Frames must have a fixed
+ destination address and must vary the source address of the frames. The
+ DUT should install flows in its flow table based on the varying source
+ addresses. Frames should then be transmitted from an external source at
+ a suitable frame rate to see if the DUT has properly learned all of the
+ addresses. If there is no frame loss and no flooding, the number of
+ addresses sent to the DUT should be increased and the test is repeated
+ until the max number of cached addresses supported by the DUT
+ determined.
+
+ **Expected Result**:
+
+ **Metrics collected**:
+
+ The following are the metrics collected for this test:
+
+ - Number of cached addresses supported by the DUT.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → 2 x physical (one receiving, one listening).
+
+.. 3.2.2.4.2
+
+Test ID: LTD.Activation.RFC2889.AddressLearningRate
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC2889 Address Learning Rate Test
+
+ **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
+
+ **Priority**:
+
+ **Description**:
+
+ Please note this test is only applicable to virtual switches that are capable of
+ MAC learning. The aim of this test is to determine the rate of address
+ learning of the DUT for a constant load (fixed length frames at a fixed
+ interval time). The selected frame sizes are those previously defined
+ under :ref:`default-test-parameters`, traffic should be
+ sent with each IPv4/IPv6 address incremented by one. The rate at which
+ the DUT learns a new address should be measured. The maximum caching
+ capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken
+ into consideration as the maximum number of addresses for which the
+ learning rate can be obtained.
+
+ **Expected Result**: It may be worthwhile to report the behaviour when
+ operating beyond address capacity - some DUTs may be more friendly to
+ new addresses than others.
+
+ **Metrics collected**:
+
+ The following are the metrics collected for this test:
+
+ - The address learning rate of the DUT.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → 2 x physical (one receiving, one listening).
+
+.. 3.2.2.5
+
+Coupling between control path and datapath Tests
+------------------------------------------------
+
+The following tests aim to determine how tightly coupled the datapath
+and the control path are within a virtual switch. The following list
+is not exhaustive but should indicate the type of tests that should be
+required. It is expected that more will be added.
+
+.. 3.2.2.5.1
+
+Test ID: LTD.CPDPCouplingFlowAddition
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: Control Path and Datapath Coupling
+
+ **Prerequisite Test**:
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand how exercising the DUT's control
+ path affects datapath performance.
+
+ Initially a certain number of flow table entries are installed in the
+ vSwitch. Then over the duration of an RFC2544 throughput test
+ flow-entries are added and removed at the rates specified below. No
+ traffic is 'hitting' these flow-entries, they are simply added and
+ removed.
+
+ The test MUST be repeated with the following initial number of
+ flow-entries installed: - < 10 - 1000 - 100,000 - 10,000,000 (or the
+ maximum supported number of flow-entries)
+
+ The test MUST be repeated with the following rates of flow-entry
+ addition and deletion per second: - 0 - 1 (i.e. 1 addition plus 1
+ deletion) - 100 - 10,000
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+.. 3.2.2.6
+
+CPU and memory consumption
+--------------------------
+
+The following tests will profile a virtual switch's CPU and memory
+utilization under various loads and circumstances. The following
+list is not exhaustive but should indicate the type of tests that
+should be required. It is expected that more will be added.
+
+.. 3.2.2.6.1
+
+.. _CPU0PacketLoss:
+
+Test ID: LTD.Stress.RFC2544.0PacketLoss
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test
+
+ **Prerequisite Test**:
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to understand the overall performance of the
+ system when a CPU or Memory intensive application is run on the same DUT as
+ the Virtual Switch. For each frame size, an
+ LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be
+ performed. Throughout the entire test a CPU or Memory intensive application
+ should be run on all cores on the system not in use by the Virtual Switch.
+ For NUMA system only cores on the same NUMA node are loaded.
+
+ It is recommended that stress-ng be used for loading the non-Virtual
+ Switch cores but any stress tool MAY be used.
+
+ **Expected Result**:
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - Memory and CPU utilization of the cores running the Virtual Switch.
+ - The number of identity of the cores allocated to the Virtual Switch.
+ - The configuration of the stress tool (for example the command line
+ parameters used to start it.)
+
+ **Note:** Stress in the test ID can be replaced with the name of the
+ component being stressed, when reporting the results:
+ LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss
+
+.. 3.2.2.7
+
+Summary List of Tests
+---------------------
+
+1. Throughput tests
+
+ - Test ID: LTD.Throughput.RFC2544.PacketLossRatio
+ - Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
+ - Test ID: LTD.Throughput.RFC2544.Profile
+ - Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
+ - Test ID: LTD.Throughput.RFC2544.BackToBackFrames
+ - Test ID: LTD.Throughput.RFC2889.Soak
+ - Test ID: LTD.Throughput.RFC2889.SoakFrameModification
+ - Test ID: LTD.Throughput.RFC6201.ResetTime
+ - Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
+ - Test ID: LTD.Throughput.RFC2889.ForwardPressure
+ - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
+ - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
+ - Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+ - Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+
+2. Packet Latency tests
+
+ - Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
+ - Test ID: LTD.PacketDelayVariation.RFC3393.Soak
+
+3. Scalability tests
+
+ - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
+ - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+ - LTD.Scalability.VNF.RFC2544.PacketLossProfile
+ - LTD.Scalability.VNF.RFC2544.PacketLossRatio
+
+4. Activation tests
+
+ - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
+ - Test ID: LTD.Activation.RFC2889.AddressLearningRate
+
+5. Coupling between control path and datapath Tests
+
+ - Test ID: LTD.CPDPCouplingFlowAddition
+
+6. CPU and memory consumption
+
+ - Test ID: LTD.Stress.RFC2544.0PacketLoss