diff options
Diffstat (limited to 'docs/testing/developer/requirements/vswitchperf_ltd.rst')
| -rw-r--r-- | docs/testing/developer/requirements/vswitchperf_ltd.rst | 1712 |
1 files changed, 0 insertions, 1712 deletions
diff --git a/docs/testing/developer/requirements/vswitchperf_ltd.rst b/docs/testing/developer/requirements/vswitchperf_ltd.rst deleted file mode 100644 index e1372520..00000000 --- a/docs/testing/developer/requirements/vswitchperf_ltd.rst +++ /dev/null @@ -1,1712 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. - -****************************** -VSPERF LEVEL TEST DESIGN (LTD) -****************************** - -.. 3.1 - -============ -Introduction -============ - -The intention of this Level Test Design (LTD) document is to specify the set of -tests to carry out in order to objectively measure the current characteristics -of a virtual switch in the Network Function Virtualization Infrastructure -(NFVI) as well as the test pass criteria. The detailed test cases will be -defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_. - -This document is currently in draft form. - -.. 3.1.1 - - -.. _doc-id-of-LTD: - -Document identifier -=================== - -The document id will be used to uniquely -identify versions of the LTD. The format for the document id will be: -OPNFV\_vswitchperf\_LTD\_REL\_STATUS, where by the -status is one of: draft, reviewed, corrected or final. The document id -for this version of the LTD is: -OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED. - -.. 3.1.2 - -.. _scope-of-LTD: - -Scope -===== - -The main purpose of this project is to specify a suite of -performance tests in order to objectively measure the current packet -transfer characteristics of a virtual switch in the NFVI. The intent of -the project is to facilitate testing of any virtual switch. Thus, a -generic suite of tests shall be developed, with no hard dependencies to -a single implementation. In addition, the test case suite shall be -architecture independent. - -The test cases developed in this project shall not form part of a -separate test framework, all of these tests may be inserted into the -Continuous Integration Test Framework and/or the Platform Functionality -Test Framework - if a vSwitch becomes a standard component of an OPNFV -release. - -.. 3.1.3 - -References -========== - -* `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ -* `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ -* `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ -* `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ -* `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ -* `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ -* `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ -* `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ - -.. 3.2 - -.. _details-of-LTD: - -================================ -Details of the Level Test Design -================================ - -This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and -identifies the sets of test cases or scenarios (TestIdentification-of-LTD_). - -.. 3.2.1 - -.. _FeaturesToBeTested-of-LTD: - -Features to be tested -===================== - -Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) -includes measuring the following performance metrics: - -- Throughput -- Packet delay -- Packet delay variation -- Packet loss -- Burst behaviour -- Packet re-ordering -- Packet correctness -- Availability and capacity of the DUT - -.. 3.2.2 - -.. _TestIdentification-of-LTD: - -Test identification -=================== - -.. 3.2.2.1 - -Throughput tests ----------------- - -The following tests aim to determine the maximum forwarding rate that -can be achieved with a virtual switch. The list is not exhaustive but -should indicate the type of tests that should be required. It is -expected that more will be added. - -.. 3.2.2.1.1 - -.. _PacketLossRatio: - -Test ID: LTD.Throughput.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.1.2 - -.. _PacketLossRatioFrameModification: - -Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss Throughput and Latency Test with - packet modification - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - During this test, the DUT must perform the following operations on the - traffic flow: - - - Perform packet parsing on the DUT's ingress port. - - Perform any relevant address look-ups on the DUT's ingress ports. - - Modify the packet header before forwarding the packet to the DUT's - egress port. Packet modifications include: - - - Modifying the Ethernet source or destination MAC address. - - Modifying/adding a VLAN tag. (**Recommended**). - - Modifying/adding a MPLS tag. - - Modifying the source or destination ip address. - - Modifying the TOS/DSCP field. - - Modifying the source or destination ports for UDP/TCP/SCTP. - - Modifying the TTL. - - **Expected Result**: The Packet parsing/modifications require some - additional degree of processing resource, therefore the - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - Throughput is expected to be somewhat lower than the Throughput level - measured without additional steps. The reduction is expected to be - greatest on tests with the smallest packet sizes (greatest header - processing rates). - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss and packet - modification operations being performed by the DUT. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.1.3 - -Test ID: LTD.Throughput.RFC2544.Profile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 Throughput and Latency Profile - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test reveals how throughput and latency degrades as the offered - rate varies in the region of the DUT's maximum forwarding rate as - determined by LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). - For example it can be used to determine if the degradation of throughput - and latency as the offered rate increases is slow and graceful or sudden - and severe. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The offered traffic rate is described as a percentage delta with respect - to the DUT's RFC 2544 Throughput as determined by - LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta - of 0% is equivalent to an offered traffic rate equal to the RFC 2544 - Maximum Throughput; A delta of +50% indicates an offered rate half-way - between the Maximum RFC2544 Throughput and line-rate, whereas a delta of - -50% indicates an offered rate of half the RFC 2544 Maximum Throughput. - Therefore the range of the delta figure is natuarlly bounded at -100% - (zero offered traffic) and +100% (traffic offered at line rate). - - The following deltas to the maximum forwarding rate should be applied: - - - -50%, -10%, 0%, +10% & +50% - - **Expected Result**: For each packet size a profile should be produced - of how throughput and latency vary with offered rate. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT - for each delta to the maximum forwarding rate and for each frame - size. - - The average latency for each delta to the maximum forwarding rate and - for each frame size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - Any failures experienced (for example if the vSwitch crashes, stops - processing packets, restarts or becomes unresponsive to commands) - when the offered load is above Maximum Throughput MUST be recorded - and reported with the results. - -.. 3.2.2.1.4 - -Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 System Recovery Time Test - - **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to determine the length of time it takes the DUT - to recover from an overload condition for a constant load (fixed length - frames at a fixed interval time). The selected frame sizes are those - previously defined under :ref:`default-test-parameters`, - traffic should be sent to the DUT under normal conditions. During the - duration of the test and while the traffic flows are passing though the - DUT, at least one situation leading to an overload condition for the DUT - should occur. The time from the end of the overload condition to when - the DUT returns to normal operations should be measured to determine - recovery time. Prior to overloading the DUT, one should record the - average latency for 10,000 packets forwarded through the DUT. - - The overload condition SHOULD be to transmit traffic at a very high - frame rate to the DUT (150% of the maximum 0% packet loss rate as - determined by LTD.Throughput.RFC2544.PacketLossRatio or line-rate - whichever is lower), for at least 60 seconds, then reduce the frame rate - to 75% of the maximum 0% packet loss rate. A number of time-stamps - should be recorded: - Record the time-stamp at which the frame rate was - reduced and record a second time-stamp at the time of the last frame - lost. The recovery time is the difference between the two timestamps. - - Record the average latency for 10,000 frames after the last frame loss - and continue to record average latency measurements for every 10,000 - frames, when latency returns to within 10% of pre-overload levels record - the time-stamp. - - **Expected Result**: - - **Metrics collected** - - The following are the metrics collected for this test: - - - The length of time it takes the DUT to recover from an overload - condition. - - The length of time it takes the DUT to recover the average latency to - pre-overload conditions. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.5 - -.. _BackToBackFrames: - -Test ID: LTD.Throughput.RFC2544.BackToBackFrames -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2544 Back To Back Frames Test - - **Prerequisite Test**: N - - **Priority**: - - **Description**: - - The aim of this test is to characterize the ability of the DUT to - process back-to-back frames. For each frame size previously defined - under :ref:`default-test-parameters`, a burst of traffic - is sent to the DUT with the minimum inter-frame gap between each frame. - If the number of received frames equals the number of frames that were - transmitted, the burst size should be increased and traffic is sent to - the DUT again. The value measured is the back-to-back value, that is the - maximum burst size the DUT can handle without any frame loss. Please note - a trial must run for a minimum of 2 seconds and should be repeated 50 - times (at a minimum). - - **Expected Result**: - - Tests of back-to-back frames with physical devices have produced - unstable results in some cases. All tests should be repeated in multiple - test sessions and results stability should be examined. - - **Metrics collected** - - The following are the metrics collected for this test: - - - The average back-to-back value across the trials, which is - the number of frames in the longest burst that the DUT will - handle without the loss of any frames. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.6 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test - - **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to understand the Max Forwarding Rate stability - over an extended test duration in order to uncover any outliers. To allow - for an extended test duration, the test should ideally run for 24 hours - or, if this is not possible, for at least 6 hours. For this test, each frame - size must be sent at the highest Throughput rate with X% packet loss, as - determined in the prerequisite test. The default loss percentages to be - tested are: - X = 0% - X = 10^-7% - - Note: Other values can be tested if required by the user. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Max Forwarding Rate stability of the DUT. - - - This means reporting the number of packets lost per time interval - and reporting any time intervals with packet loss. The - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. - - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile. - -.. 3.2.2.1.7 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification - - **Prerequisite Test**: - LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss) - - **Priority**: - - **Description**: - - The aim of this test is to understand the Max Forwarding Rate stability over an - extended test duration in order to uncover any outliers. To allow for an - extended test duration, the test should ideally run for 24 hours or, if - this is not possible, for at least 6 hour. For this test, each frame - size must be sent at the highest Throughput rate with 0% packet loss, as - determined in the prerequisite test. - - During this test, the DUT must perform the following operations on the - traffic flow: - - - Perform packet parsing on the DUT's ingress port. - - Perform any relevant address look-ups on the DUT's ingress ports. - - Modify the packet header before forwarding the packet to the DUT's - egress port. Packet modifications include: - - - Modifying the Ethernet source or destination MAC address. - - Modifying/adding a VLAN tag (**Recommended**). - - Modifying/adding a MPLS tag. - - Modifying the source or destination ip address. - - Modifying the TOS/DSCP field. - - Modifying the source or destination ports for UDP/TCP/SCTP. - - Modifying the TTL. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Max Forwarding Rate stability of the DUT. - - - This means reporting the number of packets lost per time interval - and reporting any time intervals with packet loss. The - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. - - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, using the 99th - percentile. - -.. 3.2.2.1.8 - -Test ID: LTD.Throughput.RFC6201.ResetTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 6201 Reset Time Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - The aim of this test is to determine the length of time it takes the DUT - to recover from a reset. - - Two reset methods are defined - planned and unplanned. A planned reset - requires stopping and restarting the virtual switch by the usual - 'graceful' method defined by it's documentation. An unplanned reset - requires simulating a fatal internal fault in the virtual switch - for - example by using kill -SIGKILL on a Linux environment. - - Both reset methods SHOULD be exercised. - - For each frame size previously defined under :ref:`default-test-parameters`, - traffic should be sent to the DUT under - normal conditions. During the duration of the test and while the traffic - flows are passing through the DUT, the DUT should be reset and the Reset - time measured. The Reset time is the total time that a device is - determined to be out of operation and includes the time to perform the - reset and the time to recover from it (cf. `RFC6201 - <https://www.rfc-editor.org/rfc/rfc6201.txt>`__). - - `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods - to measure the Reset time: - - - Frame-Loss Method: which requires the monitoring of the number of - lost frames and calculates the Reset time based on the number of - frames lost and the offered rate according to the following - formula: - - .. code-block:: console - - Frames_lost (packets) - Reset_time = ------------------------------------- - Offered_rate (packets per second) - - - Timestamp Method: which measures the time from which the last frame - is forwarded from the DUT to the time the first frame is forwarded - after the reset. This involves time-stamping all transmitted frames - and recording the timestamp of the last frame that was received prior - to the reset and also measuring the timestamp of the first frame that - is received after the reset. The Reset time is the difference between - these two timestamps. - - According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the - choice of method depends on the test tool's capability; the Frame-Loss - method SHOULD be used if the test tool supports: - - * Counting the number of lost frames per stream. - * Transmitting test frame despite the physical link status. - - whereas the Timestamp method SHOULD be used if the test tool supports: - - * Timestamping each frame. - * Monitoring received frame's timestamp. - * Transmitting frames only if the physical link status is up. - - **Expected Result**: - - **Metrics collected** - - The following are the metrics collected for this test: - - * Average Reset Time over the number of trials performed. - - Results of this test should include the following information: - - * The reset method used. - * Throughput in Fps and Mbps. - * Average Frame Loss over the number of trials performed. - * Average Reset Time in milliseconds over the number of trials performed. - * Number of trials performed. - * Protocol: IPv4, IPv6, MPLS, etc. - * Frame Size in Octets - * Port Media: Ethernet, Gigabit Ethernet (GbE), etc. - * Port Speed: 10 Gbps, 40 Gbps etc. - * Interface Encapsulation: Ethernet, Ethernet VLAN, etc. - - **Deployment scenario**: - - * Physical → virtual switch → physical. - -.. 3.2.2.1.9 - -Test ID: LTD.Throughput.RFC2889.MaxForwardingRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Forwarding Rate Test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - This test measures the DUT's Max Forwarding Rate when the Offered Load - is varied between the throughput and the Maximum Offered Load for fixed - length frames at a fixed time interval. The selected frame sizes are - those previously defined under :ref:`default-test-parameters`. - The throughput is the maximum offered - load with 0% frame loss (measured by the prerequisite test), and the - Maximum Offered Load (as defined by - `RFC2285 <https://www.rfc-editor.org/rfc/rfc2285.txt>`__) is *"the highest - number of frames per second that an external source can transmit to a - DUT/SUT for forwarding to a specified output interface or interfaces"*. - - Traffic should be sent to the DUT at a particular rate (TX rate) - starting with TX rate equal to the throughput rate. The rate of - successfully received frames at the destination counted (in FPS). If the - RX rate is equal to the TX rate, the TX rate should be increased by a - fixed step size and the RX rate measured again until the Max Forwarding - Rate is found. - - The trial duration for each iteration should last for the period of time - needed for the system to reach steady state for the frame size being - tested. Under `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ - (Sec. 5.6.3.1) test methodology, the test - duration should run for a minimum period of 30 seconds, regardless - whether the system reaches steady state before the minimum duration - ends. - - **Expected Result**: According to - `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding - Rate is the highest forwarding rate of a DUT taken from an iterative set of - forwarding rate measurements. The iterative set of forwarding rate measurements - are made by setting the intended load transmitted from an external source and - measuring the offered load (i.e what the DUT is capable of forwarding). If the - Throughput == the Maximum Offered Load, it follows that Max Forwarding Rate is - equal to the Maximum Offered Load. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The Max Forwarding Rate for the DUT for each packet size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. Note: Full mesh tests with - multiple ingress and egress ports are a key aspect of RFC 2889 - benchmarks, and scenarios with both 2 and 4 ports should be tested. - In any case, the number of ports used must be reported. - -.. 3.2.2.1.10 - -Test ID: LTD.Throughput.RFC2889.ForwardPressure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Forward Pressure Test - - **Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate - - **Priority**: - - **Description**: - - The aim of this test is to determine if the DUT transmits frames with an - inter-frame gap that is less than 12 bytes. This test overloads the DUT - and measures the output for forward pressure. Traffic should be - transmitted to the DUT with an inter-frame gap of 11 bytes, this will - overload the DUT by 1 byte per frame. The forwarding rate of the DUT - should be measured. - - **Expected Result**: The forwarding rate should not exceed the maximum - forwarding rate of the DUT collected by - LTD.Throughput.RFC2889.MaxForwardingRate. - - **Metrics collected** - - The following are the metrics collected for this test: - - - Forwarding rate of the DUT in FPS or Mbps. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.11 - -Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Error Frames Filtering Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - The aim of this test is to determine whether the DUT will propagate any - erroneous frames it receives or whether it is capable of filtering out - the erroneous frames. Traffic should be sent with erroneous frames - included within the flow at random intervals. Illegal frames that must - be tested include: - Oversize Frames. - Undersize Frames. - CRC Errored - Frames. - Dribble Bit Errored Frames - Alignment Errored Frames - - The traffic flow exiting the DUT should be recorded and checked to - determine if the erroneous frames where passed through the DUT. - - **Expected Result**: Broken frames are not passed! - - **Metrics collected** - - No Metrics are collected in this test, instead it determines: - - - Whether the DUT will propagate erroneous frames. - - Or whether the DUT will correctly filter out any erroneous frames - from traffic flow with out removing correct frames. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.1.12 - -Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Broadcast Frame Forwarding Test - - **Prerequisite Test**: N - - **Priority**: - - **Description**: - - The aim of this test is to determine the maximum forwarding rate of the - DUT when forwarding broadcast traffic. For each frame previously defined - under :ref:`default-test-parameters`, the traffic should - be set up as broadcast traffic. The traffic throughput of the DUT should - be measured. - - The test should be conducted with at least 4 physical ports on the DUT. - The number of ports used MUST be recorded. - - As broadcast involves forwarding a single incoming packet to several - destinations, the latency of a single packet is defined as the average - of the latencies for each of the broadcast destinations. - - The incoming packet is transmitted on each of the other physical ports, - it is not transmitted on the port on which it was received. The test MAY - be conducted using different broadcasting ports to uncover any - performance differences. - - **Expected Result**: - - **Metrics collected**: - - The following are the metrics collected for this test: - - - The forwarding rate of the DUT when forwarding broadcast traffic. - - The minimum, average & maximum packets latencies observed. - - **Deployment scenario**: - - - Physical → virtual switch 3x physical. In the Broadcast rate testing, - four test ports are required. One of the ports is connected to the test - device, so it can send broadcast frames and listen for miss-routed frames. - -.. 3.2.2.1.13 - -Test ID: LTD.Throughput.RFC2544.WorstN-BestN -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's maximum forwarding rate with X% traffic - loss for a constant load (fixed length frames at a fixed interval time). - The default loss percentages to be tested are: X = 0%, X = 10^-7% - - Modified RFC 2544 throughput benchmarking methodology aims to quantify - the throughput measurement variations observed during standard RFC 2544 - benchmarking measurements of virtual switches and VNFs. The RFC2544 - binary search algorithm is modified to use more samples per test trial - to drive the binary search and yield statistically more meaningful - results. This keeps the heart of the RFC2544 methodology, still relying - on the binary search of throughput at specified loss tolerance, while - providing more useful information about the range of results seen in - testing. Instead of using a single traffic trial per iteration step, - each traffic trial is repeated N times and the success/failure of the - iteration step is based on these N traffic trials. Two types of revised - tests are defined - *Worst-of-N* and *Best-of-N*. - - **Worst-of-N** - - *Worst-of-N* indicates the lowest expected maximum throughput for ( - packet size, loss tolerance) when repeating the test. - - 1. Repeat the same test run N times at a set packet rate, record each - result. - 2. Take the WORST result (highest packet loss) out of N result samples, - called the Worst-of-N sample. - 3. If Worst-of-N sample has loss less than the set loss tolerance, then - the step is successful - increase the test traffic rate. - 4. If Worst-of-N sample has loss greater than the set loss tolerance - then the step failed - decrease the test traffic rate. - 5. Go to step 1. - - **Best-of-N** - - *Best-of-N* indicates the highest expected maximum throughput for ( - packet size, loss tolerance) when repeating the test. - - 1. Repeat the same traffic run N times at a set packet rate, record - each result. - 2. Take the BEST result (least packet loss) out of N result samples, - called the Best-of-N sample. - 3. If Best-of-N sample has loss less than the set loss tolerance, then - the step is successful - increase the test traffic rate. - 4. If Best-of-N sample has loss greater than the set loss tolerance, - then the step failed - decrease the test traffic rate. - 5. Go to step 1. - - Performing both Worst-of-N and Best-of-N benchmark tests yields lower - and upper bounds of expected maximum throughput under the operating - conditions, giving a very good indication to the user of the - deterministic performance range for the tested setup. - - **Expected Result**: At the end of each trial series, the presence or - absence of loss determines the modification of offered load for the - next trial series, converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput - with X% loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - Following may also be collected as part of this test, to determine - the vSwitch's performance footprint on the system: - - - CPU core utilization. - - CPU cache utilization. - - Memory footprint. - - System bus (QPI, PCI, ...) utilization. - - CPU cycles consumed per packet. - -.. 3.2.2.1.14 - -Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test - - - NOTE: Throughout this test, four interchangeable overlay technologies are covered by the - same test description. They are: VXLAN, GRE, NVGRE and GENEVE. - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - This test evaluates standard switch performance benchmarks for the scenario where an - Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered - (replacing <tech> in the test name) include: - - - VXLAN - - GRE - - NVGRE - - GENEVE - - Performance will be assessed for each of the following overlay network functions: - - - Encapsulation only - - De-encapsulation only - - Both Encapsulation and De-encapsulation - - For each native packet, the DUT must perform the following operations: - - - Examine the packet and classify its correct overlay net (tunnel) assignment - - Encapsulate the packet - - Switch the packet to the correct port - - For each encapsulated packet, the DUT must perform the following operations: - - - Examine the packet and classify its correct native network assignment - - De-encapsulate the packet, if required - - Switch the packet to the correct port - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - Thus, each test comprises an overlay technology, a network function, - and a packet size *with* overlay network overhead included - (but see also the discussion at - https://etherpad.opnfv.org/p/vSwitchTestsDrafts ). - - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result for Throughput. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss (where the value of X is typically equal to zero). - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - **Metrics Collected**: - The following are the metrics collected for this test: - - - The maximum Throughput in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - and VNFs (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.3.1.15 - -Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio - - **Priority**: - - **Description**: - - The aim of this test is to determine the cost of carrying out match - action(s) on the DUT’s RFC2544 Throughput with X% traffic loss for - a constant load (fixed length frames at a fixed interval time). - - Each test case requires: - - * selection of a specific match action(s), - * specifying a percentage of total traffic that is elligible - for the match action, - * determination of the specific test configuration (number - of flows, number of test ports, presence of an external - controller, etc.), and - * measurement of the RFC 2544 Throughput level with X% packet - loss: Traffic shall be bi-directional and symmetric. - - Note: It would be ideal to verify that all match action-elligible - traffic was forwarded to the correct port, and if forwarded to - an unintended port it should be considered lost. - - A match action is an action that is typically carried on a frame - or packet that matches a set of flow classification parameters - (typically frame/packet header fields). A match action may or may - not modify a packet/frame. Match actions include [1]: - - * output : outputs a packet to a particular port. - * normal: Subjects the packet to traditional L2/L3 processing - (MAC learning). - * flood: Outputs the packet on all switch physical ports - other than the port on which it was received and any ports - on which flooding is disabled. - * all: Outputs the packet on all switch physical ports other - than the port on which it was received. - * local: Outputs the packet on the ``local port``, which - corresponds to the network device that has the same name as - the bridge. - * in_port: Outputs the packet on the port from which it was - received. - * Controller: Sends the packet and its metadata to the - OpenFlow controller as a ``packet in`` message. - * enqueue: Enqueues the packet on the specified queue - within port. - * drop: discard the packet. - - Modifications include [1]: - - * mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification - * mod_dl_src: Sets the source Ethernet address. - * mod_dl_dst: Sets the destination Ethernet address. - * mod_nw_src: Sets the IPv4 source address. - * mod_nw_dst: Sets the IPv4 destination address. - * mod_tp_src: Sets the TCP or UDP or SCTP source port. - * mod_tp_dst: Sets the TCP or UDP or SCTP destination port. - * mod_nw_tos: Sets the DSCP bits in the IPv4 ToS/DSCP or - IPv6 traffic class field. - * mod_nw_ecn: Sets the ECN bits in the appropriate IPv4 or - IPv6 field. - * mod_nw_ttl: Sets the IPv4 TTL or IPv6 hop limit field. - - Note: This comprehensive list requires extensive traffic generator - capabilities. - - The match action(s) that were applied as part of the test should be - reported in the final test report. - - During this test, the DUT must perform the following operations on - the traffic flow: - - * Perform packet parsing on the DUT’s ingress port. - * Perform any relevant address look-ups on the DUT’s ingress - ports. - * Carry out one or more of the match actions specified above. - - The default loss percentages to be tested are: - X = 0% - X = 10^-7% - Other values can be tested if required by the user. The selected - frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The test can also be used to determine the average latency of the - traffic when a match action is applied to packets in a flow. Under - the RFC2544 test methodology, the test duration will include a - number of trials; each trial should run for a minimum period of 60 - seconds. A binary search methodology must be applied for each - trial to obtain the final result. - - **Expected Result:** - - At the end of each trial, the presence or absence of loss - determines the modification of offered load for the next trial, - converging on a maximum rate, or RFC2544Throughput with X% loss. - The Throughput load is re-used in related RFC2544 tests and other - tests. - - **Metrics Collected:** - - The following are the metrics collected for this test: - - * The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps - of the DUT for each frame size with X% packet loss. - * The average latency of the traffic flow when passing through - the DUT (if testing for latency, note that this average is - different from the test specified in Section 26.3 ofRFC2544). - * CPU and memory utilization may also be collected as part of - this test, to determine the vSwitch’s performance footprint - on the system. - - The metrics collected can be compared to that of the prerequisite - test to determine the cost of the match action(s) in the pipeline. - - **Deployment scenario**: - - - Physical → virtual switch → physical (and others are possible) - - [1] ovs-ofctl - administer OpenFlow switches - [http://openvswitch.org/support/dist-docs/ovs-ofctl.8.txt ] - - -.. 3.2.2.2 - -Packet Latency tests --------------------- - -These tests will measure the store and forward latency as well as the packet -delay variation for various packet types through the virtual switch. The -following list is not exhaustive but should indicate the type of tests -that should be required. It is expected that more will be added. - -.. 3.2.2.2.1 - -Test ID: LTD.PacketLatency.InitialPacketProcessingLatency -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Initial Packet Processing Latency - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - In some virtual switch architectures, the first packets of a flow will - take the system longer to process than subsequent packets in the flow. - This test determines the latency for these packets. The test will - measure the latency of the packets as they are processed by the - flow-setup-path of the DUT. There are two methods for this test, a - recommended method and a nalternative method that can be used if it is - possible to disable the fastpath of the virtual switch. - - Recommended method: This test will send 64,000 packets to the DUT, each - belonging to a different flow. Average packet latency will be determined - over the 64,000 packets. - - Alternative method: This test will send a single packet to the DUT after - a fixed interval of time. The time interval will be equivalent to the - amount of time it takes for a flow to time out in the virtual switch - plus 10%. Average packet latency will be determined over 1,000,000 - packets. - - This test is intended only for non-learning virtual switches; For learning - virtual switches use RFC2889. - - For this test, only unidirectional traffic is required. - - **Expected Result**: The average latency for the initial packet of all - flows should be greater than the latency of subsequent traffic. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Average latency of the initial packets of all flows that are - processed by the DUT. - - **Deployment scenario**: - - - Physical → Virtual Switch → Physical. - -.. 3.2.2.2.2 - -Test ID: LTD.PacketDelayVariation.RFC3393.Soak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Packet Delay Variation Soak Test - - **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) - - **Priority**: - - **Description**: - - The aim of this test is to understand the distribution of packet delay - variation for different frame sizes over an extended test duration and - to determine if there are any outliers. To allow for an extended test - duration, the test should ideally run for 24 hours or, if this is not - possible, for at least 6 hour. For this test, each frame size must be - sent at the highest possible throughput with 0% packet loss, as - determined in the prerequisite test. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The packet delay variation value for traffic passing through the DUT. - - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ - PDV form of delay variation on the traffic flow, - using the 99th percentile, for each 60s interval during the test. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.3 - -Scalability tests ------------------ - -The general aim of these tests is to understand the impact of large flow -table size and flow lookups on throughput. The following list is not -exhaustive but should indicate the type of tests that should be required. -It is expected that more will be added. - -.. 3.2.2.3.1 - -.. _Scalability0PacketLoss: - -Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% loss Flow Scalability throughput test - - **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the - delta Throughput between the single-flow RFC2544 test and this test with - a variable number of flows is desired. - - **Priority**: - - **Description**: - - The aim of this test is to measure how throughput changes as the number - of flows in the DUT increases. The test will measure the throughput - through the fastpath, as such the flows need to be installed on the DUT - before passing traffic. - - For each frame size previously defined under :ref:`default-test-parameters` - and for each of the following number of flows: - - - 1,000 - - 2,000 - - 4,000 - - 8,000 - - 16,000 - - 32,000 - - 64,000 - - Max supported number of flows. - - This test will be conducted under two conditions following the - establishment of all flows as required by RFC 2544, regarding the flow - expiration time-out: - - 1) The time-out never expires during each trial. - - 2) The time-out expires for all flows periodically. This would require a - short time-out compared with flow re-appearance for a small number of - flows, and may not be possible for all flow conditions. - - The maximum 0% packet loss Throughput should be determined in a manner - identical to LTD.Throughput.RFC2544.PacketLossRatio. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum number of frames per second that can be forwarded at the - specified number of flows and the specified frame size, with zero - packet loss. - -.. 3.2.2.3.2 - -Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% loss Memory Bandwidth Scalability test - - **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the - delta Throughput between an undisturbed RFC2544 test and this test with - the Throughput affected by cache and memory bandwidth contention is desired. - - **Priority**: - - **Description**: - - The aim of this test is to understand how the DUT's performance is - affected by cache sharing and memory bandwidth between processes. - - During the test all cores not used by the vSwitch should be running a - memory intensive application. This application should read and write - random data to random addresses in unused physical memory. The random - nature of the data and addresses is intended to consume cache, exercise - main memory access (as opposed to cache) and exercise all memory buses - equally. Furthermore: - - - the ratio of reads to writes should be recorded. A ratio of 1:1 - SHOULD be used. - - the reads and writes MUST be of cache-line size and be cache-line aligned. - - in NUMA architectures memory access SHOULD be local to the core's node. - Whether only local memory or a mix of local and remote memory is used - MUST be recorded. - - the memory bandwidth (reads plus writes) used per-core MUST be recorded; - the test MUST be run with a per-core memory bandwidth equal to half the - maximum system memory bandwidth divided by the number of cores. The test - MAY be run with other values for the per-core memory bandwidth. - - the test MAY also be run with the memory intensive application running - on all cores. - - Under these conditions the DUT's 0% packet loss throughput is determined - as per LTD.Throughput.RFC2544.PacketLossRatio. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The DUT's 0% packet loss throughput in the presence of cache sharing and - memory bandwidth between processes. - -.. 3.2.2.3.3 - -Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and - Latency Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test determines the DUT's throughput rate with X% traffic loss for - a constant load (fixed length frames at a fixed interval time) when the - number of VNFs on the DUT increases. The default loss percentages - to be tested are: - X = 0% - X = 10^-7% . The minimum number of - VNFs to be tested are 3. - - Flow classification should be conducted with L2, L3 and L4 matching - to understand the matching and scaling capability of the vSwitch. The - matching fields which were used as part of the test should be reported - as part of the benchmark report. - - The vSwitch is responsible for forwarding frames between the VNFs - - The SUT (vSwitch and VNF daisy chain) operation should be validated - before running the test. This may be completed by running a burst or - continuous stream of traffic through the SUT to ensure proper operation - before a test. - - **Note**: The traffic rate used to validate SUT operation should be low - enough not to stress the SUT. - - **Note**: Other values can be tested if required by the user. - - **Note**: The same VNF should be used in the "daisy chain" formation. - Each addition of a VNF should be conducted in a new test setup (The DUT - is brought down, then the DUT is brought up again). An atlernative approach - would be to continue to add VNFs without bringing down the DUT. The - approach used needs to be documented as part of the test report. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - The test can also be used to determine the average latency of the traffic. - - Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ - test methodology, the test duration will - include a number of trials; each trial should run for a minimum period - of 60 seconds. A binary search methodology must be applied for each - trial to obtain the final result for Throughput. - - **Expected Result**: At the end of each trial, the presence or absence - of loss determines the modification of offered load for the next trial, - converging on a maximum rate, or - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% - loss. - The Throughput load is re-used in related - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other - tests. - - If the test VNFs are rather light-weight in terms of processing, the test - provides a view of multiple passes through the vswitch on logical - interfaces. In other words, the test produces an optimistic count of - daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is - "real" (assuming that the vSwitch has some dedicated resources, and the - effects on shared resources is understood). - - - **Metrics Collected**: - The following are the metrics collected for this test: - - - The maximum Throughput in Frames Per Second (FPS) and Mbps of - the DUT for each frame size with X% packet loss. - - The average latency of the traffic flow when passing through the DUT - and VNFs (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - -.. 3.2.2.3.4 - -Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - This test reveals how throughput and latency degrades as the number - of VNFs increases and offered rate varies in the region of the DUT's - maximum forwarding rate as determined by - LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). - For example it can be used to determine if the degradation of throughput - and latency as the number of VNFs and offered rate increases is slow - and graceful, or sudden and severe. The minimum number of VNFs to - be tested is 3. - - The selected frame sizes are those previously defined under - :ref:`default-test-parameters`. - - The offered traffic rate is described as a percentage delta with respect - to the DUT's RFC 2544 Throughput as determined by - LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta - of 0% is equivalent to an offered traffic rate equal to the RFC 2544 - Throughput; A delta of +50% indicates an offered rate half-way - between the Throughput and line-rate, whereas a delta of - -50% indicates an offered rate of half the maximum rate. Therefore the - range of the delta figure is natuarlly bounded at -100% (zero offered - traffic) and +100% (traffic offered at line rate). - - The following deltas to the maximum forwarding rate should be applied: - - - -50%, -10%, 0%, +10% & +50% - - **Note**: Other values can be tested if required by the user. - - **Note**: The same VNF should be used in the "daisy chain" formation. - Each addition of a VNF should be conducted in a new test setup (The DUT - is brought down, then the DUT is brought up again). An atlernative approach - would be to continue to add VNFs without bringing down the DUT. The - approach used needs to be documented as part of the test report. - - Flow classification should be conducted with L2, L3 and L4 matching - to understand the matching and scaling capability of the vSwitch. The - matching fields which were used as part of the test should be reported - as part of the benchmark report. - - The SUT (vSwitch and VNF daisy chain) operation should be validated - before running the test. This may be completed by running a burst or - continuous stream of traffic through the SUT to ensure proper operation - before a test. - - **Note**: the traffic rate used to validate SUT operation should be low - enough not to stress the SUT - - **Expected Result**: For each packet size a profile should be produced - of how throughput and latency vary with offered rate. - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT - for each delta to the maximum forwarding rate and for each frame - size. - - The average latency for each delta to the maximum forwarding rate and - for each frame size. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - Any failures experienced (for example if the vSwitch crashes, stops - processing packets, restarts or becomes unresponsive to commands) - when the offered load is above Maximum Throughput MUST be recorded - and reported with the results. - -.. 3.2.2.4 - -Activation tests ----------------- - -The general aim of these tests is to understand the capacity of the -and speed with which the vswitch can accommodate new flows. - -.. 3.2.2.4.1 - -Test ID: LTD.Activation.RFC2889.AddressCachingCapacity -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Address Caching Capacity Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - Please note this test is only applicable to virtual switches that are capable of - MAC learning. The aim of this test is to determine the address caching - capacity of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under :ref:`default-test-parameters`. - - In order to run this test the aging time, that is the maximum time the - DUT will keep a learned address in its flow table, and a set of initial - addresses, whose value should be >= 1 and <= the max number supported by - the implementation must be known. Please note that if the aging time is - configurable it must be longer than the time necessary to produce frames - from the external source at the specified rate. If the aging time is - fixed the frame rate must be brought down to a value that the external - source can produce in a time that is less than the aging time. - - Learning Frames should be sent from an external source to the DUT to - install a number of flows. The Learning Frames must have a fixed - destination address and must vary the source address of the frames. The - DUT should install flows in its flow table based on the varying source - addresses. Frames should then be transmitted from an external source at - a suitable frame rate to see if the DUT has properly learned all of the - addresses. If there is no frame loss and no flooding, the number of - addresses sent to the DUT should be increased and the test is repeated - until the max number of cached addresses supported by the DUT - determined. - - **Expected Result**: - - **Metrics collected**: - - The following are the metrics collected for this test: - - - Number of cached addresses supported by the DUT. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). - -.. 3.2.2.4.2 - -Test ID: LTD.Activation.RFC2889.AddressLearningRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC2889 Address Learning Rate Test - - **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity - - **Priority**: - - **Description**: - - Please note this test is only applicable to virtual switches that are capable of - MAC learning. The aim of this test is to determine the rate of address - learning of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under :ref:`default-test-parameters`, traffic should be - sent with each IPv4/IPv6 address incremented by one. The rate at which - the DUT learns a new address should be measured. The maximum caching - capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken - into consideration as the maximum number of addresses for which the - learning rate can be obtained. - - **Expected Result**: It may be worthwhile to report the behaviour when - operating beyond address capacity - some DUTs may be more friendly to - new addresses than others. - - **Metrics collected**: - - The following are the metrics collected for this test: - - - The address learning rate of the DUT. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). - -.. 3.2.2.5 - -Coupling between control path and datapath Tests ------------------------------------------------- - -The following tests aim to determine how tightly coupled the datapath -and the control path are within a virtual switch. The following list -is not exhaustive but should indicate the type of tests that should be -required. It is expected that more will be added. - -.. 3.2.2.5.1 - -Test ID: LTD.CPDPCouplingFlowAddition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: Control Path and Datapath Coupling - - **Prerequisite Test**: - - **Priority**: - - **Description**: - - The aim of this test is to understand how exercising the DUT's control - path affects datapath performance. - - Initially a certain number of flow table entries are installed in the - vSwitch. Then over the duration of an RFC2544 throughput test - flow-entries are added and removed at the rates specified below. No - traffic is 'hitting' these flow-entries, they are simply added and - removed. - - The test MUST be repeated with the following initial number of - flow-entries installed: - < 10 - 1000 - 100,000 - 10,000,000 (or the - maximum supported number of flow-entries) - - The test MUST be repeated with the following rates of flow-entry - addition and deletion per second: - 0 - 1 (i.e. 1 addition plus 1 - deletion) - 100 - 10,000 - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of - the DUT. - - The average latency of the traffic flow when passing through the DUT - (if testing for latency, note that this average is different from the - test specified in Section 26.3 of - `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__). - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → physical. - -.. 3.2.2.6 - -CPU and memory consumption --------------------------- - -The following tests will profile a virtual switch's CPU and memory -utilization under various loads and circumstances. The following -list is not exhaustive but should indicate the type of tests that -should be required. It is expected that more will be added. - -.. 3.2.2.6.1 - -.. _CPU0PacketLoss: - -Test ID: LTD.Stress.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test - - **Prerequisite Test**: - - **Priority**: - - **Description**: - - The aim of this test is to understand the overall performance of the - system when a CPU or Memory intensive application is run on the same DUT as - the Virtual Switch. For each frame size, an - LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be - performed. Throughout the entire test a CPU or Memory intensive application - should be run on all cores on the system not in use by the Virtual Switch. - For NUMA system only cores on the same NUMA node are loaded. - - It is recommended that stress-ng be used for loading the non-Virtual - Switch cores but any stress tool MAY be used. - - **Expected Result**: - - **Metrics Collected**: - - The following are the metrics collected for this test: - - - Memory and CPU utilization of the cores running the Virtual Switch. - - The number of identity of the cores allocated to the Virtual Switch. - - The configuration of the stress tool (for example the command line - parameters used to start it.) - - **Note:** Stress in the test ID can be replaced with the name of the - component being stressed, when reporting the results: - LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss - -.. 3.2.2.7 - -Summary List of Tests ---------------------- - -1. Throughput tests - - - Test ID: LTD.Throughput.RFC2544.PacketLossRatio - - Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification - - Test ID: LTD.Throughput.RFC2544.Profile - - Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime - - Test ID: LTD.Throughput.RFC2544.BackToBackFrames - - Test ID: LTD.Throughput.RFC2889.Soak - - Test ID: LTD.Throughput.RFC2889.SoakFrameModification - - Test ID: LTD.Throughput.RFC6201.ResetTime - - Test ID: LTD.Throughput.RFC2889.MaxForwardingRate - - Test ID: LTD.Throughput.RFC2889.ForwardPressure - - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering - - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding - - Test ID: LTD.Throughput.RFC2544.WorstN-BestN - - Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio - -2. Packet Latency tests - - - Test ID: LTD.PacketLatency.InitialPacketProcessingLatency - - Test ID: LTD.PacketDelayVariation.RFC3393.Soak - -3. Scalability tests - - - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss - - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability - - LTD.Scalability.VNF.RFC2544.PacketLossProfile - - LTD.Scalability.VNF.RFC2544.PacketLossRatio - -4. Activation tests - - - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity - - Test ID: LTD.Activation.RFC2889.AddressLearningRate - -5. Coupling between control path and datapath Tests - - - Test ID: LTD.CPDPCouplingFlowAddition - -6. CPU and memory consumption - - - Test ID: LTD.Stress.RFC2544.0PacketLoss |