diff options
| -rw-r--r-- | docs/requirements/vswitchperf_ltd.rst | 127 | ||||
| -rw-r--r-- | docs/requirements/vswitchperf_ltp.rst | 6 | ||||
| -rw-r--r-- | docs/results/scenario.rst | 10 |
3 files changed, 94 insertions, 49 deletions
diff --git a/docs/requirements/vswitchperf_ltd.rst b/docs/requirements/vswitchperf_ltd.rst index 23dcd3cf..0d297880 100644 --- a/docs/requirements/vswitchperf_ltd.rst +++ b/docs/requirements/vswitchperf_ltd.rst @@ -8,24 +8,25 @@ VSPERF LEVEL TEST DESIGN (LTD) .. 3.1 +============ Introduction -=============== +============ The intention of this Level Test Design (LTD) document is to specify the set of tests to carry out in order to objectively measure the current characteristics of a virtual switch in the Network Function Virtualization Infrastructure (NFVI) as well as the test pass criteria. The detailed test cases will be -defined in details-of-LTD_, preceded by the doc-id_ and the scope_. +defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_. This document is currently in draft form. .. 3.1.1 -.. _doc-id: +.. _doc-id-of-LTD: Document identifier -========================= +=================== The document id will be used to uniquely identify versions of the LTD. The format for the document id will be: @@ -36,10 +37,10 @@ OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED. .. 3.1.2 -.. _scope: +.. _scope-of-LTD: Scope -========== +===== The main purpose of this project is to specify a suite of performance tests in order to objectively measure the current packet @@ -58,7 +59,7 @@ release. .. 3.1.3 References -=============== +========== * `RFC 1242 Benchmarking Terminology for Network Interconnection Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ @@ -81,19 +82,19 @@ References .. _details-of-LTD: -=================================== +================================ Details of the Level Test Design -=================================== +================================ -This section describes the features to be tested (FeaturesToBeTested_), and -identifies the sets of test cases or scenarios (TestIdentification_). +This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and +identifies the sets of test cases or scenarios (TestIdentification-of-LTD_). .. 3.2.1 -.. _FeaturesToBeTested: +.. _FeaturesToBeTested-of-LTD: Features to be tested -========================== +===================== Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes measuring the following performance metrics: @@ -109,15 +110,16 @@ includes measuring the following performance metrics: .. 3.2.2 -.. _TestIdentification: +.. _TestIdentification-of-LTD: Test identification -========================= +=================== -.. 3.2.3.1 +.. 3.2.2.1 Throughput tests ----------------------- +---------------- + The following tests aim to determine the maximum forwarding rate that can be achieved with a virtual switch. The list is not exhaustive but should indicate the type of tests that should be required. It is @@ -126,7 +128,8 @@ expected that more will be added. .. 3.2.2.1.1 Test ID: LTD.Throughput.RFC2544.PacketLossRatio -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test **Prerequisite Test**: N/A @@ -176,7 +179,8 @@ Test ID: LTD.Throughput.RFC2544.PacketLossRatio .. 3.2.2.1.2 Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 X% packet loss Throughput and Latency Test with packet modification @@ -247,6 +251,7 @@ Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification Test ID: LTD.Throughput.RFC2544.Profile ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 Throughput and Latency Profile **Prerequisite Test**: N/A @@ -301,7 +306,8 @@ Test ID: LTD.Throughput.RFC2544.Profile .. 3.2.2.1.4 Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 System Recovery Time Test **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio @@ -353,7 +359,8 @@ Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime .. 3.2.2.1.5 Test ID: LTD.Throughput.RFC2544.BackToBackFrames -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2544 Back To Back Frames Test **Prerequisite Test**: N @@ -396,7 +403,8 @@ Test ID: LTD.Throughput.RFC2544.BackToBackFrames .. 3.2.2.1.6 Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio @@ -438,7 +446,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak .. 3.2.2.1.7 Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification **Prerequisite Test**: @@ -494,7 +503,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification .. 3.2.2.1.8 Test ID: LTD.Throughput.RFC6201.ResetTime -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 6201 Reset Time Test **Prerequisite Test**: N/A @@ -553,6 +563,7 @@ Test ID: LTD.Throughput.RFC6201.ResetTime * Transmitting test frame despite the physical link status. whereas the Timestamp method SHOULD be used if the test tool supports: + * Timestamping each frame. * Monitoring received frame's timestamp. * Transmitting frames only if the physical link status is up. @@ -585,7 +596,8 @@ Test ID: LTD.Throughput.RFC6201.ResetTime .. 3.2.2.1.9 Test ID: LTD.Throughput.RFC2889.MaxForwardingRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Forwarding Rate Test **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio @@ -647,7 +659,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRate .. 3.2.2.1.10 Test ID: LTD.Throughput.RFC2889.ForwardPressure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Forward Pressure Test **Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate @@ -682,7 +695,8 @@ Test ID: LTD.Throughput.RFC2889.ForwardPressure .. 3.2.2.1.11 Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Error Frames Filtering Test **Prerequisite Test**: N/A @@ -718,7 +732,8 @@ Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering .. 3.2.2.1.12 Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Broadcast Frame Forwarding Test **Prerequisite Test**: N @@ -763,7 +778,8 @@ Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding .. 3.2.2.1.13 Test ID: LTD.Throughput.RFC2544.WorstN-BestN -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test **Prerequisite Test**: N/A @@ -856,6 +872,7 @@ Test ID: LTD.Throughput.RFC2544.WorstN-BestN Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test @@ -935,6 +952,7 @@ Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio @@ -948,6 +966,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio a constant load (fixed length frames at a fixed interval time). Each test case requires: + * selection of a specific match action(s), * specifying a percentage of total traffic that is elligible for the match action, @@ -965,6 +984,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio or packet that matches a set of flow classification parameters (typically frame/packet header fields). A match action may or may not modify a packet/frame. Match actions include [1]: + * output : outputs a packet to a particular port. * normal: Subjects the packet to traditional L2/L3 processing (MAC learning). @@ -973,18 +993,19 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio on which flooding is disabled. * all: Outputs the packet on all switch physical ports other than the port on which it was received. - * local: Outputs the packet on the ``local port,'' which + * local: Outputs the packet on the ``local port``, which corresponds to the network device that has the same name as the bridge. * in_port: Outputs the packet on the port from which it was received. * Controller: Sends the packet and its metadata to the - OpenFlow controller as a ``packet in'' message. + OpenFlow controller as a ``packet in`` message. * enqueue: Enqueues the packet on the specified queue within port. * drop: discard the packet. Modifications include [1]: + * mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification * mod_dl_src: Sets the source Ethernet address. * mod_dl_dst: Sets the destination Ethernet address. @@ -1006,6 +1027,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio During this test, the DUT must perform the following operations on the traffic flow: + * Perform packet parsing on the DUT’s ingress port. * Perform any relevant address look-ups on the DUT’s ingress ports. @@ -1034,6 +1056,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio **Metrics Collected:** The following are the metrics collected for this test: + * The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss. * The average latency of the traffic flow when passing through @@ -1057,7 +1080,8 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio .. 3.2.2.2 Packet Latency tests ---------------------------- +-------------------- + These tests will measure the store and forward latency as well as the packet delay variation for various packet types through the virtual switch. The following list is not exhaustive but should indicate the type of tests @@ -1066,7 +1090,8 @@ that should be required. It is expected that more will be added. .. 3.2.2.2.1 Test ID: LTD.PacketLatency.InitialPacketProcessingLatency -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: Initial Packet Processing Latency **Prerequisite Test**: N/A @@ -1115,7 +1140,8 @@ Test ID: LTD.PacketLatency.InitialPacketProcessingLatency .. 3.2.2.2.2 Test ID: LTD.PacketDelayVariation.RFC3393.Soak -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: Packet Delay Variation Soak Test **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) @@ -1148,7 +1174,8 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak .. 3.2.2.3 Scalability tests ------------------------- +----------------- + The general aim of these tests is to understand the impact of large flow table size and flow lookups on throughput. The following list is not exhaustive but should indicate the type of tests that should be required. @@ -1157,7 +1184,8 @@ It is expected that more will be added. .. 3.2.2.3.1 Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 0% loss Flow Scalability throughput test **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the @@ -1212,7 +1240,8 @@ Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss .. 3.2.2.3.2 Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 0% loss Memory Bandwidth Scalability test **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the @@ -1262,6 +1291,7 @@ Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and Latency Test @@ -1342,7 +1372,8 @@ Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio .. 3.2.2.3.4 Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile **Prerequisite Test**: N/A @@ -1421,6 +1452,7 @@ Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile Activation tests ---------------- + The general aim of these tests is to understand the capacity of the and speed with which the vswitch can accommodate new flows. @@ -1428,6 +1460,7 @@ and speed with which the vswitch can accommodate new flows. Test ID: LTD.Activation.RFC2889.AddressCachingCapacity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Address Caching Capacity Test **Prerequisite Test**: N/A @@ -1479,7 +1512,8 @@ Test ID: LTD.Activation.RFC2889.AddressCachingCapacity .. 3.2.2.4.2 Test ID: LTD.Activation.RFC2889.AddressLearningRate -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC2889 Address Learning Rate Test **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity @@ -1516,7 +1550,8 @@ Test ID: LTD.Activation.RFC2889.AddressLearningRate .. 3.2.2.5 Coupling between control path and datapath Tests -------------------------------------------------------- +------------------------------------------------ + The following tests aim to determine how tightly coupled the datapath and the control path are within a virtual switch. The following list is not exhaustive but should indicate the type of tests that should be @@ -1525,7 +1560,8 @@ required. It is expected that more will be added. .. 3.2.2.5.1 Test ID: LTD.CPDPCouplingFlowAddition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: Control Path and Datapath Coupling **Prerequisite Test**: @@ -1573,7 +1609,8 @@ Test ID: LTD.CPDPCouplingFlowAddition .. 3.2.2.6 CPU and memory consumption ---------------------------------- +-------------------------- + The following tests will profile a virtual switch's CPU and memory utilization under various loads and circumstances. The following list is not exhaustive but should indicate the type of tests that @@ -1583,6 +1620,7 @@ should be required. It is expected that more will be added. Test ID: LTD.Stress.RFC2544.0PacketLoss ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test **Prerequisite Test**: @@ -1620,7 +1658,8 @@ Test ID: LTD.Stress.RFC2544.0PacketLoss .. 3.2.2.7 Summary List of Tests ----------------------------- +--------------------- + 1. Throughput tests - Test ID: LTD.Throughput.RFC2544.PacketLossRatio @@ -1650,7 +1689,7 @@ Summary List of Tests - LTD.Scalability.VNF.RFC2544.PacketLossProfile - LTD.Scalability.VNF.RFC2544.PacketLossRatio -4. Acivation tests +4. Activation tests - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity - Test ID: LTD.Activation.RFC2889.AddressLearningRate diff --git a/docs/requirements/vswitchperf_ltp.rst b/docs/requirements/vswitchperf_ltp.rst index c6692d9a..6b2ee9bc 100644 --- a/docs/requirements/vswitchperf_ltp.rst +++ b/docs/requirements/vswitchperf_ltp.rst @@ -1300,20 +1300,26 @@ vsperf CI vsperf CI jobs are broken down into: * Daily job: + * Runs everyday takes about 10 hours to complete. * TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'. * TESTPARAM_DAILY='--test-params pkt_sizes=64,128,512,1024,1518'. + * Merge job: + * Runs whenever patches are merged to master. * Runs a basic Sanity test. + * Verify job: + * Runs every time a patch is pushed to gerrit. * Builds documentation. Scripts: -------- There are 2 scripts that are part of VSPERFs CI: + * build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is used to run vsperf with the appropriate cli parameters. * vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG diff --git a/docs/results/scenario.rst b/docs/results/scenario.rst index 8de2fb59..cb42c8dc 100644 --- a/docs/results/scenario.rst +++ b/docs/results/scenario.rst @@ -50,13 +50,13 @@ Supported traffic generators: .. _PacketLossRatioFrameModification: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#test-id-ltd-throughput-rfc2544-packetlossratioframemodification -.. _Scalability0PacketLoss: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#test-id-ltd-scalability-rfc2544-0packetloss +.. _Scalability0PacketLoss: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#test-id-ltd-scalability-flows-rfc2544-0packetloss -.. _CPU0PacketLoss: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#test-id-ltd-cpu-rfc2544-0packetloss +.. _CPU0PacketLoss: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#test-id-ltd-stress-rfc2544-0packetloss -.. _Phy2Phy : http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#physical-port-vswitch-physical-port +.. _Phy2Phy : http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltp.html#physical-port-vswitch-physical-port -.. _PVP: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#physical-port-vswitch-vnf-vswitch-physical-port +.. _PVP: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltp.html#physical-port-vswitch-vnf-vswitch-physical-port -.. _PVVP: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltd.html#physical-port-vswitch-vnf-vswitch-vnf-vswitch-physical-port +.. _PVVP: http://artifacts.opnfv.org/vswitchperf/docs/requirements/vswitchperf_ltp.html#physical-port-vswitch-vnf-vswitch-vnf-vswitch-physical-port |