summaryrefslogtreecommitdiffstats
path: root/docs/requirements
diff options
context:
space:
mode:
authorMartin Klozik <martinx.klozik@intel.com>2016-09-06 06:43:18 +0000
committerGerrit Code Review <gerrit@172.30.200.206>2016-09-06 06:43:18 +0000
commit5aa0dc99f1cf9bf1a0a1d86143d6a1cbcc778691 (patch)
treeb15288334cee5119233ff9ed9351c6c7c5ee08cf /docs/requirements
parentddd9674fc7595788c58acdc564573cb228658b84 (diff)
parent3e0e7cb1be0467d7f00240796f5143f2f6e1b097 (diff)
Merge "docs: Fix links and formatting"
Diffstat (limited to 'docs/requirements')
-rw-r--r--docs/requirements/vswitchperf_ltd.rst127
-rw-r--r--docs/requirements/vswitchperf_ltp.rst6
2 files changed, 89 insertions, 44 deletions
diff --git a/docs/requirements/vswitchperf_ltd.rst b/docs/requirements/vswitchperf_ltd.rst
index 23dcd3cf..0d297880 100644
--- a/docs/requirements/vswitchperf_ltd.rst
+++ b/docs/requirements/vswitchperf_ltd.rst
@@ -8,24 +8,25 @@ VSPERF LEVEL TEST DESIGN (LTD)
.. 3.1
+============
Introduction
-===============
+============
The intention of this Level Test Design (LTD) document is to specify the set of
tests to carry out in order to objectively measure the current characteristics
of a virtual switch in the Network Function Virtualization Infrastructure
(NFVI) as well as the test pass criteria. The detailed test cases will be
-defined in details-of-LTD_, preceded by the doc-id_ and the scope_.
+defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_.
This document is currently in draft form.
.. 3.1.1
-.. _doc-id:
+.. _doc-id-of-LTD:
Document identifier
-=========================
+===================
The document id will be used to uniquely
identify versions of the LTD. The format for the document id will be:
@@ -36,10 +37,10 @@ OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED.
.. 3.1.2
-.. _scope:
+.. _scope-of-LTD:
Scope
-==========
+=====
The main purpose of this project is to specify a suite of
performance tests in order to objectively measure the current packet
@@ -58,7 +59,7 @@ release.
.. 3.1.3
References
-===============
+==========
* `RFC 1242 Benchmarking Terminology for Network Interconnection
Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
@@ -81,19 +82,19 @@ References
.. _details-of-LTD:
-===================================
+================================
Details of the Level Test Design
-===================================
+================================
-This section describes the features to be tested (FeaturesToBeTested_), and
-identifies the sets of test cases or scenarios (TestIdentification_).
+This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and
+identifies the sets of test cases or scenarios (TestIdentification-of-LTD_).
.. 3.2.1
-.. _FeaturesToBeTested:
+.. _FeaturesToBeTested-of-LTD:
Features to be tested
-==========================
+=====================
Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
includes measuring the following performance metrics:
@@ -109,15 +110,16 @@ includes measuring the following performance metrics:
.. 3.2.2
-.. _TestIdentification:
+.. _TestIdentification-of-LTD:
Test identification
-=========================
+===================
-.. 3.2.3.1
+.. 3.2.2.1
Throughput tests
-----------------------
+----------------
+
The following tests aim to determine the maximum forwarding rate that
can be achieved with a virtual switch. The list is not exhaustive but
should indicate the type of tests that should be required. It is
@@ -126,7 +128,8 @@ expected that more will be added.
.. 3.2.2.1.1
Test ID: LTD.Throughput.RFC2544.PacketLossRatio
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test
**Prerequisite Test**: N/A
@@ -176,7 +179,8 @@ Test ID: LTD.Throughput.RFC2544.PacketLossRatio
.. 3.2.2.1.2
Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss Throughput and Latency Test with
packet modification
@@ -247,6 +251,7 @@ Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
Test ID: LTD.Throughput.RFC2544.Profile
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 Throughput and Latency Profile
**Prerequisite Test**: N/A
@@ -301,7 +306,8 @@ Test ID: LTD.Throughput.RFC2544.Profile
.. 3.2.2.1.4
Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 System Recovery Time Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
@@ -353,7 +359,8 @@ Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
.. 3.2.2.1.5
Test ID: LTD.Throughput.RFC2544.BackToBackFrames
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2544 Back To Back Frames Test
**Prerequisite Test**: N
@@ -396,7 +403,8 @@ Test ID: LTD.Throughput.RFC2544.BackToBackFrames
.. 3.2.2.1.6
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
@@ -438,7 +446,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
.. 3.2.2.1.7
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
**Prerequisite Test**:
@@ -494,7 +503,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
.. 3.2.2.1.8
Test ID: LTD.Throughput.RFC6201.ResetTime
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 6201 Reset Time Test
**Prerequisite Test**: N/A
@@ -553,6 +563,7 @@ Test ID: LTD.Throughput.RFC6201.ResetTime
* Transmitting test frame despite the physical link status.
whereas the Timestamp method SHOULD be used if the test tool supports:
+
* Timestamping each frame.
* Monitoring received frame's timestamp.
* Transmitting frames only if the physical link status is up.
@@ -585,7 +596,8 @@ Test ID: LTD.Throughput.RFC6201.ResetTime
.. 3.2.2.1.9
Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Forwarding Rate Test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
@@ -647,7 +659,8 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
.. 3.2.2.1.10
Test ID: LTD.Throughput.RFC2889.ForwardPressure
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Forward Pressure Test
**Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate
@@ -682,7 +695,8 @@ Test ID: LTD.Throughput.RFC2889.ForwardPressure
.. 3.2.2.1.11
Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Error Frames Filtering Test
**Prerequisite Test**: N/A
@@ -718,7 +732,8 @@ Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
.. 3.2.2.1.12
Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Broadcast Frame Forwarding Test
**Prerequisite Test**: N
@@ -763,7 +778,8 @@ Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
.. 3.2.2.1.13
Test ID: LTD.Throughput.RFC2544.WorstN-BestN
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
**Prerequisite Test**: N/A
@@ -856,6 +872,7 @@ Test ID: LTD.Throughput.RFC2544.WorstN-BestN
Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
@@ -935,6 +952,7 @@ Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
@@ -948,6 +966,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
a constant load (fixed length frames at a fixed interval time).
Each test case requires:
+
* selection of a specific match action(s),
* specifying a percentage of total traffic that is elligible
for the match action,
@@ -965,6 +984,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
or packet that matches a set of flow classification parameters
(typically frame/packet header fields). A match action may or may
not modify a packet/frame. Match actions include [1]:
+
* output : outputs a packet to a particular port.
* normal: Subjects the packet to traditional L2/L3 processing
(MAC learning).
@@ -973,18 +993,19 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
on which flooding is disabled.
* all: Outputs the packet on all switch physical ports other
than the port on which it was received.
- * local: Outputs the packet on the ``local port,'' which
+ * local: Outputs the packet on the ``local port``, which
corresponds to the network device that has the same name as
the bridge.
* in_port: Outputs the packet on the port from which it was
received.
* Controller: Sends the packet and its metadata to the
- OpenFlow controller as a ``packet in'' message.
+ OpenFlow controller as a ``packet in`` message.
* enqueue: Enqueues the packet on the specified queue
within port.
* drop: discard the packet.
Modifications include [1]:
+
* mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification
* mod_dl_src: Sets the source Ethernet address.
* mod_dl_dst: Sets the destination Ethernet address.
@@ -1006,6 +1027,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
During this test, the DUT must perform the following operations on
the traffic flow:
+
* Perform packet parsing on the DUT’s ingress port.
* Perform any relevant address look-ups on the DUT’s ingress
ports.
@@ -1034,6 +1056,7 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
**Metrics Collected:**
The following are the metrics collected for this test:
+
* The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps
of the DUT for each frame size with X% packet loss.
* The average latency of the traffic flow when passing through
@@ -1057,7 +1080,8 @@ Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
.. 3.2.2.2
Packet Latency tests
----------------------------
+--------------------
+
These tests will measure the store and forward latency as well as the packet
delay variation for various packet types through the virtual switch. The
following list is not exhaustive but should indicate the type of tests
@@ -1066,7 +1090,8 @@ that should be required. It is expected that more will be added.
.. 3.2.2.2.1
Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Initial Packet Processing Latency
**Prerequisite Test**: N/A
@@ -1115,7 +1140,8 @@ Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
.. 3.2.2.2.2
Test ID: LTD.PacketDelayVariation.RFC3393.Soak
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Packet Delay Variation Soak Test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
@@ -1148,7 +1174,8 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak
.. 3.2.2.3
Scalability tests
-------------------------
+-----------------
+
The general aim of these tests is to understand the impact of large flow
table size and flow lookups on throughput. The following list is not
exhaustive but should indicate the type of tests that should be required.
@@ -1157,7 +1184,8 @@ It is expected that more will be added.
.. 3.2.2.3.1
Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% loss Flow Scalability throughput test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
@@ -1212,7 +1240,8 @@ Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
.. 3.2.2.3.2
Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% loss Memory Bandwidth Scalability test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
@@ -1262,6 +1291,7 @@ Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
Latency Test
@@ -1342,7 +1372,8 @@ Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
.. 3.2.2.3.4
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: VNF Scalability RFC 2544 Throughput and Latency Profile
**Prerequisite Test**: N/A
@@ -1421,6 +1452,7 @@ Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
Activation tests
----------------
+
The general aim of these tests is to understand the capacity of the
and speed with which the vswitch can accommodate new flows.
@@ -1428,6 +1460,7 @@ and speed with which the vswitch can accommodate new flows.
Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Address Caching Capacity Test
**Prerequisite Test**: N/A
@@ -1479,7 +1512,8 @@ Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
.. 3.2.2.4.2
Test ID: LTD.Activation.RFC2889.AddressLearningRate
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Address Learning Rate Test
**Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
@@ -1516,7 +1550,8 @@ Test ID: LTD.Activation.RFC2889.AddressLearningRate
.. 3.2.2.5
Coupling between control path and datapath Tests
--------------------------------------------------------
+------------------------------------------------
+
The following tests aim to determine how tightly coupled the datapath
and the control path are within a virtual switch. The following list
is not exhaustive but should indicate the type of tests that should be
@@ -1525,7 +1560,8 @@ required. It is expected that more will be added.
.. 3.2.2.5.1
Test ID: LTD.CPDPCouplingFlowAddition
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Control Path and Datapath Coupling
**Prerequisite Test**:
@@ -1573,7 +1609,8 @@ Test ID: LTD.CPDPCouplingFlowAddition
.. 3.2.2.6
CPU and memory consumption
----------------------------------
+--------------------------
+
The following tests will profile a virtual switch's CPU and memory
utilization under various loads and circumstances. The following
list is not exhaustive but should indicate the type of tests that
@@ -1583,6 +1620,7 @@ should be required. It is expected that more will be added.
Test ID: LTD.Stress.RFC2544.0PacketLoss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% Loss CPU OR Memory Stress Test
**Prerequisite Test**:
@@ -1620,7 +1658,8 @@ Test ID: LTD.Stress.RFC2544.0PacketLoss
.. 3.2.2.7
Summary List of Tests
-----------------------------
+---------------------
+
1. Throughput tests
- Test ID: LTD.Throughput.RFC2544.PacketLossRatio
@@ -1650,7 +1689,7 @@ Summary List of Tests
- LTD.Scalability.VNF.RFC2544.PacketLossProfile
- LTD.Scalability.VNF.RFC2544.PacketLossRatio
-4. Acivation tests
+4. Activation tests
- Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
- Test ID: LTD.Activation.RFC2889.AddressLearningRate
diff --git a/docs/requirements/vswitchperf_ltp.rst b/docs/requirements/vswitchperf_ltp.rst
index c6692d9a..6b2ee9bc 100644
--- a/docs/requirements/vswitchperf_ltp.rst
+++ b/docs/requirements/vswitchperf_ltp.rst
@@ -1300,20 +1300,26 @@ vsperf CI
vsperf CI jobs are broken down into:
* Daily job:
+
* Runs everyday takes about 10 hours to complete.
* TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan
phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'.
* TESTPARAM_DAILY='--test-params pkt_sizes=64,128,512,1024,1518'.
+
* Merge job:
+
* Runs whenever patches are merged to master.
* Runs a basic Sanity test.
+
* Verify job:
+
* Runs every time a patch is pushed to gerrit.
* Builds documentation.
Scripts:
--------
There are 2 scripts that are part of VSPERFs CI:
+
* build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is
used to run vsperf with the appropriate cli parameters.
* vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG