From c8cfc7346ba5d71b6a988ffff512bf1b47bdf52f Mon Sep 17 00:00:00 2001 From: Al Morton Date: Fri, 2 Oct 2015 02:18:09 +0100 Subject: test_spec: LTD: Some "Throughput" Tests belong in another category In the LTD, some of the tests with "Throughput" in the title are not actually testing throughput in the data path. Since the test names are used by TOIT/VSPERF to group similar tests for batch execution, it's important that the names accurately represent the test methods and metrics measured. JIRA: VSPERF-98 Change-Id: I620ddc0779232f2dd325379005faba8692fea18c Signed-off-by: Al Morton Reviewed-by: Maryam Tahhan Reviewed-by: Billy O'Mahony Reviewed-by: Gene Snider --- docs/to-be-reorganized/vswitchperf_ltd.rst | 204 +++++++++++++++-------------- 1 file changed, 107 insertions(+), 97 deletions(-) mode change 100644 => 100755 docs/to-be-reorganized/vswitchperf_ltd.rst (limited to 'docs') diff --git a/docs/to-be-reorganized/vswitchperf_ltd.rst b/docs/to-be-reorganized/vswitchperf_ltd.rst old mode 100644 new mode 100755 index 321ae418..4e194d0c --- a/docs/to-be-reorganized/vswitchperf_ltd.rst +++ b/docs/to-be-reorganized/vswitchperf_ltd.rst @@ -150,9 +150,9 @@ switch, the tests will be broken down into the following categories: packet and frame delay for constant loads. - **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through - the switch. + the virtual switch. - **Request/Response Performance** Tests (TCP, UDP) the measure the - transaction rate through the switch. + transaction rate through the virtual switch. - **Packet Delay Tests** to understand latency distribution for different packet sizes and over an extended test run to uncover outliers. @@ -502,7 +502,7 @@ with non default parameters will be stated explicitly**. **Note**: For throughput tests unless stated otherwise, test configurations should ensure that traffic traverses the installed flows -through the switch, i.e. flows are installed and have an appropriate +through the virtual switch, i.e. flows are installed and have an appropriate time out that doesn't expire before packet transmission starts. 2.2.3.2 Flow Classification @@ -804,7 +804,7 @@ platform should be configured for every test after this (# of vCPUs, vNICs, Memory, affinitization…) is how it should be configured for every test that uses a VNF after this. -2.2.4 RFCs for testing switch performance +2.2.4 RFCs for testing virtual switch performance ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The starting point for defining the suite of tests for benchmarking the @@ -850,7 +850,7 @@ Types of tests are: network’s ability to support real-time applications in which a large amount of frame loss will rapidly degrade service quality. -4. Burst test assesses the buffering capability of a switch. It +4. Burst test assesses the buffering capability of a virtual switch. It measures the maximum number of frames received at full line rate before a frame is lost. In carrier Ethernet networks, this measurement validates the excess information rate (EIR) as defined in @@ -1501,90 +1501,6 @@ Test ID: LTD.Throughput.RFC2889.ForwardPressure - Physical → virtual switch → physical. -Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - **Title**: RFC2889 Address Caching Capacity Test - - **Prerequisite Test**: N/A - - **Priority**: - - **Description**: - - Please note this test is only applicable to switches that are capable of - MAC learning. The aim of this test is to determine the address caching - capacity of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under `Default Test Parameters <#DefaultParams>`__. - - In order to run this test the aging time, that is the maximum time the - DUT will keep a learned address in its flow table, and a set of initial - addresses, whose value should be >= 1 and <= the max number supported by - the implementation must be known. Please note that if the aging time is - configurable it must be longer than the time necessary to produce frames - from the external source at the specified rate. If the aging time is - fixed the frame rate must be brought down to a value that the external - source can produce in a time that is less than the aging time. - - Learning Frames should be sent from an external source to the DUT to - install a number of flows. The Learning Frames must have a fixed - destination address and must vary the source address of the frames. The - DUT should install flows in its flow table based on the varying source - addresses. Frames should then be transmitted from an external source at - a suitable frame rate to see if the DUT has properly learned all of the - addresses. If there is no frame loss and no flooding, the number of - addresses sent to the DUT should be increased and the test is repeated - until the max number of cached addresses supported by the DUT - determined. - - **Expected Result**: - - **Metrics collected**: - - The following are the metrics collected for this test: - - - Number of cached addresses supported by the DUT. - - CPU and memory utilization may also be collected as part of this - test, to determine the vSwitch's performance footprint on the system. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). - -Test ID: LTD.Throughput.RFC2889.AddressLearningRate -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - **Title**: RFC2889 Address Learning Rate Test - - **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity - - **Priority**: - - **Description**: - - Please note this test is only applicable to switches that are capable of - MAC learning. The aim of this test is to determine the rate of address - learning of the DUT for a constant load (fixed length frames at a fixed - interval time). The selected frame sizes are those previously defined - under `Default Test Parameters <#DefaultParams>`__, traffic should be - sent with each IPv4/IPv6 address incremented by one. The rate at which - the DUT learns a new address should be measured. The maximum caching - capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken - into consideration as the maximum number of addresses for which the - learning rate can be obtained. - - **Expected Result**: It may be worthwhile to report the behaviour when - operating beyond address capacity - some DUTs may be more friendly to - new addresses than others. - - **Metrics collected**: - - The following are the metrics collected for this test: - - - The address learning rate of the DUT. - - **Deployment scenario**: - - - Physical → virtual switch → 2 x physical (one receiving, one listening). Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1698,8 +1614,8 @@ Test ID: LTD.PacketLatency.InitialPacketProcessingLatency plus 10%. Average packet latency will be determined over 1,000,000 packets. - This test is intended only for non-learning switches; For learning - switches use RFC2889. + This test is intended only for non-learning virtual switches; For learning + virtual switches use RFC2889. For this test, only unidirectional traffic is required. @@ -1844,6 +1760,97 @@ Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability - The DUT's 0% packet loss throughput in the presence of cache sharing and memory bandwidth between processes. +2.3.4 Activation tests +~~~~~~~~~~~~~~~~~~~~~~~~ +The general aim of these tests is to understand the capacity of the +and speed with which the vswitch can accomodate new flows. + +Test ID: LTD.Activation.RFC2889.AddressCachingCapacity +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + **Title**: RFC2889 Address Caching Capacity Test + + **Prerequisite Test**: N/A + + **Priority**: + + **Description**: + + Please note this test is only applicable to virtual switches that are capable of + MAC learning. The aim of this test is to determine the address caching + capacity of the DUT for a constant load (fixed length frames at a fixed + interval time). The selected frame sizes are those previously defined + under `Default Test Parameters <#DefaultParams>`__. + + In order to run this test the aging time, that is the maximum time the + DUT will keep a learned address in its flow table, and a set of initial + addresses, whose value should be >= 1 and <= the max number supported by + the implementation must be known. Please note that if the aging time is + configurable it must be longer than the time necessary to produce frames + from the external source at the specified rate. If the aging time is + fixed the frame rate must be brought down to a value that the external + source can produce in a time that is less than the aging time. + + Learning Frames should be sent from an external source to the DUT to + install a number of flows. The Learning Frames must have a fixed + destination address and must vary the source address of the frames. The + DUT should install flows in its flow table based on the varying source + addresses. Frames should then be transmitted from an external source at + a suitable frame rate to see if the DUT has properly learned all of the + addresses. If there is no frame loss and no flooding, the number of + addresses sent to the DUT should be increased and the test is repeated + until the max number of cached addresses supported by the DUT + determined. + + **Expected Result**: + + **Metrics collected**: + + The following are the metrics collected for this test: + + - Number of cached addresses supported by the DUT. + - CPU and memory utilization may also be collected as part of this + test, to determine the vSwitch's performance footprint on the system. + + **Deployment scenario**: + + - Physical → virtual switch → 2 x physical (one receiving, one listening). + +Test ID: LTD.Activation.RFC2889.AddressLearningRate +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + **Title**: RFC2889 Address Learning Rate Test + + **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity + + **Priority**: + + **Description**: + + Please note this test is only applicable to virtual switches that are capable of + MAC learning. The aim of this test is to determine the rate of address + learning of the DUT for a constant load (fixed length frames at a fixed + interval time). The selected frame sizes are those previously defined + under `Default Test Parameters <#DefaultParams>`__, traffic should be + sent with each IPv4/IPv6 address incremented by one. The rate at which + the DUT learns a new address should be measured. The maximum caching + capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken + into consideration as the maximum number of addresses for which the + learning rate can be obtained. + + **Expected Result**: It may be worthwhile to report the behaviour when + operating beyond address capacity - some DUTs may be more friendly to + new addresses than others. + + **Metrics collected**: + + The following are the metrics collected for this test: + + - The address learning rate of the DUT. + + **Deployment scenario**: + + - Physical → virtual switch → 2 x physical (one receiving, one listening). + + 2.3.5 Coupling between control path and datapath Tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following tests aim to determine how tightly coupled the datapath @@ -1897,7 +1904,7 @@ Test ID: LTD.CPDPCouplingFlowAddition - Physical → virtual switch → physical. -2.3.4 CPU and memory consumption +2.3.6 CPU and memory consumption ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following tests will profile a virtual switch's CPU and memory utilization under various loads and circumstances. The following @@ -1936,7 +1943,7 @@ Test ID: LTD.CPU.RFC2544.0PacketLoss - The configuration of the stress tool (for example the command line parameters used to start it.) -2.3.9 Summary List of Tests +2.3.7 Summary List of Tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Throughput tests @@ -1950,8 +1957,6 @@ Test ID: LTD.CPU.RFC2544.0PacketLoss - Test ID: LTD.Throughput.RFC6201.ResetTime - Test ID: LTD.Throughput.RFC2889.MaxForwardingRate - Test ID: LTD.Throughput.RFC2889.ForwardPressure - - Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity - - Test ID: LTD.Throughput.RFC2889.AddressLearningRate - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding @@ -1965,10 +1970,15 @@ Test ID: LTD.CPU.RFC2544.0PacketLoss - Test ID: LTD.Scalability.RFC2544.0PacketLoss - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability -4. Coupling between control path and datapath Tests +4. Acivation tests + + - Test ID: LTD.Activation.RFC2889.AddressCachingCapacity + - Test ID: LTD.Activation.RFC2889.AddressLearningRate + +5. Coupling between control path and datapath Tests - Test ID: LTD.CPDPCouplingFlowAddition -5. CPU and memory consumption +6. CPU and memory consumption - Test ID: LTD.CPU.RFC2544.0PacketLoss -- cgit 1.2.3-korg