summaryrefslogtreecommitdiffstats
path: root/docs/testing/developer/devguide/requirements
diff options
context:
space:
mode:
authorAl Morton <acmorton@att.com>2018-01-22 13:06:45 -0500
committerAl Morton <acmorton@att.com>2018-01-29 10:00:54 -0500
commit1473aa1b6bbd2d37c19b8c6ce03b9de133f33512 (patch)
tree4e8b1ea607b245e840413d29cdf013f5ff6a0359 /docs/testing/developer/devguide/requirements
parent52c154c463705b51c54d639026093ecd80c9a6b4 (diff)
LTD: Expand requirements of Soak/Long Duration Tests
In Project discussion on Jan 17, 2018 including review of RFC 8204, we identified the following five areas to improve the Soak tests. See the JIRA ticket for details. JIRA: VSPERF-557 Change-Id: I5314c283a383c57d99863fcf7262c4f28e1c12a8 Signed-off-by: Al Morton <acmorton@att.com> Reviewed-by: Martin Klozik <martinx.klozik@intel.com> Reviewed-by: Maryam Tahhan <maryam.tahhan@intel.com> Reviewed-by: Christian Trautman <ctrautma@redhat.com> Reviewed-by: Sridhar Rao <sridhar.rao@spirent.com> Reviewed-by: Trevor Cooper <trevor.cooper@intel.com> Reviewed-by: Richard Elias <richardx.elias@intel.com>
Diffstat (limited to 'docs/testing/developer/devguide/requirements')
-rw-r--r--docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst108
1 files changed, 90 insertions, 18 deletions
diff --git a/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst b/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst
index e1372520..c703ff40 100644
--- a/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst
+++ b/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst
@@ -413,7 +413,21 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
**Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
- **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
+ **Prerequisite Tests**:
+
+ LTD.Throughput.RFC2544.PacketLossRatio will determine the offered load and
+ frame size for which the maximum theoretical throughput of the interface
+ has not been achieved. As described in RFC 2544 section 24, the final
+ determination of the benchmark SHOULD be conducted using a full length
+ trial, and for this purpose the duration is 5 minutes with zero loss ratio.
+
+ It is also essential to verify that the Traffic Generator has sufficient
+ stability to conduct Soak tests. Therefore, a prerequisite is to perform
+ this test with the DUT removed and replaced with a cross-over cable (or
+ other equivalent very low overhead method such as a loopback in a HW switch),
+ so that the traffic generator (and any other network involved) can be tested
+ over the Soak period. Note that this test may be challenging for software-
+ based traffic generators.
**Priority**:
@@ -422,12 +436,19 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
The aim of this test is to understand the Max Forwarding Rate stability
over an extended test duration in order to uncover any outliers. To allow
for an extended test duration, the test should ideally run for 24 hours
- or, if this is not possible, for at least 6 hours. For this test, each frame
- size must be sent at the highest Throughput rate with X% packet loss, as
- determined in the prerequisite test. The default loss percentages to be
- tested are: - X = 0% - X = 10^-7%
+ or if this is not possible, for at least 6 hours.
- Note: Other values can be tested if required by the user.
+ For this test, one frame size must be sent at the highest frame rate with
+ X% packet loss ratio, as determined in the prerequisite test (a short trial).
+ The loss ratio shall be measured and recorded every 5 minutes during the test
+ (it may be sufficient to collect lost frame counts and divide by the number
+ of frames sent in 5 minutes to see if a threshold has been crossed,
+ and accept some small inaccuracy in the threshold evaluation, not the result).
+ The default loss ratio is X = 0% and loss ratio > 10^-7% is the default
+ threshold to terminate the test early (or inform the test operator of
+ the failure status).
+
+ Note: Other values of X and loss threshold can be tested if required by the user.
**Expected Result**:
@@ -441,13 +462,13 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
and reporting any time intervals with packet loss. The
`RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
Forwarding Rate shall be measured in each interval.
- An interval of 60s is suggested.
+ An interval of 300s is suggested.
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
- The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
PDV form of delay variation on the traffic flow,
- using the 99th percentile.
+ using the 99th percentile, may also be collected.
.. 3.2.2.1.7
@@ -457,7 +478,22 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
**Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
**Prerequisite Test**:
+
LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
+ will determine the offered load and
+ frame size for which the maximum theoretical throughput of the interface
+ has not been achieved. As described in RFC 2544 section 24, the final
+ determination of the benchmark SHOULD be conducted using a full length
+ trial, and for this purpose the duration is 5 minutes with zero loss ratio.
+
+ It is also essential to verify that the Traffic Generator has sufficient
+ stability to conduct Soak tests. Therefore, a prerequisite is to perform
+ this test with the DUT removed and replaced with a cross-over cable (or
+ other equivalent very low overhead method such as a loopback in a HW switch),
+ so that the traffic generator (and any other network involved) can be tested
+ over the Soak period. Note that this test may be challenging for software-
+ based traffic generators.
+
**Priority**:
@@ -466,9 +502,19 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
The aim of this test is to understand the Max Forwarding Rate stability over an
extended test duration in order to uncover any outliers. To allow for an
extended test duration, the test should ideally run for 24 hours or, if
- this is not possible, for at least 6 hour. For this test, each frame
- size must be sent at the highest Throughput rate with 0% packet loss, as
- determined in the prerequisite test.
+ this is not possible, for at least 6 hours.
+
+ For this test, one frame size must be sent at the highest frame rate with
+ X% packet loss ratio, as determined in the prerequisite test (a short trial).
+ The loss ratio shall be measured and recorded every 5 minutes during the test
+ (it may be sufficient to collect lost frame counts and divide by the number
+ of frames sent in 5 minutes to see if a threshold has been crossed,
+ and accept some small inaccuracy in the threshold evaluation, not the result).
+ The default loss ratio is X = 0% and loss ratio > 10^-7% is the default
+ threshold to terminate the test early (or inform the test operator of
+ the failure status).
+
+ Note: Other values of X and loss threshold can be tested if required by the user.
During this test, the DUT must perform the following operations on the
traffic flow:
@@ -498,13 +544,13 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
and reporting any time intervals with packet loss. The
`RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
Forwarding Rate shall be measured in each interval.
- An interval of 60s is suggested.
+ An interval of 300s is suggested.
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
- The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
PDV form of delay variation on the traffic flow, using the 99th
- percentile.
+ percentile, may also be collected.
.. 3.2.2.1.8
@@ -1150,7 +1196,22 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak
**Title**: Packet Delay Variation Soak Test
- **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
+ **Prerequisite Tests**:
+
+ LTD.Throughput.RFC2544.PacketLossRatio will determine the offered load and
+ frame size for which the maximum theoretical throughput of the interface
+ has not been achieved. As described in RFC 2544 section 24, the final
+ determination of the benchmark SHOULD be conducted using a full length
+ trial, and for this purpose the duration is 5 minutes with zero loss ratio.
+
+ It is also essential to verify that the Traffic Generator has sufficient
+ stability to conduct Soak tests. Therefore, a prerequisite is to perform
+ this test with the DUT removed and replaced with a cross-over cable (or
+ other equivalent very low overhead method such as a loopback in a HW switch),
+ so that the traffic generator (and any other network involved) can be tested
+ over the Soak period. Note that this test may be challenging for software-
+ based traffic generators.
+
**Priority**:
@@ -1160,9 +1221,20 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak
variation for different frame sizes over an extended test duration and
to determine if there are any outliers. To allow for an extended test
duration, the test should ideally run for 24 hours or, if this is not
- possible, for at least 6 hour. For this test, each frame size must be
- sent at the highest possible throughput with 0% packet loss, as
- determined in the prerequisite test.
+ possible, for at least 6 hours.
+
+ For this test, one frame size must be sent at the highest frame rate with
+ X% packet loss ratio, as determined in the prerequisite test (a short trial).
+ The loss ratio shall be measured and recorded every 5 minutes during the test
+ (it may be sufficient to collect lost frame counts and divide by the number
+ of frames sent in 5 minutes to see if a threshold has been crossed,
+ and accept some small inaccuracy in the threshold evaluation, not the result).
+ The default loss ratio is X = 0% and loss ratio > 10^-7% is the default
+ threshold to terminate the test early (or inform the test operator of
+ the failure status).
+
+ Note: Other values of X and loss threshold can be tested if required by the user.
+
**Expected Result**:
@@ -1173,7 +1245,7 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak
- The packet delay variation value for traffic passing through the DUT.
- The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
PDV form of delay variation on the traffic flow,
- using the 99th percentile, for each 60s interval during the test.
+ using the 99th percentile, for each 300s interval during the test.
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.