summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/04-running_the_test.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/user/userguide/04-running_the_test.rst')
-rw-r--r--docs/testing/user/userguide/04-running_the_test.rst207
1 files changed, 207 insertions, 0 deletions
diff --git a/docs/testing/user/userguide/04-running_the_test.rst b/docs/testing/user/userguide/04-running_the_test.rst
new file mode 100644
index 00000000..35404c89
--- /dev/null
+++ b/docs/testing/user/userguide/04-running_the_test.rst
@@ -0,0 +1,207 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation and others.
+
+================
+Running the test
+================
+.. _NFV-TST009: https://docbox.etsi.org/ISG/NFV/open/Publications_pdf/Specs-Reports/NFV-TST%20009v3.2.1%20-%20GS%20-%20NFVI_Benchmarks.pdf
+.. _TST009_Throughput_64B_64F.test: https://github.com/opnfv/samplevnf/blob/master/VNFs/DPPD-PROX/helper-scripts/rapid/TST009_Throughput_64B_64F.test
+.. _rapid_location: https://github.com/opnfv/samplevnf/blob/master/VNFs/DPPD-PROX/helper-scripts/rapid/
+
+Overview
+--------
+A default test will be run automatically when you launch the testing. The
+details and definition of that test is defined in file
+TST009_Throughput_64B_64F.test_.
+
+We will discuss the sections of such a test file and how this can be changed to
+accomodate the testing you want to execute. This will be done by creating your
+own test file and making sure it becomes part of your testcases.yaml, as will
+be shown below.
+
+Test File Description
+---------------------
+The test file has multiple sections. The first section is a generic section
+called TestParameters. Then there are 1 or more sections desribing the test
+machines we will be using in the test. The sections are named TestMx, where x
+is a number (starting with 1). The tests to be executed are described in a
+section called testy, where y is the number of the test to be executed,
+starting with 1. In this automated testing driven by Xtesting, we will
+typically only run 1 test.
+
+TestParameters
+^^^^^^^^^^^^^^
+In this section, the name of the test is specified. This is only used in the
+reporting and has no influence on the actual testing.
+
+.. code-block:: console
+
+ name = Rapid_ETSINFV_TST009
+
+The number of test that will be executed by this run and that will be described
+in the [testy] sections, is defined by the number_of_tests parameter. In the
+Xtesting framework that we are using here, this will typically be set to 1.
+
+.. code-block:: console
+
+ number_of_tests = 1
+
+The total number of machines to be used in this testing will be defined by the
+parameter total_number_of_test_machines. The function that these machines have
+in this test will be described in the [TestMx] section. Typically, this number
+will be set to 2, but many more machines can particiapte in a test.
+
+.. code-block:: console
+
+ total_number_of_test_machines = 2
+
+lat_percentile is a variable that is setting which percentile to use during the
+course of this test. This will be used to report the percentile round trip
+latency and is a better measurement for the high latencies during this test than
+the maximum latency which will also be reported. Note that we also report the
+total round trip latency histogram.
+
+.. code-block:: console
+
+ lat_percentile = 99
+
+
+TestMx
+^^^^^^
+In the TestMx sections, where x denotes the index of the machine, the function
+of the machine in the testing, will be described. The machine can be defined as
+a generator, or as a packet reflector (swap function). The machines can be any
+machine that is created upfront (See step 3 of the installation steps). Other
+functions can also be executed by the test machines and examples of test files
+can be found in rapid_location_.
+
+The first parameter is the name of the machine and is only used for referencing
+the machine. This will be the name of the PROX instance and will be shown in
+case you run the PROX UI. In this automated testing, this will be not be
+visible.
+
+The PROX config file is used by the PROX program and defines what PROX will be
+doing. For a generator, this will typically be gen.cfg. Multiple cfg files
+exist in the rapid_location_. dest_vm is used by a generator to find out to
+which VM he needs to send the packets. Int e example below, the packets will be
+sent to TestM2. gencores is a list of cores to be used for the generator tasks.
+Note that if you specify more than 1 core, the interface will need to support as
+many tx queues as there are generator cores. The latcores field specifies a
+list of cores to be used by the latency measurement tasks. You need as many rx
+queueus on the interface as the number of latcores. The default value for the
+bucket_size_exp parameter is 12. It is also its minimum value. In case most of
+the latency measurements in the histogram are falling in the last bucket, this
+number needs to be increased. Every time you increase this number by 1, the
+bucket size for the latency histogram is multiplied by 2. There are 128 buckets
+in the histogram.
+cores is a parameter that will be used by non-generator configurations that
+don't need a disctinction between generator and latency cores (e.g. swap.cfg).
+
+Changing these parameters requires in depth knowledge of the PROX tool and is
+not something to start with.
+
+.. code-block:: console
+
+ name = Generator
+ config_file = gen.cfg
+ dest_vm = 2
+ gencores = [1]
+ latcores = [3]
+ #bucket_size_exp = 12
+testy
+^^^^^
+In the testy sections, where y denotes the index of the test, the test that will
+be executed on the machines that were specified in the TestMx sections, will be
+described. Using Xtesting, we will typically only use 1 test.
+Parameter test is defining which test needs to be run. This is a hardcoded
+string and can only be one of the following ['flowsizetest', 'TST009test',
+'fixed_rate', 'increment_till_fail', 'corestats', 'portstats', 'impairtest',
+'irqtest', 'warmuptest']. In this project, we will use the TST009test testing.
+For examples of the other tests, please check out the other test files in
+rapid_location_.
+
+The pass_threshold parameter defines the success criterium for the test. When
+this test uses multiple combinations of packet size and flows, all combinations
+must be meeting the same threshold. The threshold is expressed in Mpps.
+
+The imixs parameter defines the pakcet sizes that will be used. Each element in
+the imix list will result in a separate test. Each element is on its turn a
+list of packet sizes which will be used during one test execution. If you only
+want to test 1 imix size, define imixs with only one element. For each element in
+the imixs list, the generator will iterate over the packet lengths and send them
+out in the order as specified in the list. An example of an imix list is [128,
+256, 64, 64, 128]. In this case, 40% of the packets will have a size of 64
+bytes, 40% will have a packet size of 128 and 20% will have a packet size of
+256. When using this with Xtesting, we will typically only use 1 imix. When
+needing results for more sizes, one should create a specific test file per size
+and launch the different tests using Xtesting.
+
+The flows parameter is a list of flow sizes. For each flow size, a test will be
+run with the specified amount of flows. The flow size needs to be powers of 2,
+max 2^30. If not a power of 2, we will use the lowest power of 2 that is larger
+than the requested number of flows. e.g. 9 will result in 16 flows.
+Same remark as for the imixs parameter: we will only use one element in the
+flows list. When more flows need to be tested, create a differnt test file and
+launch it using Xtesting.
+
+drop_rate_threshold specifies the ratio of packets than can be dropped and still
+consider the test run as succesful. Note that a value of 0 means a zero packet
+loss: even if we lose 1 packet during a certain step in a test run, it will be
+marked as failed.
+
+lat_avg_threshold, lat_perc_threshold, lat_max_threshold are threshols to define
+the maximal acceptable round trip latency to mark the test step as successful.
+You can set this threshold for the average, the percentile and the maximum
+latency. Which percentile is being used is define in the TestParameters section.
+All these thresholds are expressed in micro-seconds. You can also put the value
+to inf, which means the threshold will never be reached and hence the threshold
+value is not being used to define if the run is successful or not.
+
+MAXr, MAXz, MAXFramesPerSecondAllIngress and StepSize are defined in
+NFV-TST009_ and are used to control the binary search algorithm.
+
+ramp_step is a variable that controls the ramping of the generated traffic. When
+not specified, the requested traffic for each step in the testing will be
+applied immediately. If specified, the generator will slowly go to the requested
+speed by increasing the traffic each second with the value specified in this
+parameter till it reached the requested speed. This parameter is expressed in
+100Mb/s.
+
+.. code-block:: console
+
+ pass_threshold=0.001
+ imixs=[[64]]
+ flows=[64]
+ drop_rate_threshold = 0
+ lat_avg_threshold = inf
+ lat_perc_threshold = inf
+ lat_max_threshold = inf
+ MAXr = 3
+ MAXz = 5000
+ MAXFramesPerSecondAllIngress = 12000000
+ StepSize = 10000
+ #ramp_step = 1
+
+Modifying the test
+------------------
+In case you want to modify the parameters as specified in
+TST009_Throughput_64B_64F.test_, it is best to create your own test file. Your
+test file will need to be uploaded to the test container. Hence you will have to
+rebuild your container, and add an extra copy command to the Dockerfile so that
+your new test file will be avaialble in the container.
+Then you will need to modify the testcases.yaml file. One of the args that you
+can specify is the test_file. Put your newly created test file as the new value
+for this argument.
+Now build and publish your test container as specified in steps 5 & 6 of the
+installation procedure.
+
+Note that other arguments than test_file can be specified in testcases.yaml. For
+a list of arugments, please check out the test_params dictionary in the
+rapid_defaults.py that you can find in rapid_location_.
+It is adviced not to change these parameters unless you have an in-depth
+knowledge of the code.
+The only 2 arguments that van be changed are the test_file which was already
+discussed and the runtime argument. This argument defines how long each test run
+will take and is expressed in seconds.