1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
.. This work is licensed under a Creative Commons Attribution 4.0 International
.. License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Intel Corporation and others.
================
Running the test
================
.. _NFV-TST009: https://docbox.etsi.org/ISG/NFV/open/Publications_pdf/Specs-Reports/NFV-TST%20009v3.2.1%20-%20GS%20-%20NFVI_Benchmarks.pdf
.. _TST009_Throughput_64B_64F.test: https://github.com/opnfv/samplevnf/blob/master/VNFs/DPPD-PROX/helper-scripts/rapid/TST009_Throughput_64B_64F.test
.. _rapid_location: https://github.com/opnfv/samplevnf/blob/master/VNFs/DPPD-PROX/helper-scripts/rapid/
Overview
--------
A default test will be run automatically when you launch the testing. The
details and definition of that test are defined in file
TST009_Throughput_64B_64F.test_.
We will discuss the sections of such a test file and how this can be changed to
accomodate the testing you want to execute. This will be done by creating your
own test file and making sure it becomes part of your testcases.yaml, as will
be shown below.
As the name of the default test file suggests, the test will find the
throughput, latency and packet loss according to NFV-TST009_, for packets that
are 64 bytes long and for 64 different flows.
Test File Description
---------------------
The test file has multiple sections. The first section is a generic section
called TestParameters. Then there are 1 or more sections desribing the test
machines we will be using in the test. The sections are named TestMx, where x
is a number (starting with 1). The tests to be executed are described in a
section called testy, where y is the number of the test to be executed,
starting with 1. In this automated testing driven by Xtesting, we will
typically only run 1 test.
TestParameters
^^^^^^^^^^^^^^
In this section, the name of the test is specified. This is only used in the
reporting and has no influence on the actual testing.
.. code-block:: console
name = Rapid_ETSINFV_TST009
The number of test that will be executed by this run and that will be described
in the [testy] sections, is defined by the number_of_tests parameter. In the
Xtesting framework that we are using here, this will typically be set to 1.
.. code-block:: console
number_of_tests = 1
The total number of machines to be used in this testing will be defined by the
parameter total_number_of_test_machines. The function that these machines have
in this test will be described in the [TestMx] section. Typically, this number
will be set to 2, but many more machines can particiapte in a test.
.. code-block:: console
total_number_of_test_machines = 2
lat_percentile is a variable that is setting which percentile to use during the
course of this test. This will be used to report the percentile round trip
latency and is a better measurement for the high latencies during this test than
the maximum latency which will also be reported. Note that we also report the
total round trip latency histogram.
.. code-block:: console
lat_percentile = 99
TestMx
^^^^^^
In the TestMx sections, where x denotes the index of the machine, the function
of the machine in the testing, will be described. The machine can be defined as
a generator, or as a packet reflector (swap function). The machines can be any
machine that is created upfront (See step 3 of the installation steps). Other
functions can also be executed by the test machines and examples of test files
can be found in rapid_location_.
The first parameter is the name of the machine and is only used for referencing
the machine. This will be the name of the PROX instance and will be shown in
case you run the PROX UI. In this automated testing, this will be not be
visible.
The config_file parameter defines which PROX config file is used by the PROX
program and what PROX will be
doing. For a generator, this will typically be gen.cfg. Multiple cfg files
exist in the rapid_location_.
The dest_vm parameter is used by a generator to find out to
which VM he needs to send the packets. In the example below, the packets will be
sent to TestM2.
The gencores parameter defines a list of cores to be used for the generator tasks.
Note that if you specify more than 1 core, the interface will need to support as
many tx queues as there are generator cores.
The latcores parameter specifies a
list of cores to be used by the latency measurement tasks. You need as many rx
queues on the interface as specified in the latcores parameter.
The default value for the
bucket_size_exp parameter is 12. It is also its minimum value. In case most of
the latency measurements in the histogram are falling in the last bucket, this
number needs to be increased. Every time you increase this number by 1, the
bucket size for the latency histogram is multiplied by 2. There are 128 buckets
in the histogram.
cores is a parameter that will be used by non-generator configurations that
don't need a disctinction between generator and latency cores (e.g. swap.cfg).
Changing these parameters requires in depth knowledge of the PROX tool and is
not something to start with.
.. code-block:: console
name = Generator
config_file = gen.cfg
dest_vm = 2
gencores = [1]
latcores = [3]
#bucket_size_exp = 12
testy
^^^^^
In the testy sections, where y denotes the index of the test, the test that will
be executed on the machines that were specified in the TestMx sections, will be
described. Using Xtesting, we will typically only use 1 test.
Parameter test is defining which test needs to be run. This is a hardcoded
string and can only be one of the following ['flowsizetest', 'TST009test',
'fixed_rate', 'increment_till_fail', 'corestats', 'portstats', 'impairtest',
'irqtest', 'warmuptest']. In this project, we will use the TST009test testing.
For examples of the other tests, please check out the other test files in
rapid_location_.
The pass_threshold parameter defines the success criterium for the test. When
this test uses multiple combinations of packet size and flows, all combinations
must be meeting the same threshold. If one of the combinations fails, the test
will be reported as failed.
The threshold is expressed in Mpps.
The imixs parameter defines the pakcet sizes that will be used. Each element in
the imixs list will result in a separate test. Each element is on its turn a
list of packet sizes which will be used during one test execution. If you only
want to test 1 imix size, define imixs with only one element. For each element in
the imixs list, the generator will iterate over the packet lengths and send them
out in the order as specified in the list. An example of an imix list is [128,
256, 64, 64, 128]. In this case, 40% of the packets will have a size of 64
bytes, 40% will have a packet size of 128 and 20% will have a packet size of
256. When using this with Xtesting, we will typically only use 1 imix. When
needing results for more sizes, one should create a specific test file per size
and launch the different tests using Xtesting.
The flows parameter is a list of flow sizes. For each flow size, a test will be
run with the specified amount of flows. The flow size needs to be a power of 2,
max 2^30. If not a power of 2, we will use the lowest power of 2 that is larger
than the requested number of flows. e.g. 9 will result in 16 flows.
Same remark as for the imixs parameter: we will only use one element in the
flows list. When more flows need to be tested, create a different test file and
launch it using Xtesting.
The drop_rate_threshold parameter specifies the maximum ratio of packets than
can be dropped while still considering
the test run as succesful. Note that a value of 0 means an absolute zero packet
loss: even if we lose 1 packet during a certain step in a test run, it will be
marked as failed.
The lat_avg_threshold, lat_perc_threshold, lat_max_threshold parameters
are thresholds to define
the maximal acceptable round trip latency to mark the test step as successful.
You can set this threshold for the average, the percentile and the maximum
latency. Which percentile is being used is defined in the TestParameters section.
All these thresholds are expressed in micro-seconds. You can also put the value
to inf, which means the threshold will never be reached and hence the threshold
value is not being used to define if the run is successful or not.
The MAXr, MAXz, MAXFramesPerSecondAllIngress and StepSize parameters are defined in
NFV-TST009_ and are used to control the binary search algorithm.
The ramp_step variable controls the ramping of the generated traffic. When
not specified, the requested traffic for each step in the testing will be
applied immediately. If specified, the generator will slowly go to the requested
speed by increasing the traffic each second with the value specified in this
parameter till it reaches the requested speed. This parameter is expressed in
100Mb/s.
.. code-block:: console
pass_threshold=0.001
imixs=[[128, 256, 64, 64, 128]]
flows=[64]
drop_rate_threshold = 0
lat_avg_threshold = inf
lat_perc_threshold = inf
lat_max_threshold = inf
MAXr = 3
MAXz = 5000
MAXFramesPerSecondAllIngress = 12000000
StepSize = 10000
#ramp_step = 1
Modifying the test
------------------
In case you want to modify the parameters as specified in
TST009_Throughput_64B_64F.test_, it is best to create your own test file. Your
test file will need to be uploaded to the test container. Hence you will have to
rebuild your container, and add an extra copy command to the Dockerfile so that
your new test file will be avaialble in the container.
Then you will need to modify the testcases.yaml file. One of the args that you
can specify is the test_file. Put your newly created test file as the new value
for this argument.
Now build and publish your test container as specified in steps 5 & 6 of the
installation procedure.
Note that other arguments than test_file can be specified in testcases.yaml. For
a list of arugments, please check out the test_params dictionary in the
rapid_defaults.py that you can find in rapid_location_.
It is adviced not to change these parameters unless you have an in-depth
knowledge of the code.
The only 2 arguments that van be changed are the test_file which was already
discussed and the runtime argument. This argument defines how long each test run
will take and is expressed in seconds.
|
