aboutsummaryrefslogtreecommitdiffstats
path: root/docs/requirements/vswitchperf_ltp.rst
blob: e4e312615ef20cc8d29f7771ac409a506d267d5f (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Intel Corporation, AT&T and others.

.. 3.1

*****************************
VSPERF LEVEL TEST PLAN (LTP)
*****************************

===============
Introduction
===============

The objective of the OPNFV project titled
**Characterize vSwitch Performance for Telco NFV Use Cases**, is to
evaluate the performance of virtual switches to identify its suitability for a
Telco Network Function Virtualization (NFV) environment. The intention of this
Level Test Plan (LTP) document is to specify the scope, approach, resources,
and schedule of the virtual switch performance benchmarking activities in
OPNFV. The test cases will be identified in a separate document called the
Level Test Design (LTD) document.

This document is currently in draft form.

.. 3.1.1


.. _doc-id:

Document identifier
=========================

The document id will be used to uniquely identify versions of the LTP. The
format for the document id will be: OPNFV\_vswitchperf\_LTP\_REL\_STATUS, where
by the status is one of: draft, reviewed, corrected or final. The document id
for this version of the LTP is: OPNFV\_vswitchperf\_LTP\_Colorado\_REVIEWED.

.. 3.1.2

.. _scope:

Scope
==========

The main purpose of this project is to specify a suite of
performance tests in order to objectively measure the current packet
transfer characteristics of a virtual switch in the NFVI. The intent of
the project is to facilitate the performance testing of any virtual switch.
Thus, a generic suite of tests shall be developed, with no hard dependencies to
a single implementation. In addition, the test case suite shall be
architecture independent.

The test cases developed in this project shall not form part of a
separate test framework, all of these tests may be inserted into the
Continuous Integration Test Framework and/or the Platform Functionality
Test Framework - if a vSwitch becomes a standard component of an OPNFV
release.

.. 3.1.3

References
===============

*  `RFC 1242 Benchmarking Terminology for Network Interconnection
   Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
*  `RFC 2544 Benchmarking Methodology for Network Interconnect
   Devices <http://www.ietf.org/rfc/rfc2544.txt>`__
*  `RFC 2285 Benchmarking Terminology for LAN Switching
   Devices <http://www.ietf.org/rfc/rfc2285.txt>`__
*  `RFC 2889 Benchmarking Methodology for LAN Switching
   Devices <http://www.ietf.org/rfc/rfc2889.txt>`__
*  `RFC 3918 Methodology for IP Multicast
   Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__
*  `RFC 4737 Packet Reordering
   Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__
*  `RFC 5481 Packet Delay Variation Applicability
   Statement <http://www.ietf.org/rfc/rfc5481.txt>`__
*  `RFC 6201 Device Reset
   Characterization <http://tools.ietf.org/html/rfc6201>`__

.. 3.1.4

Level in the overall sequence
===============================
The level of testing conducted by vswitchperf in the overall testing sequence (among
all the testing projects in OPNFV) is the performance benchmarking of a
specific component (the vswitch) in the OPNFV platfrom. It's expected that this
testing will follow on from the functional and integration testing conducted by
other testing projects in OPNFV, namely Functest and Yardstick.

.. 3.1.5

Test classes and overall test conditions
=========================================
A benchmark is defined by the IETF as: A standardized test that serves as a
basis for performance evaluation and comparison. It's important to note that
benchmarks are not Functional tests. They do not provide PASS/FAIL criteria,
and most importantly ARE NOT performed on live networks, or performed with live
network traffic.

In order to determine the packet transfer characteristics of a virtual switch,
the benchmarking tests will be broken down into the following categories:

- **Throughput Tests** to measure the maximum forwarding rate (in
  frames per second or fps) and bit rate (in Mbps) for a constant load
  (as defined by `RFC1242 <https://www.rfc-editor.org/rfc/rfc1242.txt>`__)
  without traffic loss.
- **Packet and Frame Delay Tests** to measure average, min and max
  packet and frame delay for constant loads.
- **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer
  performance, i.e. how fast systems can send and receive data through
  the virtual switch.
- **Request/Response Performance** Tests (TCP, UDP) the measure the
  transaction rate through the virtual switch.
- **Packet Delay Tests** to understand latency distribution for
  different packet sizes and over an extended test run to uncover
  outliers.
- **Scalability Tests** to understand how the virtual switch performs
  as the number of flows, active ports, complexity of the forwarding
  logic's configuration... it has to deal with increases.
- **Control Path and Datapath Coupling** Tests, to understand how
  closely coupled the datapath and the control path are as well as the
  effect of this coupling on the performance of the DUT.
- **CPU and Memory Consumption Tests** to understand the virtual
  switch’s footprint on the system, this includes:

  * CPU core utilization.
  * CPU cache utilization.
  * Memory footprint.
  * System bus (QPI, PCI, ..) utilization.
  * Memory lanes utilization.
  * CPU cycles consumed per packet.
  * Time To Establish Flows Tests.

- **Noisy Neighbour Tests**, to understand the effects of resource
  sharing on the performance of a virtual switch.

**Note:** some of the tests above can be conducted simultaneously where
the combined results would be insightful, for example Packet/Frame Delay
and Scalability.



.. 3.2

.. _details-of-LTP:

===================================
Details of the Level Test Plan
===================================

This section describes the following items:
* Test items and their identifiers (TestItems_)
* Test Traceability Matrix (TestMatrix_)
* Features to be tested (FeaturesToBeTested_)
* Features not to be tested (FeaturesNotToBeTested_)
* Approach (Approach_)
* Item pass/fail criteria (PassFailCriteria_)
* Suspension criteria and resumption requirements (SuspensionResumptionReqs_)

.. 3.2.1

.. _TestItems:

Test items and their identifiers
==================================
The test item/application vsperf is trying to test are virtual switches and in
particular their performance in an nfv environment. vsperf will first try to
measure the maximum achievable performance by a virtual switch and then it will
focus in on usecases that are as close to real life deployment scenarios as
possible.

.. 3.2.2

.. _TestMatrix:

Test Traceability Matrix
==========================
vswitchperf leverages the "3x3" matrix (introduced in
https://tools.ietf.org/html/draft-ietf-bmwg-virtual-net-02) to achieve test
traceability. The matrix was expanded to 3x4 to accommodate scale metrics when
displaying the coverage of many metrics/benchmarks). Test case covreage in the
LTD is tracked using the following catagories:


+---------------+-------------+------------+---------------+-------------+
|               |             |            |               |             |
|               |   SPEED     |  ACCURACY  |  RELIABILITY  |    SCALE    |
|               |             |            |               |             |
+---------------+-------------+------------+---------------+-------------+
|               |             |            |               |             |
|  Activation   |      X      |     X      |       X       |      X      |
|               |             |            |               |             |
+---------------+-------------+------------+---------------+-------------+
|               |             |            |               |             |
|  Operation    |      X      |      X     |       X       |      X      |
|               |             |            |               |             |
+---------------+-------------+------------+---------------+-------------+
|               |             |            |               |             |
| De-activation |             |            |               |             |
|               |             |            |               |             |
+---------------+-------------+------------+---------------+-------------+

X = denotes a test catagory that has 1 or more test cases defined.

.. 3.2.3

.. _FeaturesToBeTested:

Features to be tested
==========================

Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
includes measuring the following performance metrics:

- **Throughput** as defined by `RFC1242
  <https://www.rfc-editor.org/rfc/rfc1242.txt>`__: The maximum rate at which
  **none** of the offered frames are dropped by the DUT. The maximum frame
  rate and bit rate that can be transmitted by the DUT without any error
  should be recorded. Note there is an equivalent bit rate and a specific
  layer at which the payloads contribute to the bits. Errors and
  improperly formed frames or packets are dropped.
- **Packet delay** introduced by the DUT and its cumulative effect on
  E2E networks. Frame delay can be measured equivalently.
- **Packet delay variation**: measured from the perspective of the
  VNF/application. Packet delay variation is sometimes called "jitter".
  However, we will avoid the term "jitter" as the term holds different
  meaning to different groups of people. In this document we will
  simply use the term packet delay variation. The preferred form for this
  metric is the PDV form of delay variation defined in `RFC5481
  <https://www.rfc-editor.org/rfc/rfc5481.txt>`__. The most relevant
  measurement of PDV considers the delay variation of a single user flow,
  as this will be relevant to the size of end-system buffers to compensate
  for delay variation. The measurement system's ability to store the
  delays of individual packets in the flow of interest is a key factor
  that determines the specific measurement method. At the outset, it is
  ideal to view the complete PDV distribution. Systems that can capture
  and store packets and their delays have the freedom to calculate the
  reference minimum delay and to determine various quantiles of the PDV
  distribution accurately (in post-measurement processing routines).
  Systems without storage must apply algorithms to calculate delay and
  statistical measurements on the fly. For example, a system may store
  temporary estimates of the mimimum delay and the set of (100) packets
  with the longest delays during measurement (to calculate a high quantile,
  and update these sets with new values periodically.
  In some cases, a limited number of delay histogram bins will be
  available, and the bin limits will need to be set using results from
  repeated experiments. See section 8 of `RFC5481
  <https://www.rfc-editor.org/rfc/rfc5481.txt>`__.
- **Packet loss** (within a configured waiting time at the receiver): All
  packets sent to the DUT should be accounted for.
- **Burst behaviour**: measures the ability of the DUT to buffer packets.
- **Packet re-ordering**: measures the ability of the device under test to
  maintain sending order throughout transfer to the destination.
- **Packet correctness**: packets or Frames must be well-formed, in that
  they include all required fields, conform to length requirements, pass
  integrity checks, etc.
- **Availability and capacity** of the DUT i.e. when the DUT is fully “up”
  and connected, following measurements should be captured for
  DUT without any network packet load:

  - Includes average power consumption of the CPUs (in various power states) and
    system over specified period of time. Time period should not be less
    than 60 seconds.
  - Includes average per core CPU utilization over specified period of time.
    Time period should not be less than 60 seconds.
  - Includes the number of NIC interfaces supported.
  - Includes headroom of VM workload processing cores (i.e. available
    for applications).

.. 3.2.4

.. _FeaturesNotToBeTested:

Features not to be tested
==========================
vsperf doesn't intend to define or perform any functional tests. The aim is to
focus on performance.

.. 3.2.5

.. _Approach:

Approach
==============
The testing approach adoped by the vswitchperf project is black box testing,
meaning the test inputs can be generated and the outputs captured and
completely evaluated from the outside of the System Under Test. Some metrics
can be collected on the SUT, such as cpu or memory utilization if the
collection has no/minimal impact on benchmark.
This section will look at the deployment scenarios and the general methodology
used by vswitchperf. In addition, this section will also specify the details of
the Test Report that must be collected for each of the test cases.

.. 3.2.5.1

Deployment Scenarios
--------------------------
The following represents possible deployment test scenarios which can
help to determine the performance of both the virtual switch and the
datapaths to physical ports (to NICs) and to logical ports (to VNFs):

.. 3.2.5.1.1

Physical port → vSwitch → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: console

                                                            _
       +--------------------------------------------------+  |
       |              +--------------------+              |  |
       |              |                    |              |  |
       |              |                    v              |  |  Host
       |   +--------------+            +--------------+   |  |
       |   |   phy port   |  vSwitch   |   phy port   |   |  |
       +---+--------------+------------+--------------+---+ _|
                  ^                           :
                  |                           |
                  :                           v
       +--------------------------------------------------+
       |                                                  |
       |                traffic generator                 |
       |                                                  |
       +--------------------------------------------------+

.. 3.2.5.1.2

Physical port → vSwitch → VNF → vSwitch → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: console

                                                             _
       +---------------------------------------------------+  |
       |                                                   |  |
       |   +-------------------------------------------+   |  |
       |   |                 Application               |   |  |
       |   +-------------------------------------------+   |  |
       |       ^                                  :        |  |
       |       |                                  |        |  |  Guest
       |       :                                  v        |  |
       |   +---------------+           +---------------+   |  |
       |   | logical port 0|           | logical port 1|   |  |
       +---+---------------+-----------+---------------+---+ _|
               ^                                  :
               |                                  |
               :                                  v         _
       +---+---------------+----------+---------------+---+  |
       |   | logical port 0|          | logical port 1|   |  |
       |   +---------------+          +---------------+   |  |
       |       ^                                  :       |  |
       |       |                                  |       |  |  Host
       |       :                                  v       |  |
       |   +--------------+            +--------------+   |  |
       |   |   phy port   |  vSwitch   |   phy port   |   |  |
       +---+--------------+------------+--------------+---+ _|
                  ^                           :
                  |                           |
                  :                           v
       +--------------------------------------------------+
       |                                                  |
       |                traffic generator                 |
       |                                                  |
       +--------------------------------------------------+

.. 3.2.5.1.3

Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                       _
    +----------------------+  +----------------------+  |
    |   Guest 1            |  |   Guest 2            |  |
    |   +---------------+  |  |   +---------------+  |  |
    |   |  Application  |  |  |   |  Application  |  |  |
    |   +---------------+  |  |   +---------------+  |  |
    |       ^       |      |  |       ^       |      |  |
    |       |       v      |  |       |       v      |  |  Guests
    |   +---------------+  |  |   +---------------+  |  |
    |   | logical ports |  |  |   | logical ports |  |  |
    |   |   0       1   |  |  |   |   0       1   |  |  |
    +---+---------------+--+  +---+---------------+--+ _|
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v        _
    +---+---------------+---------+---------------+--+  |
    |   |   0       1   |         |   3       4   |  |  |
    |   | logical ports |         | logical ports |  |  |
    |   +---------------+         +---------------+  |  |
    |       ^       |                 ^       |      |  |  Host
    |       |       L-----------------+       v      |  |
    |   +--------------+          +--------------+   |  |
    |   |   phy ports  | vSwitch  |   phy ports  |   |  |
    +---+--------------+----------+--------------+---+ _|
            ^       ^                 :       :
            |       |                 |       |
            :       :                 v       v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+

.. 3.2.5.1.4

Physical port → VNF → vSwitch → VNF → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                        _
    +----------------------+  +----------------------+   |
    |   Guest 1            |  |   Guest 2            |   |
    |+-------------------+ |  | +-------------------+|   |
    ||     Application   | |  | |     Application   ||   |
    |+-------------------+ |  | +-------------------+|   |
    |       ^       |      |  |       ^       |      |   |  Guests
    |       |       v      |  |       |       v      |   |
    |+-------------------+ |  | +-------------------+|   |
    ||   logical ports   | |  | |   logical ports   ||   |
    ||  0              1 | |  | | 0              1  ||   |
    ++--------------------++  ++--------------------++  _|
        ^              :          ^              :
    (PCI passthrough)  |          |     (PCI passthrough)
        |              v          :              |      _
    +--------++------------+-+------------++---------+   |
    |   |    ||        0   | |    1       ||     |   |   |
    |   |    ||logical port| |logical port||     |   |   |
    |   |    |+------------+ +------------+|     |   |   |
    |   |    |     |                 ^     |     |   |   |
    |   |    |     L-----------------+     |     |   |   |
    |   |    |                             |     |   |   |  Host
    |   |    |           vSwitch           |     |   |   |
    |   |    +-----------------------------+     |   |   |
    |   |                                        |   |   |
    |   |                                        v   |   |
    | +--------------+              +--------------+ |   |
    | | phy port/VF  |              | phy port/VF  | |   |
    +-+--------------+--------------+--------------+-+  _|
        ^                                        :
        |                                        |
        :                                        v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+

.. 3.2.5.1.5

Physical port → vSwitch → VNF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                          _
    +---------------------------------------------------+  |
    |                                                   |  |
    |   +-------------------------------------------+   |  |
    |   |                 Application               |   |  |
    |   +-------------------------------------------+   |  |
    |       ^                                           |  |
    |       |                                           |  |  Guest
    |       :                                           |  |
    |   +---------------+                               |  |
    |   | logical port 0|                               |  |
    +---+---------------+-------------------------------+ _|
            ^
            |
            :                                            _
    +---+---------------+------------------------------+  |
    |   | logical port 0|                              |  |
    |   +---------------+                              |  |
    |       ^                                          |  |
    |       |                                          |  |  Host
    |       :                                          |  |
    |   +--------------+                               |  |
    |   |   phy port   |  vSwitch                      |  |
    +---+--------------+------------ -------------- ---+ _|
               ^
               |
               :
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+

.. 3.2.5.1.6

VNF → vSwitch → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                          _
    +---------------------------------------------------+  |
    |                                                   |  |
    |   +-------------------------------------------+   |  |
    |   |                 Application               |   |  |
    |   +-------------------------------------------+   |  |
    |                                          :        |  |
    |                                          |        |  |  Guest
    |                                          v        |  |
    |                               +---------------+   |  |
    |                               | logical port  |   |  |
    +-------------------------------+---------------+---+ _|
                                               :
                                               |
                                               v         _
    +------------------------------+---------------+---+  |
    |                              | logical port  |   |  |
    |                              +---------------+   |  |
    |                                          :       |  |
    |                                          |       |  |  Host
    |                                          v       |  |
    |                               +--------------+   |  |
    |                     vSwitch   |   phy port   |   |  |
    +-------------------------------+--------------+---+ _|
                                           :
                                           |
                                           v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+

.. 3.2.5.1.7

VNF → vSwitch → VNF → vSwitch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                             _
    +-------------------------+  +-------------------------+  |
    |   Guest 1               |  |   Guest 2               |  |
    |   +-----------------+   |  |   +-----------------+   |  |
    |   |   Application   |   |  |   |   Application   |   |  |
    |   +-----------------+   |  |   +-----------------+   |  |
    |                :        |  |       ^                 |  |
    |                |        |  |       |                 |  |  Guest
    |                v        |  |       :                 |  |
    |     +---------------+   |  |   +---------------+     |  |
    |     | logical port 0|   |  |   | logical port 0|     |  |
    +-----+---------------+---+  +---+---------------+-----+ _|
                    :                    ^
                    |                    |
                    v                    :                    _
    +----+---------------+------------+---------------+-----+  |
    |    |     port 0    |            |     port 1    |     |  |
    |    +---------------+            +---------------+     |  |
    |              :                    ^                   |  |
    |              |                    |                   |  |  Host
    |              +--------------------+                   |  |
    |                                                       |  |
    |                     vswitch                           |  |
    +-------------------------------------------------------+ _|

.. 3.2.5.1.8

HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port)
→ HOST 2(Physical port → virtual switch → VNF → virtual switch → Physical port)

HOST 1 (PVP) → HOST 2 (PVP)
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: console

                                                       _
    +----------------------+  +----------------------+  |
    |   Guest 1            |  |   Guest 2            |  |
    |   +---------------+  |  |   +---------------+  |  |
    |   |  Application  |  |  |   |  Application  |  |  |
    |   +---------------+  |  |   +---------------+  |  |
    |       ^       |      |  |       ^       |      |  |
    |       |       v      |  |       |       v      |  |  Guests
    |   +---------------+  |  |   +---------------+  |  |
    |   | logical ports |  |  |   | logical ports |  |  |
    |   |   0       1   |  |  |   |   0       1   |  |  |
    +---+---------------+--+  +---+---------------+--+ _|
            ^       :                 ^       :
            |       |                 |       |
            :       v                 :       v        _
    +---+---------------+--+  +---+---------------+--+  |
    |   |   0       1   |  |  |   |   3       4   |  |  |
    |   | logical ports |  |  |   | logical ports |  |  |
    |   +---------------+  |  |   +---------------+  |  |
    |       ^       |      |  |       ^       |      |  |  Hosts
    |       |       v      |  |       |       v      |  |
    |   +--------------+   |  |   +--------------+   |  |
    |   |   phy ports  |   |  |   |   phy ports  |   |  |
    +---+--------------+---+  +---+--------------+---+ _|
            ^       :                 :       :
            |       +-----------------+       |
            :                                 v
    +--------------------------------------------------+
    |                                                  |
    |                traffic generator                 |
    |                                                  |
    +--------------------------------------------------+



**Note:** For tests where the traffic generator and/or measurement
receiver are implemented on VM and connected to the virtual switch
through vNIC, the issues of shared resources and interactions between
the measurement devices and the device under test must be considered.

**Note:** Some RFC 2889 tests require a full-mesh sending and receiving
pattern involving more than two ports. This possibility is illustrated in the
Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
diagram above (with 2 sending and 2 receiving ports, though all ports
could be used bi-directionally).

**Note:** When Deployment Scenarios are used in RFC 2889 address learning
or cache capacity testing, an additional port from the vSwitch must be
connected to the test device. This port is used to listen for flooded
frames.

.. 3.2.5.2

General Methodology:
--------------------------
To establish the baseline performance of the virtual switch, tests would
initially be run with a simple workload in the VNF (the recommended
simple workload VNF would be `DPDK <http://www.dpdk.org/>`__'s testpmd
application forwarding packets in a VM or vloop\_vnf a simple kernel
module that forwards traffic between two network interfaces inside the
virtualized environment while bypassing the networking stack).
Subsequently, the tests would also be executed with a real Telco
workload running in the VNF, which would exercise the virtual switch in
the context of higher level Telco NFV use cases, and prove that its
underlying characteristics and behaviour can be measured and validated.
Suitable real Telco workload VNFs are yet to be identified.

.. 3.2.5.2.1

Default Test Parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following list identifies the default parameters for suite of
tests:

-  Reference application: Simple forwarding or Open Source VNF.
-  Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR
   Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as
   maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>.
-  Reordering check: Tests should confirm that packets within a flow are
   not reordered.
-  Duplex: Unidirectional / Bidirectional. Default: Full duplex with
   traffic transmitting in both directions, as network traffic generally
   does not flow in a single direction. By default the data rate of
   transmitted traffic should be the same in both directions, please
   note that asymmetric traffic (e.g. downlink-heavy) tests will be
   mentioned explicitly for the relevant test cases.
-  Number of Flows: Default for non scalability tests is a single flow.
   For scalability tests the goal is to test with maximum supported
   flows but where possible will test up to 10 Million flows. Start with
   a single flow and scale up. By default flows should be added
   sequentially, tests that add flows simultaneously will explicitly
   call out their flow addition behaviour. Packets are generated across
   the flows uniformly with no burstiness. For multi-core tests should
   consider the number of packet flows based on vSwitch/VNF multi-thread
   implementation and behavior.

-  Traffic Types: UDP, SCTP, RTP, GTP and UDP traffic.
-  Deployment scenarios are:
-  Physical → virtual switch → physical.
-  Physical → virtual switch → VNF → virtual switch → physical.
-  Physical → virtual switch → VNF → virtual switch → VNF → virtual
   switch → physical.
-  Physical → VNF → virtual switch → VNF → physical.
-  Physical → virtual switch → VNF.
-  VNF → virtual switch → Physical.
-  VNF → virtual switch → VNF.

Tests MUST have these parameters unless otherwise stated. **Test cases
with non default parameters will be stated explicitly**.

**Note**: For throughput tests unless stated otherwise, test
configurations should ensure that traffic traverses the installed flows
through the virtual switch, i.e. flows are installed and have an appropriate
time out that doesn't expire before packet transmission starts.

.. 3.2.5.2.2

Flow Classification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Virtual switches classify packets into flows by processing and matching
particular header fields in the packet/frame and/or the input port where
the packets/frames arrived. The vSwitch then carries out an action on
the group of packets that match the classification parameters. Thus a
flow is considered to be a sequence of packets that have a shared set of
header field values or have arrived on the same port and have the same
action applied to them. Performance results can vary based on the
parameters the vSwitch uses to match for a flow. The recommended flow
classification parameters for L3 vSwitch performance tests are: the
input port, the source IP address, the destination IP address and the
Ethernet protocol type field. It is essential to increase the flow
time-out time on a vSwitch before conducting any performance tests that
do not measure the flow set-up time. Normally the first packet of a
particular flow will install the flow in the vSwitch which adds an
additional latency, subsequent packets of the same flow are not subject
to this latency if the flow is already installed on the vSwitch.

.. 3.2.5.2.3

Test Priority
~~~~~~~~~~~~~~~~~~~~~

Tests will be assigned a priority in order to determine which tests
should be implemented immediately and which tests implementations
can be deferred.

Priority can be of following types: - Urgent: Must be implemented
immediately. - High: Must be implemented in the next release. - Medium:
May be implemented after the release. - Low: May or may not be
implemented at all.

.. 3.2.5.2.4

SUT Setup
~~~~~~~~~~~~~~~~~~

The SUT should be configured to its "default" state. The
SUT's configuration or set-up must not change between tests in any way
other than what is required to do the test. All supported protocols must
be configured and enabled for each test set up.

.. 3.2.5.2.5

Port Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~

The DUT should be configured with n ports where
n is a multiple of 2. Half of the ports on the DUT should be used as
ingress ports and the other half of the ports on the DUT should be used
as egress ports. Where a DUT has more than 2 ports, the ingress data
streams should be set-up so that they transmit packets to the egress
ports in sequence so that there is an even distribution of traffic
across ports. For example, if a DUT has 4 ports 0(ingress), 1(ingress),
2(egress) and 3(egress), the traffic stream directed at port 0 should
output a packet to port 2 followed by a packet to port 3. The traffic
stream directed at port 1 should also output a packet to port 2 followed
by a packet to port 3.

.. 3.2.5.2.6

Frame Formats
~~~~~~~~~~~~~~~~~~~~~

**Frame formats Layer 2 (data link layer) protocols**

-  Ethernet II

.. code-block:: console

     +---------------------------+-----------+
     | Ethernet Header | Payload | Check Sum |
     +-----------------+---------+-----------+
     |_________________|_________|___________|
           14 Bytes     46 - 1500   4 Bytes
                          Bytes


**Layer 3 (network layer) protocols**

-  IPv4

.. code-block:: console

     +-----------------+-----------+---------+-----------+
     | Ethernet Header | IP Header | Payload | Checksum  |
     +-----------------+-----------+---------+-----------+
     |_________________|___________|_________|___________|
           14 Bytes       20 bytes  26 - 1480   4 Bytes
                                      Bytes

-  IPv6

.. code-block:: console

     +-----------------+-----------+---------+-----------+
     | Ethernet Header | IP Header | Payload | Checksum  |
     +-----------------+-----------+---------+-----------+
     |_________________|___________|_________|___________|
           14 Bytes       40 bytes  26 - 1460   4 Bytes
                                      Bytes

**Layer 4 (transport layer) protocols**

  - TCP
  - UDP
  - SCTP

.. code-block:: console

     +-----------------+-----------+-----------------+---------+-----------+
     | Ethernet Header | IP Header | Layer 4 Header  | Payload | Checksum  |
     +-----------------+-----------+-----------------+---------+-----------+
     |_________________|___________|_________________|_________|___________|
           14 Bytes      40 bytes      20 Bytes       6 - 1460   4 Bytes
                                                       Bytes


**Layer 5 (application layer) protocols**

  - RTP
  - GTP

.. code-block:: console

     +-----------------+-----------+-----------------+---------+-----------+
     | Ethernet Header | IP Header | Layer 4 Header  | Payload | Checksum  |
     +-----------------+-----------+-----------------+---------+-----------+
     |_________________|___________|_________________|_________|___________|
           14 Bytes      20 bytes     20 Bytes        >= 6 Bytes   4 Bytes

.. 3.2.5.2.7

Packet Throughput
~~~~~~~~~~~~~~~~~~~~~~~~~
There is a difference between an Ethernet frame,
an IP packet, and a UDP datagram. In the seven-layer OSI model of
computer networking, packet refers to a data unit at layer 3 (network
layer). The correct term for a data unit at layer 2 (data link layer) is
a frame, and at layer 4 (transport layer) is a segment or datagram.

Important concepts related to 10GbE performance are frame rate and
throughput. The MAC bit rate of 10GbE, defined in the IEEE standard 802
.3ae, is 10 billion bits per second. Frame rate is based on the bit rate
and frame format definitions. Throughput, defined in IETF RFC 1242, is
the highest rate at which the system under test can forward the offered
load, without loss.

The frame rate for 10GbE is determined by a formula that divides the 10
billion bits per second by the preamble + frame length + inter-frame
gap.

The maximum frame rate is calculated using the minimum values of the
following parameters, as described in the IEEE 802 .3ae standard:

-  Preamble: 8 bytes \* 8 = 64 bits
-  Frame Length: 64 bytes (minimum) \* 8 = 512 bits
-  Inter-frame Gap: 12 bytes (minimum) \* 8 = 96 bits

Therefore, Maximum Frame Rate (64B Frames)
= MAC Transmit Bit Rate / (Preamble + Frame Length + Inter-frame Gap)
= 10,000,000,000 / (64 + 512 + 96)
= 10,000,000,000 / 672
= 14,880,952.38 frame per second (fps)

.. 3.2.5.3

RFCs for testing virtual switch performance
--------------------------------------------------

The starting point for defining the suite of tests for benchmarking the
performance of a virtual switch is to take existing RFCs and standards
that were designed to test their physical counterparts and adapting them
for testing virtual switches. The rationale behind this is to establish
a fair comparison between the performance of virtual and physical
switches. This section outlines the RFCs that are used by this
specification.

.. 3.2.5.3.1

RFC 1242 Benchmarking Terminology for Network Interconnection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Devices RFC 1242 defines the terminology that is used in describing
performance benchmarking tests and their results. Definitions and
discussions covered include: Back-to-back, bridge, bridge/router,
constant load, data link frame size, frame loss rate, inter frame gap,
latency, and many more.

.. 3.2.5.3.2

RFC 2544 Benchmarking Methodology for Network Interconnect Devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2544 outlines a benchmarking methodology for network Interconnect
Devices. The methodology results in performance metrics such as latency,
frame loss percentage, and maximum data throughput.

In this document network “throughput” (measured in millions of frames
per second) is based on RFC 2544, unless otherwise noted. Frame size
refers to Ethernet frames ranging from smallest frames of 64 bytes to
largest frames of 9K bytes.

Types of tests are:

1. Throughput test defines the maximum number of frames per second
   that can be transmitted without any error, or 0% loss ratio.
   In some Throughput tests (and those tests with long duration),
   evaluation of an additional frame loss ratio is suggested. The
   current ratio (10^-7 %) is based on understanding the typical
   user-to-user packet loss ratio needed for good application
   performance and recognizing that a single transfer through a
   vswitch must contribute a tiny fraction of user-to-user loss.
   Further, the ratio 10^-7 % also recognizes practical limitations
   when measuring loss ratio.

2. Latency test measures the time required for a frame to travel from
   the originating device through the network to the destination device.
   Please note that RFC2544 Latency measurement will be superseded with
   a measurement of average latency over all successfully transferred
   packets or frames.

3. Frame loss test measures the network’s
   response in overload conditions - a critical indicator of the
   network’s ability to support real-time applications in which a
   large amount of frame loss will rapidly degrade service quality.

4. Burst test assesses the buffering capability of a virtual switch. It
   measures the maximum number of frames received at full line rate
   before a frame is lost. In carrier Ethernet networks, this
   measurement validates the excess information rate (EIR) as defined in
   many SLAs.

5. System recovery to characterize speed of recovery from an overload
   condition.

6. Reset to characterize speed of recovery from device or software
   reset. This type of test has been updated by `RFC6201
   <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such,
   the methodology defined by this specification will be that of RFC 6201.

Although not included in the defined RFC 2544 standard, another crucial
measurement in Ethernet networking is packet delay variation. The
definition set out by this specification comes from
`RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__.

.. 3.2.5.3.3

RFC 2285 Benchmarking Terminology for LAN Switching Devices
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2285 defines the terminology that is used to describe the
terminology for benchmarking a LAN switching device. It extends RFC
1242 and defines: DUTs, SUTs, Traffic orientation and distribution,
bursts, loads, forwarding rates, etc.

.. 3.2.5.3.4

RFC 2889 Benchmarking Methodology for LAN Switching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2889 outlines a benchmarking methodology for LAN switching, it
extends RFC 2544. The outlined methodology gathers performance
metrics for forwarding, congestion control, latency, address handling
and finally filtering.

.. 3.2.5.3.5

RFC 3918 Methodology for IP Multicast Benchmarking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 3918 outlines a methodology for IP Multicast benchmarking.

.. 3.2.5.3.6

RFC 4737 Packet Reordering Metrics
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 4737 describes metrics for identifying and counting re-ordered
packets within a stream, and metrics to measure the extent each
packet has been re-ordered.

.. 3.2.5.3.7

RFC 5481 Packet Delay Variation Applicability Statement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 5481 defined two common, but different forms of delay variation
metrics, and compares the metrics over a range of networking
circumstances and tasks. The most suitable form for vSwitch
benchmarking is the "PDV" form.

.. 3.2.5.3.8

RFC 6201 Device Reset Characterization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 6201 extends the methodology for characterizing the speed of
recovery of the DUT from device or software reset described in RFC
2544.

.. 3.2.6:

.. _PassFailCriteria:

Item pass/fail criteria
=========================

vswitchperf does not specify Pass/Fail criteria for the tests in terms of a
threshold, as benchmarks do not (and should not do this). The results/metrics
for a test are simply reported. If it had to be defined, a test is considered
to have passed if it succesfully completed and a relavent metric was
recorded/reported for the SUT.

.. 3.2.7:

.. _SuspensionResumptionReqs:

Suspension criteria and resumption requirements
================================================
In the case of a throughput test, a test should be suspended if a virtual
switch is failing to forward any traffic. A test should be restarted from a
clean state if the intention is to carry out the test again.

.. 3.2.8:

.. _TestDelierables:

Test deliverables
==================
Each test should produce a test report that details SUT information as well as
the test results. There are a number of parameters related to the system, DUT
and tests that can affect the repeatability of a test results and should be
recorded. In order to minimise the variation in the results of a test,
it is recommended that the test report includes the following information:

-  Hardware details including:

   -  Platform details.
   -  Processor details.
   -  Memory information (see below)
   -  Number of enabled cores.
   -  Number of cores used for the test.
   -  Number of physical NICs, as well as their details (manufacturer,
      versions, type and the PCI slot they are plugged into).
   -  NIC interrupt configuration.
   -  BIOS version, release date and any configurations that were
      modified.

-  Software details including:

   -  OS version (for host and VNF)
   -  Kernel version (for host and VNF)
   -  GRUB boot parameters (for host and VNF).
   -  Hypervisor details (Type and version).
   -  Selected vSwitch, version number or commit id used.
   -  vSwitch launch command line if it has been parameterised.
   -  Memory allocation to the vSwitch – which NUMA node it is using,
      and how many memory channels.
   -  Where the vswitch is built from source: compiler details including
      versions and the flags that were used to compile the vSwitch.
   -  DPDK or any other SW dependency version number or commit id used.
   -  Memory allocation to a VM - if it's from Hugpages/elsewhere.
   -  VM storage type: snapshot/independent persistent/independent
      non-persistent.
   -  Number of VMs.
   -  Number of Virtual NICs (vNICs), versions, type and driver.
   -  Number of virtual CPUs and their core affinity on the host.
   -  Number vNIC interrupt configuration.
   -  Thread affinitization for the applications (including the vSwitch
      itself) on the host.
   -  Details of Resource isolation, such as CPUs designated for
      Host/Kernel (isolcpu) and CPUs designated for specific processes
      (taskset).

-  Memory Details

   -  Total memory
   -  Type of memory
   -  Used memory
   -  Active memory
   -  Inactive memory
   -  Free memory
   -  Buffer memory
   -  Swap cache
   -  Total swap
   -  Used swap
   -  Free swap

-  Test duration.
-  Number of flows.
-  Traffic Information:

   -  Traffic type - UDP, TCP, IMIX / Other.
   -  Packet Sizes.

-  Deployment Scenario.

**Note**: Tests that require additional parameters to be recorded will
explicitly specify this.


.. 3.3:

.. _TestManagement:

Test management
=================
This section will detail the test activities that will be conducted by vsperf
as well as the infrastructure that will be used to complete the tests in OPNFV.

.. 3.3.1:

Planned activities and tasks; test progression
=================================================
A key consideration when conducting any sort of benchmark is trying to
ensure the consistency and repeatability of test results between runs.
When benchmarking the performance of a virtual switch there are many
factors that can affect the consistency of results. This section
describes these factors and the measures that can be taken to limit
their effects. In addition, this section will outline some system tests
to validate the platform and the VNF before conducting any vSwitch
benchmarking tests.

**System Isolation:**

When conducting a benchmarking test on any SUT, it is essential to limit
(and if reasonable, eliminate) any noise that may interfere with the
accuracy of the metrics collected by the test. This noise may be
introduced by other hardware or software (OS, other applications), and
can result in significantly varying performance metrics being collected
between consecutive runs of the same test. In the case of characterizing
the performance of a virtual switch, there are a number of configuration
parameters that can help increase the repeatability and stability of
test results, including:

-  OS/GRUB configuration:

   -  maxcpus = n where n >= 0; limits the kernel to using 'n'
      processors. Only use exactly what you need.
   -  isolcpus: Isolate CPUs from the general scheduler. Isolate all
      CPUs bar one which will be used by the OS.
   -  use taskset to affinitize the forwarding application and the VNFs
      onto isolated cores. VNFs and the vSwitch should be allocated
      their own cores, i.e. must not share the same cores. vCPUs for the
      VNF should be affinitized to individual cores also.
   -  Limit the amount of background applications that are running and
      set OS to boot to runlevel 3. Make sure to kill any unnecessary
      system processes/daemons.
   -  Only enable hardware that you need to use for your test – to
      ensure there are no other interrupts on the system.
   -  Configure NIC interrupts to only use the cores that are not
      allocated to any other process (VNF/vSwitch).

-  NUMA configuration: Any unused sockets in a multi-socket system
   should be disabled.
-  CPU pinning: The vSwitch and the VNF should each be affinitized to
   separate logical cores using a combination of maxcpus, isolcpus and
   taskset.
-  BIOS configuration: BIOS should be configured for performance where
   an explicit option exists, sleep states should be disabled, any
   virtualization optimization technologies should be enabled, and
   hyperthreading should also be enabled, turbo boost and overclocking
   should be disabled.

**System Validation:**

System validation is broken down into two sub-categories: Platform
validation and VNF validation. The validation test itself involves
verifying the forwarding capability and stability for the sub-system
under test. The rationale behind system validation is two fold. Firstly
to give a tester confidence in the stability of the platform or VNF that
is being tested; and secondly to provide base performance comparison
points to understand the overhead introduced by the virtual switch.

* Benchmark platform forwarding capability: This is an OPTIONAL test
  used to verify the platform and measure the base performance (maximum
  forwarding rate in fps and latency) that can be achieved by the
  platform without a vSwitch or a VNF. The following diagram outlines
  the set-up for benchmarking Platform forwarding capability:

  .. code-block:: console

                                                            __
       +--------------------------------------------------+   |
       |   +------------------------------------------+   |   |
       |   |                                          |   |   |
       |   |          l2fw or DPDK L2FWD app          |   |  Host
       |   |                                          |   |   |
       |   +------------------------------------------+   |   |
       |   |                 NIC                      |   |   |
       +---+------------------------------------------+---+ __|
                  ^                           :
                  |                           |
                  :                           v
       +--------------------------------------------------+
       |                                                  |
       |                traffic generator                 |
       |                                                  |
       +--------------------------------------------------+

* Benchmark VNF forwarding capability: This test is used to verify
  the VNF and measure the base performance (maximum forwarding rate in
  fps and latency) that can be achieved by the VNF without a vSwitch.
  The performance metrics collected by this test will serve as a key
  comparison point for NIC passthrough technologies and vSwitches. VNF
  in this context refers to the hypervisor and the VM. The following
  diagram outlines the set-up for benchmarking VNF forwarding
  capability:

  .. code-block:: console

                                                            __
       +--------------------------------------------------+   |
       |   +------------------------------------------+   |   |
       |   |                                          |   |   |
       |   |                 VNF                      |   |   |
       |   |                                          |   |   |
       |   +------------------------------------------+   |   |
       |   |          Passthrough/SR-IOV              |   |  Host
       |   +------------------------------------------+   |   |
       |   |                 NIC                      |   |   |
       +---+------------------------------------------+---+ __|
                  ^                           :
                  |                           |
                  :                           v
       +--------------------------------------------------+
       |                                                  |
       |                traffic generator                 |
       |                                                  |
       +--------------------------------------------------+


**Methodology to benchmark Platform/VNF forwarding capability**


The recommended methodology for the platform/VNF validation and
benchmark is: - Run `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
Maximum Forwarding Rate test, this test will produce maximum
forwarding rate and latency results that will serve as the
expected values. These expected values can be used in
subsequent steps or compared with in subsequent validation tests. -
Transmit bidirectional traffic at line rate/max forwarding rate
(whichever is higher) for at least 72 hours, measure throughput (fps)
and latency. - Note: Traffic should be bidirectional. - Establish a
baseline forwarding rate for what the platform can achieve. - Additional
validation: After the test has completed for 72 hours run bidirectional
traffic at the maximum forwarding rate once more to see if the system is
still functional and measure throughput (fps) and latency. Compare the
measure the new obtained values with the expected values.

**NOTE 1**: How the Platform is configured for its forwarding capability
test (BIOS settings, GRUB configuration, runlevel...) is how the
platform should be configured for every test after this

**NOTE 2**: How the VNF is configured for its forwarding capability test
(# of vCPUs, vNICs, Memory, affinitization…) is how it should be
configured for every test that uses a VNF after this.

.. 3.3.2:

Environment/infrastructure
============================
Intel is providing a hosted test-bed with nine bare-metal environments
allocated to different OPNFV projects. Currently a number of servers in
`Intel POD 3 <https://wiki.opnfv.org/display/pharos/Intel+Pod3>`__ are
allocated to vsperf:

  * pod3-wcp-node3 and pod3-wcp-node4 which are used for CI jobs.
  * pod3-node6 which is used as a vsperf sandbox environment.

vsperf CI
---------
vsperf CI jobs are broken down into:

  * Daily job:
    * Runs everyday takes about 10 hours to complete.
    * TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan
      phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back'.
    * TESTPARAM_DAILY='--test-params pkt_sizes=64,128,512,1024,1518'.
  * Merge job:
    * Runs whenever patches are merged to master.
    * Runs a basic Sanity test.
  * Verify job:
    * Runs every time a patch is pushed to gerrit.
    * Builds documentation.

Scripts:
--------
There are 2 scripts that are part of VSPERFs CI:
  * build-vsperf.sh: Lives in the VSPERF repository in the ci/ directory and is
    used to run vsperf with the appropriate cli parameters.
  * vswitchperf.yml: YAML description of our jenkins job. lives in the RELENG
    repository.

More info on vsperf CI can be found here:
https://wiki.opnfv.org/display/vsperf/VSPERF+CI

.. 3.3.3:

Responsibilities and authority
===============================
The group responsible for managing, designing, preparing and executing the
tests listed in the LTD are the vsperf committers and contributors. The vsperf
committers and contributors should work with the relavent OPNFV projects to
ensure that the infrastructure is in place for testing vswitches, and that the
results are published to common end point (a results database).