diff options
Diffstat (limited to 'docs/testing')
17 files changed, 1171 insertions, 136 deletions
diff --git a/docs/testing/developer/devguide/design/trafficgen_integration_guide.rst b/docs/testing/developer/devguide/design/trafficgen_integration_guide.rst index c88b80ed..671c7fd8 100644 --- a/docs/testing/developer/devguide/design/trafficgen_integration_guide.rst +++ b/docs/testing/developer/devguide/design/trafficgen_integration_guide.rst @@ -199,13 +199,20 @@ functions: Note: There are parameters specific to testing of tunnelling protocols, which are discussed in detail at :ref:`integration-tests` userguide. + Note: A detailed description of the ``TRAFFIC`` dictionary can be found at + :ref:`configuration-of-traffic-dictionary`. + * param **traffic_type**: One of the supported traffic types, - e.g. **rfc2544_throughput**, **rfc2544_continuous** - or **rfc2544_back2back**. - * param **frame_rate**: Defines desired percentage of frame - rate used during continuous stream tests. + e.g. **rfc2544_throughput**, **rfc2544_continuous**, + **rfc2544_back2back** or **burst**. * param **bidir**: Specifies if generated traffic will be full-duplex (true) or half-duplex (false). + * param **frame_rate**: Defines desired percentage of frame + rate used during continuous stream tests. + * param **burst_size**: Defines a number of frames in the single burst, + which is sent by burst traffic type. Burst size is applied for each + direction, i.e. the total number of tx frames will be 2*burst_size + in case of bidirectional traffic. * param **multistream**: Defines number of flows simulated by traffic generator. Value 0 disables MultiStream feature. * param **stream_type**: Stream Type defines ISO OSI network layer @@ -224,6 +231,8 @@ functions: **dstport** and l4 on/off switch **enabled**. * param **vlan**: A dictionary with vlan specific parameters, e.g. **priority**, **cfi**, **id** and vlan on/off switch **enabled**. + * param **scapy**: A dictionary with definition of the frame content for both traffic + directions. The frame content is defined by a SCAPY notation. * param **tests**: Number of times the test is executed. * param **duration**: Duration of continuous test or per iteration duration diff --git a/docs/testing/developer/devguide/design/vswitchperf_design.rst b/docs/testing/developer/devguide/design/vswitchperf_design.rst index 33051493..5fa892e0 100644 --- a/docs/testing/developer/devguide/design/vswitchperf_design.rst +++ b/docs/testing/developer/devguide/design/vswitchperf_design.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others. .. _vsperf-design: @@ -23,7 +23,7 @@ Example Connectivity to DUT Establish connectivity to the VSPERF DUT Linux host. If this is in an OPNFV lab following the steps provided by `Pharos <https://www.opnfv.org/community/projects/pharos>`_ -to `access the POD <https://wiki.opnfv.org/display/pharos/Pharos+Lab+Support>`_ +to `access the POD <https://wiki.opnfv.org/display/INF/INFRA+Lab+Support>`_ The followign steps establish the VSPERF environment. @@ -291,8 +291,8 @@ Detailed description of ``TRAFFIC`` dictionary items follows: .. code-block:: console 'traffic_type' - One of the supported traffic types. - E.g. rfc2544_throughput, rfc2544_back2back - or rfc2544_continuous + E.g. rfc2544_throughput, rfc2544_back2back, + rfc2544_continuous or burst Data type: str Default value: "rfc2544_throughput". 'bidir' - Specifies if generated traffic will be full-duplex (True) @@ -304,6 +304,12 @@ Detailed description of ``TRAFFIC`` dictionary items follows: continuous stream tests. Data type: int Default value: 100. + 'burst_size' - Defines a number of frames in the single burst, which is sent + by burst traffic type. Burst size is applied for each direction, + i.e. the total number of tx frames will be 2*burst_size in case of + bidirectional traffic. + Data type: int + Default value: 100. 'multistream' - Defines number of flows simulated by traffic generator. Value 0 disables multistream feature Data type: int @@ -326,7 +332,6 @@ Detailed description of ``TRAFFIC`` dictionary items follows: feature. If enabled, it will implicitly insert a flow for each stream. If multistream is disabled, then pre-installed flows will be ignored. - Note: It is supported only for p2p deployment scenario. Data type: str Supported values: "Yes" - flows will be inserted into OVS @@ -415,6 +420,77 @@ Detailed description of ``TRAFFIC`` dictionary items follows: congestion (DEI header field). Data type: int (NOTE: must fit to 1 bit) Default value: 0 + 'capture' - A dictionary with traffic capture configuration. + NOTE: It is supported only by T-Rex traffic generator. + 'enabled' - Specifies if traffic should be captured + Data type: bool + Default value: False + 'tx_ports' - A list of ports, where frames transmitted towards DUT will + be captured. Ports have numbers 0 and 1. TX packet capture + is disabled if list of ports is empty. + Data type: list + Default value: [0] + 'rx_ports' - A list of ports, where frames received from DUT will + be captured. Ports have numbers 0 and 1. RX packet capture + is disabled if list of ports is empty. + Data type: list + Default value: [1] + 'count' - A number of frames to be captured. The same count value + is applied to both TX and RX captures. + Data type: int + Default value: 1 + 'filter' - An expression used to filter TX and RX packets. It uses the same + syntax as pcap library. See pcap-filter man page for additional + details. + Data type: str + Default value: '' + 'scapy' - A dictionary with definition of a frame content for both traffic + directions. The frame content is defined by a SCAPY notation. + NOTE: It is supported only by the T-Rex traffic generator. + Following keywords can be used to refer to the related parts of + the TRAFFIC dictionary: + Ether_src - refers to TRAFFIC['l2']['srcmac'] + Ether_dst - refers to TRAFFIC['l2']['dstmac'] + IP_proto - refers to TRAFFIC['l3']['proto'] + IP_PROTO - refers to upper case version of TRAFFIC['l3']['proto'] + IP_src - refers to TRAFFIC['l3']['srcip'] + IP_dst - refers to TRAFFIC['l3']['dstip'] + IP_PROTO_sport - refers to TRAFFIC['l4']['srcport'] + IP_PROTO_dport - refers to TRAFFIC['l4']['dstport'] + Dot1Q_prio - refers to TRAFFIC['vlan']['priority'] + Dot1Q_id - refers to TRAFFIC['vlan']['cfi'] + Dot1Q_vlan - refers to TRAFFIC['vlan']['id'] + '0' - A string with the frame definition for the 1st direction. + Data type: str + Default value: 'Ether(src={Ether_src}, dst={Ether_dst})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_src}, dst={IP_dst})/' + '{IP_PROTO}(sport={IP_PROTO_sport}, dport={IP_PROTO_dport})' + '1' - A string with the frame definition for the 2nd direction. + Data type: str + Default value: 'Ether(src={Ether_dst}, dst={Ether_src})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_dst}, dst={IP_src})/' + '{IP_PROTO}(sport={IP_PROTO_dport}, dport={IP_PROTO_sport})', + 'latency_histogram' + - A dictionary with definition of a latency histogram provision in results. + 'enabled' - Specifies if the histogram provisioning is enabled or not. + 'type' - Defines how histogram is provided. Currenty only 'Default' is defined. + 'Default' - Default histogram as provided by the Traffic-generator. + 'imix' - A dictionary for IMIX Specification. + 'enabled' - Specifies if IMIX is enabled or NOT. + 'type' - The specification type - denotes how IMIX is specified. + Currently only 'genome' type is defined. + Other types (ex: table-of-proportions) can be added in future. + 'genome' - The Genome Encoding of Pkt-Sizes and Ratio for IMIX. + The Ratio is inferred from the number of particular geneome characters + Genome encoding is described in RFC 6985. This specification is closest + to the method described in section 6.2 of RFC 6985. + Ex: 'aaaaaaaddddg' denotes ratio of 7:4:1 of packets sizes 64:512:1518. + Note: Exact-sequence is not maintained, only the ratio of packets + is ensured. + Data type: str + Default Value: 'aaaaaaaddddg' .. _configuration-of-guest-options: @@ -719,6 +795,13 @@ As it is able to forward traffic between multiple VM NIC pairs. Note: In case of ``linux_bridge``, all NICs are connected to the same bridge inside the VM. +Note: In case that multistream feature is configured and ``pre_installed_flows`` +is set to ``Yes``, then stream specific flows will be inserted only for connections +originating at physical ports. The rest of the flows will be based on port +numbers only. The same logic applies in case that ``flow_type`` TRAFFIC option +is set to ``ip``. This configuration will avoid a testcase malfunction if frame headers +are modified inside VM (e.g. MAC swap or IP change). + VM, vSwitch, Traffic Generator Independence =========================================== @@ -762,7 +845,7 @@ ITrafficGenerator connect() disconnect() - send_burst_traffic(traffic, numpkts, time, framerate) + send_burst_traffic(traffic, time) send_cont_traffic(traffic, time, framerate) start_cont_traffic(traffic, time, framerate) @@ -854,6 +937,10 @@ Vsperf uses a standard set of routing tables in order to allow tests to easily mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and Frame Modification requirements. +The usage of routing tables is driven by configuration parameter ``OVS_ROUTING_TABLES``. +Routing tables are disabled by default (i.e. parameter is set to ``False``) for better +comparison of results among supported vSwitches (e.g. OVS vs. VPP). + .. code-block:: console +--------------+ diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 49659792..64a4758c 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -31,7 +31,7 @@ new techniques together. A new IETF benchmarking specification (RFC8204) is base 2015. VSPERF is also contributing to development of ETSI NFV test specifications through the Test and Open Source Working Group. -* Wiki: https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases +* Wiki: https://wiki.opnfv.org/display/vsperf * Repository: https://git.opnfv.org/vswitchperf * Artifacts: https://artifacts.opnfv.org/vswitchperf.html * Continuous Integration: https://build.opnfv.org/ci/view/vswitchperf/ @@ -43,7 +43,6 @@ Design Guides .. toctree:: :caption: Traffic Gen Integration, VSPERF Design, Test Design, Test Plan :maxdepth: 2 - :numbered: ./design/trafficgen_integration_guide.rst ./design/vswitchperf_design.rst @@ -75,6 +74,3 @@ VSPERF CI Test Cases :numbered: CI Test cases run daily on the VSPERF Pharos POD for master and stable branches. - - ./results/scenario.rst - ./results/results.rst diff --git a/docs/testing/developer/devguide/requirements/ietf_draft/rfc8204-vsperf-bmwg-vswitch-opnfv.rst b/docs/testing/developer/devguide/requirements/ietf_draft/rfc8204-vsperf-bmwg-vswitch-opnfv.rst index ee7f98b5..10b07d54 100644 --- a/docs/testing/developer/devguide/requirements/ietf_draft/rfc8204-vsperf-bmwg-vswitch-opnfv.rst +++ b/docs/testing/developer/devguide/requirements/ietf_draft/rfc8204-vsperf-bmwg-vswitch-opnfv.rst @@ -13,7 +13,7 @@ informational RFC published by the IETF available here https://tools.ietf.org/ht For more information about VSPERF refer to: -* Wiki: https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases +* Wiki: https://wiki.opnfv.org/display/vsperf * Repository: https://git.opnfv.org/vswitchperf * Artifacts: https://artifacts.opnfv.org/vswitchperf.html * Continuous Integration: https://build.opnfv.org/ci/view/vswitchperf/ diff --git a/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst b/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst index e1372520..1ea99f7e 100644 --- a/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst +++ b/docs/testing/developer/devguide/requirements/vswitchperf_ltd.rst @@ -62,21 +62,21 @@ References ========== * `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ + Devices <https://www.ietf.org/rfc/rfc1242.txt>`__ * `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2544.txt>`__ * `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2285.txt>`__ * `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2889.txt>`__ * `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ + Benchmarking <https://www.ietf.org/rfc/rfc3918.txt>`__ * `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ + Metrics <https://www.ietf.org/rfc/rfc4737.txt>`__ * `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ + Statement <https://www.ietf.org/rfc/rfc5481.txt>`__ * `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ + Characterization <https://tools.ietf.org/html/rfc6201>`__ .. 3.2 @@ -413,7 +413,21 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak **Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test - **Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio + **Prerequisite Tests**: + + LTD.Throughput.RFC2544.PacketLossRatio will determine the offered load and + frame size for which the maximum theoretical throughput of the interface + has not been achieved. As described in RFC 2544 section 24, the final + determination of the benchmark SHOULD be conducted using a full length + trial, and for this purpose the duration is 5 minutes with zero loss ratio. + + It is also essential to verify that the Traffic Generator has sufficient + stability to conduct Soak tests. Therefore, a prerequisite is to perform + this test with the DUT removed and replaced with a cross-over cable (or + other equivalent very low overhead method such as a loopback in a HW switch), + so that the traffic generator (and any other network involved) can be tested + over the Soak period. Note that this test may be challenging for software- + based traffic generators. **Priority**: @@ -422,12 +436,19 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24 hours - or, if this is not possible, for at least 6 hours. For this test, each frame - size must be sent at the highest Throughput rate with X% packet loss, as - determined in the prerequisite test. The default loss percentages to be - tested are: - X = 0% - X = 10^-7% + or if this is not possible, for at least 6 hours. - Note: Other values can be tested if required by the user. + For this test, one frame size must be sent at the highest frame rate with + X% packet loss ratio, as determined in the prerequisite test (a short trial). + The loss ratio shall be measured and recorded every 5 minutes during the test + (it may be sufficient to collect lost frame counts and divide by the number + of frames sent in 5 minutes to see if a threshold has been crossed, + and accept some small inaccuracy in the threshold evaluation, not the result). + The default loss ratio is X = 0% and loss ratio > 10^-7% is the default + threshold to terminate the test early (or inform the test operator of + the failure status). + + Note: Other values of X and loss threshold can be tested if required by the user. **Expected Result**: @@ -441,13 +462,13 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak and reporting any time intervals with packet loss. The `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. + An interval of 300s is suggested. - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system. - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ PDV form of delay variation on the traffic flow, - using the 99th percentile. + using the 99th percentile, may also be collected. .. 3.2.2.1.7 @@ -457,7 +478,22 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification **Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification **Prerequisite Test**: + LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss) + will determine the offered load and + frame size for which the maximum theoretical throughput of the interface + has not been achieved. As described in RFC 2544 section 24, the final + determination of the benchmark SHOULD be conducted using a full length + trial, and for this purpose the duration is 5 minutes with zero loss ratio. + + It is also essential to verify that the Traffic Generator has sufficient + stability to conduct Soak tests. Therefore, a prerequisite is to perform + this test with the DUT removed and replaced with a cross-over cable (or + other equivalent very low overhead method such as a loopback in a HW switch), + so that the traffic generator (and any other network involved) can be tested + over the Soak period. Note that this test may be challenging for software- + based traffic generators. + **Priority**: @@ -466,9 +502,19 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification The aim of this test is to understand the Max Forwarding Rate stability over an extended test duration in order to uncover any outliers. To allow for an extended test duration, the test should ideally run for 24 hours or, if - this is not possible, for at least 6 hour. For this test, each frame - size must be sent at the highest Throughput rate with 0% packet loss, as - determined in the prerequisite test. + this is not possible, for at least 6 hours. + + For this test, one frame size must be sent at the highest frame rate with + X% packet loss ratio, as determined in the prerequisite test (a short trial). + The loss ratio shall be measured and recorded every 5 minutes during the test + (it may be sufficient to collect lost frame counts and divide by the number + of frames sent in 5 minutes to see if a threshold has been crossed, + and accept some small inaccuracy in the threshold evaluation, not the result). + The default loss ratio is X = 0% and loss ratio > 10^-7% is the default + threshold to terminate the test early (or inform the test operator of + the failure status). + + Note: Other values of X and loss threshold can be tested if required by the user. During this test, the DUT must perform the following operations on the traffic flow: @@ -498,13 +544,13 @@ Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification and reporting any time intervals with packet loss. The `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ Forwarding Rate shall be measured in each interval. - An interval of 60s is suggested. + An interval of 300s is suggested. - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system. - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ PDV form of delay variation on the traffic flow, using the 99th - percentile. + percentile, may also be collected. .. 3.2.2.1.8 @@ -1150,7 +1196,22 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak **Title**: Packet Delay Variation Soak Test - **Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) + **Prerequisite Tests**: + + LTD.Throughput.RFC2544.PacketLossRatio will determine the offered load and + frame size for which the maximum theoretical throughput of the interface + has not been achieved. As described in RFC 2544 section 24, the final + determination of the benchmark SHOULD be conducted using a full length + trial, and for this purpose the duration is 5 minutes with zero loss ratio. + + It is also essential to verify that the Traffic Generator has sufficient + stability to conduct Soak tests. Therefore, a prerequisite is to perform + this test with the DUT removed and replaced with a cross-over cable (or + other equivalent very low overhead method such as a loopback in a HW switch), + so that the traffic generator (and any other network involved) can be tested + over the Soak period. Note that this test may be challenging for software- + based traffic generators. + **Priority**: @@ -1160,9 +1221,20 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak variation for different frame sizes over an extended test duration and to determine if there are any outliers. To allow for an extended test duration, the test should ideally run for 24 hours or, if this is not - possible, for at least 6 hour. For this test, each frame size must be - sent at the highest possible throughput with 0% packet loss, as - determined in the prerequisite test. + possible, for at least 6 hours. + + For this test, one frame size must be sent at the highest frame rate with + X% packet loss ratio, as determined in the prerequisite test (a short trial). + The loss ratio shall be measured and recorded every 5 minutes during the test + (it may be sufficient to collect lost frame counts and divide by the number + of frames sent in 5 minutes to see if a threshold has been crossed, + and accept some small inaccuracy in the threshold evaluation, not the result). + The default loss ratio is X = 0% and loss ratio > 10^-7% is the default + threshold to terminate the test early (or inform the test operator of + the failure status). + + Note: Other values of X and loss threshold can be tested if required by the user. + **Expected Result**: @@ -1173,7 +1245,7 @@ Test ID: LTD.PacketDelayVariation.RFC3393.Soak - The packet delay variation value for traffic passing through the DUT. - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ PDV form of delay variation on the traffic flow, - using the 99th percentile, for each 60s interval during the test. + using the 99th percentile, for each 300s interval during the test. - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system. diff --git a/docs/testing/developer/devguide/requirements/vswitchperf_ltp.rst b/docs/testing/developer/devguide/requirements/vswitchperf_ltp.rst index e5147bea..c0b63859 100644 --- a/docs/testing/developer/devguide/requirements/vswitchperf_ltp.rst +++ b/docs/testing/developer/devguide/requirements/vswitchperf_ltp.rst @@ -63,21 +63,21 @@ References =============== * `RFC 1242 Benchmarking Terminology for Network Interconnection - Devices <http://www.ietf.org/rfc/rfc1242.txt>`__ + Devices <https://www.ietf.org/rfc/rfc1242.txt>`__ * `RFC 2544 Benchmarking Methodology for Network Interconnect - Devices <http://www.ietf.org/rfc/rfc2544.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2544.txt>`__ * `RFC 2285 Benchmarking Terminology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2285.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2285.txt>`__ * `RFC 2889 Benchmarking Methodology for LAN Switching - Devices <http://www.ietf.org/rfc/rfc2889.txt>`__ + Devices <https://www.ietf.org/rfc/rfc2889.txt>`__ * `RFC 3918 Methodology for IP Multicast - Benchmarking <http://www.ietf.org/rfc/rfc3918.txt>`__ + Benchmarking <https://www.ietf.org/rfc/rfc3918.txt>`__ * `RFC 4737 Packet Reordering - Metrics <http://www.ietf.org/rfc/rfc4737.txt>`__ + Metrics <https://www.ietf.org/rfc/rfc4737.txt>`__ * `RFC 5481 Packet Delay Variation Applicability - Statement <http://www.ietf.org/rfc/rfc5481.txt>`__ + Statement <https://www.ietf.org/rfc/rfc5481.txt>`__ * `RFC 6201 Device Reset - Characterization <http://tools.ietf.org/html/rfc6201>`__ + Characterization <https://tools.ietf.org/html/rfc6201>`__ .. 3.1.4 @@ -633,7 +633,7 @@ General Methodology: -------------------------- To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended -simple workload VNF would be `DPDK <http://www.dpdk.org/>`__'s testpmd +simple workload VNF would be `DPDK <https://www.dpdk.org/>`__'s testpmd application forwarding packets in a VM or vloop\_vnf a simple kernel module that forwards traffic between two network interfaces inside the virtualized environment while bypassing the networking stack). @@ -656,7 +656,7 @@ tests: - Reference application: Simple forwarding or Open Source VNF. - Frame size (bytes): 64, 128, 256, 512, 1024, 1280, 1518, 2K, 4k OR Packet size based on use-case (e.g. RTP 64B, 256B) OR Mix of packet sizes as - maintained by the Functest project <https://wiki.opnfv.org/traffic_profile_management>. + maintained by the Functest project <https://wiki.opnfv.org/display/functest/Traffic+Profile+Management>. - Reordering check: Tests should confirm that packets within a flow are not reordered. - Duplex: Unidirectional / Bidirectional. Default: Full duplex with diff --git a/docs/testing/developer/devguide/results/scenario.rst b/docs/testing/developer/devguide/results/scenario.rst index dbdc7877..f7eadd33 100644 --- a/docs/testing/developer/devguide/results/scenario.rst +++ b/docs/testing/developer/devguide/results/scenario.rst @@ -34,7 +34,7 @@ Deployment topologies: Loopback applications in the Guest: -* `DPDK testpmd <http://dpdk.org/doc/guides/testpmd_app_ug/index.html>`_. +* `DPDK testpmd <http://doc.dpdk.org/guides/testpmd_app_ug/index.html>`_. * Linux Bridge. * :ref:`l2fwd-module` diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index 83908a97..87c32d11 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -31,7 +31,7 @@ new techniques together. A new IETF benchmarking specification (RFC8204) is base 2015. VSPERF is also contributing to development of ETSI NFV test specifications through the Test and Open Source Working Group. -* Wiki: https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases +* Wiki: https://wiki.opnfv.org/display/vsperf * Repository: https://git.opnfv.org/vswitchperf * Artifacts: https://artifacts.opnfv.org/vswitchperf.html * Continuous Integration: https://build.opnfv.org/ci/view/vswitchperf/ @@ -48,6 +48,7 @@ VSPERF Install and Configuration ./installation.rst ./upgrade.rst ./trafficgen.rst + ./tools.rst ================= VSPERF Test Guide @@ -56,10 +57,10 @@ VSPERF Test Guide .. toctree:: :caption: VSPERF Test Execution :maxdepth: 2 - :numbered: ../userguide/testusage.rst ../userguide/teststeps.rst ../userguide/integration.rst + ../userguide/trafficcapture.rst ../userguide/yardstick.rst ../userguide/testlist.rst diff --git a/docs/testing/user/configguide/installation.rst b/docs/testing/user/configguide/installation.rst index 7f4d640b..b950442e 100644 --- a/docs/testing/user/configguide/installation.rst +++ b/docs/testing/user/configguide/installation.rst @@ -53,6 +53,7 @@ Supported Operating Systems * SLES 15 * RedHat 7.2 Enterprise Linux * RedHat 7.3 Enterprise Linux +* RedHat 7.5 Enterprise Linux * Ubuntu 14.04 * Ubuntu 16.04 * Ubuntu 16.10 (kernel 4.8 requires DPDK 16.11 and newer) @@ -166,8 +167,12 @@ repository provided by Software Collections (`a link`_). The installation script will also use `virtualenv`_ to create a vsperf virtual environment, which is isolated from the default Python environment, using the Python3 package located in **/usr/bin/python3**. This environment will reside in a directory called -**vsperfenv** in $HOME. It will ensure, that system wide Python installation - is not modified or broken by VSPERF installation. The complete list of Python +**vsperfenv** in $HOME. + +It will ensure, that system wide Python installation is not modified or +broken by VSPERF installation. + +The complete list of Python packages installed inside virtualenv can be found in the file ``requirements.txt``, which is located at the vswitchperf repository. @@ -176,6 +181,11 @@ built from upstream source due to kernel incompatibilities. Please see the instructions in the vswitchperf_design document for details on configuring OVS Vanilla for binary package usage. +**NOTE:** For RHEL 7.5 Enterprise DPDK and Openvswitch are not built from +upstream sources due to kernel incompatibilities. Please use subscription +channels to obtain binary equivalents of openvswitch and dpdk packages or +build binaries using instructions from openvswitch.org and dpdk.org. + .. _vpp-installation: VPP installation @@ -202,7 +212,7 @@ new shell session. Its activation is specific to your OS: .. code:: bash - $ scl enable python33 bash + $ scl enable rh-python34 bash $ source $HOME/vsperfenv/bin/activate * Fedora and Ubuntu @@ -260,8 +270,8 @@ running any of the above. For example: export http_proxy=proxy.mycompany.com:123 export https_proxy=proxy.mycompany.com:123 -.. _a link: http://www.softwarecollections.org/en/scls/rhscl/python33/ -.. _virtualenv: https://virtualenv.readthedocs.org/en/latest/ +.. _a link: https://www.softwarecollections.org/en/scls/rhscl/python33/ +.. _virtualenv: https://virtualenv.pypa.io/en/latest/ .. _vloop-vnf-ubuntu-14.04_20160823: http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160823.qcow2 .. _vloop-vnf-ubuntu-14.04_20160804: http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160804.qcow2 .. _vloop-vnf-ubuntu-14.04_20160303: http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-14.04_20160303.qcow2 @@ -320,7 +330,7 @@ to your OS documentation to set hugepages correctly. It is recommended to set the required amount of hugepages to be allocated by default on reboots. Information on hugepage requirements for dpdk can be found at -http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html +http://doc.dpdk.org/guides/linux_gsg/sys_reqs.html You can review your hugepage amounts by executing the following command @@ -350,7 +360,7 @@ default on the Linux DUT VSPerf recommends the latest tuned-adm package, which can be downloaded from the following location: -http://www.tuned-project.org/2017/04/27/tuned-2-8-0-released/ +https://github.com/redhat-performance/tuned/releases Follow the instructions to install the latest tuned-adm onto your system. For current RHEL customers you should already have the most current version. You diff --git a/docs/testing/user/configguide/tools.rst b/docs/testing/user/configguide/tools.rst new file mode 100644 index 00000000..72e515fa --- /dev/null +++ b/docs/testing/user/configguide/tools.rst @@ -0,0 +1,227 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel Corporation, Spirent, AT&T and others. + +.. _additional-tools-configuration: + +============================================= +'vsperf' Additional Tools Configuration Guide +============================================= + +Overview +-------- + +VSPERF supports the following categories additional tools: + + * `Infrastructure Metrics Collectors`_ + * `Load Generators`_ + * `L3 Cache Management`_ + +Under each category, there are one or more tools supported by VSPERF. +This guide provides the details of how to install (if required) +and configure the above mentioned tools. + +.. _`Infrastructure Metrics Collectors`: + +Infrastructure Metrics Collection +--------------------------------- + +VSPERF supports following two tools for collecting and reporting the metrics: + +* pidstat +* collectd + +*pidstat* is a command in linux systems, which is used for monitoring individual +tasks currently being managed by Linux kernel. In VSPERF this command is used to +monitor *ovs-vswitchd*, *ovsdb-server* and *kvm* processes. + +*collectd* is linux application that collects, stores and transfers various system +metrics. For every category of metrics, there is a separate plugin in collectd. For +example, CPU plugin and Interface plugin provides all the cpu metrics and interface +metrics, respectively. CPU metrics may include user-time, system-time, etc., whereas +interface metrics may include received-packets, dropped-packets, etc. + +Installation +^^^^^^^^^^^^ + +No installation is required for *pidstat*, whereas, collectd has to be installed +separately. For installation of collectd, we recommend to follow the process described +in *OPNFV-Barometer* project, which can be found here `Barometer <https://opnfv-barometer.readthedocs.io/en/latest/release/userguide>`_ +recent release. + +VSPERF assumes that collectd is installed and configured to send metrics over localhost. +The metrics sent should be for the following categories: CPU, Processes, Interface, +OVS, DPDK, Intel-RDT. + +For multicmd, apart from collectd, installation of PROX is also necessary. +Installation steps for PROX can be found here - `DPPD-PROX <https://github.com/opnfv/samplevnf/tree/master/VNFs/DPPD-PROX>`_ + +Configuration +^^^^^^^^^^^^^ + +The configuration file for the collectors can be found in **conf/05_collector.conf**. +*pidstat* specific configuration includes: + +* ``PIDSTAT_MONITOR`` - processes to be monitored by pidstat +* ``PIDSTAT_OPTIONS`` - options which will be passed to pidstat command +* ``PIDSTAT_SAMPLE_INTERVAL`` - sampling interval used by pidstat to collect statistics +* ``LOG_FILE_PIDSTAT`` - prefix of pidstat's log file + +The *collectd* configuration option includes: + +* ``COLLECTD_IP`` - IP address where collectd is running +* ``COLLECTD_PORT`` - Port number over which collectd is sending the metrics +* ``COLLECTD_SECURITY_LEVEL`` - Security level for receiving metrics +* ``COLLECTD_AUTH_FILE`` - Authentication file for receiving metrics +* ``LOG_FILE_COLLECTD`` - Prefix for collectd's log file. +* ``COLLECTD_CPU_KEYS`` - Interesting metrics from CPU +* ``COLLECTD_PROCESSES_KEYS`` - Interesting metrics from processes +* ``COLLECTD_INTERFACE_KEYS`` - Interesting metrics from interface +* ``COLLECTD_OVSSTAT_KEYS`` - Interesting metrics from OVS +* ``COLLECTD_DPDKSTAT_KEYS`` - Interesting metrics from DPDK. +* ``COLLECTD_INTELRDT_KEYS`` - Interesting metrics from Intel-RDT +* ``COLLECTD_INTERFACE_XKEYS`` - Metrics to exclude from Interface +* ``COLLECTD_INTELRDT_XKEYS`` - Metrics to exclude from Intel-RDT +* ``MC_COLLECTD_CSV`` - Path where collectd writes its metrics as CSV. +* ``MC_COLLECTD_CMD`` - Path where Collectd is installed +* ``MC_PROX_HOME`` - Path where PROX-IRQ is installed. +* ``MC_PROX_CMD`` - Command to run PROX-IRQ +* ``MC_PROX_OUT`` - Output file generated by PROX-IRQ stats collector. +* ``MC_CRON_OUT`` - Output file path of the command run through CROND +* ``MC_BEAT_CFILE`` - Filebeat configuration file path. + + +.. _`Load Generators`: + + +Load Generation +--------------- + +In VSPERF, load generation refers to creating background cpu and memory loads to +study the impact of these loads on system under test. There are two options to +create loads in VSPERF. These options are used for different use-cases. The options are: + +* stress or stress-ng +* Stressor-VMs + +*stress and stress-ng* are linux tools to stress the system in various ways. +It can stress different subsystems such as CPU and memory. *stress-ng* is the +improvised version of *stress*. StressorVMs are custom build virtual-machines +for the noisy-neighbor use-cases. + +Installation +^^^^^^^^^^^^ + +stress and stress-ng can be installed through standard linux installation process. +Information about stress-ng, including the steps for installing can be found +here: `stress-ng <https://github.com/ColinIanKing/stress-ng>`_ + +There are two options for StressorVMs - one is VMs based on stress-ng and second +is VM based on Spirent's cloudstress. VMs based on stress-ng can be found in this +`link <https://github.com/opensource-tnbt/stressng-images>`_ . Spirent's cloudstress +based VM can be downloaded from this `site <https://github.com/spirent/cloudstress>`_ + +These stressorVMs are of OSV based VMs, which are very small in size. Download +these VMs and place it in appropriate location, and this location will used in +the configuration - as mentioned below. + +Configuration +^^^^^^^^^^^^^ + +The configuration file for loadgens can be found in **conf/07_loadgen.conf**. +There are no specific configurations for stress and stress-ng commands based +load-generation. However, for StressorVMs, following configurations apply: + +* ``NN_COUNT`` - Number of stressor VMs required. +* ``NN_MEMORY`` - Comma separated memory configuration for each VM +* ``NN_SMP`` - Comma separated configuration for each VM +* ``NN_IMAGE`` - Comma separated list of Paths for each VM image +* ``NN_SHARED_DRIVE_TYPE`` - Comma separated list of shaed drive type for each VM +* ``NN_BOOT_DRIVE_TYPE`` - Comma separated list of boot drive type for each VM +* ``NN_CORE_BINDING`` - Comma separated lists of list specifying the cores associated with each VM. +* ``NN_NICS_NR`` - Comma seprated list of number of NICS for each VM +* ``NN_BASE_VNC_PORT`` - Base VNC port Index. +* ``NN_LOG_FILE`` - Name of the log file + +.. _`L3 Cache Management`: + +Last Level Cache Management +--------------------------- + +VSPERF support last-level cache management using Intel's RDT tool(s) - the +relavant ones are `Intel CAT-CMT <https://github.com/intel/intel-cmt-cat>`_ and +`Intel RMD <https://github.com/intel/rmd>`_. RMD is a linux daemon that runs on +individual hosts, and provides a REST API for control/orchestration layer to +request LLC for the VMs/Containers/Applications. RDT receives resource policy +form orchestration layer - in this case, from VSPERF - and enforce it on the host. +It achieves this enforcement via kernel interfaces such as resctrlfs and libpqos. +The resource here refer to the last-level cache. User can configure policies to +define how much of cache a CPU can get. The policy configuration is described below. + +Installation +^^^^^^^^^^^^ + +For installation of RMD tool, please install CAT-CMT first and then install RMD. +The details of installation can be found here: `Intel CAT-CMT <https://github.com/intel/intel-cmt-cat>`_ +and `Intel RMD <https://github.com/intel/rmd>`_ + +Configuration +^^^^^^^^^^^^^ + +The configuration file for cache management can be found in **conf/08_llcmanagement.conf**. + +VSPERF provides following configuration options, for user to define and enforce policies via RMD. + +* ``LLC_ALLOCATION`` - Enable or Disable LLC management. +* ``RMD_PORT`` - RMD port (port number on which API server is listening) +* ``RMD_SERVER_IP`` - IP address where RMD is running. Currently only localhost. +* ``RMD_API_VERSION`` - RMD version. Currently it is 'v1' +* ``POLICY_TYPE`` - Specify how the policy is defined - either COS or CUSTOM +* ``VSWITCH_COS`` - Class of service (CoS for Vswitch. CoS can be gold, silver-bf or bronze-shared. +* ``VNF_COS`` - Class of service for VNF +* ``PMD_COS`` - Class of service for PMD +* ``NOISEVM_COS`` - Class of service of Noisy VM. +* ``VSWITCH_CA`` - [min-cache-value, maxi-cache-value] for vswitch +* ``VNF_CA`` - [min-cache-value, max-cache-value] for VNF +* ``PMD_CA`` - [min-cache-value, max-cache-value] for PMD +* ``NOISEVM_CA`` - [min-cache-value, max-cache-value] for Noisy VM + +VSPERF Containers +----------------- + +VSPERF containers are found in tools/docker folder. + +RESULTS CONTAINER +^^^^^^^^^^^^^^^^^ + +The results container includes multiple services - ELK Stack, Barometer-Grafana, OPNFV-TestAPI & Jupyter. + +Pre-Deployment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Set the limit on mmap counts equal to 262144 or more. + You can do this by the command - ``sysctl -w vm.max_map_count = 262144``. + Or to set it permanently, update the ``vm.max_map_count`` field in ``/etc/sysctl.conf``. + +2. You may want to modify the IP address from 0.0.0.0 to appropriate host-ip in ``docker-compose.yml`` + +3. Please add dashboards folder from OPNFV-Barometer-Grafana into the grafana folder. It can be found in `Barometer Grafana <https://github.com/opnfv/barometer/tree/master/docker/barometer-grafana` + +Build +~~~~~ + +Run ``docker-compose build`` command to build the container. + +Run +~~~ + +Run the container with ``docker-compose up`` command. + +Post-Deployment Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The directory ``resultsdb`` contains the source from Dovetail/Dovetail-webportal project. +Once the results container is deployed, please run the python script as follows, to ensure that results can be +pushed and queried correctly - ``python init_db.py host_ip_address testapi_port``. +For example, if the host on which the container is running is 10.10.120.22, and container is exposing 8000 as the port, +the command should be: ``python init_db.py 10.10.120.22 8000`` diff --git a/docs/testing/user/configguide/trafficgen.rst b/docs/testing/user/configguide/trafficgen.rst index 91c4084e..3bb09d52 100644 --- a/docs/testing/user/configguide/trafficgen.rst +++ b/docs/testing/user/configguide/trafficgen.rst @@ -39,12 +39,14 @@ and is configured as follows: TRAFFIC = { 'traffic_type' : 'rfc2544_throughput', 'frame_rate' : 100, + 'burst_size' : 100, 'bidir' : 'True', # will be passed as string in title format to tgen 'multistream' : 0, 'stream_type' : 'L4', 'pre_installed_flows' : 'No', # used by vswitch implementation 'flow_type' : 'port', # used by vswitch implementation - + 'flow_control' : False, # supported only by IxNet + 'learning_frames' : True, # supported only by IxNet 'l2': { 'framesize': 64, 'srcmac': '00:00:00:00:00:00', @@ -67,8 +69,38 @@ and is configured as follows: 'priority': 0, 'cfi': 0, }, + 'capture': { + 'enabled': False, + 'tx_ports' : [0], + 'rx_ports' : [1], + 'count': 1, + 'filter': '', + }, + 'scapy': { + 'enabled': False, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_src}, dst={IP_dst})/' + '{IP_PROTO}(sport={IP_PROTO_sport}, dport={IP_PROTO_dport})', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/' + 'Dot1Q(prio={Dot1Q_prio}, id={Dot1Q_id}, vlan={Dot1Q_vlan})/' + 'IP(proto={IP_proto}, src={IP_dst}, dst={IP_src})/' + '{IP_PROTO}(sport={IP_PROTO_dport}, dport={IP_PROTO_sport})', + }, + 'latency_histogram': { + 'enabled': False, + 'type': 'Default', + }, + 'imix': { + 'enabled': True, + 'type': 'genome', + 'genome': 'aaaaaaaddddg', + }, } +A detailed description of the ``TRAFFIC`` dictionary can be found at +:ref:`configuration-of-traffic-dictionary`. + The framesize parameter can be overridden from the configuration files by adding the following to your custom configuration file ``10_custom.conf``: @@ -92,6 +124,13 @@ commandline above to: $ ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=(x,y);TRAFFICGEN_DURATION=10;" \ "TRAFFICGEN_RFC2544_TESTS=1" $TESTNAME +If you use imix, set the TRAFFICGEN_PKT_SIZES to 0. + +.. code-block:: console + + TRAFFICGEN_PKT_SIZES = (0,) + + .. _trafficgen-dummy: Dummy @@ -368,7 +407,7 @@ Spirent Setup Spirent installation files and instructions are available on the Spirent support website at: -http://support.spirent.com +https://support.spirent.com Select a version of Spirent TestCenter software to utilize. This example will use Spirent TestCenter v4.57 as an example. Substitute the appropriate @@ -420,7 +459,7 @@ STC ReST API. Basic ReST functionality is provided by the resthttp module, and may be used for writing ReST clients independent of STC. - Project page: <https://github.com/Spirent/py-stcrestclient> -- Package download: <http://pypi.python.org/pypi/stcrestclient> +- Package download: <https://pypi.python.org/project/stcrestclient> To use REST interface, follow the instructions in the Project page to install the package. Once installed, the scripts named with 'rest' keyword @@ -543,6 +582,22 @@ Note that 'FORWARDING_RATE_FPS', 'CACHING_CAPACITY_ADDRS', 'ADDR_LEARNED_PERCENT' and 'OPTIMAL_LEARNING_RATE_FPS' are the new result-constants added to support RFC2889 tests. +4. Latency Histogram. To enable latency histogram as in results, +enable latency_histogram in conf/03_traffic.conf. + +.. code-block:: python + + 'Latency_hisotgram': + { + "enabled": True, + "tpe": "Default, + } + +Once, enabled, a 'Histogram.csv' file will be generated in the results folder. +The Histogram.csv will include latency histogram in the following order. +(a) Packet size (b) Ranges in 10ns (c) Packet counts. These set of 3 lines, +will be repeated for every packet-sizes. + .. _`Xena Networks`: Xena Networks @@ -563,13 +618,13 @@ support contract. To execute the Xena2544.exe file under Linux distributions the mono-complete package must be installed. To install this package follow the instructions below. Further information can be obtained from -http://www.mono-project.com/docs/getting-started/install/linux/ +https://www.mono-project.com/docs/getting-started/install/linux/ .. code-block:: console rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" yum-config-manager --add-repo http://download.mono-project.com/repo/centos/ - yum -y install mono-complete + yum -y install mono-complete-5.8.0.127-0.xamarin.3.epel7.x86_64 To prevent gpg errors on future yum installation of packages the mono-project repo should be disabled once installed. @@ -659,6 +714,14 @@ or modify the length of the learning by modifying the following settings. TRAFFICGEN_XENA_CONT_PORT_LEARNING_ENABLED = False TRAFFICGEN_XENA_CONT_PORT_LEARNING_DURATION = 3 +Multistream Modifier +~~~~~~~~~~~~~~~~~~~~ + +Xena has a modifier maximum value or 64k in size. For this reason when specifying +Multistream values of greater than 64k for Layer 2 or Layer 3 it will use two +modifiers that may be modified to a value that can be square rooted to create the +two modifiers. You will see a log notification for the new value that was calculated. + MoonGen ------- @@ -691,7 +754,7 @@ trafficgen.lua Follow MoonGen set up and execution instructions here: -https://github.com/atheurer/lua-trafficgen/blob/master/README.md +https://github.com/atheurer/trafficgen/blob/master/README.md Note one will need to set up ssh login to not use passwords between the server running MoonGen and the device under test (running the VSPERF test @@ -737,11 +800,14 @@ You can directly download from GitHub: git clone https://github.com/cisco-system-traffic-generator/trex-core -and use the master branch: +and use the same Trex version for both server and client API. + +**NOTE:** The Trex API version used by VSPERF is defined by variable ``TREX_TAG`` +in file ``src/package-list.mk``. .. code-block:: console - git checkout master + git checkout v2.38 or Trex latest release you can download from here: @@ -783,6 +849,17 @@ It is neccesary for proper connection between Trex server and VSPERF. cd trex-core/scripts/ ./t-rex-64 -i +**NOTE:** Please check your firewall settings at both DUT and T-Rex server. +Firewall must allow a connection from DUT (VSPERF) to the T-Rex server running +at TCP port 4501. + +**NOTE:** For high speed cards it may be advantageous to start T-Rex with more transmit queues/cores. + +.. code-block:: console + + cd trex-cores/scripts/ + ./t-rex-64 -i -c 10 + For additional information about Trex stateless mode see Trex stateless documentation: https://trex-tgn.cisco.com/trex/doc/trex_stateless.html @@ -835,6 +912,21 @@ place. This can be adjusted with the following configurations: TRAFFICGEN_TREX_LEARNING_MODE=True TRAFFICGEN_TREX_LEARNING_DURATION=5 +Latency measurements have impact on T-Rex performance. Thus vswitchperf uses a separate +latency stream for each direction with limited speed. This workaround is used for RFC2544 +**Throughput** and **Continuous** traffic types. In case of **Burst** traffic type, +the latency statistics are measured for all frames in the burst. Collection of latency +statistics is driven by configuration option ``TRAFFICGEN_TREX_LATENCY_PPS`` as follows: + + * value ``0`` - disables latency measurements + * non zero integer value - enables latency measurements; In case of Throughput + and Continuous traffic types, it specifies a speed of latency specific stream + in PPS. In case of burst traffic type, it enables latency measurements for all frames. + +.. code-block:: console + + TRAFFICGEN_TREX_LATENCY_PPS = 1000 + SR-IOV and Multistream layer 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ T-Rex by default only accepts packets on the receive side if the destination mac matches the @@ -850,6 +942,22 @@ modified. Enable Promiscuous mode when doing multistream at layer 2 testing with TRAFFICGEN_TREX_PROMISCUOUS=True +Card Bandwidth Options +~~~~~~~~~~~~~~~~~~~~~~ + +T-Rex API will attempt to retrieve the highest possible speed from the card using internal +calls to port information. If you are using two separate cards then it will take the lowest +of the two cards as the max speed. If necessary you can try to force the API to use a +specific maximum speed per port. The below configurations can be adjusted to enable this. + +.. code-block:: console + + TRAFFICGEN_TREX_FORCE_PORT_SPEED = True + TRAFFICGEN_TREX_PORT_SPEED = 40000 # 40 gig + +**Note::** Setting higher than possible speeds will result in unpredictable behavior when running +tests such as duration inaccuracy and/or complete test failure. + RFC2544 Validation ~~~~~~~~~~~~~~~~~~ @@ -869,3 +977,68 @@ The duration and maximum number of attempted verification trials can be set to c behavior of this step. If the verification step fails, it will resume the binary search with new values where the maximum output will be the last attempted frame rate minus the current set thresh hold. + +Scapy frame definition +~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to use a SCAPY frame definition to generate various network protocols +by the **T-Rex** traffic generator. In case that particular network protocol layer +is disabled by the TRAFFIC dictionary (e.g. TRAFFIC['vlan']['enabled'] = False), +then disabled layer will be removed from the scapy format definition by VSPERF. + +The scapy frame definition can refer to values defined by the TRAFFIC dictionary +by following keywords. These keywords are used in next examples. + +* ``Ether_src`` - refers to ``TRAFFIC['l2']['srcmac']`` +* ``Ether_dst`` - refers to ``TRAFFIC['l2']['dstmac']`` +* ``IP_proto`` - refers to ``TRAFFIC['l3']['proto']`` +* ``IP_PROTO`` - refers to upper case version of ``TRAFFIC['l3']['proto']`` +* ``IP_src`` - refers to ``TRAFFIC['l3']['srcip']`` +* ``IP_dst`` - refers to ``TRAFFIC['l3']['dstip']`` +* ``IP_PROTO_sport`` - refers to ``TRAFFIC['l4']['srcport']`` +* ``IP_PROTO_dport`` - refers to ``TRAFFIC['l4']['dstport']`` +* ``Dot1Q_prio`` - refers to ``TRAFFIC['vlan']['priority']`` +* ``Dot1Q_id`` - refers to ``TRAFFIC['vlan']['cfi']`` +* ``Dot1Q_vlan`` - refers to ``TRAFFIC['vlan']['id']`` + +In following examples of SCAPY frame definition only relevant parts of TRAFFIC +dictionary are shown. The rest of the TRAFFIC dictionary is set to default values +as they are defined in ``conf/03_traffic.conf``. + +Please check official documentation of SCAPY project for details about SCAPY frame +definition and supported network layers at: https://scapy.net + +#. Generate ICMP frames: + + .. code-block:: console + + 'scapy': { + 'enabled': True, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IP(proto="icmp", src={IP_src}, dst={IP_dst})/ICMP()', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IP(proto="icmp", src={IP_dst}, dst={IP_src})/ICMP()', + } + +#. Generate IPv6 ICMP Echo Request + + .. code-block:: console + + 'l3' : { + 'srcip': 'feed::01', + 'dstip': 'feed::02', + }, + 'scapy': { + 'enabled': True, + '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IPv6(src={IP_src}, dst={IP_dst})/ICMPv6EchoRequest()', + '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IPv6(src={IP_dst}, dst={IP_src})/ICMPv6EchoRequest()', + } + +#. Generate TCP frames: + + Example uses default SCAPY frame definition, which can reflect ``TRAFFIC['l3']['proto']`` settings. + + .. code-block:: console + + 'l3' : { + 'proto' : 'tcp', + }, + diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 64d91657..2c7a78ff 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -11,10 +11,10 @@ VSPERF Test Guide .. toctree:: :caption: VSPERF Test Execution :maxdepth: 2 - :numbered: ./testusage.rst ./teststeps.rst ./integration.rst + ./trafficcapture.rst ./yardstick.rst ./testlist.rst diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst index 66808400..9d847fd8 100644 --- a/docs/testing/user/userguide/integration.rst +++ b/docs/testing/user/userguide/integration.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others. .. _integration-tests: @@ -22,6 +22,12 @@ P2P (Physical to Physical scenarios). NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only. +NOTE: The overlay tests require an IxNet traffic generator. The tunneled traffic +is configured by ``ixnetrfc2544v2.tcl`` script. This script can be used +with all supported deployment scenarios for generation of frames with VXLAN, GRE +or GENEVE protocols. In that case options "Tunnel Operation" and +"TRAFFICGEN_IXNET_TCL_SCRIPT" must be properly configured at testcase definition. + Executing Integration Tests --------------------------- @@ -63,8 +69,8 @@ the following variables in you user_settings.py file: VTEP_IP2_SUBNET = '192.168.240.0/24' # Bridge names - TUNNEL_INTEGRATION_BRIDGE = 'br0' - TUNNEL_EXTERNAL_BRIDGE = 'br-ext' + TUNNEL_INTEGRATION_BRIDGE = 'vsperf-br0' + TUNNEL_EXTERNAL_BRIDGE = 'vsperf-br-ext' # IP of br-ext TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24' diff --git a/docs/testing/user/userguide/testlist.rst b/docs/testing/user/userguide/testlist.rst index 2b0e9d7f..fe8c840a 100644 --- a/docs/testing/user/userguide/testlist.rst +++ b/docs/testing/user/userguide/testlist.rst @@ -68,14 +68,13 @@ vswitch_pvvp_tput vSwitch - configure switch, two chained v vswitch_pvvp_back2back vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test vswitch_pvvp_cont vSwitch - configure switch, two chained vnfs and execute RFC2544 continuous stream test vswitch_pvvp_all vSwitch - configure switch, two chained vnfs and execute all test types -vswitch_p4vp Just configure 4 chained vnfs -vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test -vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test -vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test -vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput test -2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port -4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port -6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port +vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test, deployment pvvp4 +vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test, deployment pvvp4 +vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test, deployment pvvp4 +vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput tests, deployment pvvp4 +2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port, deployment pvpv2 +4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port, deployment pvpv4 +6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port, deployment pvpv6 vhost_numa_awareness vSwitch DPDK - verify that PMD threads are served by the same NUMA slot as QEMU instances ixnet_pvp_tput_1nic PVP Scenario with 1 port towards IXIA vswitch_vports_add_del_connection_vpp VPP: vSwitch - configure switch with vports, add and delete connection @@ -388,3 +387,46 @@ ovsdpdk_qos_p2p In a p2p setup, ensure when a QoS egres ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate. ======================================== ====================================================================================== + +Custom Statistics ++++++++++++++++++ + +A set of functional testcases for validation of Custom Statistics support by OVS. +This feature allows Custom Statistics to be accessed by VSPERF. + +These testcases require DPDK v17.11, the latest Open vSwitch(v2.9.90) +and the IxNet traffic-generator. + +======================================== ====================================================================================== +ovsdpdk_custstat_check Test if custom statistics are supported. +ovsdpdk_custstat_rx_error Test bad ethernet CRC counter 'rx_crc_errors' exposed by custom + statistics. + +======================================== ====================================================================================== + +T-Rex in VM TestCases +^^^^^^^^^^^^^^^^^^^^^ + +A set of functional testcases, which use T-Rex running in VM as a traffic generator. +These testcases require a VM image with T-Rex server installed. An example of such +image is a vloop-vnf image with T-Rex available for download at: + +http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-16.04_trex_20180209.qcow2 + +This image can be used for both T-Rex VM and loopback VM in ``vm2vm`` testcases. + +**NOTE:** The performance of T-Rex running inside the VM is lower if compared to T-Rex +execution on bare-metal. The user should perform a calibration of the VM maximum FPS +capability, to ensure this limitation is understood. + +======================================== ====================================================================================== +trex_vm_cont T-Rex VM - execute RFC2544 Continuous Stream from T-Rex VM and loop + it back through Open vSwitch. +trex_vm_tput T-Rex VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through Open vSwitch. +trex_vm2vm_cont T-Rex VM2VM - execute RFC2544 Continuous Stream from T-Rex VM and + loop it back through 2nd VM. +trex_vm2vm_tput T-Rex VM2VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through 2nd VM. + +======================================== ====================================================================================== diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst index 08c95311..cb627bc5 100644 --- a/docs/testing/user/userguide/teststeps.rst +++ b/docs/testing/user/userguide/teststeps.rst @@ -23,6 +23,13 @@ the step number by one which is indicated in the log. (testcases.integration) - Step 0 'vswitch add_vport ['br0']' start +Test steps are defined as a list of steps within a ``TestSteps`` item of test +case definition. Each step is a list with following structure: + +.. code-block:: python + + '[' [ optional-alias ',' ] test-object ',' test-function [ ',' optional-function-params ] '],' + Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue @@ -57,8 +64,14 @@ Step driven testcases can be used in two different ways: Test objects and their functions -------------------------------- -Every test step can call a function of one of the supported test objects. The list -of supported objects and their most common functions follows: +Every test step can call a function of one of the supported test objects. In general +any existing function of supported test object can be called by test step. In case +that step validation is required (valid for integration test steps, which are not +suppressed), then appropriate ``validate_`` method must be implemented. + +The list of supported objects and their most common functions is listed below. Please +check implementation of test objects for full list of implemented functions and their +parameters. * ``vswitch`` - provides functions for vSwitch configuration @@ -176,6 +189,8 @@ of supported objects and their most common functions follows: * ``getValue param`` - returns value of given ``param`` * ``setValue param value`` - sets value of ``param`` to given ``value`` + * ``resetValue param`` - if ``param`` was overridden by ``TEST_PARAMS`` (e.g. by "Parameters" + section of the test case definition), then it will be set to its original value. Examples: @@ -185,6 +200,8 @@ of supported objects and their most common functions follows: ['settings', 'setValue', 'GUEST_USERNAME', ['root']] + ['settings', 'resetValue', 'WHITELIST_NICS'], + It is possible and more convenient to access any VSPERF configuration option directly via ``$NAME`` notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of ``settings.getValue``. @@ -747,6 +764,8 @@ destination UDP port. ] }, +The same test can be written in a shorter form using "Deployment" : "pvpv". + To run the test: .. code-block:: console @@ -779,20 +798,20 @@ and available in both csv and rst report files. }, }, "TestSteps": [ - ['vswitch', 'add_vport', 'br0'], - ['vswitch', 'add_vport', 'br0'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], # priority must be higher than default 32768, otherwise flows won't match - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], 'idle_timeout': '0'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], 'idle_timeout': '0'}], - ['vswitch', 'dump_flows', 'br0'], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], ['vnf1', 'start'], ] }, diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst index 20c30a40..3dd41846 100644 --- a/docs/testing/user/userguide/testusage.rst +++ b/docs/testing/user/userguide/testusage.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, Spirent, AT&T and others. vSwitchPerf test suites userguide --------------------------------- @@ -91,55 +91,41 @@ Using a custom settings file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory -of if you want to use an alternative configuration file, the file can +or if you want to use an alternative configuration file, the file can be passed to ``vsperf`` via the ``--conf-file`` argument. .. code-block:: console $ ./vsperf --conf-file <path_to_custom_conf> ... -Note that configuration passed in via the environment (``--load-env``) -or via another command line argument will override both the default and -your custom configuration files. This "priority hierarchy" can be -described like so (1 = max priority): - -1. Testcase definition section ``Parameters`` -2. Command line arguments -3. Environment variables -4. Configuration file(s) - -Further details about configuration files evaluation and special behaviour +Evaluation of configuration parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The value of configuration parameter can be specified at various places, +e.g. at the test case definition, inside configuration files, by the command +line argument, etc. Thus it is important to understand the order of configuration +parameter evaluation. This "priority hierarchy" can be described like so +(1 = max priority): + +1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type`` +2. Parameters inside testcase definition section ``Parameters`` +3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.) +4. Environment variables (see ``--load-env`` argument) +5. Custom configuration file specified via ``--conf-file`` argument +6. Standard configuration files, where higher prefix number means higher + priority. + +For example, if the same configuration parameter is defined in custom configuration +file (specified via ``--conf-file`` argument), via ``--test-params`` argument +and also inside ``Parameters`` section of the testcase definition, then parameter +value from the ``Parameters`` section will be used. + +Further details about order of configuration files evaluation and special behaviour of options with ``GUEST_`` prefix could be found at :ref:`design document <design-configuration>`. .. _overriding-parameters-documentation: -Referencing parameter values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is possible to use a special macro ``#PARAM()`` to refer to the value of -another configuration parameter. This reference is evaluated during -access of the parameter value (by ``settings.getValue()`` call), so it -can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. -It can be used to reflect DUT HW details in the testcase definition. - -Example: - -.. code:: python - - { - ... - "Name": "testcase", - "Parameters" : { - "TRAFFIC" : { - 'l2': { - # set destination MAC to the MAC of the first - # interface from WHITELIST_NICS list - 'dstmac' : '#PARAM(NICS[0]["mac"])', - }, - }, - ... - Overriding values defined in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -155,6 +141,17 @@ Example: $ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \ "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput +The ``--test-params`` command line argument can also be used to override default +configuration values for multiple tests. Providing a list of parameters will apply each +element of the list to the test with the same index. If more tests are run than +parameters provided the last element of the list will repeat. + +.. code:: console + + $ ./vsperf --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)'," + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + pvvp_tput pvvp_tput + The second option is to override configuration items by ``Parameters`` section of the test case definition. The configuration items can be added into ``Parameters`` dictionary with their new values. These values will override values defined in @@ -186,6 +183,36 @@ parameter name is passed via ``--test-params`` CLI argument or defined in ``Para section of test case definition. It is also forbidden to redefine a value of ``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section. +**NOTE:** The new definition of the dictionary parameter, specified via ``--test-params`` +or inside ``Parameters`` section, will not override original dictionary values. Instead +the original dictionary will be updated with values from the new dictionary definition. + +Referencing parameter values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is possible to use a special macro ``#PARAM()`` to refer to the value of +another configuration parameter. This reference is evaluated during +access of the parameter value (by ``settings.getValue()`` call), so it +can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. +It can be used to reflect DUT HW details in the testcase definition. + +Example: + +.. code:: python + + { + ... + "Name": "testcase", + "Parameters" : { + "TRAFFIC" : { + 'l2': { + # set destination MAC to the MAC of the first + # interface from WHITELIST_NICS list + 'dstmac' : '#PARAM(NICS[0]["mac"])', + }, + }, + ... + vloop_vnf ^^^^^^^^^ @@ -205,6 +232,12 @@ A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd +Additional Tools Setup +^^^^^^^^^^^^^^^^^^^^^^ + +Follow the `Additional tools instructions <additional-tools-configuration>` to +install and configure additional tools such as collectors and loadgens. + Executing tests ^^^^^^^^^^^^^^^ @@ -234,6 +267,12 @@ To run a single test: Where $TESTNAME is the name of the vsperf test you would like to run. +To run a test multiple times, repeat it: + +.. code-block:: console + + $ ./vsperf $TESTNAME $TESTNAME $TESTNAME + To run a group of tests, for example all tests with a name containing 'RFC2544': @@ -256,6 +295,30 @@ Some tests allow for configurable parameters, including test duration --tests RFC2544Tput \ --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)" +To specify configurable parameters for multiple tests, use a list of +parameters. One element for each test. + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + +If the ``CUMULATIVE_PARAMS`` setting is set to True and there are different parameters +provided for each test using ``--test-params``, each test will take the parameters of +the previous test before appyling it's own. +With ``CUMULATIVE_PARAMS`` set to True the following command will be equivalent to the +previous example: + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + " + For all available options, check out the help dialog: .. code-block:: console @@ -425,10 +488,6 @@ set ``PATHS['dpdk']['bin']['modules']`` instead. **NOTE:** Please ensure your boot/grub parameters include the following: -**NOTE:** In case of VPP, it is required to explicitly define, that vfio-pci -DPDK driver should be used. It means to update dpdk part of VSWITCH_VPP_ARGS -dictionary with uio-driver section, e.g. VSWITCH_VPP_ARGS['dpdk'] = 'uio-driver vfio-pci' - .. code-block:: console iommu=pt intel_iommu=on @@ -448,6 +507,10 @@ To check that IOMMU is enabled on your platform: [ 3.335746] IOMMU: dmar1 using Queued invalidation .... +**NOTE:** In case of VPP, it is required to explicitly define, that vfio-pci +DPDK driver should be used. It means to update dpdk part of VSWITCH_VPP_ARGS +dictionary with uio-driver section, e.g. VSWITCH_VPP_ARGS['dpdk'] = 'uio-driver vfio-pci' + .. _SRIOV-support: Using SRIOV support @@ -584,7 +647,7 @@ The supported dpdk guest bind drivers are: .. code-block:: console - 'uio_pci_generic' - Use uio_pci_generic driver + 'uio_pci_generic' - Use uio_pci_generic driver 'igb_uio_from_src' - Build and use the igb_uio driver from the dpdk src files 'vfio_no_iommu' - Use vfio with no iommu option. This requires custom @@ -599,7 +662,7 @@ modified to use igb_uio_from_src instead. Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk 16.04 or greater. Using this option will also taint the kernel. -Please refer to the dpdk documents at http://dpdk.org/doc/guides for more +Please refer to the dpdk documents at https://doc.dpdk.org/guides for more information on these drivers. Guest Core and Thread Binding @@ -915,6 +978,39 @@ Example of execution of VSPERF in "trafficgen" mode: $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \ --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}" +Performance Matrix +^^^^^^^^^^^^^^^^^^ + +The ``--matrix`` command line argument analyses and displays the performance of +all the tests run. Using the metric specified by ``MATRIX_METRIC`` in the conf-file, +the first test is set as the baseline and all the other tests are compared to it. +The ``MATRIX_METRIC`` must always refer to a numeric value to enable comparision. +A table, with the test ID, metric value, the change of the metric in %, testname +and the test parameters used for each test, is printed out as well as saved into the +results directory. + +Example of 2 tests being compared using Performance Matrix: + +.. code-block:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_PKT_SIZES=(64,)',"\ + "'TRAFFICGEN_PKT_SIZES=(128,)']" \ + phy2phy_cont phy2phy_cont --matrix + +Example output: + +.. code-block:: console + + +------+--------------+---------------------+----------+---------------------------------------+ + | ID | Name | throughput_rx_fps | Change | Parameters, CUMULATIVE_PARAMS = False | + +======+==============+=====================+==========+=======================================+ + | 0 | phy2phy_cont | 23749000.000 | 0 | 'TRAFFICGEN_PKT_SIZES': [64] | + +------+--------------+---------------------+----------+---------------------------------------+ + | 1 | phy2phy_cont | 16850500.000 | -29.048 | 'TRAFFICGEN_PKT_SIZES': [128] | + +------+--------------+---------------------+----------+---------------------------------------+ + + Code change verification by pylint ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/testing/user/userguide/trafficcapture.rst b/docs/testing/user/userguide/trafficcapture.rst new file mode 100644 index 00000000..8a224dcb --- /dev/null +++ b/docs/testing/user/userguide/trafficcapture.rst @@ -0,0 +1,297 @@ +Traffic Capture +--------------- + +Tha ability to capture traffic at multiple points of the system is crucial to +many of the functional tests. It allows the verification of functionality for +both the vSwitch and the NICs using hardware acceleration for packet +manipulation and modification. + +There are three different methods of traffic capture supported by VSPERF. +Detailed descriptions of these methods as well as their pros and cons can be +found in the following chapters. + +Traffic Capture inside of a VM +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This method uses the standard PVP scenario, in which vSwitch first processes +and modifies the packet before forwarding it to the VM. Inside of the VM we +capture the traffic using **tcpdump** or a similiar technique. The capture +information is the used to verify the expected modifications to the packet done +by vSwitch. + +.. code-block:: console + + _ + +--------------------------------------------------+ | + | | | + | +------------------------------------------+ | | + | | Traffic capture and Packet Forwarding | | | + | +------------------------------------------+ | | + | ^ : | | + | | | | | Guest + | : v | | + | +---------------+ +---------------+ | | + | | logical port 0| | logical port 1| | | + +---+---------------+----------+---------------+---+ _| + ^ : + | | + : v _ + +---+---------------+----------+---------------+---+ | + | | logical port 0| | logical port 1| | | + | +---------------+ +---------------+ | | + | ^ : | | + | | | | | Host + | : v | | + | +--------------+ +--------------+ | | + | | phy port | vSwitch | phy port | | | + +---+--------------+------------+--------------+---+ _| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+ + +PROS: + +- supports testing with all traffic generators +- easy to use and implement into test +- allows testing hardware offloading on the ingress side + +CONS: + +- does not allow testing hardware offloading on the egress side + +An example of Traffic Capture in VM test: + +.. code-block:: python + + # Capture Example 1 - Traffic capture inside VM (PVP scenario) + # This TestCase will modify VLAN ID set by the traffic generator to the new value. + # Correct VLAN ID settings is verified by inspection of captured frames. + { + Name: capture_pvp_modify_vid, + Deployment: pvp, + Description: Test and verify VLAN ID modification by Open vSwitch, + Parameters : { + VSWITCH : OvsDpdkVhost, # works also for Vanilla OVS + TRAFFICGEN_DURATION : 5, + TRAFFIC : { + traffic_type : rfc2544_continuous, + frame_rate : 100, + 'vlan': { + 'enabled': True, + 'id': 8, + 'priority': 1, + 'cfi': 0, + }, + }, + GUEST_LOOPBACK : ['linux_bridge'], + }, + TestSteps: [ + # replace original flows with vlan ID modification + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['mod_vlan_vid:4','output:3']}], + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['mod_vlan_vid:4','output:4']}], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], + # verify that received frames have modified vlan ID + ['VNF0', 'execute_and_wait', 'tcpdump -i eth0 -c 5 -w dump.pcap vlan 4 &'], + ['trafficgen', 'send_traffic',{}], + ['!VNF0', 'execute_and_wait', 'tcpdump -qer dump.pcap vlan 4 2>/dev/null | wc -l','|^(\d+)$'], + ['tools', 'assert', '#STEP[-1][0] == 5'], + ], + }, + +Traffic Capture for testing NICs with HW offloading/acceleration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The NIC with hardware acceleration/offloading is inserted as an additional card +into the server. Two ports on this card are then connected together using +a patch cable as shown in the diagram. Only a single port of the tested NIC is +setup with DPDK acceleration, while the other is handled by the Linux Ip stack +allowing for traffic capture. The two NICs are then connected by vSwitch so the +original card can forward the processed packets to the traffic generator. The +ports handled by Linux IP stack allow for capturing packets, which are then +analyzed for changes done by both the vSwitch and the NIC with hardware +acceleration. + +.. code-block:: console + + _ + +------------------------------------------------+ | + | | | + | +----------------------------------------+ | | + | | vSwitch | | | + | | +----------------------------------+ | | | + | | | | | | | + | | | +------------------+ | | | | + | | | | | v | | | + | +----------------------------------------+ | | Device under Test + | ^ | ^ | | | + | | | | | | | + | | v | v | | + | +--------------+ +--------------+ | | + | | | | NIC w HW acc | | | + | | phy ports | | phy ports | | | + +---+--------------+----------+--------------+---+ _| + ^ : ^ : + | | | | + | | +-------+ + : v Patch Cable + +------------------------------------------------+ + | | + | traffic generator | + | | + +------------------------------------------------+ + +PROS: + +- allows testing hardware offloading on both the ingress and egress side +- supports testing with all traffic generators +- relatively easy to use and implement into tests + +CONS: + +- a more complex setup with two cards +- if the tested card only has one port, an additional card is needed + +An example of Traffic Capture for testing NICs with HW offloading test: + +.. code-block:: python + + # Capture Example 2 - Setup with 2 NICs, where traffic is captured after it is + # processed by NIC under the test (2nd NIC). See documentation for further details. + # This TestCase will strip VLAN headers from traffic sent by the traffic generator. + # The removal of VLAN headers is verified by inspection of captured frames. + # + # NOTE: This setup expects a DUT with two NICs with two ports each. First NIC is + # connected to the traffic generator (standard VSPERF setup). Ports of a second NIC + # are interconnected by a patch cable. PCI addresses of all four ports have to be + # properly configured in the WHITELIST_NICS parameter. + { + Name: capture_p2p2p_strip_vlan_ovs, + Deployment: clean, + Description: P2P Continuous Stream, + Parameters : { + _CAPTURE_P2P2P_OVS_ACTION : 'strip_vlan', + TRAFFIC : { + bidir : False, + traffic_type : rfc2544_continuous, + frame_rate : 100, + 'l2': { + 'srcmac': ca:fe:00:00:00:00, + 'dstmac': 00:00:00:00:00:01 + }, + 'vlan': { + 'enabled': True, + 'id': 8, + 'priority': 1, + 'cfi': 0, + }, + }, + # suppress DPDK configuration, so physical interfaces are not bound to DPDK driver + 'WHITELIST_NICS' : [], + 'NICS' : [], + }, + TestSteps: _CAPTURE_P2P2P_SETUP + [ + # capture traffic after processing by NIC under the test (after possible egress HW offloading) + ['tools', 'exec_shell_background', 'tcpdump -i [2][device] -c 5 -w capture.pcap ' + 'ether src [l2][srcmac]'], + ['trafficgen', 'send_traffic', {}], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], + ['vswitch', 'dump_flows', 'br1'], + # there must be 5 captured frames... + ['tools', 'exec_shell', 'tcpdump -r capture.pcap | wc -l', '|^(\d+)$'], + ['tools', 'assert', '#STEP[-1][0] == 5'], + # ...but no vlan headers + ['tools', 'exec_shell', 'tcpdump -r capture.pcap vlan | wc -l', '|^(\d+)$'], + ['tools', 'assert', '#STEP[-1][0] == 0'], + ], + }, + + +Traffic Capture on the Traffic Generator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using the functionality of the Traffic generator makes it possible to configure +Traffic Capture on both it's ports. With Traffic Capture enabled, VSPERF +instructs the Traffic Generator to automatically export captured data into +a pcap file. The captured packets are then sent to VSPERF for analysis and +verification, monitoring any changes done by both vSwitch and the NICs. + +Vsperf currently only supports this functionality with the **T-Rex** generator. + +.. code-block:: console + + _ + +--------------------------------------------------+ | + | | | + | +--------------------------+ | | + | | | | | + | | v | | Host + | +--------------+ +--------------+ | | + | | phy port | vSwitch | phy port | | | + +---+--------------+------------+--------------+---+ _| + ^ : + | | + : v + +--------------------------------------------------+ + | | + | traffic generator | + | | + +--------------------------------------------------+ + +PROS: + +- allows testing hardware offloading on both the ingress and egress side +- does not require an additional NIC + +CONS: + +- currently only supported by **T-Rex** traffic generator + +An example Traffic Capture on the Traffic Generator test: + +.. code-block:: python + + + # Capture Example 3 - Traffic capture by traffic generator. + # This TestCase uses OVS flow to add VLAN tag with given ID into every + # frame send by traffic generator. Correct frame modificaiton is verified by + # inspection of packet capture received by T-Rex. + { + Name: capture_p2p_add_vlan_ovs_trex, + Deployment: clean, + Description: OVS: Test VLAN tag modification and verify it by traffic capture, + vSwitch : OvsDpdkVhost, # works also for Vanilla OVS + Parameters : { + TRAFFICGEN : Trex, + TRAFFICGEN_DURATION : 5, + TRAFFIC : { + traffic_type : rfc2544_continuous, + frame_rate : 100, + # enable capture of five RX frames + 'capture': { + 'enabled': True, + 'tx_ports' : [], + 'rx_ports' : [1], + 'count' : 5, + }, + }, + }, + TestSteps : STEP_VSWITCH_P2P_INIT + [ + # replace standard L2 flows by flows, which will add VLAN tag with ID 3 + ['!vswitch', 'add_flow', 'int_br0', {'in_port': '1', 'actions': ['mod_vlan_vid:3','output:2']}], + ['!vswitch', 'add_flow', 'int_br0', {'in_port': '2', 'actions': ['mod_vlan_vid:3','output:1']}], + ['vswitch', 'dump_flows', 'int_br0'], + ['trafficgen', 'send_traffic', {}], + ['trafficgen', 'get_results'], + # verify that captured frames have vlan tag with ID 3 + ['tools', 'exec_shell', 'tcpdump -qer /#STEP[-1][0][capture_rx] vlan 3 ' + '2>/dev/null | wc -l', '|^(\d+)$'], + # number of received frames with expected VLAN id must match the number of captured frames + ['tools', 'assert', '#STEP[-1][0] == 5'], + ] + STEP_VSWITCH_P2P_FINIT, + }, + |