diff options
Diffstat (limited to 'docs/testing/user/userguide')
| -rw-r--r-- | docs/testing/user/userguide/integration.rst | 12 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testlist.rst | 58 | ||||
| -rw-r--r-- | docs/testing/user/userguide/teststeps.rst | 37 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testusage.rst | 174 | ||||
| -rw-r--r-- | docs/testing/user/userguide/trafficcapture.rst | 8 |
5 files changed, 226 insertions, 63 deletions
diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst index 66808400..9d847fd8 100644 --- a/docs/testing/user/userguide/integration.rst +++ b/docs/testing/user/userguide/integration.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others. .. _integration-tests: @@ -22,6 +22,12 @@ P2P (Physical to Physical scenarios). NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only. +NOTE: The overlay tests require an IxNet traffic generator. The tunneled traffic +is configured by ``ixnetrfc2544v2.tcl`` script. This script can be used +with all supported deployment scenarios for generation of frames with VXLAN, GRE +or GENEVE protocols. In that case options "Tunnel Operation" and +"TRAFFICGEN_IXNET_TCL_SCRIPT" must be properly configured at testcase definition. + Executing Integration Tests --------------------------- @@ -63,8 +69,8 @@ the following variables in you user_settings.py file: VTEP_IP2_SUBNET = '192.168.240.0/24' # Bridge names - TUNNEL_INTEGRATION_BRIDGE = 'br0' - TUNNEL_EXTERNAL_BRIDGE = 'br-ext' + TUNNEL_INTEGRATION_BRIDGE = 'vsperf-br0' + TUNNEL_EXTERNAL_BRIDGE = 'vsperf-br-ext' # IP of br-ext TUNNEL_EXTERNAL_BRIDGE_IP = '192.168.240.1/24' diff --git a/docs/testing/user/userguide/testlist.rst b/docs/testing/user/userguide/testlist.rst index 2b0e9d7f..fe8c840a 100644 --- a/docs/testing/user/userguide/testlist.rst +++ b/docs/testing/user/userguide/testlist.rst @@ -68,14 +68,13 @@ vswitch_pvvp_tput vSwitch - configure switch, two chained v vswitch_pvvp_back2back vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test vswitch_pvvp_cont vSwitch - configure switch, two chained vnfs and execute RFC2544 continuous stream test vswitch_pvvp_all vSwitch - configure switch, two chained vnfs and execute all test types -vswitch_p4vp Just configure 4 chained vnfs -vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test -vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test -vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test -vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput test -2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port -4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port -6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port +vswitch_p4vp_tput 4 chained vnfs, execute RFC2544 throughput test, deployment pvvp4 +vswitch_p4vp_back2back 4 chained vnfs, execute RFC2544 back2back test, deployment pvvp4 +vswitch_p4vp_cont 4 chained vnfs, execute RFC2544 continuous stream test, deployment pvvp4 +vswitch_p4vp_all 4 chained vnfs, execute RFC2544 throughput tests, deployment pvvp4 +2pvp_udp_dest_flows RFC2544 Continuous TC with 2 Parallel VMs, flows on UDP Dest Port, deployment pvpv2 +4pvp_udp_dest_flows RFC2544 Continuous TC with 4 Parallel VMs, flows on UDP Dest Port, deployment pvpv4 +6pvp_udp_dest_flows RFC2544 Continuous TC with 6 Parallel VMs, flows on UDP Dest Port, deployment pvpv6 vhost_numa_awareness vSwitch DPDK - verify that PMD threads are served by the same NUMA slot as QEMU instances ixnet_pvp_tput_1nic PVP Scenario with 1 port towards IXIA vswitch_vports_add_del_connection_vpp VPP: vSwitch - configure switch with vports, add and delete connection @@ -388,3 +387,46 @@ ovsdpdk_qos_p2p In a p2p setup, ensure when a QoS egres ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egress policer is created that the traffic is limited to the specified rate. ======================================== ====================================================================================== + +Custom Statistics ++++++++++++++++++ + +A set of functional testcases for validation of Custom Statistics support by OVS. +This feature allows Custom Statistics to be accessed by VSPERF. + +These testcases require DPDK v17.11, the latest Open vSwitch(v2.9.90) +and the IxNet traffic-generator. + +======================================== ====================================================================================== +ovsdpdk_custstat_check Test if custom statistics are supported. +ovsdpdk_custstat_rx_error Test bad ethernet CRC counter 'rx_crc_errors' exposed by custom + statistics. + +======================================== ====================================================================================== + +T-Rex in VM TestCases +^^^^^^^^^^^^^^^^^^^^^ + +A set of functional testcases, which use T-Rex running in VM as a traffic generator. +These testcases require a VM image with T-Rex server installed. An example of such +image is a vloop-vnf image with T-Rex available for download at: + +http://artifacts.opnfv.org/vswitchperf/vnf/vloop-vnf-ubuntu-16.04_trex_20180209.qcow2 + +This image can be used for both T-Rex VM and loopback VM in ``vm2vm`` testcases. + +**NOTE:** The performance of T-Rex running inside the VM is lower if compared to T-Rex +execution on bare-metal. The user should perform a calibration of the VM maximum FPS +capability, to ensure this limitation is understood. + +======================================== ====================================================================================== +trex_vm_cont T-Rex VM - execute RFC2544 Continuous Stream from T-Rex VM and loop + it back through Open vSwitch. +trex_vm_tput T-Rex VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through Open vSwitch. +trex_vm2vm_cont T-Rex VM2VM - execute RFC2544 Continuous Stream from T-Rex VM and + loop it back through 2nd VM. +trex_vm2vm_tput T-Rex VM2VM - execute RFC2544 Throughput from T-Rex VM and loop it back + through 2nd VM. + +======================================== ====================================================================================== diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst index 08c95311..cb627bc5 100644 --- a/docs/testing/user/userguide/teststeps.rst +++ b/docs/testing/user/userguide/teststeps.rst @@ -23,6 +23,13 @@ the step number by one which is indicated in the log. (testcases.integration) - Step 0 'vswitch add_vport ['br0']' start +Test steps are defined as a list of steps within a ``TestSteps`` item of test +case definition. Each step is a list with following structure: + +.. code-block:: python + + '[' [ optional-alias ',' ] test-object ',' test-function [ ',' optional-function-params ] '],' + Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue @@ -57,8 +64,14 @@ Step driven testcases can be used in two different ways: Test objects and their functions -------------------------------- -Every test step can call a function of one of the supported test objects. The list -of supported objects and their most common functions follows: +Every test step can call a function of one of the supported test objects. In general +any existing function of supported test object can be called by test step. In case +that step validation is required (valid for integration test steps, which are not +suppressed), then appropriate ``validate_`` method must be implemented. + +The list of supported objects and their most common functions is listed below. Please +check implementation of test objects for full list of implemented functions and their +parameters. * ``vswitch`` - provides functions for vSwitch configuration @@ -176,6 +189,8 @@ of supported objects and their most common functions follows: * ``getValue param`` - returns value of given ``param`` * ``setValue param value`` - sets value of ``param`` to given ``value`` + * ``resetValue param`` - if ``param`` was overridden by ``TEST_PARAMS`` (e.g. by "Parameters" + section of the test case definition), then it will be set to its original value. Examples: @@ -185,6 +200,8 @@ of supported objects and their most common functions follows: ['settings', 'setValue', 'GUEST_USERNAME', ['root']] + ['settings', 'resetValue', 'WHITELIST_NICS'], + It is possible and more convenient to access any VSPERF configuration option directly via ``$NAME`` notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of ``settings.getValue``. @@ -747,6 +764,8 @@ destination UDP port. ] }, +The same test can be written in a shorter form using "Deployment" : "pvpv". + To run the test: .. code-block:: console @@ -779,20 +798,20 @@ and available in both csv and rst report files. }, }, "TestSteps": [ - ['vswitch', 'add_vport', 'br0'], - ['vswitch', 'add_vport', 'br0'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], + ['vswitch', 'add_vport', '$VSWITCH_BRIDGE_NAME'], # priority must be higher than default 32768, otherwise flows won't match - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['output:#STEP[-2][1]'], 'idle_timeout': '0', 'dl_type':'0x0800', 'nw_proto':'17', 'tp_dst':'0', 'priority': '33000'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:1'], 'idle_timeout': '0'}], - ['vswitch', 'add_flow', 'br0', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], + ['vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '#STEP[-4][1]', 'actions': ['output:2'], 'idle_timeout': '0'}], - ['vswitch', 'dump_flows', 'br0'], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], ['vnf1', 'start'], ] }, diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst index 20c30a40..c7cc1484 100644 --- a/docs/testing/user/userguide/testusage.rst +++ b/docs/testing/user/userguide/testusage.rst @@ -91,55 +91,41 @@ Using a custom settings file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory -of if you want to use an alternative configuration file, the file can +or if you want to use an alternative configuration file, the file can be passed to ``vsperf`` via the ``--conf-file`` argument. .. code-block:: console $ ./vsperf --conf-file <path_to_custom_conf> ... -Note that configuration passed in via the environment (``--load-env``) -or via another command line argument will override both the default and -your custom configuration files. This "priority hierarchy" can be -described like so (1 = max priority): - -1. Testcase definition section ``Parameters`` -2. Command line arguments -3. Environment variables -4. Configuration file(s) - -Further details about configuration files evaluation and special behaviour +Evaluation of configuration parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The value of configuration parameter can be specified at various places, +e.g. at the test case definition, inside configuration files, by the command +line argument, etc. Thus it is important to understand the order of configuration +parameter evaluation. This "priority hierarchy" can be described like so +(1 = max priority): + +1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type`` +2. Parameters inside testcase definition section ``Parameters`` +3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.) +4. Environment variables (see ``--load-env`` argument) +5. Custom configuration file specified via ``--conf-file`` argument +6. Standard configuration files, where higher prefix number means higher + priority. + +For example, if the same configuration parameter is defined in custom configuration +file (specified via ``--conf-file`` argument), via ``--test-params`` argument +and also inside ``Parameters`` section of the testcase definition, then parameter +value from the ``Parameters`` section will be used. + +Further details about order of configuration files evaluation and special behaviour of options with ``GUEST_`` prefix could be found at :ref:`design document <design-configuration>`. .. _overriding-parameters-documentation: -Referencing parameter values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is possible to use a special macro ``#PARAM()`` to refer to the value of -another configuration parameter. This reference is evaluated during -access of the parameter value (by ``settings.getValue()`` call), so it -can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. -It can be used to reflect DUT HW details in the testcase definition. - -Example: - -.. code:: python - - { - ... - "Name": "testcase", - "Parameters" : { - "TRAFFIC" : { - 'l2': { - # set destination MAC to the MAC of the first - # interface from WHITELIST_NICS list - 'dstmac' : '#PARAM(NICS[0]["mac"])', - }, - }, - ... - Overriding values defined in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -155,6 +141,17 @@ Example: $ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \ "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput +The ``--test-params`` command line argument can also be used to override default +configuration values for multiple tests. Providing a list of parameters will apply each +element of the list to the test with the same index. If more tests are run than +parameters provided the last element of the list will repeat. + +.. code:: console + + $ ./vsperf --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)'," + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + pvvp_tput pvvp_tput + The second option is to override configuration items by ``Parameters`` section of the test case definition. The configuration items can be added into ``Parameters`` dictionary with their new values. These values will override values defined in @@ -186,6 +183,36 @@ parameter name is passed via ``--test-params`` CLI argument or defined in ``Para section of test case definition. It is also forbidden to redefine a value of ``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section. +**NOTE:** The new definition of the dictionary parameter, specified via ``--test-params`` +or inside ``Parameters`` section, will not override original dictionary values. Instead +the original dictionary will be updated with values from the new dictionary definition. + +Referencing parameter values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is possible to use a special macro ``#PARAM()`` to refer to the value of +another configuration parameter. This reference is evaluated during +access of the parameter value (by ``settings.getValue()`` call), so it +can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. +It can be used to reflect DUT HW details in the testcase definition. + +Example: + +.. code:: python + + { + ... + "Name": "testcase", + "Parameters" : { + "TRAFFIC" : { + 'l2': { + # set destination MAC to the MAC of the first + # interface from WHITELIST_NICS list + 'dstmac' : '#PARAM(NICS[0]["mac"])', + }, + }, + ... + vloop_vnf ^^^^^^^^^ @@ -205,6 +232,12 @@ A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd +Additional Tools Setup +^^^^^^^^^^^^^^^^^^^^^^ + +Follow the `Additional tools instructions <additional-tools-configuration>` to +install and configure additional tools such as collectors and loadgens. + Executing tests ^^^^^^^^^^^^^^^ @@ -234,6 +267,12 @@ To run a single test: Where $TESTNAME is the name of the vsperf test you would like to run. +To run a test multiple times, repeat it: + +.. code-block:: console + + $ ./vsperf $TESTNAME $TESTNAME $TESTNAME + To run a group of tests, for example all tests with a name containing 'RFC2544': @@ -256,6 +295,30 @@ Some tests allow for configurable parameters, including test duration --tests RFC2544Tput \ --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)" +To specify configurable parameters for multiple tests, use a list of +parameters. One element for each test. + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + +If the ``CUMULATIVE_PARAMS`` setting is set to True and there are different parameters +provided for each test using ``--test-params``, each test will take the parameters of +the previous test before appyling it's own. +With ``CUMULATIVE_PARAMS`` set to True the following command will be equivalent to the +previous example: + +.. code:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\ + "'TRAFFICGEN_PKT_SIZES=(64,)']" \ + phy2phy_cont phy2phy_cont + " + For all available options, check out the help dialog: .. code-block:: console @@ -584,7 +647,7 @@ The supported dpdk guest bind drivers are: .. code-block:: console - 'uio_pci_generic' - Use uio_pci_generic driver + 'uio_pci_generic' - Use uio_pci_generic driver 'igb_uio_from_src' - Build and use the igb_uio driver from the dpdk src files 'vfio_no_iommu' - Use vfio with no iommu option. This requires custom @@ -915,6 +978,39 @@ Example of execution of VSPERF in "trafficgen" mode: $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \ --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}" +Performance Matrix +^^^^^^^^^^^^^^^^^^ + +The ``--matrix`` command line argument analyses and displays the performance of +all the tests run. Using the metric specified by ``MATRIX_METRIC`` in the conf-file, +the first test is set as the baseline and all the other tests are compared to it. +The ``MATRIX_METRIC`` must always refer to a numeric value to enable comparision. +A table, with the test ID, metric value, the change of the metric in %, testname +and the test parameters used for each test, is printed out as well as saved into the +results directory. + +Example of 2 tests being compared using Performance Matrix: + +.. code-block:: console + + $ ./vsperf --conf-file user_settings.py \ + --test-params "['TRAFFICGEN_PKT_SIZES=(64,)',"\ + "'TRAFFICGEN_PKT_SIZES=(128,)']" \ + phy2phy_cont phy2phy_cont --matrix + +Example output: + +.. code-block:: console + + +------+--------------+---------------------+----------+---------------------------------------+ + | ID | Name | throughput_rx_fps | Change | Parameters, CUMULATIVE_PARAMS = False | + +======+==============+=====================+==========+=======================================+ + | 0 | phy2phy_cont | 23749000.000 | 0 | 'TRAFFICGEN_PKT_SIZES': [64] | + +------+--------------+---------------------+----------+---------------------------------------+ + | 1 | phy2phy_cont | 16850500.000 | -29.048 | 'TRAFFICGEN_PKT_SIZES': [128] | + +------+--------------+---------------------+----------+---------------------------------------+ + + Code change verification by pylint ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/testing/user/userguide/trafficcapture.rst b/docs/testing/user/userguide/trafficcapture.rst index fa09bfed..8a224dcb 100644 --- a/docs/testing/user/userguide/trafficcapture.rst +++ b/docs/testing/user/userguide/trafficcapture.rst @@ -92,9 +92,9 @@ An example of Traffic Capture in VM test: }, TestSteps: [ # replace original flows with vlan ID modification - ['!vswitch', 'add_flow', 'br0', {'in_port': '1', 'actions': ['mod_vlan_vid:4','output:3']}], - ['!vswitch', 'add_flow', 'br0', {'in_port': '2', 'actions': ['mod_vlan_vid:4','output:4']}], - ['vswitch', 'dump_flows', 'br0'], + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '1', 'actions': ['mod_vlan_vid:4','output:3']}], + ['!vswitch', 'add_flow', '$VSWITCH_BRIDGE_NAME', {'in_port': '2', 'actions': ['mod_vlan_vid:4','output:4']}], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], # verify that received frames have modified vlan ID ['VNF0', 'execute_and_wait', 'tcpdump -i eth0 -c 5 -w dump.pcap vlan 4 &'], ['trafficgen', 'send_traffic',{}], @@ -199,7 +199,7 @@ An example of Traffic Capture for testing NICs with HW offloading test: ['tools', 'exec_shell_background', 'tcpdump -i [2][device] -c 5 -w capture.pcap ' 'ether src [l2][srcmac]'], ['trafficgen', 'send_traffic', {}], - ['vswitch', 'dump_flows', 'br0'], + ['vswitch', 'dump_flows', '$VSWITCH_BRIDGE_NAME'], ['vswitch', 'dump_flows', 'br1'], # there must be 5 captured frames... ['tools', 'exec_shell', 'tcpdump -r capture.pcap | wc -l', '|^(\d+)$'], |