diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/user/userguide/integration.rst | 8 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testlist.rst | 16 | ||||
| -rw-r--r-- | docs/testing/user/userguide/teststeps.rst | 21 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testusage.rst | 92 |
4 files changed, 96 insertions, 41 deletions
diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst index 66808400..2b2246db 100644 --- a/docs/testing/user/userguide/integration.rst +++ b/docs/testing/user/userguide/integration.rst @@ -1,6 +1,6 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Intel Corporation, AT&T and others. +.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others. .. _integration-tests: @@ -22,6 +22,12 @@ P2P (Physical to Physical scenarios). NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only. +NOTE: The overlay tests require an IxNet traffic generator. The tunneled traffic +is configured by ``ixnetrfc2544v2.tcl`` script. This script can be used +with all supported deployment scenarios for generation of frames with VXLAN, GRE +or GENEVE protocols. In that case options "Tunnel Operation" and +"TRAFFICGEN_IXNET_TCL_SCRIPT" must be properly configured at testcase definition. + Executing Integration Tests --------------------------- diff --git a/docs/testing/user/userguide/testlist.rst b/docs/testing/user/userguide/testlist.rst index 21c4b736..3d4c28cf 100644 --- a/docs/testing/user/userguide/testlist.rst +++ b/docs/testing/user/userguide/testlist.rst @@ -389,6 +389,22 @@ ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egres traffic is limited to the specified rate. ======================================== ====================================================================================== +Custom Statistics ++++++++++++++++++ + +A set of functional testcases for validation of Custom Statistics support by OVS. +This feature allows Custom Statistics to be accessed by VSPERF. + +These testcases require DPDK v17.11, the latest Open vSwitch(v2.9.90) +and the IxNet traffic-generator. + +======================================== ====================================================================================== +ovsdpdk_custstat_check Test if custom statistics are supported. +ovsdpdk_custstat_rx_error Test bad ethernet CRC counter 'rx_crc_errors' exposed by custom + statistics. + +======================================== ====================================================================================== + T-Rex in VM TestCases ^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst index 08c95311..e7ccce9a 100644 --- a/docs/testing/user/userguide/teststeps.rst +++ b/docs/testing/user/userguide/teststeps.rst @@ -23,6 +23,13 @@ the step number by one which is indicated in the log. (testcases.integration) - Step 0 'vswitch add_vport ['br0']' start +Test steps are defined as a list of steps within a ``TestSteps`` item of test +case definition. Each step is a list with following structure: + +.. code-block:: python + + '[' [ optional-alias ',' ] test-object ',' test-function [ ',' optional-function-params ] '],' + Step driven tests can be used for both performance and integration testing. In case of integration test, each step in the test case is validated. If a step does not pass validation the test will fail and terminate. The test will continue @@ -57,8 +64,14 @@ Step driven testcases can be used in two different ways: Test objects and their functions -------------------------------- -Every test step can call a function of one of the supported test objects. The list -of supported objects and their most common functions follows: +Every test step can call a function of one of the supported test objects. In general +any existing function of supported test object can be called by test step. In case +that step validation is required (valid for integration test steps, which are not +suppressed), then appropriate ``validate_`` method must be implemented. + +The list of supported objects and their most common functions is listed below. Please +check implementation of test objects for full list of implemented functions and their +parameters. * ``vswitch`` - provides functions for vSwitch configuration @@ -176,6 +189,8 @@ of supported objects and their most common functions follows: * ``getValue param`` - returns value of given ``param`` * ``setValue param value`` - sets value of ``param`` to given ``value`` + * ``resetValue param`` - if ``param`` was overridden by ``TEST_PARAMS`` (e.g. by "Parameters" + section of the test case definition), then it will be set to its original value. Examples: @@ -185,6 +200,8 @@ of supported objects and their most common functions follows: ['settings', 'setValue', 'GUEST_USERNAME', ['root']] + ['settings', 'resetValue', 'WHITELIST_NICS'], + It is possible and more convenient to access any VSPERF configuration option directly via ``$NAME`` notation. Option evaluation is done during runtime and vsperf will automatically translate it to the appropriate call of ``settings.getValue``. diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst index 9b331869..c7cc1484 100644 --- a/docs/testing/user/userguide/testusage.rst +++ b/docs/testing/user/userguide/testusage.rst @@ -91,55 +91,41 @@ Using a custom settings file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory -of if you want to use an alternative configuration file, the file can +or if you want to use an alternative configuration file, the file can be passed to ``vsperf`` via the ``--conf-file`` argument. .. code-block:: console $ ./vsperf --conf-file <path_to_custom_conf> ... -Note that configuration passed in via the environment (``--load-env``) -or via another command line argument will override both the default and -your custom configuration files. This "priority hierarchy" can be -described like so (1 = max priority): - -1. Testcase definition section ``Parameters`` -2. Command line arguments -3. Environment variables -4. Configuration file(s) - -Further details about configuration files evaluation and special behaviour +Evaluation of configuration parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The value of configuration parameter can be specified at various places, +e.g. at the test case definition, inside configuration files, by the command +line argument, etc. Thus it is important to understand the order of configuration +parameter evaluation. This "priority hierarchy" can be described like so +(1 = max priority): + +1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type`` +2. Parameters inside testcase definition section ``Parameters`` +3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.) +4. Environment variables (see ``--load-env`` argument) +5. Custom configuration file specified via ``--conf-file`` argument +6. Standard configuration files, where higher prefix number means higher + priority. + +For example, if the same configuration parameter is defined in custom configuration +file (specified via ``--conf-file`` argument), via ``--test-params`` argument +and also inside ``Parameters`` section of the testcase definition, then parameter +value from the ``Parameters`` section will be used. + +Further details about order of configuration files evaluation and special behaviour of options with ``GUEST_`` prefix could be found at :ref:`design document <design-configuration>`. .. _overriding-parameters-documentation: -Referencing parameter values -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is possible to use a special macro ``#PARAM()`` to refer to the value of -another configuration parameter. This reference is evaluated during -access of the parameter value (by ``settings.getValue()`` call), so it -can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. -It can be used to reflect DUT HW details in the testcase definition. - -Example: - -.. code:: python - - { - ... - "Name": "testcase", - "Parameters" : { - "TRAFFIC" : { - 'l2': { - # set destination MAC to the MAC of the first - # interface from WHITELIST_NICS list - 'dstmac' : '#PARAM(NICS[0]["mac"])', - }, - }, - ... - Overriding values defined in configuration files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -197,6 +183,36 @@ parameter name is passed via ``--test-params`` CLI argument or defined in ``Para section of test case definition. It is also forbidden to redefine a value of ``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section. +**NOTE:** The new definition of the dictionary parameter, specified via ``--test-params`` +or inside ``Parameters`` section, will not override original dictionary values. Instead +the original dictionary will be updated with values from the new dictionary definition. + +Referencing parameter values +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is possible to use a special macro ``#PARAM()`` to refer to the value of +another configuration parameter. This reference is evaluated during +access of the parameter value (by ``settings.getValue()`` call), so it +can refer to parameters created during VSPERF runtime, e.g. NICS dictionary. +It can be used to reflect DUT HW details in the testcase definition. + +Example: + +.. code:: python + + { + ... + "Name": "testcase", + "Parameters" : { + "TRAFFIC" : { + 'l2': { + # set destination MAC to the MAC of the first + # interface from WHITELIST_NICS list + 'dstmac' : '#PARAM(NICS[0]["mac"])', + }, + }, + ... + vloop_vnf ^^^^^^^^^ |