summaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/user/userguide/integration.rst8
-rw-r--r--docs/testing/user/userguide/testlist.rst16
-rw-r--r--docs/testing/user/userguide/teststeps.rst21
-rw-r--r--docs/testing/user/userguide/testusage.rst92
4 files changed, 96 insertions, 41 deletions
diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst
index 66808400..2b2246db 100644
--- a/docs/testing/user/userguide/integration.rst
+++ b/docs/testing/user/userguide/integration.rst
@@ -1,6 +1,6 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. (c) OPNFV, Intel Corporation, AT&T and others.
+.. (c) OPNFV, Intel Corporation, AT&T, Tieto and others.
.. _integration-tests:
@@ -22,6 +22,12 @@ P2P (Physical to Physical scenarios).
NOTE: The configuration for overlay tests provided in this guide is for
unidirectional traffic only.
+NOTE: The overlay tests require an IxNet traffic generator. The tunneled traffic
+is configured by ``ixnetrfc2544v2.tcl`` script. This script can be used
+with all supported deployment scenarios for generation of frames with VXLAN, GRE
+or GENEVE protocols. In that case options "Tunnel Operation" and
+"TRAFFICGEN_IXNET_TCL_SCRIPT" must be properly configured at testcase definition.
+
Executing Integration Tests
---------------------------
diff --git a/docs/testing/user/userguide/testlist.rst b/docs/testing/user/userguide/testlist.rst
index 21c4b736..3d4c28cf 100644
--- a/docs/testing/user/userguide/testlist.rst
+++ b/docs/testing/user/userguide/testlist.rst
@@ -389,6 +389,22 @@ ovsdpdk_qos_pvp In a pvp setup, ensure when a QoS egres
traffic is limited to the specified rate.
======================================== ======================================================================================
+Custom Statistics
++++++++++++++++++
+
+A set of functional testcases for validation of Custom Statistics support by OVS.
+This feature allows Custom Statistics to be accessed by VSPERF.
+
+These testcases require DPDK v17.11, the latest Open vSwitch(v2.9.90)
+and the IxNet traffic-generator.
+
+======================================== ======================================================================================
+ovsdpdk_custstat_check Test if custom statistics are supported.
+ovsdpdk_custstat_rx_error Test bad ethernet CRC counter 'rx_crc_errors' exposed by custom
+ statistics.
+
+======================================== ======================================================================================
+
T-Rex in VM TestCases
^^^^^^^^^^^^^^^^^^^^^
diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst
index 08c95311..e7ccce9a 100644
--- a/docs/testing/user/userguide/teststeps.rst
+++ b/docs/testing/user/userguide/teststeps.rst
@@ -23,6 +23,13 @@ the step number by one which is indicated in the log.
(testcases.integration) - Step 0 'vswitch add_vport ['br0']' start
+Test steps are defined as a list of steps within a ``TestSteps`` item of test
+case definition. Each step is a list with following structure:
+
+.. code-block:: python
+
+ '[' [ optional-alias ',' ] test-object ',' test-function [ ',' optional-function-params ] '],'
+
Step driven tests can be used for both performance and integration testing.
In case of integration test, each step in the test case is validated. If a step
does not pass validation the test will fail and terminate. The test will continue
@@ -57,8 +64,14 @@ Step driven testcases can be used in two different ways:
Test objects and their functions
--------------------------------
-Every test step can call a function of one of the supported test objects. The list
-of supported objects and their most common functions follows:
+Every test step can call a function of one of the supported test objects. In general
+any existing function of supported test object can be called by test step. In case
+that step validation is required (valid for integration test steps, which are not
+suppressed), then appropriate ``validate_`` method must be implemented.
+
+The list of supported objects and their most common functions is listed below. Please
+check implementation of test objects for full list of implemented functions and their
+parameters.
* ``vswitch`` - provides functions for vSwitch configuration
@@ -176,6 +189,8 @@ of supported objects and their most common functions follows:
* ``getValue param`` - returns value of given ``param``
* ``setValue param value`` - sets value of ``param`` to given ``value``
+ * ``resetValue param`` - if ``param`` was overridden by ``TEST_PARAMS`` (e.g. by "Parameters"
+ section of the test case definition), then it will be set to its original value.
Examples:
@@ -185,6 +200,8 @@ of supported objects and their most common functions follows:
['settings', 'setValue', 'GUEST_USERNAME', ['root']]
+ ['settings', 'resetValue', 'WHITELIST_NICS'],
+
It is possible and more convenient to access any VSPERF configuration option directly
via ``$NAME`` notation. Option evaluation is done during runtime and vsperf will
automatically translate it to the appropriate call of ``settings.getValue``.
diff --git a/docs/testing/user/userguide/testusage.rst b/docs/testing/user/userguide/testusage.rst
index 9b331869..c7cc1484 100644
--- a/docs/testing/user/userguide/testusage.rst
+++ b/docs/testing/user/userguide/testusage.rst
@@ -91,55 +91,41 @@ Using a custom settings file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory
-of if you want to use an alternative configuration file, the file can
+or if you want to use an alternative configuration file, the file can
be passed to ``vsperf`` via the ``--conf-file`` argument.
.. code-block:: console
$ ./vsperf --conf-file <path_to_custom_conf> ...
-Note that configuration passed in via the environment (``--load-env``)
-or via another command line argument will override both the default and
-your custom configuration files. This "priority hierarchy" can be
-described like so (1 = max priority):
-
-1. Testcase definition section ``Parameters``
-2. Command line arguments
-3. Environment variables
-4. Configuration file(s)
-
-Further details about configuration files evaluation and special behaviour
+Evaluation of configuration parameters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The value of configuration parameter can be specified at various places,
+e.g. at the test case definition, inside configuration files, by the command
+line argument, etc. Thus it is important to understand the order of configuration
+parameter evaluation. This "priority hierarchy" can be described like so
+(1 = max priority):
+
+1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type``
+2. Parameters inside testcase definition section ``Parameters``
+3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.)
+4. Environment variables (see ``--load-env`` argument)
+5. Custom configuration file specified via ``--conf-file`` argument
+6. Standard configuration files, where higher prefix number means higher
+ priority.
+
+For example, if the same configuration parameter is defined in custom configuration
+file (specified via ``--conf-file`` argument), via ``--test-params`` argument
+and also inside ``Parameters`` section of the testcase definition, then parameter
+value from the ``Parameters`` section will be used.
+
+Further details about order of configuration files evaluation and special behaviour
of options with ``GUEST_`` prefix could be found at :ref:`design document
<design-configuration>`.
.. _overriding-parameters-documentation:
-Referencing parameter values
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-It is possible to use a special macro ``#PARAM()`` to refer to the value of
-another configuration parameter. This reference is evaluated during
-access of the parameter value (by ``settings.getValue()`` call), so it
-can refer to parameters created during VSPERF runtime, e.g. NICS dictionary.
-It can be used to reflect DUT HW details in the testcase definition.
-
-Example:
-
-.. code:: python
-
- {
- ...
- "Name": "testcase",
- "Parameters" : {
- "TRAFFIC" : {
- 'l2': {
- # set destination MAC to the MAC of the first
- # interface from WHITELIST_NICS list
- 'dstmac' : '#PARAM(NICS[0]["mac"])',
- },
- },
- ...
-
Overriding values defined in configuration files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -197,6 +183,36 @@ parameter name is passed via ``--test-params`` CLI argument or defined in ``Para
section of test case definition. It is also forbidden to redefine a value of
``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section.
+**NOTE:** The new definition of the dictionary parameter, specified via ``--test-params``
+or inside ``Parameters`` section, will not override original dictionary values. Instead
+the original dictionary will be updated with values from the new dictionary definition.
+
+Referencing parameter values
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is possible to use a special macro ``#PARAM()`` to refer to the value of
+another configuration parameter. This reference is evaluated during
+access of the parameter value (by ``settings.getValue()`` call), so it
+can refer to parameters created during VSPERF runtime, e.g. NICS dictionary.
+It can be used to reflect DUT HW details in the testcase definition.
+
+Example:
+
+.. code:: python
+
+ {
+ ...
+ "Name": "testcase",
+ "Parameters" : {
+ "TRAFFIC" : {
+ 'l2': {
+ # set destination MAC to the MAC of the first
+ # interface from WHITELIST_NICS list
+ 'dstmac' : '#PARAM(NICS[0]["mac"])',
+ },
+ },
+ ...
+
vloop_vnf
^^^^^^^^^