aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide
diff options
context:
space:
mode:
authorMartin Klozik <martinx.klozik@intel.com>2017-08-31 15:01:18 +0200
committerMartin Klozik <martinx.klozik@intel.com>2017-11-03 08:36:29 +0000
commitb1534957e463b5e34957a8d48ce5c6b0552ffbb4 (patch)
tree10985b181d62cffb4ea36355de66dc8ea4edbf8a /docs/testing/user/userguide
parent87f6e48ca1b17361955f0d31551b0c6360028688 (diff)
teststeps: Improvements and bugfixing of teststeps
This patch introduces several improvements and small bugfixes of teststeps. These changes were identified during implementation of OVS/DPDK regression tests. Patch content: * teststeps: step aliases were implemented * teststeps: improved filtering by regex for any step, which returns string or list of stings; filter will process all lines * teststeps: support for log object * teststeps: support for trafficgen get_results call * teststeps: configurable suppression of step validation * trafficgen: remove old results before traffic is executed * trafficgen: support for flow control on/off (IxNet) * trafficgen: support for configurable learning frames (IxNet) * trafficgen: support for runtime changes of TRAFFICGEN_PKT_SIZES, _DURATION and _LOSSRATE * vnf: flush pexpect output of previous commands * vnf: use execute_and_wait() to ensure correct cmds order * vnf: dpdk vHost User interface name set according to its type, e.g. dpdkvhostuserclient * vswitch: support for OVS restart * decap: simplify configuration of tunneling decapsulation tests * settings: values of all configuration options are restored after TC execution * modified formatting of test description used by --list * testcase name and description is logged before its execution * small bugfixes JIRA: VSPERF-539 Change-Id: I550ba0d897ece89abd3f33d6d66f545c4d863e7b Signed-off-by: Martin Klozik <martinx.klozik@intel.com> Reviewed-by: Al Morton <acmorton@att.com> Reviewed-by: Christian Trautman <ctrautma@redhat.com> Reviewed-by: Sridhar Rao <sridhar.rao@spirent.com> Reviewed-by: Trevor Cooper <trevor.cooper@intel.com>
Diffstat (limited to 'docs/testing/user/userguide')
-rw-r--r--docs/testing/user/userguide/integration.rst39
-rw-r--r--docs/testing/user/userguide/teststeps.rst88
2 files changed, 90 insertions, 37 deletions
diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst
index 249a03c4..66808400 100644
--- a/docs/testing/user/userguide/integration.rst
+++ b/docs/testing/user/userguide/integration.rst
@@ -140,13 +140,7 @@ To run VXLAN decapsulation tests:
1. Set the variables used in "Executing Tunnel encapsulation tests"
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
- .. code-block:: python
-
- DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
.. code-block:: console
@@ -181,13 +175,7 @@ To run GRE decapsulation tests:
1. Set the variables used in "Executing Tunnel encapsulation tests"
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
- .. code-block:: python
-
- DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
.. code-block:: console
@@ -238,13 +226,7 @@ To run GENEVE decapsulation tests:
1. Set the variables used in "Executing Tunnel encapsulation tests"
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
- .. code-block:: python
-
- DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
.. code-block:: console
@@ -289,8 +271,6 @@ To run VXLAN decapsulation tests:
'datapath/linux/openvswitch.ko',
]
- DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
@@ -302,7 +282,8 @@ To run VXLAN decapsulation tests:
VXLAN_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
- 'dstmac': DUT_NIC1_MAC
+ 'dstmac':
+ '06:05:04:03:02:01',
}
VXLAN_FRAME_L3 = {'proto': 'udp',
@@ -349,8 +330,6 @@ To run GRE decapsulation tests:
'datapath/linux/openvswitch.ko',
]
- DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
@@ -362,7 +341,8 @@ To run GRE decapsulation tests:
GRE_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
- 'dstmac': DUT_NIC1_MAC
+ 'dstmac':
+ '06:05:04:03:02:01',
}
GRE_FRAME_L3 = {'proto': 'udp',
@@ -408,8 +388,6 @@ To run GENEVE decapsulation tests:
'datapath/linux/openvswitch.ko',
]
- DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
TRAFFICGEN_PORT1_IP = '172.16.1.2'
TRAFFICGEN_PORT2_IP = '192.168.1.11'
@@ -421,7 +399,8 @@ To run GENEVE decapsulation tests:
GENEVE_FRAME_L2 = {'srcmac':
'01:02:03:04:05:06',
- 'dstmac': DUT_NIC1_MAC
+ 'dstmac':
+ '06:05:04:03:02:01',
}
GENEVE_FRAME_L3 = {'proto': 'udp',
diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst
index 5349d2ea..8be67310 100644
--- a/docs/testing/user/userguide/teststeps.rst
+++ b/docs/testing/user/userguide/teststeps.rst
@@ -29,6 +29,14 @@ does not pass validation the test will fail and terminate. The test will continu
until a failure is detected or all steps pass. A csv report file is generated after
a test completes with an OK or FAIL result.
+**NOTE**: It is possible to suppress validation process of given step by prefixing
+it by ``!`` (exclamation mark).
+In following example test execution won't fail if all traffic is dropped:
+
+.. code-block:: python
+
+ ['!trafficgen', 'send_traffic', {}]
+
In case of performance test, the validation of steps is not performed and
standard output files with results from traffic generator and underlying OS
details are generated by vsperf.
@@ -75,6 +83,7 @@ of supported objects and their most common functions follows:
* ``disable_stp br_name`` - disables Spanning Tree Protocol for bridge ``br_name``
* ``enable_rstp br_name`` - enables Rapid Spanning Tree Protocol for bridge ``br_name``
* ``disable_rstp br_name`` - disables Rapid Spanning Tree Protocol for bridge ``br_name``
+ * ``restart`` - restarts switch, which is useful for failover testcases
Examples:
@@ -101,15 +110,26 @@ of supported objects and their most common functions follows:
* ``start`` - starts a VNF based on VSPERF configuration
* ``stop`` - gracefully terminates given VNF
+ * ``execute command [delay]`` - executes command `cmd` inside VNF; Optional
+ delay defines number of seconds to wait before next step is executed. Method
+ returns command output as a string.
+ * ``execute_and_wait command [timeout] [prompt]`` - executes command `cmd` inside
+ VNF; Optional timeout defines number of seconds to wait until ``prompt`` is detected.
+ Optional ``prompt`` defines a string, which is used as detection of successful command
+ execution. In case that prompt is not defined, then content of ``GUEST_PROMPT_LOGIN``
+ parameter will be used. Method returns command output as a string.
Examples:
.. code-block:: python
- ['vnf1', 'start']
- ['vnf2', 'start']
- ['vnf2', 'stop']
- ['vnf1', 'stop']
+ ['vnf1', 'start'],
+ ['vnf2', 'start'],
+ ['vnf1', 'execute_and_wait', 'ifconfig eth0 5.5.5.1/24 up'],
+ ['vnf2', 'execute_and_wait', 'ifconfig eth0 5.5.5.2/24 up', 120, 'root.*#'],
+ ['vnf2', 'execute_and_wait', 'ping -c1 5.5.5.1'],
+ ['vnf2', 'stop'],
+ ['vnf1', 'stop'],
* ``trafficgen`` - triggers traffic generation
@@ -119,6 +139,8 @@ of supported objects and their most common functions follows:
and given ``traffic`` dictionary. More details about ``traffic`` dictionary
and its possible values are available at :ref:`Traffic Generator Integration Guide
<step-5-supported-traffic-types>`
+ * ``get_results`` - returns dictionary with results collected from previous execution
+ of ``send_traffic``
Examples:
@@ -126,7 +148,12 @@ of supported objects and their most common functions follows:
['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}]
- ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}]
+ ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}],
+ ['trafficgen', 'get_results'],
+ ['tools', 'assert', '#STEP[-1][0]["frame_loss_percent"] < 0.05'],
+
+
+.. _step-driven-tests-variable-usage:
* ``settings`` - reads or modifies VSPERF configuration
@@ -229,8 +256,7 @@ of supported objects and their most common functions follows:
in case that condition is not ``True``
* ``Eval expression`` - evaluates given expression as a python code and returns
its result
- * ``Exec_Shell command [regex]`` - executes a shell command and filters its output by
- (optional) regular expression
+ * ``Exec_Shell command`` - executes a shell command
* ``Exec_Python code`` - executes a python code
@@ -260,6 +286,23 @@ of supported objects and their most common functions follows:
['sleep', '60']
+ * ``log level message`` - is used to log ``message`` of given ``level`` into vsperf output.
+ Level is one of info, debug, warning or error.
+
+ Examples:
+
+ .. code-block:: python
+
+ ['log', 'error', 'tools $TOOLS']
+
+ * ``pdb`` - executes python debugger
+
+ Examples:
+
+ .. code-block:: python
+
+ ['pdb']
+
Test Macros
-----------
@@ -296,6 +339,15 @@ functionality:
['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'], # STEP 2
+Another option to refer to previous values, is to define an alias for given step
+by its first argument with '#' prefix. Alias must be unique and it can't be a number.
+Example of step alias usage:
+
+.. code-block:: python
+
+ ['#port1', 'vswitch', 'add_vport', 'int_br0'],
+ ['vswitch', 'del_port', 'int_br0', '#STEP[port1][0]'],
+
Also commonly used steps can be created as a separate profile.
.. code-block:: python
@@ -324,6 +376,28 @@ This profile can then be used inside other testcases
STEP_VSWITCH_PVP_FINIT
}
+It is possible to refer to vsperf configuration parameters within step macros. Please
+see :ref:`step-driven-tests-variable-usage` for more details.
+
+In case that step returns a string or list of strings, then it is possible to
+filter such output by regular expression. This optional filter can be specified
+as a last step parameter with prefix '|'. Output will be split into separate lines
+and only matching records will be returned. It is also possible to return a specified
+group of characters from the matching lines, e.g. by regex ``|ID (\d+)``.
+
+Examples:
+
+.. code-block:: python
+
+ ['tools', 'exec_shell', "sudo $TOOLS['ovs-appctl'] dpif-netdev/pmd-rxq-show",
+ '|dpdkvhostuser0\s+queue-id: \d'],
+ ['tools', 'assert', 'len(#STEP[-1])==1'],
+
+ ['vnf', 'execute_and_wait', 'ethtool -L eth0 combined 2'],
+ ['vnf', 'execute_and_wait', 'ethtool -l eth0', '|Combined:\s+2'],
+ ['tools', 'assert', 'len(#STEP[-1])==2']
+
+
HelloWorld and other basic Testcases
------------------------------------