teststeps: Improvements and bugfixing of teststeps

This patch introduces several improvements and small bugfixes
of teststeps. These changes were identified during implementation
of OVS/DPDK regression tests.

Patch content:
* teststeps: step aliases were implemented
* teststeps: improved filtering by regex for any step, which returns string
  or list of stings; filter will process all lines
* teststeps: support for log object
* teststeps: support for trafficgen get_results call
* teststeps: configurable suppression of step validation
* trafficgen: remove old results before traffic is executed
* trafficgen: support for flow control on/off (IxNet)
* trafficgen: support for configurable learning frames (IxNet)
* trafficgen: support for runtime changes of TRAFFICGEN_PKT_SIZES, _DURATION
  and _LOSSRATE
* vnf: flush pexpect output of previous commands
* vnf: use execute_and_wait() to ensure correct cmds order
* vnf: dpdk vHost User interface name set according to its type,
  e.g. dpdkvhostuserclient
* vswitch: support for OVS restart
* decap: simplify configuration of tunneling decapsulation tests
* settings: values of all configuration options are restored after TC execution
* modified formatting of test description used by --list
* testcase name and description is logged before its execution
* small bugfixes

JIRA: VSPERF-539

Change-Id: I550ba0d897ece89abd3f33d6d66f545c4d863e7b
Signed-off-by: Martin Klozik <martinx.klozik@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>
Reviewed-by: Christian Trautman <ctrautma@redhat.com>
Reviewed-by: Sridhar Rao <sridhar.rao@spirent.com>
Reviewed-by: Trevor Cooper <trevor.cooper@intel.com>

2 files changed, 90 insertions, 37 deletions

diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst
index 249a03c4..66808400 100644
--- a/docs/testing/user/userguide/integration.rst
+++ b/docs/testing/user/userguide/integration.rst
@@ -140,13 +140,7 @@ To run VXLAN decapsulation tests:
 
 1. Set the variables used in "Executing Tunnel encapsulation tests"
 
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
-  .. code-block:: python
-
-    DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
 
   .. code-block:: console
 
@@ -181,13 +175,7 @@ To run GRE decapsulation tests:
 
 1. Set the variables used in "Executing Tunnel encapsulation tests"
 
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
-  .. code-block:: python
-
-    DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
 
   .. code-block:: console
 
@@ -238,13 +226,7 @@ To run GENEVE decapsulation tests:
 
 1. Set the variables used in "Executing Tunnel encapsulation tests"
 
-2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT
-
-  .. code-block:: python
-
-    DUT_NIC2_MAC = '<DUT NIC2 MAC>'
-
-3. Run test:
+2. Run test:
 
   .. code-block:: console
 
@@ -289,8 +271,6 @@ To run VXLAN decapsulation tests:
         'datapath/linux/openvswitch.ko',
     ]
 
-    DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
     TRAFFICGEN_PORT1_IP = '172.16.1.2'
     TRAFFICGEN_PORT2_IP = '192.168.1.11'
 
@@ -302,7 +282,8 @@ To run VXLAN decapsulation tests:
 
     VXLAN_FRAME_L2 = {'srcmac':
                       '01:02:03:04:05:06',
-                      'dstmac': DUT_NIC1_MAC
+                      'dstmac':
+                      '06:05:04:03:02:01',
                      }
 
     VXLAN_FRAME_L3 = {'proto': 'udp',
@@ -349,8 +330,6 @@ To run GRE decapsulation tests:
         'datapath/linux/openvswitch.ko',
     ]
 
-    DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
     TRAFFICGEN_PORT1_IP = '172.16.1.2'
     TRAFFICGEN_PORT2_IP = '192.168.1.11'
 
@@ -362,7 +341,8 @@ To run GRE decapsulation tests:
 
     GRE_FRAME_L2 = {'srcmac':
                     '01:02:03:04:05:06',
-                    'dstmac': DUT_NIC1_MAC
+                    'dstmac':
+                    '06:05:04:03:02:01',
                    }
 
     GRE_FRAME_L3 = {'proto': 'udp',
@@ -408,8 +388,6 @@ To run GENEVE decapsulation tests:
         'datapath/linux/openvswitch.ko',
     ]
 
-    DUT_NIC1_MAC = '<DUT NIC1 MAC ADDRESS>'
-
     TRAFFICGEN_PORT1_IP = '172.16.1.2'
     TRAFFICGEN_PORT2_IP = '192.168.1.11'
 
@@ -421,7 +399,8 @@ To run GENEVE decapsulation tests:
 
     GENEVE_FRAME_L2 = {'srcmac':
                        '01:02:03:04:05:06',
-                       'dstmac': DUT_NIC1_MAC
+                       'dstmac':
+                       '06:05:04:03:02:01',
                       }
 
     GENEVE_FRAME_L3 = {'proto': 'udp',
diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst
index 5349d2ea..8be67310 100644
--- a/docs/testing/user/userguide/teststeps.rst
+++ b/docs/testing/user/userguide/teststeps.rst
@@ -29,6 +29,14 @@ does not pass validation the test will fail and terminate. The test will continu
 until a failure is detected or all steps pass. A csv report file is generated after
 a test completes with an OK or FAIL result.
 
+**NOTE**: It is possible to suppress validation process of given step by prefixing
+it by ``!`` (exclamation mark).
+In following example test execution won't fail if all traffic is dropped:
+
+.. code-block:: python
+
+    ['!trafficgen', 'send_traffic', {}]
+
 In case of performance test, the validation of steps is not performed and
 standard output files with results from traffic generator and underlying OS
 details are generated by vsperf.
@@ -75,6 +83,7 @@ of supported objects and their most common functions follows:
         * ``disable_stp br_name`` - disables Spanning Tree Protocol for bridge ``br_name``
         * ``enable_rstp br_name`` - enables Rapid Spanning Tree Protocol for bridge ``br_name``
         * ``disable_rstp br_name`` - disables Rapid Spanning Tree Protocol for bridge ``br_name``
+        * ``restart`` - restarts switch, which is useful for failover testcases
 
         Examples:
 
@@ -101,15 +110,26 @@ of supported objects and their most common functions follows:
 
         * ``start`` - starts a VNF based on VSPERF configuration
         * ``stop`` - gracefully terminates given VNF
+        * ``execute command [delay]`` - executes command `cmd` inside VNF; Optional
+          delay defines number of seconds to wait before next step is executed. Method
+          returns command output as a string.
+        * ``execute_and_wait command [timeout] [prompt]`` - executes command `cmd` inside
+          VNF; Optional timeout defines number of seconds to wait until ``prompt`` is detected.
+          Optional ``prompt`` defines a string, which is used as detection of successful command
+          execution. In case that prompt is not defined, then content of ``GUEST_PROMPT_LOGIN``
+          parameter will be used. Method returns command output as a string.
 
         Examples:
 
         .. code-block:: python
 
-            ['vnf1', 'start']
-            ['vnf2', 'start']
-            ['vnf2', 'stop']
-            ['vnf1', 'stop']
+            ['vnf1', 'start'],
+            ['vnf2', 'start'],
+            ['vnf1', 'execute_and_wait', 'ifconfig eth0 5.5.5.1/24 up'],
+            ['vnf2', 'execute_and_wait', 'ifconfig eth0 5.5.5.2/24 up', 120, 'root.*#'],
+            ['vnf2', 'execute_and_wait', 'ping -c1 5.5.5.1'],
+            ['vnf2', 'stop'],
+            ['vnf1', 'stop'],
 
     * ``trafficgen`` - triggers traffic generation
 
@@ -119,6 +139,8 @@ of supported objects and their most common functions follows:
           and given ``traffic`` dictionary. More details about ``traffic`` dictionary
           and its possible values are available at :ref:`Traffic Generator Integration Guide
           <step-5-supported-traffic-types>`
+        * ``get_results`` - returns dictionary with results collected from previous execution
+          of ``send_traffic``
 
         Examples:
 
@@ -126,7 +148,12 @@ of supported objects and their most common functions follows:
 
             ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}]
 
-            ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}]
+            ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}],
+            ['trafficgen', 'get_results'],
+            ['tools', 'assert', '#STEP[-1][0]["frame_loss_percent"] < 0.05'],
+
+
+.. _step-driven-tests-variable-usage:
 
     * ``settings`` - reads or modifies VSPERF configuration
 
@@ -229,8 +256,7 @@ of supported objects and their most common functions follows:
           in case that condition is not ``True``
         * ``Eval expression`` - evaluates given expression as a python code and returns
           its result
-        * ``Exec_Shell command [regex]`` - executes a shell command and filters its output by
-          (optional) regular expression
+        * ``Exec_Shell command`` - executes a shell command
         * ``Exec_Python code`` - executes a python code
 
 
@@ -260,6 +286,23 @@ of supported objects and their most common functions follows:
 
         ['sleep', '60']
 
+    * ``log level message`` - is used to log ``message`` of given ``level`` into vsperf output.
+      Level is one of info, debug, warning or error.
+
+      Examples:
+
+      .. code-block:: python
+
+        ['log', 'error', 'tools $TOOLS']
+
+    * ``pdb`` - executes python debugger
+
+      Examples:
+
+      .. code-block:: python
+
+        ['pdb']
+
 Test Macros
 -----------
 
@@ -296,6 +339,15 @@ functionality:
 
                 ['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'],  # STEP 2
 
+Another option to refer to previous values, is to define an alias for given step
+by its first argument with '#' prefix. Alias must be unique and it can't be a number.
+Example of step alias usage:
+
+.. code-block:: python
+
+                ['#port1', 'vswitch', 'add_vport', 'int_br0'],
+                ['vswitch', 'del_port', 'int_br0', '#STEP[port1][0]'],
+
 Also commonly used steps can be created as a separate profile.
 
 .. code-block:: python
@@ -324,6 +376,28 @@ This profile can then be used inside other testcases
                      STEP_VSWITCH_PVP_FINIT
     }
 
+It is possible to refer to vsperf configuration parameters within step macros. Please
+see :ref:`step-driven-tests-variable-usage` for more details.
+
+In case that step returns a string or list of strings, then it is possible to
+filter such output by regular expression. This optional filter can be specified
+as a last step parameter with prefix '|'. Output will be split into separate lines
+and only matching records will be returned. It is also possible to return a specified
+group of characters from the matching lines, e.g. by regex ``|ID (\d+)``.
+
+Examples:
+
+.. code-block:: python
+
+   ['tools', 'exec_shell', "sudo $TOOLS['ovs-appctl'] dpif-netdev/pmd-rxq-show",
+    '|dpdkvhostuser0\s+queue-id: \d'],
+   ['tools', 'assert', 'len(#STEP[-1])==1'],
+
+   ['vnf', 'execute_and_wait', 'ethtool -L eth0 combined 2'],
+   ['vnf', 'execute_and_wait', 'ethtool -l eth0', '|Combined:\s+2'],
+   ['tools', 'assert', 'len(#STEP[-1])==2']
+
+
 HelloWorld and other basic Testcases
 ------------------------------------