tests: Improvement of step driven testcases

A set of improvements was introduced to enhance step driven
testcases capabilities.

Details:
* delay among test steps is configurable by TEST_STEP_DELAY
  parameter
* step driven tool function exec was renamed to exec_shell
* new step driven tool function exec_python was introduced to execute
  a python code
* new step driven object sleep was introduced to pause test execution
  for defined number of seconds.
* fixed bug in settings.validate_getValue() to correctly validate
  access of parameters modified by TEST_PARAMS
* new #PARAM() macro was introduced to allow references among
  configuration parameters
* multistream support has been added into ixnetrfc2544v2.tcl,
  which is used for tunneling protocols test (op2p deployment)
* fixed bug in op2p deployment to list interfaces and flows
  from both bridges involved in the test
* test report updated to state exact rfcxxxx type of traffic
  type, e.g. rfc2544_continuous
* test report of step driven testcases was updated to contain
  measured values from traffic generator in CSV report
* method for ovs flow comparison was modified to normalize
  IPv4 CIDR network addr (e.g. 10.0.0.5/8 => 10.0.0.0/8)

JIRA: VSPERF-512

Change-Id: Ib4f38dcdfbf3820dd766b25520da0ad0c81f3293
Signed-off-by: Martin Klozik <martinx.klozik@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>
Reviewed-by: Christian Trautman <ctrautma@redhat.com>
Reviewed-by: Sridhar Rao <sridhar.rao@spirent.com>
Reviewed-by: Trevor Cooper <trevor.cooper@intel.com>
Reviewed-by: Ciara Loftus <ciara.loftus@intel.com>

4 files changed, 62 insertions, 23 deletions

diff --git a/conf/00_common.conf b/conf/00_common.conf
index 7f30deb2..4c25b0b8 100644
--- a/conf/00_common.conf
+++ b/conf/00_common.conf
@@ -1,4 +1,4 @@
-# Copyright 2015 Intel Corporation.
+# Copyright 2015-2017 Intel Corporation.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -116,6 +116,9 @@ VERBOSITY = 'debug'
 # conventions
 TEST_PARAMS = {}
 
+# delay enforced after every step to allow system to process changes
+TEST_STEP_DELAY = 5
+
 # ############################
 # Modules
 # ############################
diff --git a/conf/01_testcases.conf b/conf/01_testcases.conf
index 2d5ab93e..df582df9 100755
--- a/conf/01_testcases.conf
+++ b/conf/01_testcases.conf
@@ -75,6 +75,16 @@
 #                                  # override any values defined by TEST_PARAMS option
 #                                  # stated in configuration files or values specified
 #                                  # on command line through --test-params parameter.
+#
+# "TestSteps": []                  # Definition of detailed test steps.
+#                                  # In case that this list is defined, then
+#                                  # vsperf will execute defined test steps
+#                                  # one by one. It can be used to configure
+#                                  # vswitch, insert flows and transmit traffic.
+#                                  # It is possible to refer to result of any
+#                                  # previous step through #STEP[i][j] macro.
+#                                  # Where i is a number of step (starts from 0)
+#                                  # and j is index of result returned by step i.
 # "Test Modifier": [FrameMod|Other],
 # "Dependency": [Test_Case_Name |None],
 
diff --git a/conf/__init__.py b/conf/__init__.py
index 2a2586ff..e24111dc 100644
--- a/conf/__init__.py
+++ b/conf/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2015-2016 Intel Corporation.
+# Copyright 2015-2017 Intel Corporation.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -49,6 +49,40 @@ class Settings(object):
     def __init__(self):
         pass
 
+    def _eval_param(self, param):
+        # pylint: disable=invalid-name
+        """ Helper function for expansion of references to vsperf parameters
+        """
+        if isinstance(param, str):
+            # evaluate every #PARAM reference inside parameter itself
+            macros = re.findall(r'#PARAM\((([\w\-]+)(\[[\w\[\]\-\'\"]+\])*)\)', param)
+            if macros:
+                for macro in macros:
+                    # pylint: disable=eval-used
+                    try:
+                        tmp_val = str(eval("self.getValue('{}'){}".format(macro[1], macro[2])))
+                        param = param.replace('#PARAM({})'.format(macro[0]), tmp_val)
+                    # silently ignore that option required by PARAM macro can't be evaluated;
+                    # It is possible, that referred parameter will be constructed during runtime
+                    # and re-read later.
+                    except IndexError:
+                        pass
+                    except AttributeError:
+                        pass
+            return param
+        elif isinstance(param, list) or isinstance(param, tuple):
+            tmp_list = []
+            for item in param:
+                tmp_list.append(self._eval_param(item))
+            return tmp_list
+        elif isinstance(param, dict):
+            tmp_dict = {}
+            for (key, value) in param.items():
+                tmp_dict[key] = self._eval_param(value)
+            return tmp_dict
+        else:
+            return param
+
     def getValue(self, attr):
         """Return a settings item value
         """
@@ -65,11 +99,11 @@ class Settings(object):
                     if attr == 'TRAFFIC':
                         tmp_value = copy.deepcopy(master_value)
                         tmp_value = merge_spec(tmp_value, cli_value)
-                        return tmp_value
+                        return self._eval_param(tmp_value)
                     else:
-                        return cli_value
+                        return self._eval_param(cli_value)
                 else:
-                    return master_value
+                    return self._eval_param(master_value)
         else:
             raise AttributeError("%r object has no attribute %r" %
                                  (self.__class__, attr))
@@ -189,7 +223,7 @@ class Settings(object):
         for key in self.__dict__:
             if key.startswith('GUEST_'):
                 value = self.getValue(key)
-                if isinstance(value, str) and value.find('#') >= 0:
+                if isinstance(value, str) and str(value).find('#') >= 0:
                     self._expand_vm_settings(key, 1)
 
                 if isinstance(value, list):
@@ -266,7 +300,9 @@ class Settings(object):
     def validate_getValue(self, result, attr):
         """Verifies, that correct value was returned
         """
-        assert result == self.__dict__[attr]
+        # getValue must be called to expand macros and apply
+        # values from TEST_PARAM option
+        assert result == self.getValue(attr)
         return True
 
     def validate_setValue(self, dummy_result, name, value):
diff --git a/conf/integration/01_testcases.conf b/conf/integration/01_testcases.conf
index 61766e33..b58fa965 100644
--- a/conf/integration/01_testcases.conf
+++ b/conf/integration/01_testcases.conf
@@ -1,4 +1,4 @@
-# Copyright 2015-2016 Intel Corporation.
+# Copyright 2015-2017 Intel Corporation.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -34,16 +34,6 @@ SUPPORTED_TUNNELING_PROTO = ['vxlan', 'gre', 'geneve']
 #
 # bidirectional testing for OP2P is not yet supported.
 # TRAFFIC['bidir'] must be set to 'False'.
-#
-# "TestSteps": []                   # Definition of integration test steps.
-#                                   # In case that this list is defined, then
-#                                   # vsperf will execute defined test steps
-#                                   # one by one. It can be used to configure
-#                                   # vswitch, insert flows and transmit traffic.
-#                                   # It is possible to refer to result of any
-#                                   # previous step through #STEP[i][j] macro.
-#                                   # Where i is a number of step (starts from 0)
-#                                   # and j is index of result returned by step i.
 
 #
 # Common TestSteps parts ("macros")
@@ -894,12 +884,12 @@ INTEGRATION_TESTS = [
         "TestSteps": STEP_VSWITCH_PVVP_INIT +                                       # STEP 0-6
             [
                 # check that at least 2 numa slots are available
-                ['tools', 'exec', 'numactl -H', 'available: ([0-9]+)'],             # STEP 7
+                ['tools', 'exec_shell', 'numactl -H', 'available: ([0-9]+)'],       # STEP 7
                 ['tools', 'assert', '#STEP[-1][0]>1'],                              # STEP 8
                 # store last 2 cores from numa slot 0
-                ['tools', 'exec', 'numactl -H', 'node 0 cpus:.*\s+(\\d+) (\\d+)$'], # STEP 9
+                ['tools', 'exec_shell', 'numactl -H', 'node 0 cpus:.*\s+(\\d+) (\\d+)$'], # STEP 9
                 # store last 2 cores from numa slot 1
-                ['tools', 'exec', 'numactl -H', 'node 1 cpus:.*\s+(\\d+) (\\d+)$'], # STEP 10
+                ['tools', 'exec_shell', 'numactl -H', 'node 1 cpus:.*\s+(\\d+) (\\d+)$'], # STEP 10
                 # pin VNF1 to 1st NUMA slot and VNF2 to 2nd NUMA slot
                 ['settings', 'setValue', 'GUEST_CORE_BINDING',                      # STEP 11
                     [("#STEP[-2][0][0]", "#STEP[-2][0][1]"),
@@ -912,7 +902,7 @@ INTEGRATION_TESTS = [
                 ['settings', 'getValue', 'TOOLS'],                                  # STEP 14
                 # check that PMD thread serving VNF1 runs at NUMA slot 0
                 ## i.e. get numa slot ID serving dpdhvhostuser0...
-                ['tools', 'exec', "sudo #STEP[-1]['ovs-appctl'] "                   # STEP 15
+                ['tools', 'exec_shell', "sudo #STEP[-1]['ovs-appctl'] "             # STEP 15
                     "dpif-netdev/pmd-rxq-show | "
                     "sed -e '/dpdkvhostuser0/,$d' | tac",
                     'pmd thread numa_id ([0-9])+'
@@ -921,7 +911,7 @@ INTEGRATION_TESTS = [
                 ['tools', 'assert', '#STEP[-1][0]==0'],                             # STEP 16
                 # check that PMD thread serving VNF2 runs at NUMA slot 1
                 ## i.e. get numa slot ID serving dpdhvhostuser2...
-                ['tools', 'exec', "sudo #STEP[-3]['ovs-appctl'] "                   # STEP 17
+                ['tools', 'exec_shell', "sudo #STEP[-3]['ovs-appctl'] "             # STEP 17
                     "dpif-netdev/pmd-rxq-show | "
                     "sed -e '/dpdkvhostuser2/,$d' | tac",
                     'pmd thread numa_id ([0-9])+'