From b1534957e463b5e34957a8d48ce5c6b0552ffbb4 Mon Sep 17 00:00:00 2001 From: Martin Klozik Date: Thu, 31 Aug 2017 15:01:18 +0200 Subject: teststeps: Improvements and bugfixing of teststeps This patch introduces several improvements and small bugfixes of teststeps. These changes were identified during implementation of OVS/DPDK regression tests. Patch content: * teststeps: step aliases were implemented * teststeps: improved filtering by regex for any step, which returns string or list of stings; filter will process all lines * teststeps: support for log object * teststeps: support for trafficgen get_results call * teststeps: configurable suppression of step validation * trafficgen: remove old results before traffic is executed * trafficgen: support for flow control on/off (IxNet) * trafficgen: support for configurable learning frames (IxNet) * trafficgen: support for runtime changes of TRAFFICGEN_PKT_SIZES, _DURATION and _LOSSRATE * vnf: flush pexpect output of previous commands * vnf: use execute_and_wait() to ensure correct cmds order * vnf: dpdk vHost User interface name set according to its type, e.g. dpdkvhostuserclient * vswitch: support for OVS restart * decap: simplify configuration of tunneling decapsulation tests * settings: values of all configuration options are restored after TC execution * modified formatting of test description used by --list * testcase name and description is logged before its execution * small bugfixes JIRA: VSPERF-539 Change-Id: I550ba0d897ece89abd3f33d6d66f545c4d863e7b Signed-off-by: Martin Klozik Reviewed-by: Al Morton Reviewed-by: Christian Trautman Reviewed-by: Sridhar Rao Reviewed-by: Trevor Cooper --- .../devguide/design/vswitchperf_design.rst | 12 +++ docs/testing/user/userguide/integration.rst | 39 +++------- docs/testing/user/userguide/teststeps.rst | 88 ++++++++++++++++++++-- 3 files changed, 102 insertions(+), 37 deletions(-) (limited to 'docs/testing') diff --git a/docs/testing/developer/devguide/design/vswitchperf_design.rst b/docs/testing/developer/devguide/design/vswitchperf_design.rst index 55614d33..33051493 100644 --- a/docs/testing/developer/devguide/design/vswitchperf_design.rst +++ b/docs/testing/developer/devguide/design/vswitchperf_design.rst @@ -339,6 +339,18 @@ Detailed description of ``TRAFFIC`` dictionary items follows: "IP" - flow is defined by ingress ports and src and dst IP addresses Default value: "port" + 'flow_control' - Controls flow control support by traffic generator. + Supported values: + False - flow control is disabled + True - flow control is enabled + Default value: False + Note: Currently it is supported by IxNet only + 'learning_frames' - Controls learning frames support by traffic generator. + Supported values: + False - learning frames are disabled + True - learning frames are enabled + Default value: True + Note: Currently it is supported by IxNet only 'l2' - A dictionary with l2 network layer details. Supported values are: 'srcmac' - Specifies source MAC address filled by traffic generator. diff --git a/docs/testing/user/userguide/integration.rst b/docs/testing/user/userguide/integration.rst index 249a03c4..66808400 100644 --- a/docs/testing/user/userguide/integration.rst +++ b/docs/testing/user/userguide/integration.rst @@ -140,13 +140,7 @@ To run VXLAN decapsulation tests: 1. Set the variables used in "Executing Tunnel encapsulation tests" -2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT - - .. code-block:: python - - DUT_NIC2_MAC = '' - -3. Run test: +2. Run test: .. code-block:: console @@ -181,13 +175,7 @@ To run GRE decapsulation tests: 1. Set the variables used in "Executing Tunnel encapsulation tests" -2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT - - .. code-block:: python - - DUT_NIC2_MAC = '' - -3. Run test: +2. Run test: .. code-block:: console @@ -238,13 +226,7 @@ To run GENEVE decapsulation tests: 1. Set the variables used in "Executing Tunnel encapsulation tests" -2. Set dstmac of DUT_NIC2_MAC to the MAC adddress of the 2nd NIC of your DUT - - .. code-block:: python - - DUT_NIC2_MAC = '' - -3. Run test: +2. Run test: .. code-block:: console @@ -289,8 +271,6 @@ To run VXLAN decapsulation tests: 'datapath/linux/openvswitch.ko', ] - DUT_NIC1_MAC = '' - TRAFFICGEN_PORT1_IP = '172.16.1.2' TRAFFICGEN_PORT2_IP = '192.168.1.11' @@ -302,7 +282,8 @@ To run VXLAN decapsulation tests: VXLAN_FRAME_L2 = {'srcmac': '01:02:03:04:05:06', - 'dstmac': DUT_NIC1_MAC + 'dstmac': + '06:05:04:03:02:01', } VXLAN_FRAME_L3 = {'proto': 'udp', @@ -349,8 +330,6 @@ To run GRE decapsulation tests: 'datapath/linux/openvswitch.ko', ] - DUT_NIC1_MAC = '' - TRAFFICGEN_PORT1_IP = '172.16.1.2' TRAFFICGEN_PORT2_IP = '192.168.1.11' @@ -362,7 +341,8 @@ To run GRE decapsulation tests: GRE_FRAME_L2 = {'srcmac': '01:02:03:04:05:06', - 'dstmac': DUT_NIC1_MAC + 'dstmac': + '06:05:04:03:02:01', } GRE_FRAME_L3 = {'proto': 'udp', @@ -408,8 +388,6 @@ To run GENEVE decapsulation tests: 'datapath/linux/openvswitch.ko', ] - DUT_NIC1_MAC = '' - TRAFFICGEN_PORT1_IP = '172.16.1.2' TRAFFICGEN_PORT2_IP = '192.168.1.11' @@ -421,7 +399,8 @@ To run GENEVE decapsulation tests: GENEVE_FRAME_L2 = {'srcmac': '01:02:03:04:05:06', - 'dstmac': DUT_NIC1_MAC + 'dstmac': + '06:05:04:03:02:01', } GENEVE_FRAME_L3 = {'proto': 'udp', diff --git a/docs/testing/user/userguide/teststeps.rst b/docs/testing/user/userguide/teststeps.rst index 5349d2ea..8be67310 100644 --- a/docs/testing/user/userguide/teststeps.rst +++ b/docs/testing/user/userguide/teststeps.rst @@ -29,6 +29,14 @@ does not pass validation the test will fail and terminate. The test will continu until a failure is detected or all steps pass. A csv report file is generated after a test completes with an OK or FAIL result. +**NOTE**: It is possible to suppress validation process of given step by prefixing +it by ``!`` (exclamation mark). +In following example test execution won't fail if all traffic is dropped: + +.. code-block:: python + + ['!trafficgen', 'send_traffic', {}] + In case of performance test, the validation of steps is not performed and standard output files with results from traffic generator and underlying OS details are generated by vsperf. @@ -75,6 +83,7 @@ of supported objects and their most common functions follows: * ``disable_stp br_name`` - disables Spanning Tree Protocol for bridge ``br_name`` * ``enable_rstp br_name`` - enables Rapid Spanning Tree Protocol for bridge ``br_name`` * ``disable_rstp br_name`` - disables Rapid Spanning Tree Protocol for bridge ``br_name`` + * ``restart`` - restarts switch, which is useful for failover testcases Examples: @@ -101,15 +110,26 @@ of supported objects and their most common functions follows: * ``start`` - starts a VNF based on VSPERF configuration * ``stop`` - gracefully terminates given VNF + * ``execute command [delay]`` - executes command `cmd` inside VNF; Optional + delay defines number of seconds to wait before next step is executed. Method + returns command output as a string. + * ``execute_and_wait command [timeout] [prompt]`` - executes command `cmd` inside + VNF; Optional timeout defines number of seconds to wait until ``prompt`` is detected. + Optional ``prompt`` defines a string, which is used as detection of successful command + execution. In case that prompt is not defined, then content of ``GUEST_PROMPT_LOGIN`` + parameter will be used. Method returns command output as a string. Examples: .. code-block:: python - ['vnf1', 'start'] - ['vnf2', 'start'] - ['vnf2', 'stop'] - ['vnf1', 'stop'] + ['vnf1', 'start'], + ['vnf2', 'start'], + ['vnf1', 'execute_and_wait', 'ifconfig eth0 5.5.5.1/24 up'], + ['vnf2', 'execute_and_wait', 'ifconfig eth0 5.5.5.2/24 up', 120, 'root.*#'], + ['vnf2', 'execute_and_wait', 'ping -c1 5.5.5.1'], + ['vnf2', 'stop'], + ['vnf1', 'stop'], * ``trafficgen`` - triggers traffic generation @@ -119,6 +139,8 @@ of supported objects and their most common functions follows: and given ``traffic`` dictionary. More details about ``traffic`` dictionary and its possible values are available at :ref:`Traffic Generator Integration Guide ` + * ``get_results`` - returns dictionary with results collected from previous execution + of ``send_traffic`` Examples: @@ -126,7 +148,12 @@ of supported objects and their most common functions follows: ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_throughput'}] - ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}] + ['trafficgen', 'send_traffic', {'traffic_type' : 'rfc2544_back2back', 'bidir' : 'True'}], + ['trafficgen', 'get_results'], + ['tools', 'assert', '#STEP[-1][0]["frame_loss_percent"] < 0.05'], + + +.. _step-driven-tests-variable-usage: * ``settings`` - reads or modifies VSPERF configuration @@ -229,8 +256,7 @@ of supported objects and their most common functions follows: in case that condition is not ``True`` * ``Eval expression`` - evaluates given expression as a python code and returns its result - * ``Exec_Shell command [regex]`` - executes a shell command and filters its output by - (optional) regular expression + * ``Exec_Shell command`` - executes a shell command * ``Exec_Python code`` - executes a python code @@ -260,6 +286,23 @@ of supported objects and their most common functions follows: ['sleep', '60'] + * ``log level message`` - is used to log ``message`` of given ``level`` into vsperf output. + Level is one of info, debug, warning or error. + + Examples: + + .. code-block:: python + + ['log', 'error', 'tools $TOOLS'] + + * ``pdb`` - executes python debugger + + Examples: + + .. code-block:: python + + ['pdb'] + Test Macros ----------- @@ -296,6 +339,15 @@ functionality: ['vswitch', 'del_port', 'int_br0', '#STEP[-1][0]'], # STEP 2 +Another option to refer to previous values, is to define an alias for given step +by its first argument with '#' prefix. Alias must be unique and it can't be a number. +Example of step alias usage: + +.. code-block:: python + + ['#port1', 'vswitch', 'add_vport', 'int_br0'], + ['vswitch', 'del_port', 'int_br0', '#STEP[port1][0]'], + Also commonly used steps can be created as a separate profile. .. code-block:: python @@ -324,6 +376,28 @@ This profile can then be used inside other testcases STEP_VSWITCH_PVP_FINIT } +It is possible to refer to vsperf configuration parameters within step macros. Please +see :ref:`step-driven-tests-variable-usage` for more details. + +In case that step returns a string or list of strings, then it is possible to +filter such output by regular expression. This optional filter can be specified +as a last step parameter with prefix '|'. Output will be split into separate lines +and only matching records will be returned. It is also possible to return a specified +group of characters from the matching lines, e.g. by regex ``|ID (\d+)``. + +Examples: + +.. code-block:: python + + ['tools', 'exec_shell', "sudo $TOOLS['ovs-appctl'] dpif-netdev/pmd-rxq-show", + '|dpdkvhostuser0\s+queue-id: \d'], + ['tools', 'assert', 'len(#STEP[-1])==1'], + + ['vnf', 'execute_and_wait', 'ethtool -L eth0 combined 2'], + ['vnf', 'execute_and_wait', 'ethtool -l eth0', '|Combined:\s+2'], + ['tools', 'assert', 'len(#STEP[-1])==2'] + + HelloWorld and other basic Testcases ------------------------------------ -- cgit 1.2.3-korg