pkt_gen: support of standalone execution of traffic generator

Support for multiple modes of VSPERF operation has been added.
These modes can be used for standalone execution of traffic
generator or for manual testing or for execution of unsupported
traffic generator. Supported modes are: "normal" - execute vSwitch,
VNF and traffic generator; "trafficgen" - execute only traffic
generator; "trafficgen-off" - execute vSwitch and VNF.
Normal mode is selected by default.
In case that trafficgen mode is selected, then various
--test-params could be specified to affect traffic generator
configuration. These parameters include traffic type, frame rate,
bidirectional and scalability settings. Selection of transport
protocol is not supported by IxNet yet (UDP is enforced), thus
modification of transport protocol from command line is not
supported too.
Fixes of testpmd and qemu warning patches are inclduded.

Change-Id: Idac10fe03e724075268a01ec3eb0817fba830aec
JIRA: VSPERF-173
Signed-off-by: Martin Klozik <martinx.klozik@intel.com>
Reviewed-by: Maryam Tahhan <maryam.tahhan@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>

6 files changed, 226 insertions, 134 deletions

diff --git a/conf/01_testcases.conf b/conf/01_testcases.conf
index b855405d..77072eeb 100755
--- a/conf/01_testcases.conf
+++ b/conf/01_testcases.conf
@@ -20,14 +20,18 @@
 # "Name": "phy2phy_burst",         # A human-readable string identifying the
 #                                  # test.
 # "Traffic Type": "rfc2544",       # One of the supported traffic types.
+#                                  # It can be overridden by cli option traffic_type.
+#                                  # Default value is "rfc2544".
 # "Deployment": "p2p",             # One of the supported deployment scenarios.
 # "Description": "Lorem ipsum..."  # Optional. A human-readable string
 #                                  # describing the test.
 # "Frame Modification": "vlan"     # One of the supported frame modifications:
 #                                  # vlan, mpls, mac, dscp, ttl, ip_addr,
 #                                  # ip_port.
-# "biDirectional": [true|false],   # Specifies if genearted traffic will be
+# "biDirectional": [true|false],   # Specifies if generated traffic will be
 #                                  # full-duplex (true) or half-duplex (false)
+#                                  # It can be overridden by cli option bidirectional.
+#                                  # Default value is "false".
 # "MultiStream": 0-65535           # Optional. Defines number of flows simulated
 #                                  # by traffic generator. Value 0 disables
 #                                  # MultiStream feature
diff --git a/core/pktfwd_controller.py b/core/pktfwd_controller.py
index 4f300ce8..40565504 100644
--- a/core/pktfwd_controller.py
+++ b/core/pktfwd_controller.py
@@ -17,9 +17,6 @@
 
 import logging
 
-from conf import settings
-
-
 class PktFwdController(object):
     """Packet forwarder controller for P2P deployment scenario.
 
@@ -66,3 +63,10 @@ class PktFwdController(object):
         :return: The controlled IPktFwd
         """
         return self._pktfwd
+
+    def dump_vswitch_flows(self):
+        """ Dumps flows from vswitch
+        """
+        raise NotImplementedError(
+            "The PktFwdController does not implement the "
+            "\"dump_vswitch_flows\" function.")
diff --git a/docs/userguides/quickstart.rst b/docs/userguides/quickstart.rst
index caf3be04..fa951e0e 100755
--- a/docs/userguides/quickstart.rst
+++ b/docs/userguides/quickstart.rst
@@ -294,9 +294,9 @@ To run tests using Vanilla OVS:
 
    or use --test-param
 
-   ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
-            --test-param "vanilla_tgen_tx_ip=n.n.n.n;
-                          vanilla_tgen_tx_mac=nn:nn:nn:nn:nn:nn"
+   $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
+              --test-param "vanilla_tgen_tx_ip=n.n.n.n;
+                            vanilla_tgen_tx_mac=nn:nn:nn:nn:nn:nn"
 
 
 2. Recompile src for Vanilla OVS testing
@@ -356,7 +356,7 @@ following configuration parameter should be configured:
 
      or use --vswitch and --fwdapp
 
-     ./vsperf --conf-file user_settings.py
+     $ ./vsperf --conf-file user_settings.py
               --vswitch none
               --fwdapp TestPMD
 
@@ -386,8 +386,55 @@ for selected Packet Forwarder:
 
   .. code-block:: console
 
-     ./vsperf --conf-file <path_to_settings_py>
+     $ ./vsperf --conf-file <path_to_settings_py>
 
+VSPERF modes of operation
+-------------------------
+VSPERF can be run in different modes. By default it will configure vSwitch,
+traffic generator and VNF. However it can be used just for configuration
+and execution of traffic generator. Another option is execution of all
+components except traffic generator itself.
+
+Mode of operation is driven by configuration parameter -m or --mode
+
+.. code-block:: console
+
+    -m MODE, --mode MODE  vsperf mode of operation;
+        Values:
+            "normal" - execute vSwitch, VNF and traffic generator
+            "trafficgen" - execute only traffic generator
+            "trafficgen-off" - execute vSwitch and VNF
+
+In case, that VSPERF is executed in "trafficgen" mode, then configuration
+of traffic generator should be configured through --test-param option.
+Supported CLI options useful for traffic generator configuration are:
+
+.. code-block:: console
+
+    'traffic_type'  - One of the supported traffic types. E.g. rfc2544,
+                      back2back or continuous
+                      Default value is "rfc2544".
+    'bidirectional' - Specifies if generated traffic will be full-duplex (true)
+                      or half-duplex (false)
+                      Default value is "false".
+    'iload'         - Defines desired percentage of frame rate used during
+                      continuous stream tests.
+                      Default value is 100.
+    'multistream'   - Defines number of flows simulated by traffic generator.
+                      Value 0 disables MultiStream feature
+                      Default value is 0.
+    'stream_type'   - Stream Type is an extension of the "MultiStream" feature.
+                      If MultiStream is disabled, then Stream Type will be
+                      ignored. Stream Type defines ISO OSI network layer used
+                      for simulation of multiple streams.
+                      Default value is "L4".
+
+Example of execution of VSPERF in "trafficgen" mode:
+
+.. code-block:: console
+
+    $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf
+        --test-params "traffic_type=continuous;bidirectional=True;iload=60"
 
 Code change verification by pylint
 ----------------------------------
diff --git a/testcases/testcase.py b/testcases/testcase.py
index 7d5162e6..9c755ea7 100644
--- a/testcases/testcase.py
+++ b/testcases/testcase.py
@@ -19,6 +19,7 @@ import os
 import logging
 import subprocess
 import copy
+import time
 from collections import OrderedDict
 
 from core.results.results_constants import ResultsConstants
@@ -45,14 +46,23 @@ class TestCase(object):
         :param results_dir: Where the csv formatted results are written.
         """
         self._hugepages_mounted = False
+
+        # set test parameters; CLI options take precedence to testcase settings
         self._logger = logging.getLogger(__name__)
         self.name = cfg['Name']
         self.desc = cfg.get('Description', 'No description given.')
+
+        bidirectional = cfg.get('biDirectional', False)
+        bidirectional = get_test_param('bidirectional', bidirectional)
+
+        traffic_type = cfg.get('Traffic Type', 'rfc2544')
+        traffic_type = get_test_param('traffic_type', traffic_type)
+
+        framerate = cfg.get('iLoad', 100)
+        framerate = get_test_param('iload', framerate)
+
         self.deployment = cfg['Deployment']
         self._frame_mod = cfg.get('Frame Modification', None)
-        framerate = get_test_param('iload', None)
-        if framerate == None:
-            framerate = cfg.get('iLoad', 100)
 
         # identify guest loopback method, so it can be added into reports
         self.guest_loopback = []
@@ -75,7 +85,6 @@ class TestCase(object):
         pre_installed_flows = cfg.get('Pre-installed Flows', 'No')
         pre_installed_flows = get_test_param('pre-installed_flows', pre_installed_flows)
 
-
         # check if test requires background load and which generator it uses
         self._load_cfg = cfg.get('Load', None)
         if self._load_cfg and 'tool' in self._load_cfg:
@@ -90,9 +99,9 @@ class TestCase(object):
 
         # set traffic details, so they can be passed to vswitch and traffic ctls
         self._traffic = copy.deepcopy(TRAFFIC_DEFAULTS)
-        self._traffic.update({'traffic_type': cfg['Traffic Type'],
+        self._traffic.update({'traffic_type': traffic_type,
                               'flow_type': cfg.get('Flow Type', 'port'),
-                              'bidir': cfg['biDirectional'],
+                              'bidir': bidirectional,
                               'multistream': int(multistream),
                               'stream_type': stream_type,
                               'pre_installed_flows' : pre_installed_flows,
@@ -152,29 +161,36 @@ class TestCase(object):
                 if not self._vswitch_none:
                     self._add_flows(vswitch_ctl)
 
-                with traffic_ctl:
-                    traffic_ctl.send_traffic(self._traffic)
+                # run traffic generator if requested, otherwise wait for manual termination
+                if S.getValue('mode') == 'trafficgen-off':
+                    time.sleep(2)
+                    self._logger.debug("All is set. Please run traffic generator manually.")
+                    input(os.linesep + "Press Enter to terminate vswitchperf..." + os.linesep + os.linesep)
+                else:
+                    with traffic_ctl:
+                        traffic_ctl.send_traffic(self._traffic)
 
-                # dump vswitch flows before they are affected by VNF termination
-                if not self._vswitch_none:
-                    vswitch_ctl.dump_vswitch_flows()
+                    # dump vswitch flows before they are affected by VNF termination
+                    if not self._vswitch_none:
+                        vswitch_ctl.dump_vswitch_flows()
 
         # umount hugepages if mounted
         self._umount_hugepages()
 
-        self._logger.debug("Traffic Results:")
-        traffic_ctl.print_results()
-
         self._logger.debug("Collector Results:")
         collector.print_results()
 
-        output_file = os.path.join(self._results_dir, "result_" + self.name +
-                                   "_" + self.deployment + ".csv")
+        if S.getValue('mode') != 'trafficgen-off':
+            self._logger.debug("Traffic Results:")
+            traffic_ctl.print_results()
+
+            output_file = os.path.join(self._results_dir, "result_" + self.name +
+                                       "_" + self.deployment + ".csv")
 
-        tc_results = self._append_results(traffic_ctl.get_results())
-        TestCase._write_result_to_file(tc_results, output_file)
+            tc_results = self._append_results(traffic_ctl.get_results())
+            TestCase._write_result_to_file(tc_results, output_file)
 
-        report.generate(output_file, tc_results, collector.get_results())
+            report.generate(output_file, tc_results, collector.get_results())
 
     def _append_results(self, results):
         """
@@ -287,7 +303,7 @@ class TestCase(object):
         return list(result.keys())
 
 
-    def _add_flows(vswitch_ctl):
+    def _add_flows(self, vswitch_ctl):
         """Add flows to the vswitch
 
         :param vswitch_ctl vswitch controller
diff --git a/vnfs/qemu/qemu.py b/vnfs/qemu/qemu.py
index 536f1fd0..38394974 100644
--- a/vnfs/qemu/qemu.py
+++ b/vnfs/qemu/qemu.py
@@ -83,7 +83,7 @@ class IVnfQemu(IVnf):
                      '-nographic', '-vnc', str(vnc), '-name', name,
                      '-snapshot', '-net none', '-no-reboot',
                      '-drive',
-                     'if=scsi,type=raw,file=fat:rw:%s,snapshot=off' %
+                     'if=scsi,format=raw,file=fat:rw:%s,snapshot=off' %
                      S.getValue('GUEST_SHARE_DIR')[self._number],
                     ]
         self._configure_logging()
diff --git a/vsperf b/vsperf
index 50f0996a..93772326 100755
--- a/vsperf
+++ b/vsperf
@@ -27,6 +27,7 @@ import shutil
 import unittest
 import xmlrunner
 import locale
+import copy
 
 sys.dont_write_bytecode = True
 
@@ -37,6 +38,9 @@ from testcases import TestCase
 from tools import tasks
 from tools.pkt_gen import trafficgen
 from tools.opnfvdashboard import opnfvdashboard
+from tools.pkt_gen.trafficgen.trafficgenhelper import TRAFFIC_DEFAULTS
+from conf import get_test_param
+import core.component_factory as component_factory
 
 VERBOSITY_LEVELS = {
     'debug': logging.DEBUG,
@@ -133,6 +137,11 @@ def parse_arguments():
             To run all tests omit both positional args and --tests arg.')
 
     group = parser.add_argument_group('test selection options')
+    group.add_argument('-m', '--mode', help='vsperf mode of operation;\
+            Values: "normal" - execute vSwitch, VNF and traffic generator;\
+            "trafficgen" - execute only traffic generator; "trafficgen-off" \
+            - execute vSwitch and VNF', default='normal')
+
     group.add_argument('-f', '--test-spec', help='test specification file')
     group.add_argument('-d', '--test-dir', help='directory containing tests')
     group.add_argument('-t', '--tests', help='Comma-separated list of terms \
@@ -332,7 +341,6 @@ def main():
     check_and_set_locale()
 
     # configure trafficgens
-
     if args['trafficgen']:
         trafficgens = Loader().get_trafficgens()
         if args['trafficgen'] not in trafficgens:
@@ -387,119 +395,132 @@ def main():
             tmp_gl.append(guest_loopback)
         settings.setValue('GUEST_LOOPBACK', tmp_gl)
 
+    settings.setValue('mode', args['mode'])
+
     # generate results directory name
     date = datetime.datetime.fromtimestamp(time.time())
     results_dir = "results_" + date.strftime('%Y-%m-%d_%H-%M-%S')
     results_path = os.path.join(settings.getValue('LOG_DIR'), results_dir)
 
-    # configure tests
-    testcases = settings.getValue('PERFORMANCE_TESTS')
-    all_tests = []
-    for cfg in testcases:
-        try:
-            all_tests.append(TestCase(cfg, results_path))
-        except (Exception) as _:
-            logger.exception("Failed to create test: %s",
-                             cfg.get('Name', '<Name not set>'))
-            raise
-
-    # if required, handle list-* operations
-
-    if args['list']:
-        print("Available Tests:")
-        print("======")
-        for test in all_tests:
-            print('* %-18s%s' % ('%s:' % test.name, test.desc))
-        exit()
-
-    if args['list_trafficgens']:
-        print(Loader().get_trafficgens_printable())
-        exit()
-
-    if args['list_collectors']:
-        print(Loader().get_collectors_printable())
-        exit()
-
-    if args['list_vswitches']:
-        print(Loader().get_vswitches_printable())
-        exit()
-
-    if args['list_fwdapps']:
-        print(Loader().get_pktfwds_printable())
-        exit()
-
-    if args['list_vnfs']:
-        print(Loader().get_vnfs_printable())
-        exit()
-
-    if args['list_settings']:
-        print(str(settings))
-        exit()
-
-    # select requested tests
-    if args['exact_test_name'] and args['tests']:
-        logger.error("Cannot specify tests with both positional args and --test.")
-        sys.exit(1)
-
-    if args['exact_test_name']:
-        exact_names = args['exact_test_name']
-        # positional args => exact matches only
-        selected_tests = [test for test in all_tests if test.name in exact_names]
-    elif args['tests']:
-        # --tests => apply filter to select requested tests
-        selected_tests = apply_filter(all_tests, args['tests'])
-    else:
-        # Default - run all tests
-        selected_tests = all_tests
-
-    if not selected_tests:
-        logger.error("No tests matched --test option or positional args. Done.")
-        sys.exit(1)
-
     # create results directory
     if not os.path.exists(results_path):
         logger.info("Creating result directory: "  + results_path)
         os.makedirs(results_path)
 
-    # run tests
-    suite = unittest.TestSuite()
-    for test in selected_tests:
-        try:
-            if vswitch_none:
-                if test.deployment.lower() != 'p2p':
-                    logging.error('\'none\' vswitch option supported only'
-                                  ' for p2p deployment.')
-                    sys.exit(1)
-            test.run()
-            suite.addTest(MockTestCase('', True, test.name))
-        #pylint: disable=broad-except
-        except (Exception) as ex:
-            logger.exception("Failed to run test: %s", test.name)
-            suite.addTest(MockTestCase(str(ex), False, test.name))
-            logger.info("Continuing with next test...")
-
-    if settings.getValue('XUNIT'):
-        xmlrunner.XMLTestRunner(
-            output=settings.getValue('XUNIT_DIR'), outsuffix="",
-            verbosity=0).run(suite)
-
-    if args['opnfvpod']:
-        pod_name = args['opnfvpod']
-        installer_name = settings.getValue('OPNFV_INSTALLER')
-        opnfv_url = settings.getValue('OPNFV_URL')
-        pkg_list = settings.getValue('PACKAGE_LIST')
-
-        int_data = {'cuse': False,
-                    'vanilla': False,
-                    'pod': pod_name,
-                    'installer': installer_name,
-                    'pkg_list': pkg_list,
-                    'db_url': opnfv_url}
-        if settings.getValue('VSWITCH').endswith('Vanilla'):
-            int_data['vanilla'] = True
-        if settings.getValue('VNF').endswith('Cuse'):
-            int_data['cuse'] = True
-        opnfvdashboard.results2opnfv_dashboard(results_path, int_data)
+    if settings.getValue('mode') == 'trafficgen':
+        # execute only traffic generator
+        logging.debug("Executing traffic generator:")
+        loader = Loader()
+        # set traffic details, so they can be passed to traffic ctl
+        traffic = copy.deepcopy(TRAFFIC_DEFAULTS)
+        traffic.update({'traffic_type': get_test_param('traffic_type', 'rfc2544'),
+                        'bidir': get_test_param('bidirectional', False),
+                        'multistream': int(get_test_param('multistream', 0)),
+                        'stream_type': get_test_param('stream_type', 'L4'),
+                        'frame_rate': int(get_test_param('iload', 100))})
+
+        traffic_ctl = component_factory.create_traffic(
+            traffic['traffic_type'],
+            loader.get_trafficgen_class())
+        with traffic_ctl:
+            traffic_ctl.send_traffic(traffic)
+        logging.debug("Traffic Results:")
+        traffic_ctl.print_results()
+    else:
+        # configure tests
+        testcases = settings.getValue('PERFORMANCE_TESTS')
+        all_tests = []
+        for cfg in testcases:
+            try:
+                all_tests.append(TestCase(cfg, results_path))
+            except (Exception) as _:
+                logger.exception("Failed to create test: %s",
+                                 cfg.get('Name', '<Name not set>'))
+                raise
+
+        # if required, handle list-* operations
+
+        if args['list']:
+            print("Available Tests:")
+            print("======")
+            for test in all_tests:
+                print('* %-18s%s' % ('%s:' % test.name, test.desc))
+            exit()
+
+        if args['list_trafficgens']:
+            print(Loader().get_trafficgens_printable())
+            exit()
+
+        if args['list_collectors']:
+            print(Loader().get_collectors_printable())
+            exit()
+
+        if args['list_vswitches']:
+            print(Loader().get_vswitches_printable())
+            exit()
+
+        if args['list_vnfs']:
+            print(Loader().get_vnfs_printable())
+            exit()
+
+        if args['list_settings']:
+            print(str(settings))
+            exit()
+
+        # select requested tests
+        if args['exact_test_name'] and args['tests']:
+            logger.error("Cannot specify tests with both positional args and --test.")
+            sys.exit(1)
+
+        if args['exact_test_name']:
+            exact_names = args['exact_test_name']
+            # positional args => exact matches only
+            selected_tests = [test for test in all_tests if test.name in exact_names]
+        elif args['tests']:
+            # --tests => apply filter to select requested tests
+            selected_tests = apply_filter(all_tests, args['tests'])
+        else:
+            # Default - run all tests
+            selected_tests = all_tests
+
+        if not selected_tests:
+            logger.error("No tests matched --test option or positional args. Done.")
+            sys.exit(1)
+
+        # run tests
+        suite = unittest.TestSuite()
+        for test in selected_tests:
+            try:
+                test.run()
+                suite.addTest(MockTestCase('', True, test.name))
+            #pylint: disable=broad-except
+            except (Exception) as ex:
+                logger.exception("Failed to run test: %s", test.name)
+                suite.addTest(MockTestCase(str(ex), False, test.name))
+                logger.info("Continuing with next test...")
+
+        if settings.getValue('XUNIT'):
+            xmlrunner.XMLTestRunner(
+                output=settings.getValue('XUNIT_DIR'), outsuffix="",
+                verbosity=0).run(suite)
+
+        if args['opnfvpod']:
+            pod_name = args['opnfvpod']
+            installer_name = settings.getValue('OPNFV_INSTALLER')
+            opnfv_url = settings.getValue('OPNFV_URL')
+            pkg_list = settings.getValue('PACKAGE_LIST')
+
+            int_data = {'cuse': False,
+                        'vanilla': False,
+                        'pod': pod_name,
+                        'installer': installer_name,
+                        'pkg_list': pkg_list,
+                        'db_url': opnfv_url}
+            if settings.getValue('VSWITCH').endswith('Vanilla'):
+                int_data['vanilla'] = True
+            if settings.getValue('VNF').endswith('Cuse'):
+                int_data['cuse'] = True
+            opnfvdashboard.results2opnfv_dashboard(results_path, int_data)
 
     #remove directory if no result files were created.
     if os.path.exists(results_path):