framework: Add reworked framework to repo

This commit adds the vSwitch Integration Test Framework whose
design, based off TOIT, is outlined in the HLD previously made
availiable to the community for review.

The design of this framework allows developers to add different
implementations of components, specifically vSwitches, Traffic
Generators, Metrics Collectors and VNFs, easily.

The goal of this design is that all testcases should run regardless of
what is "under the hood".

This commit adds support for running the framework for a phy to phy
RFC2544 testcase only. More testcases will be added by the community.

vSwitches supported at this time:
* Intel DPDK (r) accelerated OpenvSwitch

Traffic Generators supported at this time:
* IxNet - IxNetwork Implementation
* Ixia - IxExplorer Implementation
* Dummy - Manual Implementation

Metrics Collectors supported at this time:
* Linux Metrics

No VNFs are supported at this time but the framework outlines how they
should be integrated and provides APIs for them to adhere to.

JIRA: VSPERF-27
Change-Id: I312e1a1199487ffee8f824be06cd97d4f793eee0
Signed-off-by: Stephen Finucane <Stephen.Finucane@intel.com>
Signed-off-by: Meghan Halton <Meghan.Halton@intel.com>
Signed-off-by: Christopher Nolan <Christopher.Nolan@intel.com>
Signed-off-by: Maryam Tahhan <Maryam.Tahhan@intel.com>
Signed-off-by: Ciara Loftus <Ciara.Loftus@intel.com>
Signed-off-by: Mark Kavanagh <Mark.B.Kavanagh@intel.com>
Signed-off-by: Cian Ferriter <Cian.Ferriter@intel.com>
Signed-off-by: Timo Puha <TimoX.Puha@intel.com>
Signed-off-by: Billy O'Mahony <billy.o.mahony@intel.com>
Signed-off-by: Michal Weglicki <MichalX.Weglicki@intel.com>
Signed-off-by: Rory Sexton <Rory.Sexton@intel.com>
Signed-off-by: Ian Stokes <Ian.Stokes@intel.com>
Signed-off-by: Kevin Traynor <Kevin.Traynor@intel.com>
Signed-off-by: Dino Simeon Madarang <dino.simeonx.madarang@intel.com>
Reviewed-by: Eugene Snider <Eugene.Snider@huawei.com>
Reviewed-by: Aihua Li <aihua.li@huawei.com>

8 files changed, 510 insertions, 0 deletions

diff --git a/conf/00_common.conf b/conf/00_common.conf
new file mode 100644
index 00000000..ef1c6280
--- /dev/null
+++ b/conf/00_common.conf
@@ -0,0 +1,61 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ###########################
+# Common setting potentially accessed by all components.
+# ###########################
+
+import os
+
+# ############################
+# Directories
+# ############################
+
+ROOT_DIR = os.path.normpath(os.path.join(
+    os.path.dirname(os.path.realpath(__file__)), '../'))
+TRAFFICGEN_DIR = os.path.join(ROOT_DIR, 'tools/pkt_gen')
+SYSMETRICS_DIR = os.path.join(ROOT_DIR, 'tools/collectors')
+
+# ############################
+# Process configuration
+# ############################
+
+# shell command to use when running commands through Pexpect
+SHELL_CMD = ['/bin/bash', '-c']
+
+# ############################
+# Logging configuration
+# ############################
+
+# default log output directory for all logs
+LOG_DIR = '/tmp'
+
+# default log for all "small" executables
+LOG_FILE_DEFAULT = 'overall.log'
+
+# log file for all commands executed on host
+LOG_FILE_HOST_CMDS = 'host-cmds.log'
+
+# ############################
+# Test configuration
+# ############################
+
+# verbosity of output to 'stdout'
+# NOTE: output to log files is always 'debug' level
+VERBOSITY = 'debug'
+
+# dictionary of test-specific parameters. These values are accessible
+# from anywhere in the test framework so be careful with naming
+# conventions
+TEST_PARAMS = {}
diff --git a/conf/01_testcases.conf b/conf/01_testcases.conf
new file mode 100755
index 00000000..42090df4
--- /dev/null
+++ b/conf/01_testcases.conf
@@ -0,0 +1,40 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file describes a list of testcases.  Each testcase is described as a
+# dictionary in a list of dictionaries.
+#
+# The dictionary keys, their meanings and available values are:
+#
+# "Name": "phy2phy_burst",         # A human-readable string identifying the
+#                                  # test.
+# "Traffic Type": "rfc2544",       # One of the supported traffic types.
+# "Collector": "cpu, memory",      # Comma-separated list of Collectors to
+#                                  # be activated during this test.
+# "Deployment": "p2p",             # One of the supported deployment scenarios.
+# "Description": "Lorem ipsum..."  # Optional. A human-readable string
+#                                  # describing the test.
+# "MultiStream": [true|false],
+# "Test Modifier": [FrameMod|Other],
+# "Dependency": [Test_Case_Name |None],
+
+PERFORMANCE_TESTS = [
+    {
+        "Name": "phy2phy_tput",
+        "Traffic Type": "rfc2544",
+        "Collector": "cpu",
+        "Deployment": "p2p",
+        "Description": "RFC2544 Throughput Phy2Phy Loopback",
+    },
+]
diff --git a/conf/02_vswitch.conf b/conf/02_vswitch.conf
new file mode 100644
index 00000000..8b6a80a0
--- /dev/null
+++ b/conf/02_vswitch.conf
@@ -0,0 +1,78 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ############################
+# vswitch configuration
+# ############################
+RTE_SDK = '~/dpdk'
+OVS_DIR = '~/openvswitch'
+
+# ############################
+# DPDK configuration
+# ############################
+
+# DPDK target used when builing DPDK
+RTE_TARGET = 'x86_64-ivshmem-linuxapp-gcc'
+
+# list of NIC HWIDs which will be bound to the 'igb_uio' driver on
+# system init
+WHITELIST_NICS = ['05:00.0', '05:00.1']
+
+# list of NIC HWIDs which will be ignored by the 'igb_uio' driver on
+# system init
+BLACKLIST_NICS = ['0000:09:00.0', '0000:09:00.1', '0000:09:00.2',
+                  '0000:09:00.3']
+
+# These are DPDK EAL parameters and they may need to be changed depending on
+# hardware configuration, like cpu numbering and NUMA.
+VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4', '--socket-mem 1024,0']
+
+# directory where hugepages will be mounted on system init
+HUGEPAGE_DIR = '/dev/hugepages'
+
+# list of tuples of format (path, module_name), which will be inserted
+# using 'insmod' on system init
+
+# for OVS modules the path is in reference to the OVS directory.
+OVS_MODULES = []
+
+# for DPDK_MODULES the path is in reference to the build directory
+DPDK_MODULES = [
+    ('kmod', 'igb_uio'),
+]
+
+VHOST_MODULE = [
+    ('eventfd_link', 'eventfd_link')
+]
+
+# list of modules that will be inserted using 'modprobe' on system init
+SYS_MODULES = ['uio', 'cuse']
+
+# vhost character device file used by dpdkvhostport QemuWrap cases
+VHOST_DEV_FILE = 'ovs-vhost-net'
+
+# location of vhost-user sockets
+VHOST_USER_SOCKS = ['/tmp/dpdkvhostuser0', '/tmp/dpdkvhostuser1',
+                    '/tmp/dpdkvhostuser2', '/tmp/dpdkvhostuser3',
+                    '/tmp/myport0', '/tmp/helloworld123', '/tmp/abcstuff0']
+
+# log file for ovs-vswitchd
+LOG_FILE_VSWITCHD = 'vswitchd.log'
+
+# log file for ovs-dpdk
+LOG_FILE_OVS = 'ovs.log'
+
+VSWITCH_DIR = os.path.join(ROOT_DIR, 'vswitches')
+VSWITCH = "OvsDpdkVhost"
+
diff --git a/conf/03_traffic.conf b/conf/03_traffic.conf
new file mode 100644
index 00000000..efc1b2f6
--- /dev/null
+++ b/conf/03_traffic.conf
@@ -0,0 +1,57 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ############################
+# Traffic gen configuration
+# ############################
+
+# log file for all traffic generator related commands
+LOG_FILE_TRAFFIC_GEN = 'traffic-gen.log'
+
+#path to traffic generators directory.
+TRAFFICGEN_DIR = os.path.join(ROOT_DIR, 'tools/pkt_gen')
+
+# traffic generator to use in tests
+TRAFFICGEN = 'IxNet'
+
+# List of packet sizes to send.
+# Expand like this: (64, 128, 256, 512, 1024)
+TRAFFICGEN_PKT_SIZES = (64,)
+
+# path to 'ixos' install path
+TRAFFICGEN_IXIA_ROOT_DIR = '/opt/ixos'
+
+# network address of IXIA chassis
+TRAFFICGEN_IXIA_HOST = ''
+
+TRAFFICGEN_IXIA_CARD = ''
+
+TRAFFICGEN_IXIA_PORT1 = ''
+
+TRAFFICGEN_IXIA_PORT2 = ''
+
+TRAFFICGEN_IXNET_LIB_PATH = '/opt/ixnetwork/lib/IxTclNetwork'
+
+# IxNetwork host IP address
+TRAFFICGEN_IXNET_MACHINE = ''
+TRAFFICGEN_IXNET_PORT = ''
+TRAFFICGEN_IXNET_USER = ''
+TRAFFICGEN_IXNET_CHASSIS = ''
+
+# The result directory on $TRAFFICGEN_IXNET_MACHINE
+TRAFFICGEN_IXNET_TESTER_RESULT_DIR = ''
+
+# The result directory on DUT. This needs to map to the same directory
+# as the previous one
+TRAFFICGEN_IXNET_DUT_RESULT_DIR = ''
diff --git a/conf/04_vnf.conf b/conf/04_vnf.conf
new file mode 100644
index 00000000..2603d589
--- /dev/null
+++ b/conf/04_vnf.conf
@@ -0,0 +1,58 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ############################
+# VNF configuration
+# ############################
+QEMU_DIR = ''
+
+# ############################
+# Executables
+# ############################
+
+QEMU_BIN = 'qemu-system-x86_64'
+
+# ############################
+# Guest configuration
+# ############################
+
+# directory which is shared to QEMU guests. Useful for exchanging files
+# between host and guest
+GUEST_SHARE_DIR = '/tmp/qemu_share'
+
+# location of guest disk image
+GUEST_IMAGE = ''
+
+# username for guest image
+GUEST_USERNAME = ''
+
+# password for guest image
+GUEST_PASSWORD = ''
+
+# login username prompt for guest image
+GUEST_PROMPT_LOGIN = ''
+
+# login password prompt for guest image
+GUEST_PROMPT_PASSWORD = ''
+
+# standard prompt for guest image
+GUEST_PROMPT = ''
+
+# log file for qemu
+LOG_FILE_QEMU = 'qemu.log'
+
+# log file for all commands executed on guest(s)
+# multiple guests will result in log files with the guest number appended
+LOG_FILE_GUEST_CMDS = 'guest-cmds.log'
+
diff --git a/conf/05_collector.conf b/conf/05_collector.conf
new file mode 100644
index 00000000..4dcd8162
--- /dev/null
+++ b/conf/05_collector.conf
@@ -0,0 +1,31 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ############################
+# Collector configuration
+# ############################
+
+# ############################
+# Sysmetrics configuration
+# ############################
+
+COLLECTOR = 'LinuxMetrics'
+COLLECTOR_DIR = os.path.join(ROOT_DIR, 'tools/collectors')
+
+# the number of seconds between samples when calculating CPU percentage
+SYSMETRICS_LINUX_METRICS_CPU_SAMPLES_INTERVAL = 5
+
+# log file for sysmetrics
+LOG_FILE_SYS_METRICS = 'system-metrics.log'
+
diff --git a/conf/10_custom.conf b/conf/10_custom.conf
new file mode 100644
index 00000000..95118753
--- /dev/null
+++ b/conf/10_custom.conf
@@ -0,0 +1,44 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# traffic generator to use in tests
+RTE_SDK = ''    # full path to DPDK src dir
+OVS_DIR = ''    # full path to Open vSwitch src dir
+RTE_TARGET = '' # the relevant DPDK build target
+
+# traffic generator to use in tests
+TRAFFICGEN = 'Dummy'
+#TRAFFICGEN = 'IxNet'
+#TRAFFICGEN = 'Ixia'
+
+# Ixia/IxNet configuration
+TRAFFICGEN_IXIA_CARD = ''
+TRAFFICGEN_IXIA_PORT1 = ''
+TRAFFICGEN_IXIA_PORT2 = ''
+TRAFFICGEN_IXIA_LIB_PATH = '/opt/ixos/lib/ixTcl1.0'
+TRAFFICGEN_IXNET_LIB_PATH = '/opt/ixnet/IxTclNetwork'
+
+# Ixia traffic generator
+TRAFFICGEN_IXIA_HOST = ''      # quad dotted ip address
+
+# host where IxNetwork GUI/daemon runs
+TRAFFICGEN_IXNET_MACHINE = ''  # quad dotted ip address
+TRAFFICGEN_IXNET_PORT = ''
+TRAFFICGEN_IXNET_USER = ''
+
+# paths to shared directory for IXIA_HOST and DUT (localhost)
+TRAFFICGEN_IXNET_TESTER_RESULT_DIR = ''
+TRAFFICGEN_IXNET_DUT_RESULT_DIR = ''
+
+TEST_PARAMS = {'packet_sizes':'64'}
diff --git a/conf/__init__.py b/conf/__init__.py
new file mode 100644
index 00000000..0af47adb
--- /dev/null
+++ b/conf/__init__.py
@@ -0,0 +1,141 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Settings and configuration handlers.
+
+Settings will be loaded from several .conf files
+and any user provided settings file.
+"""
+
+# pylint: disable=invalid-name
+
+import os
+import re
+import pprint
+
+class Settings(object):
+    """Holding class for settings.
+    """
+    def __init__(self):
+        pass
+
+    def getValue(self, attr):
+        """Return a settings item value
+        """
+        if attr in self.__dict__:
+            return getattr(self, attr)
+        else:
+            raise AttributeError("%r object has no attribute %r" %
+                                 (self.__class__, attr))
+
+    def __setattr__(self, name, value):
+        """Set a value
+        """
+        # skip non-settings. this should exclude built-ins amongst others
+        if not name.isupper():
+            return
+
+        # we can assume all uppercase keys are valid settings
+        super(Settings, self).__setattr__(name, value)
+
+    def load_from_file(self, path):
+        """Update ``settings`` with values found in module at ``path``.
+        """
+        import imp
+
+        custom_settings = imp.load_source('custom_settings', path)
+
+        for key in dir(custom_settings):
+            if getattr(custom_settings, key) is not None:
+                setattr(self, key, getattr(custom_settings, key))
+
+    def load_from_dir(self, dir_path):
+        """Update ``settings`` with contents of the .conf files at ``path``.
+
+        Each file must be named Nfilename.conf, where N is a single or
+        multi-digit decimal number.  The files are loaded in ascending order of
+        N - so if a configuration item exists in more that one file the setting
+        in the file with the largest value of N takes precedence.
+
+        :param dir_path: The full path to the dir from which to load the .conf
+            files.
+
+        :returns: None
+        """
+        regex = re.compile("^(?P<digit_part>[0-9]+).*.conf$")
+
+        def get_prefix(filename):
+            """
+            Provide a suitable function for sort's key arg
+            """
+            match_object = regex.search(os.path.basename(filename))
+            return int(match_object.group('digit_part'))
+
+        # get full file path to all files & dirs in dir_path
+        file_paths = os.listdir(dir_path)
+        file_paths = [os.path.join(dir_path, x) for x in file_paths]
+
+        # filter to get only those that are a files, with a leading
+        # digit and end in '.conf'
+        file_paths = [x for x in file_paths if os.path.isfile(x) and
+                      regex.search(os.path.basename(x))]
+
+        # sort ascending on the leading digits
+        file_paths.sort(key=get_prefix)
+
+        # load settings from each file in turn
+        for filepath in file_paths:
+            self.load_from_file(filepath)
+
+    def load_from_dict(self, conf):
+        """
+        Update ``settings`` with values found in ``conf``.
+
+        Unlike the other loaders, this is case insensitive.
+        """
+        for key in conf:
+            if conf[key] is not None:
+                setattr(self, key.upper(), conf[key])
+
+    def load_from_env(self):
+        """
+        Update ``settings`` with values found in the environment.
+        """
+        for key in os.environ:
+            setattr(self, key, os.environ[key])
+
+    def __str__(self):
+        """Provide settings as a human-readable string.
+
+        This can be useful for debug.
+
+        Returns:
+            A human-readable string.
+        """
+        return pprint.pformat(self.__dict__)
+
+
+settings = Settings()
+
+
+def get_test_param(key, default=None):
+    """Retrieve value for test param ``key`` if available.
+
+    :param key: Key to retrieve from test params.
+    :param default: Default to return if key not found.
+
+    :returns: Value for ``key`` if found, else ``default``.
+    """
+    test_params = settings.getValue('TEST_PARAMS')
+    return test_params.get(key, default) if test_params else default