Initial reporting implemenation

Reporting from TOIT was merged and improved. Default template was modified
to support any testcase and to show more details about system environment.
Affected files:
* docs/NEWS.md
* testcases/testcase.py
* tools/report/__init__.py
* tools/report/report.jinja
* tools/report/report.py
* tools/systeminfo.py
* vsperf

JIRA: VSPERF-71
Change-Id: I4dc84ca69e5c292eae1f8dede1411c06ae3ef8af
Signed-off-by: Martin Klozik (martinx.klozik@intel.com)
Reviewed-by: Maryam Tahhan <maryam.tahhan@intel.com>
Reviewed-by: Billy O Mahony <billy.o.mahony@intel.com>

7 files changed, 454 insertions, 10 deletions

diff --git a/docs/NEWS.md b/docs/NEWS.md
index 618dbebb..604328ab 100644
--- a/docs/NEWS.md
+++ b/docs/NEWS.md
@@ -1,3 +1,10 @@
+#August 2015
+
+## New
+
+* Backport and enhancement of reporting
+
+
 #July 2015
 
 ## New
@@ -40,6 +47,5 @@ once the community has digested the initial release.
 
 ## Missing
 
-* Report generation is currently disabled
 * xmlunit output is  currently disabled
 * VNF support.
diff --git a/testcases/testcase.py b/testcases/testcase.py
index 77d5992d..6191a117 100644
--- a/testcases/testcase.py
+++ b/testcases/testcase.py
@@ -19,6 +19,7 @@ import os
 import logging
 from collections import OrderedDict
 
+from core.results.results_constants import ResultsConstants
 import core.component_factory as component_factory
 from core.loader import Loader
 
@@ -39,7 +40,7 @@ class TestCase(object):
         self.name = cfg['Name']
         self.desc = cfg.get('Description', 'No description given.')
         self._traffic_type = cfg['Traffic Type']
-        self._deployment = cfg['Deployment']
+        self.deployment = cfg['Deployment']
         self._collector = cfg['Collector']
         self._bidir = cfg['biDirectional']
         self._frame_mod = cfg.get('Frame Modification', None)
@@ -61,10 +62,10 @@ class TestCase(object):
             self._traffic_type,
             loader.get_trafficgen_class())
         vnf_ctl = component_factory.create_vnf(
-            self._deployment,
+            self.deployment,
             loader.get_vnf_class())
         vswitch_ctl = component_factory.create_vswitch(
-            self._deployment,
+            self.deployment,
             loader.get_vswitch_class(),
             self._bidir)
         collector_ctl = component_factory.create_collector(
@@ -98,13 +99,28 @@ class TestCase(object):
         self._logger.debug("Collector Results:")
         self._logger.debug(collector_ctl.get_results())
 
+        output_file = "result_" + self.name + "_" + self.deployment +".csv"
 
-        output_file = "result_" + self.name + "_" + self._deployment +".csv"
-
-        self._write_result_to_file(
-            traffic_ctl.get_results(),
+        TestCase._write_result_to_file(
+            self._append_results(traffic_ctl.get_results()),
             os.path.join(self._results_dir, output_file))
 
+    def _append_results(self, results):
+        """
+        Method appends mandatory Test Case results to list of dictionaries.
+
+        :param results: list of dictionaries which contains results from
+                traffic generator.
+
+        :returns: modified list of dictionaries.
+        """
+        for item in results:
+            item[ResultsConstants.ID] = self.name
+            item[ResultsConstants.DEPLOYMENT] = self.deployment
+
+        return results
+
+
     @staticmethod
     def _write_result_to_file(results, output):
         """Write list of dictionaries to a CSV file.
diff --git a/tools/report/__init__.py b/tools/report/__init__.py
new file mode 100644
index 00000000..e69de29b
--- /dev/null
+++ b/tools/report/__init__.py
diff --git a/tools/report/report.jinja b/tools/report/report.jinja
new file mode 100644
index 00000000..63d30fc0
--- /dev/null
+++ b/tools/report/report.jinja
@@ -0,0 +1,132 @@
+# CHARACTERIZE VSWITCH PERFORMANCE FOR TELCO NFV USE CASES LEVEL TEST REPORT
+
+## Table of Contents
+
+- [1. Introduction](#Introduction)
+  - [1.1. Document identifier](#DocId)
+  - [1.2. Scope](#Scope)
+  - [1.3. References](#References)
+- [2. Details of the Level Test Report](#DetailsoftheLevelTestReport)
+  - [2.1. Overview of test results](#OverviewOfTestResults)
+  - [2.2. Detailed test results](#DetailedTestResults)
+  - [2.3. Rationale for decisions](#RationaleForDecisions)
+  - [2.4. Conclusions and recommendations](#ConclusionsandRecommendations)
+- [3. General](#General)
+  - [3.1. Glossary](#Glossary)
+  - [3.2. Document change procedures and history](#DocChangeProceduresandHistory)
+
+---
+
+<a name="Introduction"></a>
+## 1. Introduction
+
+The objective of the OPNFV project titled **“Characterise vSwitch Performance for Telco NFV Use Cases”**, is to evaluate a virtual switch to identify its suitability for a Telco Network Function Virtualization (NFV) environment. As well as this, the project aims to identify any gaps or bottlenecks in order to drive architectural changes to improve virtual switch performance and determinism. The purpose of this document is to summarize the results of the tests carried out on the virtual switch in the Network Function Virtualization Infrastructure (NFVI) and, from these results, provide evaluations and recommendations for the virtual switch. Test results will be outlined in [Details of the Level Test Report](#DetailsoftheLevelTestReport), preceded by the [Document Identifier](#DocId), [Scope](#Scope) and [References](#References).
+
+This document is currently in draft form.
+
+<a name="DocId"></a>
+### 1.1. Document identifier
+
+The document id will be used to uniquely identify versions of the LTR. The format for the document id will be: `OPNFV_vswitchperf_LTR_ver_NUM_MONTH_YEAR_AUTHOR_STATUS`, where by the AUTHOR field should be replaced with the initials of the author and the status is one of: DRAFT, REVIEWED, CORRECTED or FINAL. The document id for this version of the LTR is: `OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT`.
+
+<a name="Scope"></a>
+### 1.2. Scope
+
+The scope of this report is to detail the results of the tests that have been performed on the virtual switch. This report will also evaluate the results of these tests and, based on these evaluations, provide recommendations on the suitability of the virtual switch for use in a Telco NFV environment.
+
+<a name="References"></a>
+### 1.3. References
+
+- `OPNFV_vswitchperf_LTD_ver_1.6_Jan_15_DRAFT`
+
+---
+
+<a name="DetailsoftheLevelTestReport"></a>
+## 2. Details of the Level Test Report
+
+This section provides an overview of the test results ([Section 2.1.](#OveriewOfTestResults)) as well as detailed test results for each test ([Section 2.2.](#DetailedTestResults)). Also included are the rationale used to evaluate each test ([Section 2.3.](#RationaleForDecisions)) and the conclusions and recommendations for each test ([Section 2.4.](#ConclusionsandRecommendations)).
+
+<a name="OverviewOfTestResults"></a>
+### 2.1. Overview of test results
+
+##### Test Environment
+
+Below is the environment that the test was performed in:
+
+- OS: {{tests[0].env.os}}
+- Kernel Version: {{tests[0].env.kernel}}
+- NIC(s): {{tests[0].env.nic}}
+- Board: {{tests[0].env.platform}}
+- CPU: {{tests[0].env.cpu}}
+- CPU cores: {{tests[0].env.cpu_cores}}
+- Memory: {{tests[0].env.memory}}
+- Virtual Switch Set-up: {{tests[0].deployment}}
+- IxNetwork: {{tests[0].env.ixnetwork_ver}}
+
+For each test, a summary of the key test results is provided.
+{% for test in tests %}
+#### Test ID: {{ test.ID }}
+
+Below are test details:
+
+- Test ID: {{ "%s"|format(test.id) }}
+- Description: {{ "%s"|format(test.conf['Description']) }}
+- Deployment: {{ "%s"|format(test.deployment) }}
+- Traffic type: {{ "%s"|format(test.result['type']) }}
+- Packet size: {{ "%s"|format(test.result['packet_size']) }}
+- Bidirectional : {{ "%s"|format(test.conf['biDirectional']) }}
+{% endfor %}
+
+<a name="DetailedTestResults"></a>
+### 2.2. Detailed test results
+
+A detailed summary of the main results for each test is outlined below.
+{% for test in tests %}
+#### Test ID: {{ test.ID }}
+
+##### Results/Metrics Collected
+
+The following are the metrics obtained during this test:
+
+|              Metric            |             Result             |
+| ------------------------------ | ------------------------------ |
+{%- for item, value in test.result.items() %}
+| {{ "%-30s | %30s |"|format(item,value)}}
+{%- endfor %}
+
+##### Anomalies
+
+No anomalies were detected during the course of this test.
+
+##### Testing Activities/Events
+
+There were no significant testing activities for this test.
+{% endfor %}
+<a name="RationaleForDecisions"></a>
+### 2.3. Rationale for decisions
+
+TODO.
+
+<a name="ConclusionsandRecommendations"></a>
+### 2.4. Conclusions and recommendations
+
+TODO.
+
+-----
+
+<a name="General"></a>
+## 3. General
+
+<a name="Glossary"></a>
+### 3.1. Glossary
+
+- NFV - Network Function Virtualization
+- Mbps - 1,000,000bps
+
+<a name="DocChangeProceduresandHistory"></a>
+### 3.2. Document change procedures and history
+
+| Document ID | Author | Date Modified |
+| ----------- |------- | ------------- |
+| `OPNFV_vswitchperf_LTR_ver_1.0_Jan_15_CN_DRAFT` | Christopher Nolan | 23/01/2015
+| `OPNFV_vswitchperf_LTR_ver_1.1_Jan_15_CN_DRAFT` | Christopher Nolan | 28/01/2015
diff --git a/tools/report/report.py b/tools/report/report.py
new file mode 100644
index 00000000..8d213297
--- /dev/null
+++ b/tools/report/report.py
@@ -0,0 +1,142 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+vSwitch Characterization Report Generation.
+
+Generate reports in format defined by X.
+"""
+
+import sys
+import os
+import jinja2
+import csv
+import logging
+
+from collections import OrderedDict
+from core.results.results_constants import ResultsConstants
+from conf import settings
+from tools import systeminfo
+
+_TEMPLATE_FILE = 'report.jinja'
+_ROOT_DIR = os.path.normpath(os.path.dirname(os.path.realpath(__file__)))
+
+
+def _get_env():
+    """
+    Get system configuration.
+
+    :returns: Return a dictionary of the test environment.
+              The following is an example return value:
+               {'kernel': '3.10.0-229.4.2.el7.x86_64',
+                'os': 'OS Version',
+                'cpu': ' CPU 2.30GHz',
+                'platform': '[2 sockets]',
+                'nic': 'NIC'}
+
+    """
+
+    env = {
+        'os': systeminfo.get_os(),
+        'kernel': systeminfo.get_kernel(),
+        'nic': systeminfo.get_nic(),
+        'cpu': systeminfo.get_cpu(),
+        'cpu_cores': systeminfo.get_cpu_cores(),
+        'memory' : systeminfo.get_memory(),
+        'platform': systeminfo.get_platform(),
+    }
+
+    return env
+
+
+def _get_results(results_file):
+    """Get results from tests.
+
+    Get test results from a CSV file and return it as a list
+    of dictionaries for each row of data.
+
+    :param results_file: Path of the CSV results file
+
+    :returns: List of test results
+    """
+    with open(results_file, 'r') as csvfile:
+        reader = csv.reader(csvfile, delimiter=',')
+        result = []
+        res_head = next(reader)
+        for res_row in reader:
+            result.append(OrderedDict(zip(list(res_head), list(res_row))))
+
+    return result
+
+
+def generate(testcases, input_file):
+    """Generate actual report.
+
+    Generate a Markdown-formatted file using results of tests and some
+    parsed system info.
+
+    :param input_file: Path to CSV results file
+
+    :returns: Path to generated report
+    """
+    output_file = '.'.join([os.path.splitext(input_file)[0], 'md'])
+
+    template_loader = jinja2.FileSystemLoader(searchpath=_ROOT_DIR)
+    template_env = jinja2.Environment(loader=template_loader)
+    template = template_env.get_template(_TEMPLATE_FILE)
+
+    tests = []
+    try:
+        for result in _get_results(input_file):
+            test_config = {}
+            for tc_conf in testcases:
+                if tc_conf['Name'] == result[ResultsConstants.ID]:
+                    test_config = tc_conf
+                    break
+
+            # remove id and deployment from results but store their values
+            tc_id = result[ResultsConstants.ID]
+            tc_deployment = result[ResultsConstants.DEPLOYMENT]
+            del result[ResultsConstants.ID]
+            del result[ResultsConstants.DEPLOYMENT]
+
+            # pass test results, env details and configuration to template
+            tests.append({
+                'ID': tc_id.upper(),
+                'id': tc_id,
+                'deployment': tc_deployment,
+                'conf': test_config,
+                'result': result,
+                'env': _get_env(),
+            })
+
+        template_vars = {
+            'tests': tests,
+        }
+
+        output_text = template.render(template_vars)
+        with open(output_file, 'w') as file_:
+            file_.write(output_text)
+            logging.info('Test report written to "%s"', output_file)
+
+    except KeyError:
+        logging.info("Report: Ignoring file (Wrongly defined columns): %s", (input_file))
+        raise
+    return output_file
+
+
+if __name__ == '__main__':
+    settings.load_from_dir('conf')
+    OUT = generate(sys.argv[1])
+    print('Test report written to "%s"...' % OUT)
diff --git a/tools/systeminfo.py b/tools/systeminfo.py
new file mode 100644
index 00000000..19c5d16e
--- /dev/null
+++ b/tools/systeminfo.py
@@ -0,0 +1,139 @@
+# Copyright 2015 Intel Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tools for access to OS details
+"""
+
+import os
+import platform
+import subprocess
+import locale
+
+from conf import settings
+
+def get_os():
+    """Get distro name.
+
+    :returns: Return distro name as a string
+    """
+    return ' '.join(platform.dist())
+
+def get_kernel():
+    """Get kernel version.
+
+    :returns: Return kernel version as a string
+    """
+    return platform.release()
+
+def get_cpu():
+    """Get CPU information.
+
+    :returns: Return CPU information as a string
+    """
+    with open('/proc/cpuinfo') as file_:
+        for line in file_:
+            if not line.strip():
+                continue
+            if not line.rstrip('\n').startswith('model name'):
+                continue
+
+            return line.rstrip('\n').split(':')[1]
+
+def get_nic():
+    """Get NIC(s) information.
+
+    :returns: Return NIC(s) information as a string
+    """
+    nics = []
+    output = subprocess.check_output('lspci', shell=True)
+    output = output.decode(locale.getdefaultlocale()[1])
+    for line in output.split('\n'):
+        for nic_pciid in settings.getValue('WHITELIST_NICS'):
+            if line.startswith(nic_pciid):
+                nics.append(''.join(line.split(':')[2:]).strip())
+    return ', '.join(nics).strip()
+
+def get_platform():
+    """Get platform information.
+
+    Currently this is the motherboard vendor, name and socket
+    count.
+
+    :returns: Return platform information as a string
+    """
+    output = []
+
+    with open('/sys/class/dmi/id/board_vendor', 'r') as file_:
+        output.append(file_.readline().rstrip())
+
+    with open('/sys/class/dmi/id/board_name', 'r') as file_:
+        output.append(file_.readline().rstrip())
+
+    num_nodes = len([name for name in os.listdir(
+        '/sys/devices/system/node/') if name.startswith('node')])
+    output.append(''.join(['[', str(num_nodes), ' sockets]']))
+
+    return ' '.join(output).strip()
+
+def get_cpu_cores():
+    """Get number of CPU cores.
+
+    :returns: Return number of CPU cores
+    """
+    cores = 0
+    with open('/proc/cpuinfo') as file_:
+        for line in file_:
+            if line.rstrip('\n').startswith('processor'):
+                cores += 1
+            continue
+
+    # this code must be executed by at leat one core...
+    if cores < 1:
+        cores = 1
+    return cores
+
+def get_memory():
+    """Get memory information.
+
+    :returns: amount of system memory as string together with unit
+    """
+    with open('/proc/meminfo') as file_:
+        for line in file_:
+            if not line.strip():
+                continue
+            if not line.rstrip('\n').startswith('MemTotal'):
+                continue
+
+            return line.rstrip('\n').split(':')[1].strip()
+
+def get_memory_bytes():
+    """Get memory information in bytes
+
+    :returns: amount of system memory
+    """
+    mem_list = get_memory().split(' ')
+    mem = float(mem_list[0].strip())
+    if mem_list.__len__() > 1:
+        unit = mem_list[1].strip().lower()
+        if unit == 'kb':
+            mem *= 1024
+        elif unit == 'mb':
+            mem *= 1024 ** 2
+        elif unit == 'gb':
+            mem *= 1024 ** 3
+        elif unit == 'tb':
+            mem *= 1024 ** 4
+
+    return int(mem)
+
diff --git a/vsperf b/vsperf
index 67141a7e..4510e121 100755
--- a/vsperf
+++ b/vsperf
@@ -32,6 +32,7 @@ sys.dont_write_bytecode = True
 from conf import settings
 from core.loader import Loader
 from testcases import TestCase
+from tools.report import report
 from tools import tasks
 from tools.collectors import collector
 from tools.pkt_gen import trafficgen
@@ -373,7 +374,7 @@ def main():
         exit()
 
     # create results directory
-    if not os.path.exists(results_dir):
+    if not os.path.exists(results_path):
         logger.info("Creating result directory: "  + results_path)
         os.makedirs(results_path)
 
@@ -397,8 +398,16 @@ def main():
 
     #remove directory if no result files were created.
     if os.path.exists(results_path):
-        if os.listdir(results_path) == []:
+        files_list = os.listdir(results_path)
+        if files_list == []:
             shutil.rmtree(results_path)
+        else:
+            for file in files_list:
+                # generate report from all csv files
+                if file[-3:] == 'csv':
+                    results_csv = os.path.join(results_path, file)
+                    if os.path.isfile(results_csv) and os.access(results_csv, os.R_OK):
+                        report.generate(testcases, results_csv)
 
 if __name__ == "__main__":
     main()