Initiate packetization of Testing reporting

Testing reporting provides
- static reporting pages [1] (jinja2)
- landing pages including test case catalogue [2] (angular)

It consumes the Test API to build web pages providing status for
the testing projects (so far functest, yardstick, storperf and
qtip).

Dockerization has been initiated [3]. The goal is to replace the
static page hosted on testresults.opnfv.org by a docker regenerated
and redeployed when reporting code is changed

But the docker and more generally the testing reporting directory
must be refactored to
- manage dependencies properly
- leverage tox (py27, docs, pylint, pep8)

This patch
- setups tox (py27, pep8, pylint, docs)
- integrate the requirement management
- fix pep8 errors
- introduce 1 dummy unit test

This patch does not
- fix pylint errors
- create any doc

[1]: http://testresults.opnfv.org/reporting2/display/index.html
[2]: http://testresults.opnfv.org/reporting2/reporting/index.html
[3]: https://gerrit.opnfv.org/gerrit/#/c/36735/

Change-Id: I4613de7ca7036d6c6bbb8f58ade492b1d673599b
Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>

23 files changed, 0 insertions, 308 deletions

diff --git a/utils/test/reporting/yardstick/img/gauge_0.png b/utils/test/reporting/yardstick/img/gauge_0.png
deleted file mode 100644
index ecefc0e66..000000000
--- a/utils/test/reporting/yardstick/img/gauge_0.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_100.png b/utils/test/reporting/yardstick/img/gauge_100.png
deleted file mode 100644
index e199e1561..000000000
--- a/utils/test/reporting/yardstick/img/gauge_100.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_16.7.png b/utils/test/reporting/yardstick/img/gauge_16.7.png
deleted file mode 100644
index 3e3993c3b..000000000
--- a/utils/test/reporting/yardstick/img/gauge_16.7.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_25.png b/utils/test/reporting/yardstick/img/gauge_25.png
deleted file mode 100644
index 4923659b9..000000000
--- a/utils/test/reporting/yardstick/img/gauge_25.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_33.3.png b/utils/test/reporting/yardstick/img/gauge_33.3.png
deleted file mode 100644
index 364574b4a..000000000
--- a/utils/test/reporting/yardstick/img/gauge_33.3.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_41.7.png b/utils/test/reporting/yardstick/img/gauge_41.7.png
deleted file mode 100644
index 8c3e910fa..000000000
--- a/utils/test/reporting/yardstick/img/gauge_41.7.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_50.png b/utils/test/reporting/yardstick/img/gauge_50.png
deleted file mode 100644
index 2874b9fcf..000000000
--- a/utils/test/reporting/yardstick/img/gauge_50.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_58.3.png b/utils/test/reporting/yardstick/img/gauge_58.3.png
deleted file mode 100644
index beedc8aa9..000000000
--- a/utils/test/reporting/yardstick/img/gauge_58.3.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_66.7.png b/utils/test/reporting/yardstick/img/gauge_66.7.png
deleted file mode 100644
index 93f44d133..000000000
--- a/utils/test/reporting/yardstick/img/gauge_66.7.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_75.png b/utils/test/reporting/yardstick/img/gauge_75.png
deleted file mode 100644
index 9fc261ff8..000000000
--- a/utils/test/reporting/yardstick/img/gauge_75.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_8.3.png b/utils/test/reporting/yardstick/img/gauge_8.3.png
deleted file mode 100644
index 59f86571e..000000000
--- a/utils/test/reporting/yardstick/img/gauge_8.3.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_83.3.png b/utils/test/reporting/yardstick/img/gauge_83.3.png
deleted file mode 100644
index 27ae4ec54..000000000
--- a/utils/test/reporting/yardstick/img/gauge_83.3.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/gauge_91.7.png b/utils/test/reporting/yardstick/img/gauge_91.7.png
deleted file mode 100644
index 280865714..000000000
--- a/utils/test/reporting/yardstick/img/gauge_91.7.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/icon-nok.png b/utils/test/reporting/yardstick/img/icon-nok.png
deleted file mode 100644
index 526b5294b..000000000
--- a/utils/test/reporting/yardstick/img/icon-nok.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/icon-ok.png b/utils/test/reporting/yardstick/img/icon-ok.png
deleted file mode 100644
index 3a9de2e89..000000000
--- a/utils/test/reporting/yardstick/img/icon-ok.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/weather-clear.png b/utils/test/reporting/yardstick/img/weather-clear.png
deleted file mode 100644
index a0d967750..000000000
--- a/utils/test/reporting/yardstick/img/weather-clear.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/weather-few-clouds.png b/utils/test/reporting/yardstick/img/weather-few-clouds.png
deleted file mode 100644
index acfa78398..000000000
--- a/utils/test/reporting/yardstick/img/weather-few-clouds.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/weather-overcast.png b/utils/test/reporting/yardstick/img/weather-overcast.png
deleted file mode 100644
index 4296246d0..000000000
--- a/utils/test/reporting/yardstick/img/weather-overcast.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/img/weather-storm.png b/utils/test/reporting/yardstick/img/weather-storm.png
deleted file mode 100644
index 956f0e20f..000000000
--- a/utils/test/reporting/yardstick/img/weather-storm.png
+++ /dev/null
diff --git a/utils/test/reporting/yardstick/index.html b/utils/test/reporting/yardstick/index.html
deleted file mode 100644
index 488f1421d..000000000
--- a/utils/test/reporting/yardstick/index.html
+++ /dev/null
@@ -1,51 +0,0 @@
- <html>
-  <head>
-    <meta charset="utf-8">
-    <!-- Bootstrap core CSS -->
-    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css" rel="stylesheet">
-    <link href="default.css" rel="stylesheet">
-    <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>
-    <script type="text/javascript" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
-    <script type="text/javascript">
-    $(document).ready(function (){
-        $(".btn-more").click(function() {
-            $(this).hide();
-            $(this).parent().find(".panel-default").show();
-        });
-    })
-    </script>
-  </head>
-    <body>
-    <div class="container">
-      <div class="masthead">
-        <h3 class="text-muted">Yardstick reporting page</h3>
-        <nav>
-          <ul class="nav nav-justified">
-            <li class="active"><a href="http://testresults.opnfv.org/reporting/index.html">Home</a></li>
-            <li><a href="index-status-apex.html">Apex</a></li>
-            <li><a href="index-status-compass.html">Compass</a></li>
-            <li><a href="index-status-fuel.html">Fuel</a></li>
-            <li><a href="index-status-joid.html">Joid</a></li>
-          </ul>
-        </nav>
-      </div>
-<div class="row">
-    <div class="col-md-1"></div>
-    <div class="col-md-10">
-        <div class="page-main">
-            <h2>Yardstick</h2>
-            Yardstick is used in OPNFV for verifying the OPNFV infrastructure and some of the OPNFV features.
-            <br>The Yardstick framework is deployed in several OPNFV community labs.
-            <br>It is installer, infrastructure and application independent.
-
-            <h2>Useful Links</h2>
-            <li><a href="https://wiki.opnfv.org/download/attachments/5734608/yardstick%20in%20depth.pdf?version=1&modificationDate=1463410431000&api=v2">Yardstick in Depth</a></li>
-            <li><a href="https://git.opnfv.org/cgit/yardstick">Yardstick Repo</a></li>
-            <li><a href="https://wiki.opnfv.org/display/yardstick">Yardstick Project</a></li>
-            <li><a href="https://build.opnfv.org/ci/view/yardstick/">Yardstick Jenkins page</a></li>
-            <li><a href="https://jira.opnfv.org/browse/YARDSTICK-119?jql=project%20%3D%20YARDSTICK">JIRA</a></li>
-
-        </div>
-    </div>
-    <div class="col-md-1"></div>
-</div>
diff --git a/utils/test/reporting/yardstick/reporting-status.py b/utils/test/reporting/yardstick/reporting-status.py
deleted file mode 100644
index 12f42ca31..000000000
--- a/utils/test/reporting/yardstick/reporting-status.py
+++ /dev/null
@@ -1,120 +0,0 @@
-#!/usr/bin/python
-#
-# This program and the accompanying materials
-# are made available under the terms of the Apache License, Version 2.0
-# which accompanies this distribution, and is available at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-import datetime
-import jinja2
-import os
-
-import utils.scenarioResult as sr
-from scenarios import config as cf
-
-# manage conf
-import utils.reporting_utils as rp_utils
-
-installers = rp_utils.get_config('general.installers')
-versions = rp_utils.get_config('general.versions')
-PERIOD = rp_utils.get_config('general.period')
-
-# Logger
-logger = rp_utils.getLogger("Yardstick-Status")
-reportingDate = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")
-
-logger.info("*******************************************")
-logger.info("*   Generating reporting scenario status  *")
-logger.info("*   Data retention = %s days              *" % PERIOD)
-logger.info("*                                         *")
-logger.info("*******************************************")
-
-
-# For all the versions
-for version in versions:
-    # For all the installers
-    for installer in installers:
-        # get scenarios results data
-        scenario_results = rp_utils.getScenarioStatus(installer, version)
-        if 'colorado' == version:
-            stable_result = rp_utils.getScenarioStatus(installer,
-                                                       'stable/colorado')
-            for k, v in stable_result.items():
-                if k not in scenario_results.keys():
-                    scenario_results[k] = []
-                scenario_results[k] += stable_result[k]
-        scenario_result_criteria = {}
-
-        for s in scenario_results.keys():
-            if installer in cf.keys() and s in cf[installer].keys():
-                scenario_results.pop(s)
-
-        # From each scenarios get results list
-        for s, s_result in scenario_results.items():
-            logger.info("---------------------------------")
-            logger.info("installer %s, version %s, scenario %s", installer,
-                        version, s)
-
-            ten_criteria = len(s_result)
-            ten_score = 0
-            for v in s_result:
-                ten_score += v
-
-            LASTEST_TESTS = rp_utils.get_config(
-                'general.nb_iteration_tests_success_criteria')
-            four_result = s_result[:LASTEST_TESTS]
-            four_criteria = len(four_result)
-            four_score = 0
-            for v in four_result:
-                four_score += v
-
-            s_status = str(rp_utils.get_percent(four_result, s_result))
-            s_four_score = str(four_score) + '/' + str(four_criteria)
-            s_ten_score = str(ten_score) + '/' + str(ten_criteria)
-            s_score_percent = rp_utils.get_percent(four_result, s_result)
-
-            if '100' == s_status:
-                logger.info(">>>>> scenario OK, save the information")
-            else:
-                logger.info(">>>> scenario not OK, last 4 iterations = %s, \
-                            last 10 days = %s" % (s_four_score, s_ten_score))
-
-            # Save daily results in a file
-            path_validation_file = ("./display/" + version +
-                                    "/yardstick/scenario_history.txt")
-
-            if not os.path.exists(path_validation_file):
-                with open(path_validation_file, 'w') as f:
-                    info = 'date,scenario,installer,details,score\n'
-                    f.write(info)
-
-            with open(path_validation_file, "a") as f:
-                info = (reportingDate + "," + s + "," + installer +
-                        "," + s_ten_score + "," +
-                        str(s_score_percent) + "\n")
-                f.write(info)
-
-            scenario_result_criteria[s] = sr.ScenarioResult(s_status,
-                                                            s_four_score,
-                                                            s_ten_score,
-                                                            s_score_percent)
-
-            logger.info("--------------------------")
-
-        templateLoader = jinja2.FileSystemLoader(".")
-        templateEnv = jinja2.Environment(loader=templateLoader,
-                                         autoescape=True)
-
-        TEMPLATE_FILE = "./yardstick/template/index-status-tmpl.html"
-        template = templateEnv.get_template(TEMPLATE_FILE)
-
-        outputText = template.render(scenario_results=scenario_result_criteria,
-                                     installer=installer,
-                                     period=PERIOD,
-                                     version=version,
-                                     date=reportingDate)
-
-        with open("./display/" + version +
-                  "/yardstick/status-" + installer + ".html", "wb") as fh:
-            fh.write(outputText)
diff --git a/utils/test/reporting/yardstick/scenarios.py b/utils/test/reporting/yardstick/scenarios.py
deleted file mode 100644
index 26e8c8bb0..000000000
--- a/utils/test/reporting/yardstick/scenarios.py
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/usr/bin/python
-#
-# This program and the accompanying materials
-# are made available under the terms of the Apache License, Version 2.0
-# which accompanies this distribution, and is available at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-import requests
-import yaml
-
-import utils.reporting_utils as rp_utils
-
-yardstick_conf = rp_utils.get_config('yardstick.test_conf')
-response = requests.get(yardstick_conf)
-yaml_file = yaml.safe_load(response.text)
-reporting = yaml_file.get('reporting')
-
-config = {}
-
-for element in reporting:
-    name = element['name']
-    scenarios = element['scenario']
-    for s in scenarios:
-        if name not in config:
-            config[name] = {}
-        config[name][s] = True
diff --git a/utils/test/reporting/yardstick/template/index-status-tmpl.html b/utils/test/reporting/yardstick/template/index-status-tmpl.html
deleted file mode 100644
index 77ba9502f..000000000
--- a/utils/test/reporting/yardstick/template/index-status-tmpl.html
+++ /dev/null
@@ -1,110 +0,0 @@
- <html>
-  <head>
-    <meta charset="utf-8">
-    <!-- Bootstrap core CSS -->
-    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css" rel="stylesheet">
-    <link href="../../css/default.css" rel="stylesheet">
-    <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>
-    <script type="text/javascript" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js"></script>
-    <script type="text/javascript" src="http://d3js.org/d3.v2.min.js"></script>
-    <script type="text/javascript" src="../../js/gauge.js"></script>
-    <script type="text/javascript" src="../../js/trend.js"></script>
-    <script>
-        function onDocumentReady() {
-            // Gauge management
-            {% for scenario in scenario_results.keys() -%}
-            var gaugeScenario{{loop.index}} = gauge('#gaugeScenario{{loop.index}}');
-            {%- endfor %}
-            // assign success rate to the gauge
-            function updateReadings() {
-                {% for scenario in scenario_results.keys() -%}
-                 gaugeScenario{{loop.index}}.update({{scenario_results[scenario].getScorePercent()}});
-                 {%- endfor %}
-            }
-            updateReadings();
-        }
-
-        // trend line management
-        d3.csv("./scenario_history.csv", function(data) {
-            // ***************************************
-            // Create the trend line
-            {% for scenario in scenario_results.keys() -%}
-            // for scenario {{scenario}}
-            // Filter results
-                var trend{{loop.index}} = data.filter(function(row) {
-                    return row["scenario"]=="{{scenario}}" && row["installer"]=="{{installer}}";
-                })
-            // Parse the date
-            trend{{loop.index}}.forEach(function(d) {
-                d.date = parseDate(d.date);
-                d.score = +d.score
-            });
-            // Draw the trend line
-            var mytrend = trend("#trend_svg{{loop.index}}",trend{{loop.index}})
-            // ****************************************
-            {%- endfor %}
-        });
-        if ( !window.isLoaded ) {
-            window.addEventListener("load", function() {
-            onDocumentReady();
-            }, false);
-        } else {
-            onDocumentReady();
-        }
-    </script>
-    <script type="text/javascript">
-    $(document).ready(function (){
-        $(".btn-more").click(function() {
-            $(this).hide();
-            $(this).parent().find(".panel-default").show();
-        });
-    })
-    </script>
-  </head>
-    <body>
-    <div class="container">
-      <div class="masthead">
-          <h3 class="text-muted">Yardstick status page ({{version}}, {{date}})</h3>
-        <nav>
-          <ul class="nav nav-justified">
-            <li class="active"><a href="http://testresults.opnfv.org/reporting/index.html">Home</a></li>
-            <li><a href="status-apex.html">Apex</a></li>
-            <li><a href="status-compass.html">Compass</a></li>
-            <li><a href="status-fuel.html">Fuel</a></li>
-            <li><a href="status-joid.html">Joid</a></li>
-          </ul>
-        </nav>
-      </div>
-<div class="row">
-    <div class="col-md-1"></div>
-    <div class="col-md-10">
-        <div class="page-header">
-            <h2>{{installer}}</h2>
-        </div>
-
-        <div class="scenario-overview">
-            <div class="panel-heading"><h4><b>List of last scenarios ({{version}}) run over the last {{period}} days </b></h4></div>
-                <table class="table">
-                    <tr>
-                        <th width="40%">Scenario</th>
-                        <th width="20%">Status</th>
-                        <th width="20%">Trend</th>
-                        <th width="10%">Last 4 Iterations</th>
-                        <th width="10%">Last 10 Days</th>
-                    </tr>
-                        {% for scenario,result in scenario_results.iteritems() -%}
-                            <tr class="tr-ok">
-                                <td>{{scenario}}</td>
-                                <td><div id="gaugeScenario{{loop.index}}"></div></td>
-                                <td><div id="trend_svg{{loop.index}}"></div></td>
-                                <td>{{scenario_results[scenario].getFourDaysScore()}}</td>
-                                <td>{{scenario_results[scenario].getTenDaysScore()}}</td>
-                            </tr>
-                        {%- endfor %}
-                </table>
-        </div>
-
-
-    </div>
-    <div class="col-md-1"></div>
-</div>