Added doc directories

Change-Id: I671d7c3ad4f4e5e476c98f53780d867dc94b3089
Signed-off-by: Stuart Mackie <wsmackie@juniper.net>

5 files changed, 757 insertions, 0 deletions

diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
new file mode 100644
index 0000000..8668859
--- /dev/null
+++ b/docs/testing/developer/devguide/index.rst
@@ -0,0 +1,353 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
+***********************
+Testing developer guide
+***********************
+
+.. toctree::
+   :numbered:
+   :maxdepth: 2
+
+
+============
+Introduction
+============
+
+The OPNFV testing ecosystem is wide.
+
+The goal of this guide consists in providing some guidelines for new developers
+involved in test areas.
+
+For the description of the ecosystem, see `[1]`_.
+
+
+=================
+Developer journey
+=================
+
+Be involved in the testing group
+================================
+
+Best practices
+==============
+
+Unit tests
+----------
+
+Dockerization
+-------------
+
+API
+---
+
+CLI
+---
+
+Traffic generators
+------------------
+
+Towards a pseudo micro services approach
+----------------------------------------
+
+======================================
+Testing group configuration parameters
+======================================
+
+
+Testing categories
+==================
+
+The testing group defined several categories also known as tiers. These
+categories can be used to group test suites.
+
++----------------+-------------------------------------------------------------+
+| Healthcheck    | Simple and quick healthcheck tests case                     |
++----------------+-------------------------------------------------------------+
+| Smoke          | Set of smoke test cases/suites to validate the release      |
++----------------+-------------------------------------------------------------+
+| Features       | Test cases that validate a specific feature on top of OPNFV.|
+|                | Those come from Feature projects and need a bit of support  |
+|                | for integration                                             |
++----------------+-------------------------------------------------------------+
+| Components     | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)|
+|                | It may extend smoke tests                                   |
++----------------+-------------------------------------------------------------+
+| Performance    | Performance qualification                                   |
++----------------+-------------------------------------------------------------+
+| VNF            | Test cases related to deploy an open source VNF including   |
+|                | an orchestrator                                             |
++----------------+-------------------------------------------------------------+
+| Stress         | Stress and robustness tests                                 |
++----------------+-------------------------------------------------------------+
+| In Service     | In service testing                                          |
++----------------+-------------------------------------------------------------+
+
+Testing domains
+===============
+
+The domains deal with the technical scope of the tests. It shall correspond to
+domains defined for the certification program:
+
+ * compute
+ * network
+ * storage
+ * hypervisor
+ * container
+ * vim
+ * mano
+ * vnf
+ * ...
+
+Testing coverage
+=================
+One of the goals of the testing working group is to identify the poorly covered
+areas and avoid testing overlap.
+Ideally based on the declaration of the test cases, through the tags, domains
+and tier fields, it shall be possible to create heuristic maps.
+
+
+==============================
+Testing group generic enablers
+==============================
+
+
+TestAPI framework
+=================
+
+The OPNFV testing group created a test collection database to collect
+the test results from CI:
+
+
+ http://testresults.opnfv.org/test/swagger/spec.html
+
+Any test project running on any lab integrated in CI can push the
+results to this database.
+This database can be used to see the evolution of the tests and compare
+the results versus the installers, the scenarios or the labs.
+It is used to produce a dashboard with the current test status of the project.
+
+
+Overall Architecture
+--------------------
+The Test result management can be summarized as follows::
+
+  +-------------+    +-------------+    +-------------+
+  |             |    |             |    |             |
+  |   Test      |    |   Test      |    |   Test      |
+  | Project #1  |    | Project #2  |    | Project #N  |
+  |             |    |             |    |             |
+  +-------------+    +-------------+    +-------------+
+           |               |               |
+           V               V               V
+       +-----------------------------------------+
+       |                                         |
+       |         Test Rest API front end         |
+       |                                         |
+       +-----------------------------------------+
+           A                |
+           |                V
+           |     +-------------------------+
+           |     |                         |
+           |     |    Test Results DB      |
+           |     |         Mongo DB        |
+           |     |                         |
+           |     +-------------------------+
+           |
+           |
+     +----------------------+
+     |                      |
+     |    test Dashboard    |
+     |                      |
+     +----------------------+
+
+TestAPI description
+-------------------
+The TestAPI is used to declare pods, projects, test cases and test
+results. Pods are the sets of bare metal or virtual servers and networking
+equipments used to run the tests.
+
+The results pushed in the database are related to pods, projects and test cases.
+If you try to push results of test done on non referenced pod, the API will
+return an error message.
+
+An additional method dashboard has been added to post-process
+the raw results in release Brahmaputra (deprecated in Colorado).
+
+The data model is very basic, 5 objects are created:
+
+  * Pods
+  * Projects
+  * Testcases
+  * Results
+  * Scenarios
+
+The code of the API is hosted in the releng repository `[6]`_.
+The static documentation of the API can be found at `[7]`_.
+The TestAPI has been dockerized and may be installed locally in your
+lab. See `[15]`_ for details.
+
+The deployment of the TestAPI has been automated.
+A jenkins job manages:
+
+  * the unit tests of the TestAPI
+  * the creation of a new docker file
+  * the deployment of the new TestAPI
+  * the archive of the old TestAPI
+  * the backup of the Mongo DB
+
+TestAPI Authorization
+~~~~~~~~~~~~~~~~~~~~~
+
+PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs
+to be added in the request using a header 'X-Auth-Token' for access to the database.
+
+e.g::
+    headers['X-Auth-Token']
+
+The value of the header i.e the token can be accessed in the jenkins environment variable
+*TestApiToken*. The token value is added as a masked password.
+
+.. code-block:: python
+
+    headers['X-Auth-Token'] = os.environ.get('TestApiToken')
+
+The above example is in Python. Token based authentication has been added so that only ci pods
+jenkins job can have access to the database.
+
+Please note that currently token authorization is implemented but is not yet enabled.
+
+Reporting
+=========
+
+An automatic reporting page has been created in order to provide a
+consistent view of the scenarios.
+
+In this page, each scenario is evaluated according to test criteria.
+The code for the automatic reporting is available at `[8]`_.
+
+The results are collected from the centralized database every day and,
+per scenario. A score is calculated based on the results from the last
+10 days.
+
+Dashboard
+=========
+
+Dashboard is used to provide a consistent view of the results collected in CI.
+The results showed on the dashboard are post processed from the Database,
+which only contains raw results.
+
+It can be used in addition of the reporting page (high level view) to allow
+the creation of specific graphs according to what the test owner wants to show.
+
+In Brahmaputra, a basic home made dashboard was created in Functest.
+In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex
+graphs).
+Since Danube, the testing community decided to adopt ELK framework and to rely
+on bitergia. It was not implemented for Danube but it is planned for Euphrates.
+
+Bitergia already provides a dashboard for code and infrastructure.
+A new Test tab will be added. The dataset will be built by consuming
+the TestAPI.
+
+See `[3]`_ for details.
+
+
+=======
+How TOs
+=======
+
+Where can I find information on the different test projects?
+===========================================================
+
+
+How can I contribute to a test project?
+=======================================
+
+
+Where can I find hardware resources?
+====================================
+
+
+How do I integrate my tests in CI?
+==================================
+
+
+How to declare my tests in the test Database?
+=============================================
+
+
+How to push your results into the Test Database?
+================================================
+
+The test database is used to collect test results. By default it is
+enabled only for CI tests from Production CI pods.
+
+Please note that it is possible to create your own local database.
+
+A dedicated database is for instance created for each plugfest.
+
+The architecture and associated API is described in previous chapter.
+If you want to push your results from CI, you just have to call the API
+at the end of your script.
+
+You can also reuse a python function defined in functest_utils.py `[5]`_
+
+
+Where can I find the documentation on the test API?
+===================================================
+
+http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+
+I have tests, to which category should I declare them?
+======================================================
+
+
+
+The main ambiguity could be between features and VNF.
+In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
+feature you introduced.
+We recommend to declare your test in the feature category.
+
+VNF category is really dedicated to test including:
+
+ * creation of resources
+ * deployement of an orchestrator/VNFM
+ * deployment of the VNF
+ * test of the VNFM
+ * free resources
+
+The goal is not to study a particular feature on the infrastructure but to have
+a whole end to end test of a VNF automatically deployed in CI.
+Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
+jobs and use to get a scenario score.
+
+Where are the logs of CI runs?
+==============================
+
+Logs and configuration files can be pushed to artifact server from the CI under
+http://artifacts.opnfv.org/<project name>
+
+
+==========
+References
+==========
+
+_`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html
+
+_`[2]`: http://www.opnfv.org
+
+_`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
+
+_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
+
+_`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+
+_`[6]`: https://git.opnfv.org/functest/tree/releng
+
+_`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+IRC support chan: #opnfv-testperf
diff --git a/docs/testing/ecosystem/index.rst b/docs/testing/ecosystem/index.rst
new file mode 100644
index 0000000..f51fa19
--- /dev/null
+++ b/docs/testing/ecosystem/index.rst
@@ -0,0 +1,14 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Christopher Price (Ericsson AB)
+
+========================
+Test Framework Overview
+========================
+
+.. toctree::
+   :maxdepth: 2
+
+   ./abstract
+   ./overview
+
diff --git a/docs/testing/ecosystem/overview.rst b/docs/testing/ecosystem/overview.rst
new file mode 100644
index 0000000..ed1657c
--- /dev/null
+++ b/docs/testing/ecosystem/overview.rst
@@ -0,0 +1,274 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
+=============
+OPNFV testing
+=============
+
+Introduction
+============
+
+Testing is one of the key activities in OPNFV and includes unit, feature, component, system
+level testing for development, automated deployment, performance characterization or stress
+testing.
+
+Test projects are dedicated to provide frameworks, tooling and test-cases categorized as
+functional, performance or compliance testing. Test projects fulfill different roles such as
+verifying VIM functionality, benchmarking components and platforms or analysis of measured
+KPIs for the scenarios released in OPNFV.
+
+Feature projects also provide their own test suites that either run independently or within a
+test project.
+
+This document details the OPNFV testing ecosystem, describes common test components used
+by individual OPNFV projects and provides links to project specific documentation.
+
+
+OPNFV testing ecosystem
+=======================
+
+The testing projects
+--------------------
+
+The OPNFV testing projects may be summarized as follows:
+
+.. figure:: ../../images/OPNFV_testing_working_group.png
+   :align: center
+   :alt: Overview of OPNFV Testing projects
+
+The major testing projects are described in the table below:
+
++----------------+---------------------------------------------------------+
+|  Project       |   Description                                           |
++================+=========================================================+
+|  Bottlenecks   | This project aims to find system bottlenecks by testing |
+|                | and verifying OPNFV infrastructure in a staging         |
+|                | environment before committing it to a production        |
+|                | environment. Instead of debugging a deployment in       |
+|                | production environment, an automatic method for         |
+|                | executing benchmarks which plans to validate the        |
+|                | deployment during staging is adopted. This project      |
+|                | forms a staging framework to find bottlenecks and to do |
+|                | analysis of the OPNFV infrastructure.                   |
++----------------+---------------------------------------------------------+
+| CPerf          | SDN Controller benchmarks and performance testing,      |
+|                | applicable to controllers in general. Collaboration of  |
+|                | upstream controller testing experts, external test tool |
+|                | developers and the standards community. Primarily       |
+|                | contribute to upstream/external tooling, then add jobs  |
+|                | to run those tools on OPNFV's infrastructure.           |
++----------------+---------------------------------------------------------+
+| Dovetail       | This project intends to define and provide a set of     |
+|                | OPNFV related validation criteria that will provide     |
+|                | input for the evaluation of the use of OPNFV trademarks.|
+|                | The dovetail project is executed with the guidance and  |
+|                | oversight of the Compliance and Certification committee |
+|                | and work to secure the goals of the C&C committee for   |
+|                | each release. The project intends to incrementally      |
+|                | define qualification criteria that establish the        |
+|                | foundations of how we are able to measure the ability to|
+|                | utilize the OPNFV platform, how the platform itself     |
+|                | should behave, and how applications may be deployed on  |
+|                | the platform.                                           |
++----------------+---------------------------------------------------------+
+| Functest       | This project deals with the functional testing of the   |
+|                | VIM and NFVI. It leverages several upstream test suites |
+|                | (OpenStack, ODL, ONOS, etc.) and can be used by feature |
+|                | project to launch feature test suites in CI/CD.         |
+|                | The project is used for scenario validation.            |
++----------------+---------------------------------------------------------+
+| Qtip           | QTIP as the project for "Platform Performance           |
+|                | Benchmarking" in OPNFV aims to provide user a simple    |
+|                | indicator for performance, supported by comprehensive   |
+|                | testing data and transparent calculation formula.       |
+|                | It provides a platform with common services for         |
+|                | performance benchmarking which helps users to build     |
+|                | indicators by themselves with ease.                     |
++----------------+---------------------------------------------------------+
+| Storperf       | The purpose of this project is to provide a tool to     |
+|                | measure block and object storage performance in an NFVI.|
+|                | When complemented with a characterization of typical VF |
+|                | storage performance requirements, it can provide        |
+|                | pass/fail thresholds for test, staging, and production  |
+|                | NFVI environments.                                      |
++----------------+---------------------------------------------------------+
+| VSperf         | This project provides a framework for automation of NFV |
+|                | data-plane performance testing and benchmarking. The    |
+|                | NFVI fast-path includes switch technology and network   |
+|                | with physical and virtual interfaces. VSperf can be     |
+|                | used to evaluate the suitability of different Switch    |
+|                | implementations and features, quantify data-path        |
+|                | performance and optimize platform configurations.       |
++----------------+---------------------------------------------------------+
+| Yardstick      | The goal of the Project is to verify the infrastructure |
+|                | compliance when running VNF applications. NFV Use Cases |
+|                | described in ETSI GS NFV 001 show a large variety of    |
+|                | applications, each defining specific requirements and   |
+|                | complex configuration on the underlying infrastructure  |
+|                | and test tools.The Yardstick concept decomposes typical |
+|                | VNF work-load performance metrics into a number of      |
+|                | characteristics/performance vectors, which each of them |
+|                | can be represented by distinct test-cases.              |
++----------------+---------------------------------------------------------+
+
+
+===================================
+The testing working group resources
+===================================
+
+The assets
+==========
+
+Overall Architecture
+--------------------
+The Test result management can be summarized as follows::
+
+  +-------------+    +-------------+    +-------------+
+  |             |    |             |    |             |
+  |   Test      |    |   Test      |    |   Test      |
+  | Project #1  |    | Project #2  |    | Project #N  |
+  |             |    |             |    |             |
+  +-------------+    +-------------+    +-------------+
+           |               |               |
+           V               V               V
+       +---------------------------------------------+
+       |                                             |
+       |           Test Rest API front end           |
+       |    http://testresults.opnfv.org/test        |
+       |                                             |
+       +---------------------------------------------+
+           ^                |                     ^
+           |                V                     |
+           |     +-------------------------+      |
+           |     |                         |      |
+           |     |    Test Results DB      |      |
+           |     |         Mongo DB        |      |
+           |     |                         |      |
+           |     +-------------------------+      |
+           |                                      |
+           |                                      |
+     +----------------------+        +----------------------+
+     |                      |        |                      |
+     | Testing Dashboards   |        |      Landing page    |
+     |                      |        |                      |
+     +----------------------+        +----------------------+
+
+
+The testing databases
+---------------------
+A Mongo DB Database has been introduced for the Brahmaputra release.
+The following collections are declared in this database:
+ * pods: the list of pods used for production CI
+ * projects: the list of projects providing test cases
+ * testcases: the test cases related to a given project
+ * results: the results of the test cases
+ * scenarios: the OPNFV scenarios tested in CI
+
+This database can be used by any project through the testapi.
+Please note that projects may also use additional databases. This database is
+mainly use to colelct CI results and scenario trust indicators.
+
+This database is also cloned for OPNFV Plugfest.
+
+
+The test API
+------------
+The Test API is used to declare pods, projects, test cases and test results.
+Pods correspond to the cluster of machines (3 controller and 2 compute nodes in
+HA mode) used to run the tests and defined in Pharos project.
+The results pushed in the database are related to pods, projects and cases.
+If you try to push results of test done on non referenced pod, the API will
+return an error message.
+
+An additional method dashboard has been added to post-process the raw results in
+the Brahmaputra release (deprecated in Colorado release).
+
+The data model is very basic, 5 objects are available:
+  * Pods
+  * Projects
+  * Testcases
+  * Results
+  * Scenarios
+
+For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+The reporting
+-------------
+The reporting page for the test projects is http://testresults.opnfv.org/reporting/
+
+.. figure:: ../../images/reporting_page.png
+   :align: center
+   :alt: Testing group reporting page
+
+This page provides a reporting per OPNFV release and per testing project.
+
+.. figure:: ../../images/reporting_danube_page.png
+   :align: center
+   :alt: Testing group Danube reporting page
+
+An evolution of this page is planned.
+It was decided to unify the reporting by creating a landing page that should give
+the scenario status in one glance (it was previously consolidated manually
+on a wiki page).
+
+The landing page (planned for Danube 2.0) will be displayed per scenario:
+ * the status of the deployment
+ * the score of the test projectS
+ * a trust indicator
+
+Additional filters (version, installer, test collection time window,... ) are
+included.
+
+The test case catalog
+---------------------
+Until the Colorado release, each testing project was managing the list of its
+test cases. It was very hard to have a global view of the available test cases
+among the different test projects. A common view was possible through the API
+but it was not very user friendly.
+In fact you may know all the cases per project calling:
+
+ http://testresults.opnfv.org/test/api/v1/projects/<project_name>/cases
+
+with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick
+
+It was decided to build a web site providing a consistent view of the test cases
+per project and allow any scenario owner to build his/her custom list of tests
+(Danube 2.0).
+
+Other resources
+===============
+
+wiki: https://wiki.opnfv.org/testing
+
+mailing list: test-wg@lists.opnfv.org
+
+IRC chan: #opnfv-testperf
+
+weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf):
+ * Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST
+ * APAC time: 2nd Wednesday of the month 8:00-9:00 UTC
+
+=======================
+Reference documentation
+=======================
+
++----------------+---------------------------------------------------------+
+|  Project       |   Documentation links                                   |
++================+=========================================================+
+|  Bottlenecks   | https://wiki.opnfv.org/display/bottlenecks/Bottlenecks  |
++----------------+---------------------------------------------------------+
+| CPerf          | https://wiki.opnfv.org/display/cperf                    |
++----------------+---------------------------------------------------------+
+| Dovetail       | https://wiki.opnfv.org/display/dovetail                 |
++----------------+---------------------------------------------------------+
+| Functest       | https://wiki.opnfv.org/display/functest/                |
++----------------+---------------------------------------------------------+
+| Qtip           | https://wiki.opnfv.org/display/qtip                     |
++----------------+---------------------------------------------------------+
+| Storperf       | https://wiki.opnfv.org/display/storperf/Storperf        |
++----------------+---------------------------------------------------------+
+| VSperf         | https://wiki.opnfv.org/display/vsperf                   |
++----------------+---------------------------------------------------------+
+| Yardstick      | https://wiki.opnfv.org/display/yardstick/Yardstick      |
++----------------+---------------------------------------------------------+
diff --git a/docs/testing/testing-dev.rst b/docs/testing/testing-dev.rst
new file mode 100644
index 0000000..e7b6800
--- /dev/null
+++ b/docs/testing/testing-dev.rst
@@ -0,0 +1,51 @@
+.. _testing-dev:
+
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+========================
+Testing Developer Guides
+========================
+
+Testing group
+-------------
+.. include:: ./developer/devguide/index.rst
+
+Bottlenecks
+------------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/bottlenecks/docs/testing/developer/devguide/index
+
+
+Functest
+---------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/functest/docs/testing/developer/devguide/index
+
+
+QTIP
+-----
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/qtip/docs/testing/developer/devguide/index
+
+
+VSPERF
+-------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/vswitchperf/docs/testing/developer/devguide/index
+
+
+Yardstick
+---------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/yardstick/docs/testing/developer/devguide/index
diff --git a/docs/testing/testing-user.rst b/docs/testing/testing-user.rst
new file mode 100644
index 0000000..198b090
--- /dev/null
+++ b/docs/testing/testing-user.rst
@@ -0,0 +1,65 @@
+.. _testing-userguide:
+
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+===================
+Testing User Guides
+===================
+
+Bottlenecks
+------------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/bottlenecks/docs/testing/user/configguide/index
+   ../submodules/bottlenecks/docs/testing/user/userguide/index
+
+
+Functest
+---------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/functest/docs/testing/user/configguide/index
+   ../submodules/functest/docs/testing/user/userguide/index
+
+
+QTIP
+-----
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/qtip/docs/testing/user/configguide/index
+   ../submodules/qtip/docs/testing/user/userguide/index
+
+
+Storperf
+--------
+
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/storperf/docs/testing/user/index
+
+
+VSPERF
+------
+
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/vswitchperf/docs/testing/user/configguide/index
+   ../submodules/vswitchperf/docs/testing/user/userguide/index
+
+
+Yardstick
+----------
+.. toctree::
+   :maxdepth: 1
+
+   ../submodules/yardstick/docs/testing/user/configguide/index
+   ../submodules/yardstick/docs/testing/user/userguide/index
+
+
+