26 files changed, 353 insertions, 0 deletions
diff --git a/docs/submodules/apex b/docs/submodules/apex
-Subproject 1f346dd0052a8a24309a55058c54299803d95e0
+Subproject 68bb8c986bfdca528a06e18e58507b4750fdf40
diff --git a/docs/submodules/armband b/docs/submodules/armband
-Subproject 7dca6641608ea4085cedaf25107c9f702b4cdeb
+Subproject de0b0f160f6ac408c2ba302507ede4d0a537b73
diff --git a/docs/submodules/barometer b/docs/submodules/barometer
-Subproject 477bc4fd49e4eb2481809cc99297eac02d0d6db
+Subproject f56d04d684f5ccae7815960e9282c394545e385
diff --git a/docs/submodules/bottlenecks b/docs/submodules/bottlenecks
-Subproject e4a17446f0f02d6ea950278074d88db6c09c5bb
+Subproject b39f17a60e91552ffe94c520a30e6b271077f29
diff --git a/docs/submodules/compass4nfv b/docs/submodules/compass4nfv
-Subproject 8ed47ae6ab2e57a37cd90707adcd0b2ea53362a
+Subproject a0edd1b2b4d4df6fc5c3e773eba66224930fdb4
diff --git a/docs/submodules/daisy b/docs/submodules/daisy
-Subproject e0d6c8a4e5c84bedbcbe91ca51c8bbabf1d8fdf
+Subproject 69a1af39f02bb5c6a471768648c15f8ae1e8577
diff --git a/docs/submodules/doctor b/docs/submodules/doctor
-Subproject d182b8a69eeef01f37b123e7d331972b7c57911
+Subproject 7e0ab6e2149df22455e58eb363eb291a985514d
diff --git a/docs/submodules/dovetail b/docs/submodules/dovetail
-Subproject 2c170360711d8307ffab3fe8070a771650e4fd4
+Subproject cc034a7135cf70f137915f8b55c8ef00ab61e96
diff --git a/docs/submodules/fds b/docs/submodules/fds
-Subproject 4c8ccbb8594b7091f498dcf83b9b8c28835fe71
+Subproject 5363f1bd4b2007bc66f4171069251f813ff2b41
diff --git a/docs/submodules/fuel b/docs/submodules/fuel
-Subproject d433573b98d288feaff518dd3ac9a5b19e35f1e
+Subproject d676c91a67684a7efb317d58429c49db7c56247
diff --git a/docs/submodules/functest b/docs/submodules/functest
-Subproject a48bf0c8306e566f15877f3bd72882cf8b7663d
+Subproject 0323218afb7500bac979e46aafc9c33d614c99d
diff --git a/docs/submodules/joid b/docs/submodules/joid
-Subproject 703cfe7ce2639d3074cf39bffd748e59b5ed7b4
+Subproject aa443647aa36a04cf34358647310ca3baa4ed21
diff --git a/docs/submodules/octopus b/docs/submodules/octopus
-Subproject b41f1567f781aebc5fcd9c95c050978ddc95936
+Subproject 3b530194ef79082aebb5afc4cf61cedd83dfc03
diff --git a/docs/submodules/onosfw b/docs/submodules/onosfw
-Subproject ee2ebb70cd79c10cb9322e04172b0a0903c6656
+Subproject 0570a96f65db547dcf2f88ac71dfad9b17a0764
diff --git a/docs/submodules/openretriever b/docs/submodules/openretriever
-Subproject ddaccb38f85dba6bec1fcd79b0b4cce45d92192
+Subproject b4804ed0b8d88fc91324dfbbf9e285d0691bb0f
diff --git a/docs/submodules/parser b/docs/submodules/parser
-Subproject a1eebb743002a10194d8f7c5448c02a99f1f0fe
+Subproject 5b3f19f30ff02faa80ad02762ae23ad020b0421
diff --git a/docs/submodules/pharos b/docs/submodules/pharos
-Subproject d4d63f63f1ab4cd636499adee319f2a0b851302
+Subproject 7cbe4e7447280d39fdb5a56b3d9a0fea437b5b1
diff --git a/docs/submodules/qtip b/docs/submodules/qtip
-Subproject 26c2a8f2d556a8777d443b02d931e7df7fec8f6
+Subproject d218f94274ac94c36d2c843f73b83290f761a8f
diff --git a/docs/submodules/releng b/docs/submodules/releng
-Subproject 26f47542741f17b0d534565fc1cc6a6e6948dd3
+Subproject 1bfef523c2abd5024e2ec95a2fec3262178d194
diff --git a/docs/submodules/sdnvpn b/docs/submodules/sdnvpn
-Subproject de80ee56c458b127be3497f3882bc1c962f42df
+Subproject 45df38fe0cd7a604827a5cde6ad6480971b0456
diff --git a/docs/submodules/sfc b/docs/submodules/sfc
-Subproject a29f1fcf91f130f74dbd9d7c83bb7445faec613
+Subproject d0260c5d35d753e2bce23c74d877da648d47c48
diff --git a/docs/submodules/snaps b/docs/submodules/snaps
-Subproject c8212122569c2dbf6290b43a0fbde0171c2ffdc
+Subproject 2a6d3a0f73eb58762bdbe10567cfbb6edec54ab
diff --git a/docs/submodules/storperf b/docs/submodules/storperf
-Subproject c7806a0f08f6114d8b1f037a77af041a2b0364d
+Subproject 7602a54309adbe5c5346ee6befecc2e59697650
diff --git a/docs/submodules/vswitchperf b/docs/submodules/vswitchperf
-Subproject 80c9f96f5e60ee6a2a93494a0aae82014c2311b
+Subproject d3b124a22bf3aa2c05a5cb030f37b97db3d27db
diff --git a/docs/submodules/yardstick b/docs/submodules/yardstick
-Subproject 0509cf3e141b550653e642083907a584450a7fd
+Subproject 29484c3071028df6c897797706117082a4b8c3f
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
new file mode 100644
index 000000000..a23231b03
--- /dev/null
+++ b/docs/testing/developer/devguide/index.rst
@@ -0,0 +1,353 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
+*********************
+Test developer guide
+*********************
+
+.. toctree::
+   :numbered:
+   :maxdepth: 2
+
+
+============
+Introduction
+============
+
+The OPNFV testing ecosystem is wide.
+
+The goal of this guide consists in providing some guidelines for new developers
+involved in test areas.
+
+For the description of the ecosystem, see `[1]`_.
+
+
+=================
+Developer journey
+=================
+
+Be involved in the testing group
+================================
+
+Best practices
+==============
+
+Unit tests
+----------
+
+Dockerization
+-------------
+
+API
+---
+
+CLI
+---
+
+Traffic generators
+------------------
+
+Towards a pseudo micro services approach
+----------------------------------------
+
+======================================
+Testing group configuration parameters
+======================================
+
+
+Testing categories
+==================
+
+The testing group defined several categories also known as tiers. These
+categories can be used to group test suites.
+
++----------------+-------------------------------------------------------------+
+| healthcheck    | Simple and quick healthcheck tests case                     |
++----------------+-------------------------------------------------------------+
+| Smoke          | Set of smoke test cases/suites to validate the release      |
++----------------+-------------------------------------------------------------+
+| Features       | Test cases that validate a specific feature on top of OPNFV.|
+|                | Those come from Feature projects and need a bit of support  |
+|                | for integration                                             |
++----------------+-------------------------------------------------------------+
+| Components     | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)|
+|                | It may extend smoke tests                                   |
++----------------+-------------------------------------------------------------+
+| Performance    | Performance qualification                                   |
++----------------+-------------------------------------------------------------+
+| VNF            | Test cases related to deploy an open source VNF including   |
+|                | an orchestrator                                             |
++----------------+-------------------------------------------------------------+
+| Stress         | Stress and robustness tests                                 |
++----------------+-------------------------------------------------------------+
+| In Service     | In service testing                                          |
++----------------+-------------------------------------------------------------+
+
+Testing domains
+===============
+
+The domains deal with the technical scope of the tests. It shall correspond to
+domains defined for the certification program:
+
+ * compute
+ * network
+ * storage
+ * hypervisor
+ * container
+ * vim
+ * mano
+ * vnf
+ * ...
+
+Testing coverage
+=================
+One of the goals of the testing working group is to identify the poorly covered
+areas and avoid testing overlap.
+Ideally based on the declaration of the test cases, through the tags, domains
+and tier fields, it shall be possible to create heuristic maps.
+
+
+==============================
+Testing group generic enablers
+==============================
+
+
+TestAPI framework
+=================
+
+The OPNFV testing group created a test collection database to collect
+the test results from CI:
+
+
+ http://testresults.opnfv.org/test/swagger/spec.html
+
+Any test project running on any lab integrated in CI can push the
+results to this database.
+This database can be used to see the evolution of the tests and compare
+the results versus the installers, the scenarios or the labs.
+It is used to produce a dashboard with the current test status of the project.
+
+
+Overall Architecture
+--------------------
+The Test result management can be summarized as follows::
+
+  +-------------+    +-------------+    +-------------+
+  |             |    |             |    |             |
+  |   Test      |    |   Test      |    |   Test      |
+  | Project #1  |    | Project #2  |    | Project #N  |
+  |             |    |             |    |             |
+  +-------------+    +-------------+    +-------------+
+           |               |               |
+           V               V               V
+       +-----------------------------------------+
+       |                                         |
+       |         Test Rest API front end         |
+       |                                         |
+       +-----------------------------------------+
+           A                |
+           |                V
+           |     +-------------------------+
+           |     |                         |
+           |     |    Test Results DB      |
+           |     |         Mongo DB        |
+           |     |                         |
+           |     +-------------------------+
+           |
+           |
+     +----------------------+
+     |                      |
+     |    test Dashboard    |
+     |                      |
+     +----------------------+
+
+TestAPI description
+-------------------
+The TestAPI is used to declare pods, projects, test cases and test
+results. Pods are the sets of bare metal or virtual servers and networking
+equipments used to run the tests.
+
+The results pushed in the database are related to pods, projects and test cases.
+If you try to push results of test done on non referenced pod, the API will
+return an error message.
+
+An additional method dashboard has been added to post-process
+the raw results in release Brahmaputra (deprecated in Colorado).
+
+The data model is very basic, 5 objects are created:
+
+  * Pods
+  * Projects
+  * Testcases
+  * Results
+  * Scenarios
+
+The code of the API is hosted in the releng repository `[6]`_.
+The static documentation of the API can be found at `[7]`_.
+The TestAPI has been dockerized and may be installed locally in your
+lab. See `[15]`_ for details.
+
+The deployment of the TestAPI has been automated.
+A jenkins job manages:
+
+  * the unit tests of the TestAPI
+  * the creation of a new docker file
+  * the deployment of the new TestAPI
+  * the archive of the old TestAPI
+  * the backup of the Mongo DB
+
+TestAPI Authorization
+~~~~~~~~~~~~~~~~~~~~~
+
+PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs
+to be added in the request using a header 'X-Auth-Token' for access to the database.
+
+e.g::
+    headers['X-Auth-Token']
+
+The value of the header i.e the token can be accessed in the jenkins environment variable
+*TestApiToken*. The token value is added as a masked password.
+
+.. code-block:: python
+
+    headers['X-Auth-Token'] = os.environ.get('TestApiToken')
+
+The above example is in Python. Token based authentication has been added so that only ci pods
+jenkins job can have access to the database.
+
+Please note that currently token authorization is implemented but is not yet enabled.
+
+Reporting
+=========
+
+An automatic reporting page has been created in order to provide a
+consistent view of the scenarios.
+
+In this page, each scenario is evaluated according to test criteria.
+The code for the automatic reporting is available at `[8]`_.
+
+The results are collected from the centralized database every day and,
+per scenario. A score is calculated based on the results from the last
+10 days.
+
+Dashboard
+=========
+
+Dashboard is used to provide a consistent view of the results collected in CI.
+The results showed on the dashboard are post processed from the Database,
+which only contains raw results.
+
+It can be used in addition of the reporting page (high level view) to allow
+the creation of specific graphs according to what the test owner wants to show.
+
+In Brahmaputra, a basic home made dashboard was created in Functest.
+In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex
+graphs).
+Since Danube, the testing community decided to adopt ELK framework and to rely
+on bitergia. It was not implemented for Danube but it is planned for Euphrates.
+
+Bitergia already provides a dashboard for code and infrastructure.
+A new Test tab will be added. The dataset will be built by consuming
+the TestAPI.
+
+See `[3]`_ for details.
+
+
+=======
+How TOs
+=======
+
+Where can I find information on the different test projects?
+===========================================================
+
+
+How can I contribute to a test project?
+=======================================
+
+
+Where can I find hardware resources?
+====================================
+
+
+How do I integrate my tests in CI?
+==================================
+
+
+How to declare my tests in the test Database?
+=============================================
+
+
+How to push your results into the Test Database?
+================================================
+
+The test database is used to collect test results. By default it is
+enabled only for CI tests from Production CI pods.
+
+Please note that it is possible to create your own local database.
+
+A dedicated database is for instance created for each plugfest.
+
+The architecture and associated API is described in previous chapter.
+If you want to push your results from CI, you just have to call the API
+at the end of your script.
+
+You can also reuse a python function defined in functest_utils.py `[5]`_
+
+
+Where can I find the documentation on the test API?
+===================================================
+
+http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+
+I have tests, to which category should I declare them?
+======================================================
+
+
+
+The main ambiguity could be between features and VNF.
+In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
+feature you introduced.
+We recommend to declare your test in the feature category.
+
+VNF category is really dedicated to test including:
+
+ * creation of resources
+ * deployement of an orchestrator/VNFM
+ * deployment of the VNF
+ * test of the VNFM
+ * free resources
+
+The goal is not to study a particular feature on the infrastructure but to have
+a whole end to end test of a VNF automatically deployed in CI.
+Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
+jobs and use to get a scenario score.
+
+Where are the logs of CI runs?
+==============================
+
+Logs and configuration files can be pushed to artifact server from the CI under
+http://artifacts.opnfv.org/<project name>
+
+
+==========
+References
+==========
+
+_`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html
+
+_`[2]`: http://www.opnfv.org
+
+_`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
+
+_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
+
+_`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+
+_`[6]`: https://git.opnfv.org/functest/tree/releng
+
+_`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+IRC support chan: #opnfv-testperf