Merge "Testing group documentation update for Euphrates"

1 files changed, 137 insertions, 190 deletions

diff --git a/docs/testing/developer/devguide/dev-guide.rst b/docs/testing/developer/devguide/dev-guide.rst
index 95d3ac574..494c21e18 100644
--- a/docs/testing/developer/devguide/dev-guide.rst
+++ b/docs/testing/developer/devguide/dev-guide.rst
@@ -19,7 +19,7 @@ The OPNFV testing ecosystem is wide.
 The goal of this guide consists in providing some guidelines for new developers
 involved in test areas.
 
-For the description of the ecosystem, see `[1]`_.
+For the description of the ecosystem, see `[DEV1]`_.
 
 =================
 Developer journey
@@ -30,6 +30,8 @@ There are several ways to join test projects as a developer. In fact you may:
  * Develop new test cases
  * Develop frameworks
  * Develop tooling (reporting, dashboards, graphs, middleware,...)
+ * Troubleshoot results
+ * Post-process results
 
 These different tasks may be done within a specific project or as a shared
 resource accross the different projects.
@@ -48,6 +50,11 @@ Tooling may be specific to a project or generic to all the projects. For
 specific tooling, please report to the test project user guide. The tooling used
 by several test projects will be detailed in this document.
 
+The best event to meet the testing community is probably the plugfest. Such an
+event is organized after each release. Most of the test projects are present.
+
+The summit is also a good opportunity to meet most of the actors `[DEV4]`_.
+
 Be involved in the testing group
 ================================
 
@@ -57,8 +64,8 @@ consistant test strategy (test case definition, scope of projects, resources for
 long duration, documentation, ...) and align tooling or best practices.
 
 A weekly meeting is organized, the agenda may be amended by any participant.
-2 slots have been defined (US and APAC). Agendas and minutes are public.See
-`[8]`_ for details.
+2 slots have been defined (US/Europe and APAC). Agendas and minutes are public.
+See `[DEV3]`_ for details.
 The testing group IRC channel is #opnfv-testperf
 
 Best practices
@@ -71,7 +78,7 @@ indicative. Contact the testing group for further details.
 
 
 Repository structure
----------------------
+--------------------
 
 Most of the projects have a similar structure, which can be defined as follows::
 
@@ -110,7 +117,7 @@ projects. This pseudo micro service approach should allow a flexible use of
 the different projects and reduce the risk of overlapping. In fact if project A
 provides an API to deploy a traffic generator, it is better to reuse it rather
 than implementing a new way to deploy it. This approach has not been implemented
-yet but the prerequisite consiting in exposing and API has already been done by
+yet but the prerequisites consiting in exposing and API has already been done by
 several test projects.
 
 
@@ -122,25 +129,76 @@ possible to prepare the environement and run tests through a CLI.
 
 Dockerization
 -------------
-Dockerization has been introduced in Brahmaputra and adopted by the test
+Dockerization has been introduced in Brahmaputra and adopted by most of the test
 projects. Docker containers are pulled on the jumphost of OPNFV POD.
-
-
-
-
-Unit tests
-----------
-
-
-Traffic generators
-------------------
+<TODO Jose/Mark/Alec>
+
+Code quality
+------------
+
+It is recommended to control the quality of the code of the testing projects,
+and more precisely to implement some verifications before any merge:
+  * pep8
+  * pylint
+  * unit tests (python 2.7)
+  * unit tests (python 3.5)
+
+
+The code of the test project must be covered by unit tests. The coverage
+shall be reasonable and not decrease when adding new features to the framework.
+The use of tox is recommended.
+It is possible to implement strict rules (no decrease of pylint score, unit
+test coverages) on critical python classes.
+
+
+Third party tooling
+-------------------
+
+Several test projects integrate third party tooling for code quality check
+and/or traffic generation. Some of the tools can be listed as follows:
+
++---------------+----------------------+------------------------------------+
+| Project       | Tool                 | Comments                           |
++===============+======================+====================================+
+| Bottlenecks   | TODO                 |                                    |
++---------------+----------------------+------------------------------------+
+| Functest      | Tempest              | OpenStack test tooling             |
+|               | Rally                | OpenStack test tooling             |
+|               | Refstack             | OpenStack test tooling             |
+|               | RobotFramework       | Used for ODL tests                 |
++---------------+----------------------+------------------------------------+
+| QTIP          | Unixbench            |                                    |
+|               | RAMSpeed             |                                    |
+|               | nDPI                 |                                    |
+|               | openSSL              |                                    |
+|               | inxi                 |                                    |
++---------------+----------------------+------------------------------------+
+| Storperf      | TODO                 |                                    |
++---------------+----------------------+------------------------------------+
+| VSPERF        | TODO                 |                                    |
++---------------+----------------------+------------------------------------+
+| Yardstick     | Moongen              | Traffic generator                  |
+|               | Trex                 | Traffic generator                  |
+|               | Pktgen               | Traffic generator                  |
+|               | IxLoad, IxNet        | Traffic generator                  |
+|               | SPEC                 | Compute                            |
+|               | Unixbench            | Compute                            |
+|               | RAMSpeed             | Compute                            |
+|               | LMBench              | Compute                            |
+|               | Iperf3               | Network                            |
+|               | Netperf              | Network                            |
+|               | Pktgen-DPDK          | Network                            |
+|               | Testpmd              | Network                            |
+|               | L2fwd                | Network                            |
+|               | Fio                  | Storage                            |
+|               | Bonnie++             | Storage                            |
++---------------+----------------------+------------------------------------+
 
 
 ======================================
 Testing group configuration parameters
 ======================================
 
-
 Testing categories
 ==================
 
@@ -195,184 +253,76 @@ Ideally based on the declaration of the test cases, through the tags, domains
 and tier fields, it shall be possible to create heuristic maps.
 
 
-=================
-TestAPI framework
-=================
-
-The OPNFV testing group created a test collection database to collect
-the test results from CI:
-
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
-Any test project running on any lab integrated in CI can push the
-results to this database.
-This database can be used to see the evolution of the tests and compare
-the results versus the installers, the scenarios or the labs.
-It is used to produce a dashboard with the current test status of the project.
-
-
-Overall Architecture
-====================
-The Test result management can be summarized as follows::
-
-  +-------------+    +-------------+    +-------------+
-  |             |    |             |    |             |
-  |   Test      |    |   Test      |    |   Test      |
-  | Project #1  |    | Project #2  |    | Project #N  |
-  |             |    |             |    |             |
-  +-------------+    +-------------+    +-------------+
-           |               |               |
-           V               V               V
-       +-----------------------------------------+
-       |                                         |
-       |         Test Rest API front end         |
-       |                                         |
-       +-----------------------------------------+
-           |                |
-           |                V
-           |     +-------------------------+
-           |     |                         |
-           |     |    Test Results DB      |
-           |     |         Mongo DB        |
-           |     |                         |
-           |     +-------------------------+
-           |
-           |
-     +----------------------+
-     |                      |
-     |    test Dashboard    |
-     |                      |
-     +----------------------+
-
-TestAPI description
-===================
-The TestAPI is used to declare pods, projects, test cases and test
-results. Pods are the sets of bare metal or virtual servers and networking
-equipments used to run the tests.
-
-The results pushed in the database are related to pods, projects and test cases.
-If you try to push results of test done on non referenced pod, the API will
-return an error message.
-
-An additional method dashboard has been added to post-process
-the raw results in release Brahmaputra (deprecated in Colorado).
-
-The data model is very basic, 5 objects are created:
-
-  * Pods
-  * Projects
-  * Testcases
-  * Results
-  * Scenarios
-
-The code of the API is hosted in the releng repository `[6]`_.
-The static documentation of the API can be found at `[7]`_.
-The TestAPI has been dockerized and may be installed locally in your
-lab. See `[15]`_ for details.
-
-The deployment of the TestAPI has been automated.
-A jenkins job manages:
-
-  * the unit tests of the TestAPI
-  * the creation of a new docker file
-  * the deployment of the new TestAPI
-  * the archive of the old TestAPI
-  * the backup of the Mongo DB
-
-TestAPI Authorization
----------------------
-
-PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs
-to be added in the request using a header 'X-Auth-Token' for access to the database.
-
-e.g::
-    headers['X-Auth-Token']
-
-The value of the header i.e the token can be accessed in the jenkins environment variable
-*TestApiToken*. The token value is added as a masked password.
-
-.. code-block:: python
-
-    headers['X-Auth-Token'] = os.environ.get('TestApiToken')
-
-The above example is in Python. Token based authentication has been added so that only ci pods
-jenkins job can have access to the database.
-
-Please note that currently token authorization is implemented but is not yet enabled.
-
-===============================
-Feedback from the testing group
-================================
-
-Test case catalog
-===================
-
-A test case catalog has been realized. Roll over the project then click to get
-the list of test cases, click on the case to get more details.
-
-.. raw:: html
-   :url: http://testresults.opnfv.org/reporting2/reporting/index.html#!/select/visual
-
-Reporting
-=========
-
-An automatic reporting page has been created in order to provide a
-consistent view of the scenarios.
-
-In this page, each scenario is evaluated according to test criteria.
-The code for the automatic reporting is available at `[8]`_.
-
-The results are collected from the centralized database every day and,
-per scenario. A score is calculated based on the results from the last
-10 days.
-
-Dashboard
-=========
-
-Dashboard is used to provide a consistent view of the results collected in CI.
-The results showed on the dashboard are post processed from the Database,
-which only contains raw results.
-
-It can be used in addition of the reporting page (high level view) to allow
-the creation of specific graphs according to what the test owner wants to show.
-
-In Brahmaputra, a basic home made dashboard was created in Functest.
-In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex
-graphs).
-Since Danube, the testing community decided to adopt ELK framework and to rely
-on bitergia. It was not implemented for Danube but it is planned for Euphrates.
-
-Bitergia already provides a dashboard for code and infrastructure.
-A new Test tab will be added. The dataset will be built by consuming
-the TestAPI.
-
-See `[3]`_ for details.
-
-
 =======
 How TOs
 =======
 
 Where can I find information on the different test projects?
 ===========================================================
+On http://docs.opnfv.org! A section is dedicated to the testing projects. You
+will find the overview of the ecosystem and the links to the project documents.
+
+Another source is the testing wiki on https://wiki.opnfv.org/display/testing
+
+You may also contact the testing group on the IRC channel #opnfv-testperf or by
+mail at test-wg AT lists.opnfv.org (testing group) or opnfv-tech-discuss AT
+lists.opnfv.org (generic technical discussions).
 
 
 How can I contribute to a test project?
 =======================================
+As any project, the best solution is to contact the project. The project
+members with their email address can be found under
+https://git.opnfv.org/<project>/tree/INFO
+
+You may also send a mail to the testing mailing list or use the IRC channel
+#opnfv-testperf
 
 
 Where can I find hardware resources?
 ====================================
+You should discuss this topic with the project you are working with. If you need
+access to an OPNFV community POD, it is possible to contact the infrastructure
+group. Depending on your needs (scenario/installer/tooling), it should be
+possible to find free time slots on one OPNFV community POD from the Pharos
+federation. Create a JIRA ticket to describe your needs on
+https://jira.opnfv.org/projects/INFRA.
+You must already be an OPNFV contributor. See
+https://wiki.opnfv.org/display/DEV/Developer+Getting+Started.
+
+Please note that lots of projects have their own "how to contribute" or
+"get started" page on the OPNFV wiki.
 
 
 How do I integrate my tests in CI?
 ==================================
-
+It shall be discussed directly with the project you are working with. It is
+done through jenkins jobs calling testing project files but the way to onboard
+cases differ from one project to another.
 
 How to declare my tests in the test Database?
 =============================================
+If you have access to the test API swagger (access granted to contributors), you
+may use the swagger interface of the test API to declare your project.
+The URL is http://testresults.opnfv.org/test/swagger/spec.html.
+
+.. figure:: ../../../images/swaggerUI.png
+   :align: center
+   :alt: Testing Group Test API swagger
 
+Click on *Spec*, the list of available methods must be displayed.
+
+.. figure:: ../../../images/API-operations.png
+   :align: center
+   :alt: Testing Group Test API swagger
+
+For the declaration of a new project use the POST /api/v1/projects method.
+For the declaration of new test cases in an existing project, use the POST
+ /api/v1/projects/{project_name}/cases method
+
+ .. figure:: ../../../images/CreateCase.png
+    :align: center
+    :alt: Testing group declare new test case
 
 How to push your results into the Test Database?
 ================================================
@@ -388,20 +338,21 @@ The architecture and associated API is described in previous chapter.
 If you want to push your results from CI, you just have to call the API
 at the end of your script.
 
-You can also reuse a python function defined in functest_utils.py `[5]`_
+You can also reuse a python function defined in functest_utils.py `[DEV2]`_
 
 
 Where can I find the documentation on the test API?
 ===================================================
 
+The Test API is now documented in this document (see sections above).
+You may also find autogenerated documentation in
 http://artifacts.opnfv.org/releng/docs/testapi.html
-
-
+A web protal is also under construction for certification at
+http://testresults.opnfv.org/test/#/
 
 I have tests, to which category should I declare them?
 ======================================================
-
-
+See table above.
 
 The main ambiguity could be between features and VNF.
 In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
@@ -432,21 +383,17 @@ http://artifacts.opnfv.org/<project name>
 References
 ==========
 
-_`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html
-
-_`[2]`: http://www.opnfv.org
-
-_`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
-
-_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
-
-_`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+`[DEV1]`_: OPNFV Testing Ecosystem
 
-_`[6]`: https://git.opnfv.org/functest/tree/releng
+`[DEV2]`_: Python code sample to push results into the Database
 
-_`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+`[DEV3]`_: Testing group wiki page
 
-_`[8]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting
+`[DEV4]`_: Conversation with the testing community, OPNFV Beijing Summit
 
+.. _`[DEV1]`: http://docs.opnfv.org/en/latest/testing/ecosystem/index.html
+.. _`[DEV2]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
+.. _`[DEV3]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting
+.. _`[DEV4]`: https://www.youtube.com/watch?v=f9VAUdEqHoA
 
 IRC support chan: #opnfv-testperf