Merge "Functest Doc: first update for Euphrates - move part of the dev group to the testing document - move reporting into user guide - update logs - introduce alpine - introduce functest API - split documents per chapter"

1 files changed, 90 insertions, 0 deletions

diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst
new file mode 100644
index 00000000..14d22f23
--- /dev/null
+++ b/docs/testing/user/userguide/reporting.rst
@@ -0,0 +1,90 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Test reporting
+==============
+
+An automatic reporting page has been created in order to provide a consistent
+view of the Functest tests on the different scenarios.
+
+In this page, each scenario is evaluated according to test criteria.
+
+The results are collected from the centralized database every day and, per
+scenario. A score is calculated based on the results from the last 10 days.
+This score is the addition of single test scores. Each test case has a success
+criteria reflected in the criteria field from the results.
+
+As an illustration, let's consider the scenario
+os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the
+scores of all the runnable tests from the categories (tiers, healthcheck, smoke
+and features) corresponding to this scenario.
+
+   +---------------------+---------+---------+---------+---------+
+   | Test                | Apex    | Compass | Fuel    |  Joid   |
+   +=====================+=========+=========+=========+=========+
+   | vPing_ssh           |    X    |    X    |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | vPing_userdata      |    X    |    X    |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | tempest_smoke_serial|    X    |    X    |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | rally_sanity        |    X    |    X    |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | odl                 |    X    |    X    |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | promise             |         |         |    X    |    X    |
+   +---------------------+---------+---------+---------+---------+
+   | doctor              |    X    |         |    X    |         |
+   +---------------------+---------+---------+---------+---------+
+   | security_scan       |    X    |         |         |         |
+   +---------------------+---------+---------+---------+---------+
+   | parser              |         |         |    X    |         |
+   +---------------------+---------+---------+---------+---------+
+   | copper              |    X    |         |         |    X    |
+   +---------------------+---------+---------+---------+---------+
+   src:  os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version)
+
+All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature
+scenarios.
+Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration
+addons and, as a consequence, specific scenario.
+There are not considered as runnable on the generic odl_l2 scenario.
+
+
+If no result is available or if all the results are failed, the test case get 0
+point.
+If it was successful at least once but not anymore during the 4 runs, the case
+get 1 point (it worked once).
+If at least 3 of the last 4 runs were successful, the case get 2 points.
+If the last 4 runs of the test are successful, the test get 3 points.
+
+In the example above, the target score for fuel/os-odl_l2-nofeature-ha is
+3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .
+
+The scenario is validated per installer when we got 3 points for all individual
+test cases (e.g 24/24 for fuel, 15/15 for compass).
+
+Please note that complex or long duration tests are not considered yet for the
+scoring. In fact the success criteria are not always easy to define and may
+require specific hardware configuration.
+
+Please also note that all the test cases have the same "weight" for the score
+calculation whatever the complexity of the test case. Concretely a vping has the
+same weith than the 200 tempst tests.
+Moreover some installers support more features than others. The more cases your
+scenario is dealing with, the most difficult to rich a good scoring.
+
+Therefore the scoring provides 3 types of indicators:
+
+  * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features
+  * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS
+  * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI)
+
+In any case, the scoring is used to give feedback to the other projects and
+does not represent an absolute value of the scenario.
+
+See `reporting page`_ for details. For the status, click on the version,
+Functest then the Status menu.
+
+
+_`reporting page`: http://testresults.opnfv.org/reporting/