summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/userguide/reporting.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/user/userguide/reporting.rst')
-rw-r--r--docs/testing/user/userguide/reporting.rst90
1 files changed, 90 insertions, 0 deletions
diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst
new file mode 100644
index 000000000..14d22f230
--- /dev/null
+++ b/docs/testing/user/userguide/reporting.rst
@@ -0,0 +1,90 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Test reporting
+==============
+
+An automatic reporting page has been created in order to provide a consistent
+view of the Functest tests on the different scenarios.
+
+In this page, each scenario is evaluated according to test criteria.
+
+The results are collected from the centralized database every day and, per
+scenario. A score is calculated based on the results from the last 10 days.
+This score is the addition of single test scores. Each test case has a success
+criteria reflected in the criteria field from the results.
+
+As an illustration, let's consider the scenario
+os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the
+scores of all the runnable tests from the categories (tiers, healthcheck, smoke
+and features) corresponding to this scenario.
+
+ +---------------------+---------+---------+---------+---------+
+ | Test | Apex | Compass | Fuel | Joid |
+ +=====================+=========+=========+=========+=========+
+ | vPing_ssh | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | vPing_userdata | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | tempest_smoke_serial| X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | rally_sanity | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | odl | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | promise | | | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | doctor | X | | X | |
+ +---------------------+---------+---------+---------+---------+
+ | security_scan | X | | | |
+ +---------------------+---------+---------+---------+---------+
+ | parser | | | X | |
+ +---------------------+---------+---------+---------+---------+
+ | copper | X | | | X |
+ +---------------------+---------+---------+---------+---------+
+ src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version)
+
+All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature
+scenarios.
+Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration
+addons and, as a consequence, specific scenario.
+There are not considered as runnable on the generic odl_l2 scenario.
+
+
+If no result is available or if all the results are failed, the test case get 0
+point.
+If it was successful at least once but not anymore during the 4 runs, the case
+get 1 point (it worked once).
+If at least 3 of the last 4 runs were successful, the case get 2 points.
+If the last 4 runs of the test are successful, the test get 3 points.
+
+In the example above, the target score for fuel/os-odl_l2-nofeature-ha is
+3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .
+
+The scenario is validated per installer when we got 3 points for all individual
+test cases (e.g 24/24 for fuel, 15/15 for compass).
+
+Please note that complex or long duration tests are not considered yet for the
+scoring. In fact the success criteria are not always easy to define and may
+require specific hardware configuration.
+
+Please also note that all the test cases have the same "weight" for the score
+calculation whatever the complexity of the test case. Concretely a vping has the
+same weith than the 200 tempst tests.
+Moreover some installers support more features than others. The more cases your
+scenario is dealing with, the most difficult to rich a good scoring.
+
+Therefore the scoring provides 3 types of indicators:
+
+ * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features
+ * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS
+ * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI)
+
+In any case, the scoring is used to give feedback to the other projects and
+does not represent an absolute value of the scenario.
+
+See `reporting page`_ for details. For the status, click on the version,
+Functest then the Status menu.
+
+
+_`reporting page`: http://testresults.opnfv.org/reporting/