diff options
Diffstat (limited to 'docs/testing/user/userguide/reporting.rst')
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 90 |
1 files changed, 0 insertions, 90 deletions
diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst deleted file mode 100644 index 88f5e865..00000000 --- a/docs/testing/user/userguide/reporting.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Test reporting -============== - -An automatic reporting page has been created in order to provide a consistent -view of the Functest tests on the different scenarios. - -In this page, each scenario is evaluated according to test criteria. - -The results are collected from the centralized database every day and, per -scenario. A score is calculated based on the results from the last 10 days. -This score is the addition of single test scores. Each test case has a success -criteria reflected in the criteria field from the results. - -As an illustration, let's consider the scenario -os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the -scores of all the runnable tests from the categories (tiers, healthcheck, smoke -and features) corresponding to this scenario. - - +---------------------+---------+---------+---------+---------+ - | Test | Apex | Compass | Fuel | Joid | - +=====================+=========+=========+=========+=========+ - | vPing_ssh | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | vPing_userdata | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | rally_sanity | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | odl | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | promise | | | X | X | - +---------------------+---------+---------+---------+---------+ - | doctor | X | | X | | - +---------------------+---------+---------+---------+---------+ - | security_scan | X | | | | - +---------------------+---------+---------+---------+---------+ - | parser | | | X | | - +---------------------+---------+---------+---------+---------+ - | copper | X | | | X | - +---------------------+---------+---------+---------+---------+ - src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version) - -All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature -scenarios. -Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. -There are not considered as runnable on the generic odl_l2 scenario. - - -If no result is available or if all the results are failed, the test case get 0 -point. -If it was successful at least once but not anymore during the 4 runs, the case -get 1 point (it worked once). -If at least 3 of the last 4 runs were successful, the case get 2 points. -If the last 4 runs of the test are successful, the test get 3 points. - -In the example above, the target score for fuel/os-odl_l2-nofeature-ha is -3 x 8 = 24 points and for compass it is 3 x 5 = 15 points . - -The scenario is validated per installer when we got 3 points for all individual -test cases (e.g 24/24 for fuel, 15/15 for compass). - -Please note that complex or long duration tests are not considered yet for the -scoring. In fact the success criteria are not always easy to define and may -require specific hardware configuration. - -Please also note that all the test cases have the same "weight" for the score -calculation whatever the complexity of the test case. Concretely a vping has the -same weith than the 200 tempst tests. -Moreover some installers support more features than others. The more cases your -scenario is dealing with, the most difficult to rich a good scoring. - -Therefore the scoring provides 3 types of indicators: - - * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features - * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS - * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) - -In any case, the scoring is used to give feedback to the other projects and -does not represent an absolute value of the scenario. - -See `reporting page`_ for details. For the status, click on the version, -Functest then the Status menu. - - -.. _`reporting page`: http://testresults.opnfv.org/reporting/ |