summaryrefslogtreecommitdiffstats
path: root/reporting
diff options
context:
space:
mode:
authorMorgan Richomme <morgan.richomme@orange.com>2017-10-06 20:39:36 +0200
committerMorgan Richomme <morgan.richomme@orange.com>2017-10-07 08:20:01 +0200
commita437b42018b9b7c2cd5173ebca429b7b9c03449b (patch)
tree0c2206d84f408cd377db99477cad1a45ba61378b /reporting
parentac00a280e6b1f7d69d8dbc37f96145624c650b88 (diff)
Change score calculation for functest reporting
the goal is not to penalize scenario with few iteration due to CI pipeline constraints until now the top score was calculated on the base on 4 iterations, it means whatever the number of iterations, we always consider the possible top score based on at least 4 iterations each case can get - 3 points (last 4 iterations PASS) - 2 points (more than 1 PASS over the last 10 days but 1 FAIL of the last 4) - 1 point (at least 1 PASS over the last 10 days) - no point (never work) so if we consider a scenario with 10 cases, the top score is 30 = 10*3 (max points) the coefficient to calculate the top score is not the number of iterations - 1 iteration => possible top score 10 - 2 iterations => top score 20 - 3 iterations => top score 20 - 4 or more iterations => top score 30 previously top score was systematically 30 https://wiki.opnfv.org/plugins/servlet/mobile#content/view/6828617 Change-Id: I01102bc00766b7f5b58e73befa7d5328fcc895cc Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>
Diffstat (limited to 'reporting')
-rwxr-xr-xreporting/reporting/functest/reporting-status.py24
1 files changed, 19 insertions, 5 deletions
diff --git a/reporting/reporting/functest/reporting-status.py b/reporting/reporting/functest/reporting-status.py
index 267803e..c71e00f 100755
--- a/reporting/reporting/functest/reporting-status.py
+++ b/reporting/reporting/functest/reporting-status.py
@@ -230,12 +230,26 @@ for version in versions:
# Evaluate the results for scenario validation
# **********************************************
# the validation criteria = nb runnable tests x 3
- # because each test case = 0,1,2 or 3
- scenario_criteria = nb_test_runnable_for_this_scenario * 3
- # if 0 runnable tests set criteria at a high value
- if scenario_criteria < 1:
- scenario_criteria = 50 # conf.MAX_SCENARIO_CRITERIA
+ # because each test case can get
+ # 0 point (never PASS)
+ # 1 point at least (PASS once over the time window)
+ # 2 points (PASS more than once but 1 FAIL on the last 4)
+ # 3 points PASS on the last 4 iterations
+ # e.g. 1 scenario = 10 cases
+ # 1 iteration : max score = 10 (10x1)
+ # 2 iterations : max score = 20 (10x2)
+ # 3 iterations : max score = 20
+ # 4 or more iterations : max score = 30 (1x30)
+ if len(s_result) > 3:
+ k_score = 3
+ elif len(s_result) < 2:
+ k_score = 1
+ else:
+ k_score = 2
+
+ scenario_criteria = nb_test_runnable_for_this_scenario*k_score
+ # score for reporting
s_score = str(scenario_score) + "/" + str(scenario_criteria)
s_score_percent = rp_utils.getScenarioPercent(
scenario_score,