blob: cf9881844013aaee98ac6038cffc7d0b05ed4f8c (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) 2016 ZTE Corp.
*********
Dashboard
*********
The dashboard gives user an intuitive view of benchmark result.
Purpose
=======
The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it
is also important to show user
#. How is the final score calculated?
#. Under what condition is the test plan executed?
#. How many runs of a performance tests have been executed and is there any deviation?
#. Comparison of benchmark result from different PODs or configuration
Templates
=========
Different board templates are created to satisfy the above requirements.
Composition
-----------
QTIP gives a simple score but there must be a complex formula behind it. This
view explains the composition of the QPI.
Condition
---------
The condition of a benchmark result includes
* System Under Test
* Hardware environment
* Hypervisor version
* Operation System release version
* System Configuration
* Test Tools
* Release version
* Configuration
* Test Facility
* Laboratory
* Engineer
* Date
Conditions that do NOT have an obvious affect on the test result may be ignored,
e.g. temperature, power supply.
Stats
-----
Performance tests are actually measurement of specific metrics. All measurement
comes with uncertainty. The final result is normally one or a group of metrics
calculated from many repeats.
For each metric, the stats board shall consist of a diagram of all measured
values and a box of stats::
^ +------------+
| | count: ? |
| |average: ? |
| | min: ? |
| X | max: ? |
| XXXX XXXX X XXXXX | |
|X XX XX XX XXX XXX XX | |
| XXXXXX X XXXXX XX | |
| | |
| | |
| | |
| | |
| | |
+---------------------------------------------> +------------+
The type of diagram and selection of stats shall depend on what metric to show.
Comparison
----------
Comparison can be done between different PODs or different configuration on the
same PODs.
|
