blob: 60c4720d1e3584f9bd60b93e5da101acfb48dd13 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) 2016 ZTE Corp.
*********
Dashboard
*********
The dashboard gives user an intuitive view of benchmark result.
Purpose
=======
The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it
is also important to show user
#. How is the final score calculated?
#. Under what condition is the test plan executed?
#. How many runs of a performance tests have been executed and is there any deviation?
#. Comparison of benchmark result from different PODs or configuration
Templates
=========
Different board templates are created to satisfy the above requirements.
Composition
-----------
QTIP gives a simple score but there must be a complex formula behind it. This
view explains the composition of the QPI.
Condition
---------
The condition of a benchmark result includes
* System Under Test
* Hardware environment
* Hypervisor version
* Operation System release version
* System Configuration
* Test Tools
* Release version
* Configuration
* Test Facility
* Laboratory
* Engineer
* Date
Conditions that do NOT have an obvious affect on the test result may be ignored,
e.g. temperature, power supply.
Stats
-----
Performance tests are actually measurement of specific metrics. All measurement
comes with uncertainty. The final result is normally one or a group of metrics
calculated from many repeats.
For each metric, the stats board shall consist of a diagram of all measured
values and a box of stats::
^ +------------+
| | count: ? |
| |average: ? |
| | min: ? |
| X | max: ? |
| XXXX XXXX X XXXXX | |
|X XX XX XX XXX XXX XX | |
| XXXXXX X XXXXX XX | |
| | |
| | |
| | |
| | |
| | |
+---------------------------------------------> +------------+
The type of diagram and selection of stats shall depend on what metric to show.
Comparison
----------
Comparison can be done between different PODs or different configuration on the
same PODs.
In a comparison view, the metrics are displayed in the same diagram. And the
parameters are listed side by side.
Both common parameters and different parameters are listed. Common values are
merged to the same cell. And user may configure the view to hide common rows.
A draft design is as following::
^
|
|
|
| XXXXXXXX
| XXX XX+-+ XXXXXXXXXX
| XXX +XXXX XXXXX
+-+XX X +--+ ++ XXXXXX +-+
| X+-+X +----+ +-+ +----+X
|X +--+ +---+ XXXXXX X
| +-------+ X
|
|
+----------------------------------------------------->
+--------------------+----------------+---------------+
| different param 1 | | |
| | | |
+-----------------------------------------------------+
| different param 2 | | |
| | | |
+-------------------------------------+---------------+
| common param 1 | |
| | |
+-------------------------------------+---------------+
| different param 3 | | |
| | | |
+-------------------------------------+---------------+
| common param 2 | |
| | |
+--------------------+--------------------------------+
+------------+
| HIDE COMMON|
+------------+
Time line
---------
Time line diagram for analysis of time critical performance test::
+-----------------+-----------+-------------+-------------+-----+
| | | | | |
+-----------------> | | | |
| +-----------> | | |
| ? ms +-------------> | |
| ? ms +------------>+ |
| ? ms ? ms |
| |
+---------------------------------------------------------------+
The time cost between checkpoints shall be displayed in the diagram.
|
