1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
.. (c) 2015 Dell Inc.
.. (c) 2016 ZTE Corp.
********************************
Compute Performance Benchmarking
********************************
The compute QPI aims to benchmark the compute components of an OPNFV platform.
Such components include, the CPU performance, the memory performance.
The compute QPI consists of both synthetic and application specific benchmarks to
test compute components.
All the compute benchmarks could be run in the scenario:
On Baremetal Machines provisioned by an OPNFV installer (Host machines)
On Virtual machines provisioned by OpenStack deployed by an OPNFV installer
Note: The Compute benchmank constains relatively old benchmarks such as dhrystone
and whetstone. The suite would be updated for better benchmarks such as Linbench for
the OPNFV future release.
Getting started
===============
Notice: All descriptions are based on QTIP container.
Inventory File
--------------
QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to
determine what hosts to work against. QTIP can automatically generate a inventory
file via OPNFV installer. Users also can write their own inventory information into
``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host
IP addresses. For example:
::
[hosts]
10.20.0.11
10.20.0.12
QTIP key Pair
-------------
QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI,
QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass
public key to remote hosts.
If environment variable *CI_DEBUG* is set to *true*, users should delete it by
manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from
remote hosts before the execution ends. Please make sure the key deleted from remote
hosts or it can introduce a security flaw.
Execution
---------
There are two ways to execute compute QPI:
* Script
You can run compute QPI with docker exec:
::
# run with baremetal machines provisioned by an OPNFV installer
docker exec <qtip container> bash -x /home/opnfv/repos/qtip/qtip/scripts/quickstart.sh -q compute
# run with virtual machines provisioned by OpenStack
docker exec <qtip container> bash -x /home/opnfv/repos/qtip/qtip/scripts/quickstart.sh -q compute -u vnf
* Commands
In a QTIP container, you can run compute QPI by using QTIP CLI. You can get more details from
*userguide/cli.rst*.
Test result
------------
QTIP generates results in the ``/home/opnfv/<project_name>/results/`` directory are listed down under the
timestamp name.
Metrics
-------
The benchmarks include:
Dhrystone 2.1
^^^^^^^^^^^^^
Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer
calculations to evaluate CPU capabilities. Both Single CPU performance is measured
along multi-cpu performance.
Dhrystone, however, is a dated benchmark and has some short comings.
Written in C, it is a small program that doesn't test the CPU memory subsystem.
Additionally, dhrystone results could be modified by optimizing the compiler and
insome cases hardware configuration.
References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf
Whetstone
^^^^^^^^^
Whetstone is a synthetic benchmark to measure CPU floating point operation performance.
Both Single CPU performance is measured along multi-cpu performance.
Like Dhrystone, Whetstone is a dated benchmark and has short comings.
References:
http://www.netlib.org/benchmark/whetstone.c
OpenSSL Speed
^^^^^^^^^^^^^
OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP,
two OpenSSL Speed benchmarks are incorporated:
1. RSA signatunes/sec signed by a machine
2. AES 128-bit encryption throughput for a machine for cipher block sizes
References:
https://www.openssl.org/docs/manmaster/apps/speed.html
RAMSpeed
^^^^^^^^
RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is
large enough to ensure Cache Misses so that the main machine memory is used.
INTmem and FLOATmem benchmarks are executed in 4 different scenarios:
a. Copy: a(i)=b(i)
b. Add: a(i)=b(i)+c(i)
c. Scale: a(i)=b(i)*d
d. Tniad: a(i)=b(i)+c(i)*d
INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points
for these benchmarks.
References:
http://alasir.com/software/ramspeed/
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements
DPI
^^^
nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that
is maintained by ntop. An example application called *pcapreader* has been developed
and is available for use along nDPI.
A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic
in the pcap file into different categories based on string matching. The *pcapreader*
application provides a throughput number for the rate at which traffic was classified,
indicating a machine's computational performance. The results are run 10 times and an
average is taken for the obtained number.
*nDPI may provide non consistent results and was added to Brahmaputra for experimental
purposes*
References:
http://www.ntop.org/products/deep-packet-inspection/ndpi/
http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf
|
