diff options
Diffstat (limited to 'docs/testing/user/userguide')
| -rw-r--r-- | docs/testing/user/userguide/cli.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/testing/user/userguide/overview.rst | 32 | ||||
| -rw-r--r-- | docs/testing/user/userguide/qpi-compute.rst | 100 |
4 files changed, 116 insertions, 23 deletions
diff --git a/docs/testing/user/userguide/cli.rst b/docs/testing/user/userguide/cli.rst index 19420bd1..96026c5b 100644 --- a/docs/testing/user/userguide/cli.rst +++ b/docs/testing/user/userguide/cli.rst @@ -38,6 +38,12 @@ as above. An important thing to remember is to provide absolute path of result d qtip plan run <plan_name> -p $PWD/result Similarly, the same commands can be used for the other two components making up the plans, i.e QPI's and metrics. +For example, in order to run a single metric +:: + + qtip metric run <metric_name> -p $PWD/result + +The same can be applied for a QPI. QTIP also provides the utility to view benchmarking results on the console. One just need to provide to where the results are stored. Extending the example above diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 78c5d117..96d077c0 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -11,5 +11,6 @@ QTIP User Guide .. toctree:: :maxdepth: 2 + overview.rst cli.rst qpi-compute.rst diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst new file mode 100644 index 00000000..1ad0b670 --- /dev/null +++ b/docs/testing/user/userguide/overview.rst @@ -0,0 +1,32 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2017 ZTE Corp. + + +############ +Introduction +############ + +`QTIP`_ is the project for **Platform Performance Benchmarking** in `OPNFV`_. It aims to provide user a simple indicator +for performance, simple but supported by comprehensive testing data and transparent calculation formula. + +QTIP introduces a concept called **QPI**, a.k.a. QTIP Performance Index, which aims to be a **TRUE** indicator of +performance. **TRUE** reflects the core value of QPI in four aspects + +- *Transparent*: being an open source project, user can inspect all details behind QPI, e.g. formulas, metrics, raw data +- *Reliable*: the integrity of QPI will be guaranteed by traceability in each step back to raw test result +- *Understandable*: QPI is broke down into section scores, and workload scores in report to help user to understand +- *Extensible*: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics + +########## +Benchmarks +########## + +The builtin benchmarks of QTIP are located in ``<package_root>/benchmarks`` folder + +- *QPI*: specifications about how an QPI is calculated and sources of metrics +- *metric*: performance metrics referred in QPI, currently it is categorized by performance testing tools +- *plan*: executable benchmarking plan which collects metrics and calculate QPI + +.. _QTIP: https://wiki.opnfv.org/display/qtip +.. _OPNFV: https://www.opnfv.org/ diff --git a/docs/testing/user/userguide/qpi-compute.rst b/docs/testing/user/userguide/qpi-compute.rst index 369240c9..d64925bd 100644 --- a/docs/testing/user/userguide/qpi-compute.rst +++ b/docs/testing/user/userguide/qpi-compute.rst @@ -4,24 +4,69 @@ .. (c) 2016 ZTE Corp. -Compute Suite -============= +Compute QPI +=========== Introduction ------------ -The QTIP testing suite aims to benchmark the compute components of an OPNFV platform. +The compute QPI aims to benchmark the compute components of an OPNFV platform. Such components include, the CPU performance, the memory performance. -Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too. -The test suite consists of both synthetic and application specific benchmarks to test compute components. +The compute QPI consists of both synthetic and application specific benchmarks to +test compute components. -All the compute benchmarks could be run in 2 scenarios: +All the compute benchmarks could be run in the scenario: +On Baremetal Machines provisioned by an OPNFV installer (Host machines) -1. On Baremetal Machines provisioned by an OPNFV installer (Host machines) -2. On Virtual Machines brought up through OpenStack on an OPNFV platform +Note: The Compute benchmank constains relatively old benchmarks such as dhrystone +and whetstone. The suite would be updated for better benchmarks such as Linbench for +the OPNFV E release. -Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release. +Getting start with compute QPI +------------------------------ + +Notice: All descriptions are based on QTIP container. + +Inventory File +^^^^^^^^^^^^^^ + +QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to +determine what hosts to work against. QTIP can automatically generate a inventory +file via OPNFV installer. Users also can write their own inventory infomation into +``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host +IP addresses. For example: +:: + + [hosts] + 10.20.0.11 + 10.20.0.12 + +QTIP key Pair +^^^^^^^^^^^^^ + +QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, +QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass +public key to remote hosts. + +If environment variable *CI_DEBUG* is set to *true*, users should delete it by +manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from +remote hosts before the execution ends. Please make sure the key deleted from remote +hosts or it can introduce a security flaw. + +Commands to run compute QPI +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In a QTIP container, you can run compute QPI by using QTIP CLI: +:: + + mkdir result + qtip plan run <plan_name> -p $PWD/result + +QTIP generates results in the ``$PWD/result`` directory are listed down under the +timestamp name. + +you can get more details from *userguide/cli.rst*. Benchmarks ---------- @@ -31,13 +76,15 @@ The benchmarks include: Dhrystone 2.1 ^^^^^^^^^^^^^ -Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. -Both Single CPU performance is measured along multi-cpu performance. +Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer +calculations to evaluate CPU capabilities. Both Single CPU performance is measured +along multi-cpu performance. Dhrystone, however, is a dated benchmark and has some short comings. Written in C, it is a small program that doesn't test the CPU memory subsystem. -Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration. +Additionally, dhrystone results could be modified by optimizing the compiler and +insome cases hardware configuration. References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf @@ -56,7 +103,9 @@ http://www.netlib.org/benchmark/whetstone.c OpenSSL Speed ^^^^^^^^^^^^^ -OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated: +OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, +two OpenSSL Speed benchmarks are incorporated: + 1. RSA signatunes/sec signed by a machine 2. AES 128-bit encryption throughput for a machine for cipher block sizes @@ -67,8 +116,9 @@ https://www.openssl.org/docs/manmaster/apps/speed.html RAMSpeed ^^^^^^^^ -RAMSpeed is used to measure a machine's memory perfomace. -The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used. +RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is +large enough to ensure Cache Misses so that the main machine memory is used. + INTmem and FLOATmem benchmarks are executed in 4 different scenarios: a. Copy: a(i)=b(i) @@ -76,7 +126,8 @@ b. Add: a(i)=b(i)+c(i) c. Scale: a(i)=b(i)*d d. Tniad: a(i)=b(i)+c(i)*d -INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks. +INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points +for these benchmarks. References: @@ -87,15 +138,18 @@ https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf DPI ^^^ -nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. -An example application called *pcapreader* has been developed and is available for use along nDPI. +nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that +is maintained by ntop. An example application called *pcapreader* has been developed +and is available for use along nDPI. -A sample .pcap file is passed to the *pcapreader* application. -nDPI classifies traffic in the pcap file into different categories based on string matching. -The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance. -The results are run 10 times and an average is taken for the obtained number. +A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic +in the pcap file into different categories based on string matching. The *pcapreader* +application provides a throughput number for the rate at which traffic was classified, +indicating a machine's computational performance. The results are run 10 times and an +average is taken for the obtained number. -*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes* +*nDPI may provide non consistent results and was added to Brahmaputra for experimental +purposes* References: |