aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing/user
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/user')
-rw-r--r--docs/testing/user/configguide/configuration.rst98
-rw-r--r--docs/testing/user/userguide/api.rst91
-rw-r--r--docs/testing/user/userguide/cli.rst6
-rw-r--r--docs/testing/user/userguide/index.rst2
-rw-r--r--docs/testing/user/userguide/overview.rst32
-rw-r--r--docs/testing/user/userguide/qpi-compute.rst100
6 files changed, 234 insertions, 95 deletions
diff --git a/docs/testing/user/configguide/configuration.rst b/docs/testing/user/configguide/configuration.rst
index 78e96492..8cc891f0 100644
--- a/docs/testing/user/configguide/configuration.rst
+++ b/docs/testing/user/configguide/configuration.rst
@@ -18,66 +18,52 @@ to configure OPNFV with this specific installer
Installing QTIP using Docker
============================
+QTIP docker image
+-----------------
+
QTIP has a Docker images on the docker hub. Pulling opnfv/qtip docker image
from docker hub:
::
- docker pull opnfv/qtip
+ docker pull opnfv/qtip:stable
-Verify that opnfv/qtip has been downloaded. It should be listed as an image by
+Verify that ``opnfv/qtip`` has been downloaded. It should be listed as an image by
running the following command.
::
docker images
-Make dir to store the QTIP image which will be used to create vm in cloud.
-::
-
- mkdir $HOME/imgstore
-Run and enter the Docker instance:
-::
- envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP}
--e NODE_NAME={NODE_NAME}"
- docker run --name qtip -id -e $envs -v "$HOME/imgstore:/home/opnfv/imgstore" opnfv/qtip
- docker exec -i -t qtip /bin/bash
+Run and enter the docker instance
+---------------------------------
-Now you are in the container and QTIP can be found in the /repos/qtip and can
-be navigated to using the following command.
+1. If you want to run benchmarks:
::
- cd repos/qtip
-
-
-OpenStack parameters and credentials
-====================================
+ envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP}"
+ docker run --name qtip -id -e $envs opnfv/qtip
+ docker exec -i -t qtip /bin/bash
+``INSTALLER_TYPE`` should be one of OPNFV installer, e.g. apex, compass, daisy, fuel
+and joid. Currenty, QTIP only supports installer fuel.
-Environment variables
----------------------
+``INSTALLER_IP`` is the ip address of the installer that can be accessed by QTIP.
-Before running QTIP it is necessary to export OpenStack environment variables
-from the OpenStack *openrc* file. This can be done by running the following
-command.
+2. If you do not want to run any benchmarks:
::
- source scripts/get_env_info.sh -n {INSTALLER_TYPE} -i {INSTALLER_IP}
- source opnfv-creds.sh
-
-This provides a ``opnfv-creds.sh`` file which can be sources to get the
-environment variables.
-
-
-QTIP default key pair
-----------------------
+ docker run --name qtip -id opnfv/qtip
+ docker exec -i -t qtip /bin/bash
-QTIP uses a SSH key pair to connect to the guest image. You should generate key pair
-before running QTIP test. And put key pair in the ``config/`` directory.
+Now you are in the container and QTIP can be found in the ``/repos/qtip`` and can
+be navigated to using the following command.
::
- ssh-keygen -t rsa -N "" -f config/QtipKey -q
+ cd repos/qtip
+Environment configuration
+=========================
Hardware configuration
----------------------
@@ -91,46 +77,14 @@ Jumphost configuration
Installer Docker on Jumphost, which is used for running Qtip image.
-The first step is to install docker:
-::
+You can refer to these links:
- sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80
- --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
-
-
-Add an entry for your Ubuntu operating system:
-::
-
- Open the /etc/apt/sources.list.d/docker.list file in your favorite editor.
-
-If the file doesn’t exist, create it.
-
-Remove any existing entries.
-
-Add an entry for your Ubuntu operating system.
-
-On Ubuntu Trusty 14.04 (LTS)
-::
-
- deb https://apt.dockerproject.org/repo ubuntu-trusty main
-
-Update the package manager
-::
-
- sudo apt-get update
-
-Install Docker:
-::
-
- sudo apt-get install docker-engine
-
-Starting Docker Daemon:
-::
+Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntu/
- sudo service docker start
+Centos: https://docs.docker.com/engine/installation/linux/centos/
Platform components configuration
---------------------------------
-Describe the configuration of each component in the installer
+Describe the configuration of each component in the installer.
diff --git a/docs/testing/user/userguide/api.rst b/docs/testing/user/userguide/api.rst
new file mode 100644
index 00000000..080fef5f
--- /dev/null
+++ b/docs/testing/user/userguide/api.rst
@@ -0,0 +1,91 @@
+**************
+QTIP API Usage
+**************
+
+QTIP consists of a number of benchmarking tools or metrics, grouped under QPI's. QPI's map to the different
+components of an NFVI ecosystem, such as compute, network and storage. Depending on the type of application,
+a user may group them under plans.
+
+QTIP API provides a RESTful interface to all of the above components. User can retrieve list of plans, QPIs
+and metrics and their individual information.
+
+
+Running
+=======
+
+After installing QTIP. API server can be run using command ``qtip-api`` on the local machine.
+
+All the resources and their corresponding operation details can be seen at ``/v1.0/ui``,
+on hosting server(``0.0.0.0:5000`` for the local machine).
+
+The whole API specification in json format can be seen at ``/v1.0/swagger.json``.
+
+The data models are given below:
+
+ * Plan
+ * Metric
+ * QPI
+
+Plan::
+
+ {
+ "name": <plan name>,
+ "description": <plan profile>,
+ "info": <{plan info}>,
+ "config": <{plan configuration}>,
+ "QPIs": <[list of qpis]>,
+ },
+
+Metric::
+
+ {
+ "name": <metric name>,
+ "description": <metric description>,
+ "links": <[links with metric information]>,
+ "workloads": <[cpu workloads(single_cpu, multi_cpu]>,
+ },
+
+QPI::
+
+ {
+ "name": <qpi name>,
+ "description": <qpi description>,
+ "formula": <formula>,
+ "sections": <[list of sections with different metrics and formulaes]>,
+ }
+
+The API can be described as follows
+
+Plans:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /v1.0/plans | Get the list of of all plans |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /v1.0/plans/{name} | Get details of the specified plan |
+ +--------+----------------------------+-----------------------------------------+
+
+Metrics:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /v1.0/metrics | Get the list of all metrics |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /v1.0/metrics/{name} | Get details of specified metric |
+ +--------+----------------------------+-----------------------------------------+
+
+QPIs:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /v1.0/qpis | Get the list of all QPIs |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /v1.0/qpis/{name} | Get details of specified QPI |
+ +--------+----------------------------+-----------------------------------------+
+
+
+*Note:*
+ *running API with connexion cli does not require base path (/v1.0/) in url*
diff --git a/docs/testing/user/userguide/cli.rst b/docs/testing/user/userguide/cli.rst
index 19420bd1..96026c5b 100644
--- a/docs/testing/user/userguide/cli.rst
+++ b/docs/testing/user/userguide/cli.rst
@@ -38,6 +38,12 @@ as above. An important thing to remember is to provide absolute path of result d
qtip plan run <plan_name> -p $PWD/result
Similarly, the same commands can be used for the other two components making up the plans, i.e QPI's and metrics.
+For example, in order to run a single metric
+::
+
+ qtip metric run <metric_name> -p $PWD/result
+
+The same can be applied for a QPI.
QTIP also provides the utility to view benchmarking results on the console. One just need to provide to where
the results are stored. Extending the example above
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
index 78c5d117..04a12f08 100644
--- a/docs/testing/user/userguide/index.rst
+++ b/docs/testing/user/userguide/index.rst
@@ -11,5 +11,7 @@ QTIP User Guide
.. toctree::
:maxdepth: 2
+ overview.rst
cli.rst
+ api.rst
qpi-compute.rst
diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst
new file mode 100644
index 00000000..1ad0b670
--- /dev/null
+++ b/docs/testing/user/userguide/overview.rst
@@ -0,0 +1,32 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) 2017 ZTE Corp.
+
+
+############
+Introduction
+############
+
+`QTIP`_ is the project for **Platform Performance Benchmarking** in `OPNFV`_. It aims to provide user a simple indicator
+for performance, simple but supported by comprehensive testing data and transparent calculation formula.
+
+QTIP introduces a concept called **QPI**, a.k.a. QTIP Performance Index, which aims to be a **TRUE** indicator of
+performance. **TRUE** reflects the core value of QPI in four aspects
+
+- *Transparent*: being an open source project, user can inspect all details behind QPI, e.g. formulas, metrics, raw data
+- *Reliable*: the integrity of QPI will be guaranteed by traceability in each step back to raw test result
+- *Understandable*: QPI is broke down into section scores, and workload scores in report to help user to understand
+- *Extensible*: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics
+
+##########
+Benchmarks
+##########
+
+The builtin benchmarks of QTIP are located in ``<package_root>/benchmarks`` folder
+
+- *QPI*: specifications about how an QPI is calculated and sources of metrics
+- *metric*: performance metrics referred in QPI, currently it is categorized by performance testing tools
+- *plan*: executable benchmarking plan which collects metrics and calculate QPI
+
+.. _QTIP: https://wiki.opnfv.org/display/qtip
+.. _OPNFV: https://www.opnfv.org/
diff --git a/docs/testing/user/userguide/qpi-compute.rst b/docs/testing/user/userguide/qpi-compute.rst
index 369240c9..d64925bd 100644
--- a/docs/testing/user/userguide/qpi-compute.rst
+++ b/docs/testing/user/userguide/qpi-compute.rst
@@ -4,24 +4,69 @@
.. (c) 2016 ZTE Corp.
-Compute Suite
-=============
+Compute QPI
+===========
Introduction
------------
-The QTIP testing suite aims to benchmark the compute components of an OPNFV platform.
+The compute QPI aims to benchmark the compute components of an OPNFV platform.
Such components include, the CPU performance, the memory performance.
-Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too.
-The test suite consists of both synthetic and application specific benchmarks to test compute components.
+The compute QPI consists of both synthetic and application specific benchmarks to
+test compute components.
-All the compute benchmarks could be run in 2 scenarios:
+All the compute benchmarks could be run in the scenario:
+On Baremetal Machines provisioned by an OPNFV installer (Host machines)
-1. On Baremetal Machines provisioned by an OPNFV installer (Host machines)
-2. On Virtual Machines brought up through OpenStack on an OPNFV platform
+Note: The Compute benchmank constains relatively old benchmarks such as dhrystone
+and whetstone. The suite would be updated for better benchmarks such as Linbench for
+the OPNFV E release.
-Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release.
+Getting start with compute QPI
+------------------------------
+
+Notice: All descriptions are based on QTIP container.
+
+Inventory File
+^^^^^^^^^^^^^^
+
+QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to
+determine what hosts to work against. QTIP can automatically generate a inventory
+file via OPNFV installer. Users also can write their own inventory infomation into
+``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host
+IP addresses. For example:
+::
+
+ [hosts]
+ 10.20.0.11
+ 10.20.0.12
+
+QTIP key Pair
+^^^^^^^^^^^^^
+
+QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI,
+QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass
+public key to remote hosts.
+
+If environment variable *CI_DEBUG* is set to *true*, users should delete it by
+manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from
+remote hosts before the execution ends. Please make sure the key deleted from remote
+hosts or it can introduce a security flaw.
+
+Commands to run compute QPI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In a QTIP container, you can run compute QPI by using QTIP CLI:
+::
+
+ mkdir result
+ qtip plan run <plan_name> -p $PWD/result
+
+QTIP generates results in the ``$PWD/result`` directory are listed down under the
+timestamp name.
+
+you can get more details from *userguide/cli.rst*.
Benchmarks
----------
@@ -31,13 +76,15 @@ The benchmarks include:
Dhrystone 2.1
^^^^^^^^^^^^^
-Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities.
-Both Single CPU performance is measured along multi-cpu performance.
+Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer
+calculations to evaluate CPU capabilities. Both Single CPU performance is measured
+along multi-cpu performance.
Dhrystone, however, is a dated benchmark and has some short comings.
Written in C, it is a small program that doesn't test the CPU memory subsystem.
-Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration.
+Additionally, dhrystone results could be modified by optimizing the compiler and
+insome cases hardware configuration.
References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf
@@ -56,7 +103,9 @@ http://www.netlib.org/benchmark/whetstone.c
OpenSSL Speed
^^^^^^^^^^^^^
-OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated:
+OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP,
+two OpenSSL Speed benchmarks are incorporated:
+
1. RSA signatunes/sec signed by a machine
2. AES 128-bit encryption throughput for a machine for cipher block sizes
@@ -67,8 +116,9 @@ https://www.openssl.org/docs/manmaster/apps/speed.html
RAMSpeed
^^^^^^^^
-RAMSpeed is used to measure a machine's memory perfomace.
-The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used.
+RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is
+large enough to ensure Cache Misses so that the main machine memory is used.
+
INTmem and FLOATmem benchmarks are executed in 4 different scenarios:
a. Copy: a(i)=b(i)
@@ -76,7 +126,8 @@ b. Add: a(i)=b(i)+c(i)
c. Scale: a(i)=b(i)*d
d. Tniad: a(i)=b(i)+c(i)*d
-INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks.
+INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points
+for these benchmarks.
References:
@@ -87,15 +138,18 @@ https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf
DPI
^^^
-nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop.
-An example application called *pcapreader* has been developed and is available for use along nDPI.
+nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that
+is maintained by ntop. An example application called *pcapreader* has been developed
+and is available for use along nDPI.
-A sample .pcap file is passed to the *pcapreader* application.
-nDPI classifies traffic in the pcap file into different categories based on string matching.
-The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance.
-The results are run 10 times and an average is taken for the obtained number.
+A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic
+in the pcap file into different categories based on string matching. The *pcapreader*
+application provides a throughput number for the rate at which traffic was classified,
+indicating a machine's computational performance. The results are run 10 times and an
+average is taken for the obtained number.
-*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes*
+*nDPI may provide non consistent results and was added to Brahmaputra for experimental
+purposes*
References: