From 0c8fdfafa8a85dd6e25497a9f9af9694e75ee4a1 Mon Sep 17 00:00:00 2001 From: Yujun Zhang Date: Thu, 30 Mar 2017 10:14:04 +0800 Subject: Normalize all section headers - folder with `index.rst` for parts, using # with overline for heading - `*.rst` file for chapter, using * with overline for heading In each *.rst file, use the following token for heading - =, for sections - -, for subsections - ^, for subsubsections - ", for paragraphs Change-Id: I6c247c1f1e27e891d0dade099195ef521f7dbb72 Signed-off-by: Yujun Zhang --- docs/release/release-notes/index.rst | 4 +- docs/testing/developer/devguide/api.rst | 11 +- docs/testing/developer/devguide/arch.rst | 6 +- docs/testing/developer/devguide/cli.rst | 103 ++++++++++++++--- docs/testing/developer/devguide/index.rst | 6 +- docs/testing/developer/devguide/overview.rst | 31 +++--- docs/testing/user/configguide/index.rst | 6 +- docs/testing/user/userguide/api.rst | 10 +- docs/testing/user/userguide/cli.rst | 10 +- docs/testing/user/userguide/compute.rst | 157 ++++++++++++++++++++++++++ docs/testing/user/userguide/index.rst | 6 +- docs/testing/user/userguide/overview.rst | 10 +- docs/testing/user/userguide/qpi-compute.rst | 158 --------------------------- 13 files changed, 297 insertions(+), 221 deletions(-) create mode 100644 docs/testing/user/userguide/compute.rst delete mode 100644 docs/testing/user/userguide/qpi-compute.rst diff --git a/docs/release/release-notes/index.rst b/docs/release/release-notes/index.rst index 8c52b5c8..e18699a6 100644 --- a/docs/release/release-notes/index.rst +++ b/docs/release/release-notes/index.rst @@ -6,9 +6,9 @@ .. (c) 2016 ZTE Corp. -################## +****************** QTIP Release Notes -################## +****************** .. toctree:: :maxdepth: 2 diff --git a/docs/testing/developer/devguide/api.rst b/docs/testing/developer/devguide/api.rst index 48ae3ae4..491c70f8 100644 --- a/docs/testing/developer/devguide/api.rst +++ b/docs/testing/developer/devguide/api.rst @@ -1,9 +1,10 @@ -********************************************** -QTIP RESTful Application Programming Interface -********************************************** +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 -Abstract -######## + +*************************************** +API - Application Programming Interface +*************************************** QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI subsystems(QPI's) such as compute, storage and network. diff --git a/docs/testing/developer/devguide/arch.rst b/docs/testing/developer/devguide/arch.rst index d95faba6..6b9208e9 100644 --- a/docs/testing/developer/devguide/arch.rst +++ b/docs/testing/developer/devguide/arch.rst @@ -3,9 +3,9 @@ .. (c) 2017 ZTE Corp. -######################## -QTIP Architecture Design -######################## +************ +Architecture +************ In Danube, QTIP releases its standalone mode, which is also know as ``solo``: diff --git a/docs/testing/developer/devguide/cli.rst b/docs/testing/developer/devguide/cli.rst index 487bdec5..7c681a64 100644 --- a/docs/testing/developer/devguide/cli.rst +++ b/docs/testing/developer/devguide/cli.rst @@ -1,24 +1,93 @@ -- Which framework has been used and why -- How to extend to more commands +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 -Original content from DEVELOP.md - ### CLI +**************************** +CLI - Command Line Interface +**************************** - Click currently supports Bash completion. The prerequisite for this is that the program - needs to be installed correctly. To install Qtip, execute the following command in root - folder of Qtip: +QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics fall under different NFVI +subsystems(QPI's) such as compute, storage and network. A plan consists of one or more QPI's, depending upon how +the end user would want to measure performance. CLI is designed to help the user, execute benchmarks and +view respective scores. - ``` - cd - pip install -e . - ``` +Framework +========= - Once the installation has been completed successfully, the following needs to be added to - the `.bashrc` file: +QTIP CLI has been created using the Python package `Click`_, Command Line Interface Creation Kit. It has been +chosen for number of reasons. It presents the user with a very simple yet powerful API to build complex +applications. One of the most striking features is command nesting. - ``` - eval "$(_QTIP_COMPLETE=source qtip)" - ``` +As explained, QTIP consists of metrics, QPI's and plans. CLI is designed to provide interface to all +these components. It is responsible for execution, as well as provide listing and details of each individual +element making up these components. - The above would activate command completion for Qtip. +Design +====== + +CLI's entry point extends Click's built in MultiCommand class object. It provides two methods, which are +overridden to provide custom configurations. + +.. code-block:: python + + class QtipCli(click.MultiCommand): + + def list_commands(self, ctx): + rv = [] + for filename in os.listdir(cmd_folder): + if filename.endswith('.py') and \ + filename.startswith('cmd_'): + rv.append(filename[4:-3]) + rv.sort() + return rv + + def get_command(self, ctx, name): + try: + if sys.version_info[0] == 2: + name = name.encode('ascii', 'replace') + mod = __import__('qtip.cli.commands.cmd_' + name, + None, None, ['cli']) + except ImportError: + return + return mod.cli + +Commands and subcommands will then be loaded by the ``get_command`` method above. + +Extending the Framework +======================= + +Framework can be easily extended, as per the users requirements. One such example can be to override the builtin +configurations with user defined ones. These can be written in a file, loaded via a Click Context and passed +through to all the commands. + +.. code-block:: python + + class Context: + + def __init__(): + + self.config = ConfigParser.ConfigParser() + self.config.read('path/to/configuration_file') + + def get_paths(): + + paths = self.config.get('section', 'path') + return paths + +The above example loads configuration from user defined paths, which then need to be provided to the actual +command definitions. + +.. code-block:: python + + from qtip.cli.entry import Context + + pass_context = click.make_pass_decorator(Context, ensure=False) + + @cli.command('list', help='List the Plans') + @pass_context + def list(ctx): + plans = Plan.list_all(ctx.paths()) + table = utils.table('Plans', plans) + click.echo(table) + +.. _Click: http://click.pocoo.org/5/ diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 89113e56..1c993dc9 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -3,9 +3,9 @@ .. (c) 2016 ZTE Corp. -########################## -QTIP Design Specifications -########################## +*************** +Developer Guide +*************** .. toctree:: :maxdepth: 2 diff --git a/docs/testing/developer/devguide/overview.rst b/docs/testing/developer/devguide/overview.rst index d394574e..1d7e22fe 100644 --- a/docs/testing/developer/devguide/overview.rst +++ b/docs/testing/developer/devguide/overview.rst @@ -3,9 +3,9 @@ .. (c) 2017 ZTE Corporation -######## +******** Overview -######## +******** QTIP uses Python as primary programming language and build the framework from the following packages @@ -19,16 +19,15 @@ docs `sphinx`_ - a tool that makes it easy to create intelligent and beautif testing `pytest`_ - a mature full-featured Python testing tool that helps you write better programs ======== =============================================================================================================== -*********** + Source Code -*********** +=========== - The structure of repository is based on the recommended sample in -`The Hitchhiker's Guide to Python`_ +The structure of repository is based on the recommended sample in `The Hitchhiker's Guide to Python`_ -================== ======================================================================================================== +================== ==================================================================================================== Path Content -================== ======================================================================================================== +================== ==================================================================================================== ``./benchmarks/`` builtin benchmark assets including plan, QPI and metrics ``./contrib/`` independent project/plugin/code contributed to QTIP ``./docker/`` configuration for building Docker image for QTIP deployment @@ -38,11 +37,11 @@ Path Content ``./qtip/`` the actual package ``./tests/`` package functional and unit tests ``./third-party/`` third part included in QTIP project -================== ======================================================================================================== +================== ==================================================================================================== + -************ Coding Style -************ +============ QTIP follows `OpenStack Style Guidelines`_ for source code and commit message. @@ -52,19 +51,19 @@ Specially, it is recommended to link each patch set with a JIRA issue. Put:: in commit message to create an automatic link. -******* + Testing -******* +======= All testing related code are stored in ``./tests/`` -================== ======================================================================================================== +================== ==================================================================================================== Path Content -================== ======================================================================================================== +================== ==================================================================================================== ``./tests/data/`` data fixtures for testing ``./tests/unit/`` unit test for each module, follow the same layout as ./qtip/ ``./conftest.py`` pytest configuration in project scope -================== ======================================================================================================== +================== ==================================================================================================== `tox`_ is used to automate the testing tasks diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index d5e05d63..43c32cab 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -4,9 +4,9 @@ .. (c) 2016 ZTE Corp. -################# -QTIP Config Guide -################# +********************************* +QTIP Installation & Configuration +********************************* .. toctree:: :maxdepth: 2 diff --git a/docs/testing/user/userguide/api.rst b/docs/testing/user/userguide/api.rst index 080fef5f..7e1d7b1c 100644 --- a/docs/testing/user/userguide/api.rst +++ b/docs/testing/user/userguide/api.rst @@ -1,6 +1,10 @@ -************** -QTIP API Usage -************** +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + + +*************** +API User Manual +*************** QTIP consists of a number of benchmarking tools or metrics, grouped under QPI's. QPI's map to the different components of an NFVI ecosystem, such as compute, network and storage. Depending on the type of application, diff --git a/docs/testing/user/userguide/cli.rst b/docs/testing/user/userguide/cli.rst index 96026c5b..99efd930 100644 --- a/docs/testing/user/userguide/cli.rst +++ b/docs/testing/user/userguide/cli.rst @@ -1,6 +1,10 @@ -************** -QTIP CLI Usage -************** +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + + +*************** +CLI User Manual +*************** QTIP consists of a number of benchmarking tools or metrics, grouped under QPI's. QPI's map to the different components of a NFVI ecosystem, such as compute, network and storage. Depending on the type of application, diff --git a/docs/testing/user/userguide/compute.rst b/docs/testing/user/userguide/compute.rst new file mode 100644 index 00000000..f889bfe6 --- /dev/null +++ b/docs/testing/user/userguide/compute.rst @@ -0,0 +1,157 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2015 Dell Inc. +.. (c) 2016 ZTE Corp. + + +******************************** +Compute Performance Benchmarking +******************************** + +The compute QPI aims to benchmark the compute components of an OPNFV platform. +Such components include, the CPU performance, the memory performance. + +The compute QPI consists of both synthetic and application specific benchmarks to +test compute components. + +All the compute benchmarks could be run in the scenario: +On Baremetal Machines provisioned by an OPNFV installer (Host machines) + +Note: The Compute benchmank constains relatively old benchmarks such as dhrystone +and whetstone. The suite would be updated for better benchmarks such as Linbench for +the OPNFV E release. + + +Getting started +=============== + +Notice: All descriptions are based on QTIP container. + +Inventory File +-------------- + +QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to +determine what hosts to work against. QTIP can automatically generate a inventory +file via OPNFV installer. Users also can write their own inventory infomation into +``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host +IP addresses. For example: +:: + + [hosts] + 10.20.0.11 + 10.20.0.12 + +QTIP key Pair +------------- + +QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, +QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass +public key to remote hosts. + +If environment variable *CI_DEBUG* is set to *true*, users should delete it by +manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from +remote hosts before the execution ends. Please make sure the key deleted from remote +hosts or it can introduce a security flaw. + +Commands +-------- + +In a QTIP container, you can run compute QPI by using QTIP CLI: +:: + + mkdir result + qtip plan run -p $PWD/result + +QTIP generates results in the ``$PWD/result`` directory are listed down under the +timestamp name. + +you can get more details from *userguide/cli.rst*. + +Metrics +------- + +The benchmarks include: + +Dhrystone 2.1 +^^^^^^^^^^^^^ + +Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer +calculations to evaluate CPU capabilities. Both Single CPU performance is measured +along multi-cpu performance. + + +Dhrystone, however, is a dated benchmark and has some short comings. +Written in C, it is a small program that doesn't test the CPU memory subsystem. +Additionally, dhrystone results could be modified by optimizing the compiler and +insome cases hardware configuration. + +References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf + +Whetstone +^^^^^^^^^ + +Whetstone is a synthetic benchmark to measure CPU floating point operation performance. +Both Single CPU performance is measured along multi-cpu performance. + +Like Dhrystone, Whetstone is a dated benchmark and has short comings. + +References: + +http://www.netlib.org/benchmark/whetstone.c + +OpenSSL Speed +^^^^^^^^^^^^^ + +OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, +two OpenSSL Speed benchmarks are incorporated: + +1. RSA signatunes/sec signed by a machine +2. AES 128-bit encryption throughput for a machine for cipher block sizes + +References: + +https://www.openssl.org/docs/manmaster/apps/speed.html + +RAMSpeed +^^^^^^^^ + +RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is +large enough to ensure Cache Misses so that the main machine memory is used. + +INTmem and FLOATmem benchmarks are executed in 4 different scenarios: + +a. Copy: a(i)=b(i) +b. Add: a(i)=b(i)+c(i) +c. Scale: a(i)=b(i)*d +d. Tniad: a(i)=b(i)+c(i)*d + +INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points +for these benchmarks. + +References: + +http://alasir.com/software/ramspeed/ + +https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements + +DPI +^^^ + +nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that +is maintained by ntop. An example application called *pcapreader* has been developed +and is available for use along nDPI. + +A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic +in the pcap file into different categories based on string matching. The *pcapreader* +application provides a throughput number for the rate at which traffic was classified, +indicating a machine's computational performance. The results are run 10 times and an +average is taken for the obtained number. + +*nDPI may provide non consistent results and was added to Brahmaputra for experimental +purposes* + +References: + +http://www.ntop.org/products/deep-packet-inspection/ndpi/ + +http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 04a12f08..d0d555f8 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -4,9 +4,9 @@ .. (c) 2016 ZTE Corp. -############### +*************** QTIP User Guide -############### +*************** .. toctree:: :maxdepth: 2 @@ -14,4 +14,4 @@ QTIP User Guide overview.rst cli.rst api.rst - qpi-compute.rst + compute.rst diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst index 1ad0b670..726d70bc 100644 --- a/docs/testing/user/userguide/overview.rst +++ b/docs/testing/user/userguide/overview.rst @@ -3,9 +3,9 @@ .. (c) 2017 ZTE Corp. -############ -Introduction -############ +******** +Overview +******** `QTIP`_ is the project for **Platform Performance Benchmarking** in `OPNFV`_. It aims to provide user a simple indicator for performance, simple but supported by comprehensive testing data and transparent calculation formula. @@ -18,9 +18,9 @@ performance. **TRUE** reflects the core value of QPI in four aspects - *Understandable*: QPI is broke down into section scores, and workload scores in report to help user to understand - *Extensible*: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics -########## + Benchmarks -########## +========== The builtin benchmarks of QTIP are located in ``/benchmarks`` folder diff --git a/docs/testing/user/userguide/qpi-compute.rst b/docs/testing/user/userguide/qpi-compute.rst deleted file mode 100644 index d64925bd..00000000 --- a/docs/testing/user/userguide/qpi-compute.rst +++ /dev/null @@ -1,158 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - - -Compute QPI -=========== - -Introduction ------------- - -The compute QPI aims to benchmark the compute components of an OPNFV platform. -Such components include, the CPU performance, the memory performance. - -The compute QPI consists of both synthetic and application specific benchmarks to -test compute components. - -All the compute benchmarks could be run in the scenario: -On Baremetal Machines provisioned by an OPNFV installer (Host machines) - -Note: The Compute benchmank constains relatively old benchmarks such as dhrystone -and whetstone. The suite would be updated for better benchmarks such as Linbench for -the OPNFV E release. - -Getting start with compute QPI ------------------------------- - -Notice: All descriptions are based on QTIP container. - -Inventory File -^^^^^^^^^^^^^^ - -QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to -determine what hosts to work against. QTIP can automatically generate a inventory -file via OPNFV installer. Users also can write their own inventory infomation into -``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host -IP addresses. For example: -:: - - [hosts] - 10.20.0.11 - 10.20.0.12 - -QTIP key Pair -^^^^^^^^^^^^^ - -QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, -QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass -public key to remote hosts. - -If environment variable *CI_DEBUG* is set to *true*, users should delete it by -manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from -remote hosts before the execution ends. Please make sure the key deleted from remote -hosts or it can introduce a security flaw. - -Commands to run compute QPI -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In a QTIP container, you can run compute QPI by using QTIP CLI: -:: - - mkdir result - qtip plan run -p $PWD/result - -QTIP generates results in the ``$PWD/result`` directory are listed down under the -timestamp name. - -you can get more details from *userguide/cli.rst*. - -Benchmarks ----------- - -The benchmarks include: - -Dhrystone 2.1 -^^^^^^^^^^^^^ - -Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer -calculations to evaluate CPU capabilities. Both Single CPU performance is measured -along multi-cpu performance. - - -Dhrystone, however, is a dated benchmark and has some short comings. -Written in C, it is a small program that doesn't test the CPU memory subsystem. -Additionally, dhrystone results could be modified by optimizing the compiler and -insome cases hardware configuration. - -References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf - -Whetstone -^^^^^^^^^ - -Whetstone is a synthetic benchmark to measure CPU floating point operation performance. -Both Single CPU performance is measured along multi-cpu performance. - -Like Dhrystone, Whetstone is a dated benchmark and has short comings. - -References: - -http://www.netlib.org/benchmark/whetstone.c - -OpenSSL Speed -^^^^^^^^^^^^^ - -OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, -two OpenSSL Speed benchmarks are incorporated: - -1. RSA signatunes/sec signed by a machine -2. AES 128-bit encryption throughput for a machine for cipher block sizes - -References: - -https://www.openssl.org/docs/manmaster/apps/speed.html - -RAMSpeed -^^^^^^^^ - -RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is -large enough to ensure Cache Misses so that the main machine memory is used. - -INTmem and FLOATmem benchmarks are executed in 4 different scenarios: - -a. Copy: a(i)=b(i) -b. Add: a(i)=b(i)+c(i) -c. Scale: a(i)=b(i)*d -d. Tniad: a(i)=b(i)+c(i)*d - -INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points -for these benchmarks. - -References: - -http://alasir.com/software/ramspeed/ - -https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Untangling+memory+access+measurements - -DPI -^^^ - -nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that -is maintained by ntop. An example application called *pcapreader* has been developed -and is available for use along nDPI. - -A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic -in the pcap file into different categories based on string matching. The *pcapreader* -application provides a throughput number for the rate at which traffic was classified, -indicating a machine's computational performance. The results are run 10 times and an -average is taken for the obtained number. - -*nDPI may provide non consistent results and was added to Brahmaputra for experimental -purposes* - -References: - -http://www.ntop.org/products/deep-packet-inspection/ndpi/ - -http://www.ntop.org/wp-content/uploads/2013/12/nDPI_QuickStartGuide.pdf -- cgit 1.2.3-korg