diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/developer/devguide/api.rst | 239 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/overview.rst | 27 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configuration.rst | 98 | ||||
| -rw-r--r-- | docs/testing/user/userguide/api.rst | 91 | ||||
| -rw-r--r-- | docs/testing/user/userguide/cli.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 2 | ||||
| -rw-r--r-- | docs/testing/user/userguide/overview.rst | 32 | ||||
| -rw-r--r-- | docs/testing/user/userguide/qpi-compute.rst | 100 |
9 files changed, 499 insertions, 97 deletions
diff --git a/docs/testing/developer/devguide/api.rst b/docs/testing/developer/devguide/api.rst index eb2b0d67..48ae3ae4 100644 --- a/docs/testing/developer/devguide/api.rst +++ b/docs/testing/developer/devguide/api.rst @@ -1,2 +1,237 @@ -- Which framework has been used and why -- How to extend to more api +********************************************** +QTIP RESTful Application Programming Interface +********************************************** + +Abstract +######## + +QTIP consists of different tools(metrics) to benchmark the NFVI. These metrics +fall under different NFVI subsystems(QPI's) such as compute, storage and network. +A plan consists of one or more QPI's, depending upon how the end-user would want +to measure performance. API is designed to expose a RESTful interface to the user +for executing benchmarks and viewing respective scores. + +Framework +========= + +QTIP API has been created using the Python package `Connexion`_. It has been chosen +for a number of reasons. It follows API First approach to create micro-services. +Hence, firstly the API specifications are defined from the client side perspective, +followed by the implementation of the micro-service. It decouples the business logic +from routing and resource mapping making design and implementation cleaner. + +It has two major components: + +API Specifications + + The API specification is defined in a yaml or json file. Connexion follows + `Open API specification`_ to determine the design and maps the endpoints to methods in python. + +Micro-service Implementation + Connexion maps the ``operationId`` corresponding to every operation in API + Specification to methods in python which handles request and responses. + +As explained, QTIP consists of metrics, QPI's and plans. The API is designed to provide +a RESTful interface to all these components. It is responsible to provide listing and details of +each individual element making up these components. + +Design +====== + +Specification +------------- + +API's entry point (``main``) runs connexion ``App`` class object after adding API Specification +using ``App.add_api`` method. It loads specification from ``swagger.yaml`` file by specifying +``specification_dir``. + +Connexion reads API's endpoints(paths), operations, their request and response parameter +details and response definitions from the API specification i.e. ``swagger.yaml`` in this case. + +Following example demonstrates specification for the resource ``plans``. + +:: + + paths: + /plans/{name}: + get: + summary: Get a plan by plan name + operationId: qtip.api.controllers.plan.get_plan + tags: + - Plan + - Standalone + parameters: + - name: name + in: path + description: Plan name + required: true + type: string + responses: + 200: + description: Plan information + schema: + $ref: '#/definitions/Plan' + 404: + description: Plan not found + schema: + $ref: '#/definitions/Error' + 501: + description: Resource not implemented + schema: + $ref: '#/definitions/Error' + default: + description: Unexpected error + schema: + $ref: '#/definitions/Error' + definitions: + Plan: + type: object + required: + - name + properties: + name: + type: string + description: + type: string + info: + type: object + config: + type: object + +Every ``operationId`` in above operations corresponds to a method in controllers. +QTIP has three controller modules each for plan, QPI and metric. Connexion will +read these mappings and automatically route endpoints to business logic. + +`Swagger Editor`_ can be explored to play with more such examples and to validate +the specification. + +Controllers +----------- + +The request is handled through these methods and response is sent back to the client. +Connexion takes care of data validation. + +.. code-block:: python + + @common.check_endpoint_for_error(resource='Plan') + def get_plan(name): + plan_spec = plan.Plan(name) + return plan_spec.content + +In above code ``get_plan`` takes a plan name and return its content. + +The decorator ``check_endpoint_for_error`` defined in ``common`` is used to handle error +and return a suitable error response. + + +During Development the server can be run by passing specification file(``swagger.yaml`` +in this case) to connexion cli - + +:: + + connexion run <path_to_specification_file> -v + + +Extending the Framework +======================= + +Modifying Existing API: +----------------------- + API can be modified by adding entries in ``swagger.yaml`` and adding the corresponding + controller mapped from ``operationID``. + + Adding endpoints: + + New endpoints can be defined in ``paths`` section in ``swagger.yaml``. To add a new resource *dummy* - + + :: + + paths: + /dummies: + get: + summary: Get all dummies + operationId: qtip.api.controllers.dummy.get_dummies + tags: + - dummy + responses: + 200: + description: Foo information + schema: + $ref: '#/definitions/Dummy + default: + description: Unexpected error + schema: + $ref: '#/definitions/Error' + + + And then model of the resource can be defined in the ``definitions`` section. + + :: + + definitions: + Dummy: + type: object + required: + - name + properties: + name: + type: string + description: + type: string + id: + type: string + + + Adding controller methods: + Methods for handling requests and responses for every operation for the endpoint added can be + implemented in ``controller``. + + In ``controllers.dummy`` + + .. code-block:: python + + def get_dummies(): + all_dummies = [<code to get all dummies>] + return all_dummies, httplib.OK + + Adding error responses + Decorators for handling errors are defined in ``common.py`` in ``api``. + + .. code-block:: python + + from qtip.api import common + + @common.check_endpoint_for_error(resource='dummy',operation='get') + def get_dummies() + all_dummies = [<code to get all dummies>] + return all_dummies + +Adding new API: +--------------- + + API can easily be extended by adding more APIs to ``Connexion.App`` class object using + ``add_api`` class method. + + In ``__main__`` + + .. code-block:: python + + def get_app(): + app = connexion.App(__name__, specification_dir=swagger_dir) + app.add_api('swagger.yaml', base_path='/v1.0', strict_validation=True) + return app + + + Extending it to add new APIs. The new API should have all endpoints mapped using ``operationId``. + + .. code-block:: python + + from qtip.api import __main__ + my_app = __main__.get_app() + my_app.add_api('new_api.yaml',base_path'api2',strict_validation=True) + my_app.run(host="0.0.0.0", port=5000) + + +.. _Connexion: https://connexion.readthedocs.io/en/latest/ +.. _Open API specification: https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md +.. _Swagger Editor: http://editor.swagger.io/ diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 2b4bd9b0..89113e56 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -10,6 +10,7 @@ QTIP Design Specifications .. toctree:: :maxdepth: 2 + overview.rst arch.rst cli.rst api.rst diff --git a/docs/testing/developer/devguide/overview.rst b/docs/testing/developer/devguide/overview.rst new file mode 100644 index 00000000..4ccaae20 --- /dev/null +++ b/docs/testing/developer/devguide/overview.rst @@ -0,0 +1,27 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2017 ZTE Corp. + + +######## +Overview +######## + +QTIP uses Python as primary programming language. The structure of repository is based on the recommended sample in +`The Hitchhiker's Guide to Python`_ + +============== ========================================================================== +Path Content +============== ========================================================================== +./benchmarks/ builtin benchmark assets including plan, QPI and metrics +./contrib/ independent project/plugin/code contributed to QTIP +./docker/ configuration for building Docker image for QTIP deployment +./docs/ release notes, user and developer documentation, design proposals +./legacy/ legacy obsoleted code that is unmaintained but kept for reference +./opt/ optional component, e.g. scripts to setup infrastructure services for QTIP +./qtip/ the actual package +./tests/ package functional and unit tests +./third-party/ third part included in QTIP project +============== ========================================================================== + +.. _The Hitchhiker's Guide to Python: http://python-guide-pt-br.readthedocs.io/en/latest/writing/structure/ diff --git a/docs/testing/user/configguide/configuration.rst b/docs/testing/user/configguide/configuration.rst index 78e96492..8cc891f0 100644 --- a/docs/testing/user/configguide/configuration.rst +++ b/docs/testing/user/configguide/configuration.rst @@ -18,66 +18,52 @@ to configure OPNFV with this specific installer Installing QTIP using Docker ============================ +QTIP docker image +----------------- + QTIP has a Docker images on the docker hub. Pulling opnfv/qtip docker image from docker hub: :: - docker pull opnfv/qtip + docker pull opnfv/qtip:stable -Verify that opnfv/qtip has been downloaded. It should be listed as an image by +Verify that ``opnfv/qtip`` has been downloaded. It should be listed as an image by running the following command. :: docker images -Make dir to store the QTIP image which will be used to create vm in cloud. -:: - - mkdir $HOME/imgstore -Run and enter the Docker instance: -:: - envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP} --e NODE_NAME={NODE_NAME}" - docker run --name qtip -id -e $envs -v "$HOME/imgstore:/home/opnfv/imgstore" opnfv/qtip - docker exec -i -t qtip /bin/bash +Run and enter the docker instance +--------------------------------- -Now you are in the container and QTIP can be found in the /repos/qtip and can -be navigated to using the following command. +1. If you want to run benchmarks: :: - cd repos/qtip - - -OpenStack parameters and credentials -==================================== + envs="INSTALLER_TYPE={INSTALLER_TYPE} -e INSTALLER_IP={INSTALLER_IP}" + docker run --name qtip -id -e $envs opnfv/qtip + docker exec -i -t qtip /bin/bash +``INSTALLER_TYPE`` should be one of OPNFV installer, e.g. apex, compass, daisy, fuel +and joid. Currenty, QTIP only supports installer fuel. -Environment variables ---------------------- +``INSTALLER_IP`` is the ip address of the installer that can be accessed by QTIP. -Before running QTIP it is necessary to export OpenStack environment variables -from the OpenStack *openrc* file. This can be done by running the following -command. +2. If you do not want to run any benchmarks: :: - source scripts/get_env_info.sh -n {INSTALLER_TYPE} -i {INSTALLER_IP} - source opnfv-creds.sh - -This provides a ``opnfv-creds.sh`` file which can be sources to get the -environment variables. - - -QTIP default key pair ----------------------- + docker run --name qtip -id opnfv/qtip + docker exec -i -t qtip /bin/bash -QTIP uses a SSH key pair to connect to the guest image. You should generate key pair -before running QTIP test. And put key pair in the ``config/`` directory. +Now you are in the container and QTIP can be found in the ``/repos/qtip`` and can +be navigated to using the following command. :: - ssh-keygen -t rsa -N "" -f config/QtipKey -q + cd repos/qtip +Environment configuration +========================= Hardware configuration ---------------------- @@ -91,46 +77,14 @@ Jumphost configuration Installer Docker on Jumphost, which is used for running Qtip image. -The first step is to install docker: -:: +You can refer to these links: - sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 - --recv-keys 58118E89F3A912897C070ADBF76221572C52609D - - -Add an entry for your Ubuntu operating system: -:: - - Open the /etc/apt/sources.list.d/docker.list file in your favorite editor. - -If the file doesn’t exist, create it. - -Remove any existing entries. - -Add an entry for your Ubuntu operating system. - -On Ubuntu Trusty 14.04 (LTS) -:: - - deb https://apt.dockerproject.org/repo ubuntu-trusty main - -Update the package manager -:: - - sudo apt-get update - -Install Docker: -:: - - sudo apt-get install docker-engine - -Starting Docker Daemon: -:: +Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntu/ - sudo service docker start +Centos: https://docs.docker.com/engine/installation/linux/centos/ Platform components configuration --------------------------------- -Describe the configuration of each component in the installer +Describe the configuration of each component in the installer. diff --git a/docs/testing/user/userguide/api.rst b/docs/testing/user/userguide/api.rst new file mode 100644 index 00000000..080fef5f --- /dev/null +++ b/docs/testing/user/userguide/api.rst @@ -0,0 +1,91 @@ +************** +QTIP API Usage +************** + +QTIP consists of a number of benchmarking tools or metrics, grouped under QPI's. QPI's map to the different +components of an NFVI ecosystem, such as compute, network and storage. Depending on the type of application, +a user may group them under plans. + +QTIP API provides a RESTful interface to all of the above components. User can retrieve list of plans, QPIs +and metrics and their individual information. + + +Running +======= + +After installing QTIP. API server can be run using command ``qtip-api`` on the local machine. + +All the resources and their corresponding operation details can be seen at ``/v1.0/ui``, +on hosting server(``0.0.0.0:5000`` for the local machine). + +The whole API specification in json format can be seen at ``/v1.0/swagger.json``. + +The data models are given below: + + * Plan + * Metric + * QPI + +Plan:: + + { + "name": <plan name>, + "description": <plan profile>, + "info": <{plan info}>, + "config": <{plan configuration}>, + "QPIs": <[list of qpis]>, + }, + +Metric:: + + { + "name": <metric name>, + "description": <metric description>, + "links": <[links with metric information]>, + "workloads": <[cpu workloads(single_cpu, multi_cpu]>, + }, + +QPI:: + + { + "name": <qpi name>, + "description": <qpi description>, + "formula": <formula>, + "sections": <[list of sections with different metrics and formulaes]>, + } + +The API can be described as follows + +Plans: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /v1.0/plans | Get the list of of all plans | + +--------+----------------------------+-----------------------------------------+ + | GET | /v1.0/plans/{name} | Get details of the specified plan | + +--------+----------------------------+-----------------------------------------+ + +Metrics: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /v1.0/metrics | Get the list of all metrics | + +--------+----------------------------+-----------------------------------------+ + | GET | /v1.0/metrics/{name} | Get details of specified metric | + +--------+----------------------------+-----------------------------------------+ + +QPIs: + + +--------+----------------------------+-----------------------------------------+ + | Method | Path | Description | + +========+============================+=========================================+ + | GET | /v1.0/qpis | Get the list of all QPIs | + +--------+----------------------------+-----------------------------------------+ + | GET | /v1.0/qpis/{name} | Get details of specified QPI | + +--------+----------------------------+-----------------------------------------+ + + +*Note:* + *running API with connexion cli does not require base path (/v1.0/) in url* diff --git a/docs/testing/user/userguide/cli.rst b/docs/testing/user/userguide/cli.rst index 19420bd1..96026c5b 100644 --- a/docs/testing/user/userguide/cli.rst +++ b/docs/testing/user/userguide/cli.rst @@ -38,6 +38,12 @@ as above. An important thing to remember is to provide absolute path of result d qtip plan run <plan_name> -p $PWD/result Similarly, the same commands can be used for the other two components making up the plans, i.e QPI's and metrics. +For example, in order to run a single metric +:: + + qtip metric run <metric_name> -p $PWD/result + +The same can be applied for a QPI. QTIP also provides the utility to view benchmarking results on the console. One just need to provide to where the results are stored. Extending the example above diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 78c5d117..04a12f08 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -11,5 +11,7 @@ QTIP User Guide .. toctree:: :maxdepth: 2 + overview.rst cli.rst + api.rst qpi-compute.rst diff --git a/docs/testing/user/userguide/overview.rst b/docs/testing/user/userguide/overview.rst new file mode 100644 index 00000000..1ad0b670 --- /dev/null +++ b/docs/testing/user/userguide/overview.rst @@ -0,0 +1,32 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2017 ZTE Corp. + + +############ +Introduction +############ + +`QTIP`_ is the project for **Platform Performance Benchmarking** in `OPNFV`_. It aims to provide user a simple indicator +for performance, simple but supported by comprehensive testing data and transparent calculation formula. + +QTIP introduces a concept called **QPI**, a.k.a. QTIP Performance Index, which aims to be a **TRUE** indicator of +performance. **TRUE** reflects the core value of QPI in four aspects + +- *Transparent*: being an open source project, user can inspect all details behind QPI, e.g. formulas, metrics, raw data +- *Reliable*: the integrity of QPI will be guaranteed by traceability in each step back to raw test result +- *Understandable*: QPI is broke down into section scores, and workload scores in report to help user to understand +- *Extensible*: users may create their own QPI by composing the existed metrics in QTIP or extend new metrics + +########## +Benchmarks +########## + +The builtin benchmarks of QTIP are located in ``<package_root>/benchmarks`` folder + +- *QPI*: specifications about how an QPI is calculated and sources of metrics +- *metric*: performance metrics referred in QPI, currently it is categorized by performance testing tools +- *plan*: executable benchmarking plan which collects metrics and calculate QPI + +.. _QTIP: https://wiki.opnfv.org/display/qtip +.. _OPNFV: https://www.opnfv.org/ diff --git a/docs/testing/user/userguide/qpi-compute.rst b/docs/testing/user/userguide/qpi-compute.rst index 369240c9..d64925bd 100644 --- a/docs/testing/user/userguide/qpi-compute.rst +++ b/docs/testing/user/userguide/qpi-compute.rst @@ -4,24 +4,69 @@ .. (c) 2016 ZTE Corp. -Compute Suite -============= +Compute QPI +=========== Introduction ------------ -The QTIP testing suite aims to benchmark the compute components of an OPNFV platform. +The compute QPI aims to benchmark the compute components of an OPNFV platform. Such components include, the CPU performance, the memory performance. -Additionally virtual computing performance provided by the Hypervisor (KVM) installed as part of OPNFV platforms would be benchmarked too. -The test suite consists of both synthetic and application specific benchmarks to test compute components. +The compute QPI consists of both synthetic and application specific benchmarks to +test compute components. -All the compute benchmarks could be run in 2 scenarios: +All the compute benchmarks could be run in the scenario: +On Baremetal Machines provisioned by an OPNFV installer (Host machines) -1. On Baremetal Machines provisioned by an OPNFV installer (Host machines) -2. On Virtual Machines brought up through OpenStack on an OPNFV platform +Note: The Compute benchmank constains relatively old benchmarks such as dhrystone +and whetstone. The suite would be updated for better benchmarks such as Linbench for +the OPNFV E release. -Note: The Compute benchmank suite constains relatively old benchmarks such as dhrystone and whetstone. The suite would be updated for better benchmarks such as Linbench for the OPNFV C release. +Getting start with compute QPI +------------------------------ + +Notice: All descriptions are based on QTIP container. + +Inventory File +^^^^^^^^^^^^^^ + +QTIP uses Ansible to trigger benchmark test. Ansible uses an inventory file to +determine what hosts to work against. QTIP can automatically generate a inventory +file via OPNFV installer. Users also can write their own inventory infomation into +``/home/opnfv/qtip/hosts``. This file is just a text file containing a list of host +IP addresses. For example: +:: + + [hosts] + 10.20.0.11 + 10.20.0.12 + +QTIP key Pair +^^^^^^^^^^^^^ + +QTIP use a SSH key pair to connect to remote hosts. When users execute compute QPI, +QTIP will generate a key pair named *QtipKey* under ``/home/opnfv/qtip/`` and pass +public key to remote hosts. + +If environment variable *CI_DEBUG* is set to *true*, users should delete it by +manual. If *CI_DEBUG* is not set or set to *false*, QTIP will delete the key from +remote hosts before the execution ends. Please make sure the key deleted from remote +hosts or it can introduce a security flaw. + +Commands to run compute QPI +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In a QTIP container, you can run compute QPI by using QTIP CLI: +:: + + mkdir result + qtip plan run <plan_name> -p $PWD/result + +QTIP generates results in the ``$PWD/result`` directory are listed down under the +timestamp name. + +you can get more details from *userguide/cli.rst*. Benchmarks ---------- @@ -31,13 +76,15 @@ The benchmarks include: Dhrystone 2.1 ^^^^^^^^^^^^^ -Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer calculations to evaluate CPU capabilities. -Both Single CPU performance is measured along multi-cpu performance. +Dhrystone is a synthetic benchmark for measuring CPU performance. It uses integer +calculations to evaluate CPU capabilities. Both Single CPU performance is measured +along multi-cpu performance. Dhrystone, however, is a dated benchmark and has some short comings. Written in C, it is a small program that doesn't test the CPU memory subsystem. -Additionally, dhrystone results could be modified by optimizing the compiler and insome cases hardware configuration. +Additionally, dhrystone results could be modified by optimizing the compiler and +insome cases hardware configuration. References: http://www.eembc.org/techlit/datasheets/dhrystone_wp.pdf @@ -56,7 +103,9 @@ http://www.netlib.org/benchmark/whetstone.c OpenSSL Speed ^^^^^^^^^^^^^ -OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, two OpenSSL Speed benchmarks are incorporated: +OpenSSL Speed can be used to benchmark compute performance of a machine. In QTIP, +two OpenSSL Speed benchmarks are incorporated: + 1. RSA signatunes/sec signed by a machine 2. AES 128-bit encryption throughput for a machine for cipher block sizes @@ -67,8 +116,9 @@ https://www.openssl.org/docs/manmaster/apps/speed.html RAMSpeed ^^^^^^^^ -RAMSpeed is used to measure a machine's memory perfomace. -The problem(array)size is large enough to ensure Cache Misses so that the main machine memory is used. +RAMSpeed is used to measure a machine's memory perfomace. The problem(array)size is +large enough to ensure Cache Misses so that the main machine memory is used. + INTmem and FLOATmem benchmarks are executed in 4 different scenarios: a. Copy: a(i)=b(i) @@ -76,7 +126,8 @@ b. Add: a(i)=b(i)+c(i) c. Scale: a(i)=b(i)*d d. Tniad: a(i)=b(i)+c(i)*d -INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points for these benchmarks. +INTmem uses integers in these four benchmarks whereas FLOATmem uses floating points +for these benchmarks. References: @@ -87,15 +138,18 @@ https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf DPI ^^^ -nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that is maintained by ntop. -An example application called *pcapreader* has been developed and is available for use along nDPI. +nDPI is a modified variant of OpenDPI, Open source Deep packet Inspection, that +is maintained by ntop. An example application called *pcapreader* has been developed +and is available for use along nDPI. -A sample .pcap file is passed to the *pcapreader* application. -nDPI classifies traffic in the pcap file into different categories based on string matching. -The *pcapreader* application provides a throughput number for the rate at which traffic was classified, indicating a machine's computational performance. -The results are run 10 times and an average is taken for the obtained number. +A sample .pcap file is passed to the *pcapreader* application. nDPI classifies traffic +in the pcap file into different categories based on string matching. The *pcapreader* +application provides a throughput number for the rate at which traffic was classified, +indicating a machine's computational performance. The results are run 10 times and an +average is taken for the obtained number. -*nDPI may provide non consistent results and was added to Brahmaputra for experimental purposes* +*nDPI may provide non consistent results and was added to Brahmaputra for experimental +purposes* References: |