diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/release/release-notes/release-notes.rst | 42 | ||||
| -rwxr-xr-x | docs/testing/user/userguide/01-introduction.rst | 20 | ||||
| -rw-r--r-- | docs/testing/user/userguide/02-methodology.rst | 22 | ||||
| -rwxr-xr-x | docs/testing/user/userguide/03-architecture.rst | 46 | ||||
| -rw-r--r-- | docs/testing/user/userguide/04-installation.rst | 32 | ||||
| -rw-r--r-- | docs/testing/user/userguide/06-yardstick-plugin.rst | 26 | ||||
| -rw-r--r-- | docs/testing/user/userguide/07-result-store-InfluxDB.rst | 19 | ||||
| -rw-r--r-- | docs/testing/user/userguide/08-grafana.rst | 24 | ||||
| -rw-r--r-- | docs/testing/user/userguide/09-api.rst | 63 | ||||
| -rw-r--r-- | docs/testing/user/userguide/10-yardstick-user-interface.rst | 18 | ||||
| -rw-r--r-- | docs/testing/user/userguide/12-nsb-overview.rst | 99 | ||||
| -rw-r--r-- | docs/testing/user/userguide/13-nsb-installation.rst | 53 | ||||
| -rw-r--r-- | docs/testing/user/userguide/14-nsb-operation.rst | 10 |
13 files changed, 307 insertions, 167 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index 914daa3a4..920e90fbc 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -36,6 +36,9 @@ Version History | December 14, 2018 | 7.1.0 | Yardstick for Gambia release | | | | | +-------------------+-----------+---------------------------------+ +| January 25, 2019 | 7.2.0 | Yardstick for Gambia release | +| | | | ++-------------------+-----------+---------------------------------+ Important Notes @@ -114,19 +117,19 @@ Release Data | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ -| **Repo/tag** | yardstick/opnfv-7.1.0 | +| **Repo/tag** | yardstick/opnfv-7.2.0 | | | | +--------------------------------+-----------------------+ -| **Yardstick Docker image tag** | opnfv-7.1.0 | +| **Yardstick Docker image tag** | opnfv-7.2.0 | | | | +--------------------------------+-----------------------+ -| **Release designation** | Gambia 7.1 | +| **Release designation** | Gambia 7.2 | | | | +--------------------------------+-----------------------+ -| **Release date** | December 14, 2018 | +| **Release date** | January 25, 2019 | | | | +--------------------------------+-----------------------+ -| **Purpose of the delivery** | OPNFV Gambia 7.1.0 | +| **Purpose of the delivery** | OPNFV Gambia 7.2.0 | | | | +--------------------------------+-----------------------+ @@ -272,7 +275,7 @@ List of Scenarios New Test cases -------------- -.. note:: Yardstick Gambia 7.1.0 adds no new test cases. +.. note:: Yardstick Gambia 7.2.0 adds no new test cases. * Generic NFVI test cases @@ -329,7 +332,7 @@ Feature additions Scenario Matrix =============== -For Gambia 7.1.0, Yardstick was tested on the following scenarios: +For Gambia 7.2.0, Yardstick was tested on the following scenarios: +-------------------------+------+---------+----------+------+ | Scenario | Apex | Compass | Fuel-arm | Fuel | @@ -373,33 +376,16 @@ Known Issues/Faults Corrected Faults ---------------- -Gambia 7.1.0: +Gambia 7.2.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ -| YARDSTICK-1241 | Update NSB PROX devguide. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1458 | NSB NFVi PROX Should report realtime port activity not historical data. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1471 | Add Testcase Prox Standalone SRIOV. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1475 | Adding Testcase for Prox Stanalone OvS-DPDK. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1500 | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1517 | Missing opnfv "os-ovn-nofeature-ha" scenario test suite. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-l526 | Run testcase 074 result overridden by job status. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1547 | Adding scale up test case for l3fwd OvS-DPDK. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1560 | Fix pip environment. | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1561 | L3FWD Gradana Dashboards Out-of-date and incorrect. | +| YARDSTICK-1512 | [dovetail] split the sla check results into process recovery and service | +| | recovery for HA test cases. | +--------------------+--------------------------------------------------------------------------+ -Gambia 7.1.0 known restrictions/issues +Gambia 7.2.0 known restrictions/issues ====================================== diff --git a/docs/testing/user/userguide/01-introduction.rst b/docs/testing/user/userguide/01-introduction.rst index 5fc2e8d0f..2a3ab4ea7 100755 --- a/docs/testing/user/userguide/01-introduction.rst +++ b/docs/testing/user/userguide/01-introduction.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Introduction @@ -32,7 +42,7 @@ independent. About This Document -=================== +------------------- This document consists of the following chapters: @@ -79,7 +89,7 @@ This document consists of the following chapters: cases. Contact Yardstick -================= +----------------- Feedback? `Contact us`_ diff --git a/docs/testing/user/userguide/02-methodology.rst b/docs/testing/user/userguide/02-methodology.rst index 34d271095..b1eee9781 100644 --- a/docs/testing/user/userguide/02-methodology.rst +++ b/docs/testing/user/userguide/02-methodology.rst @@ -1,20 +1,30 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. =========== Methodology =========== Abstract -======== +-------- This chapter describes the methodology implemented by the Yardstick project for verifying the :term:`NFVI` from the perspective of a :term:`VNF`. ETSI-NFV -======== +-------- .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf .. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2 @@ -53,7 +63,7 @@ The methodology includes five steps: .. seealso:: Yardsticktst_ for material on alignment ETSI TST001 and Yardstick. Metrics -======= +------- The metrics, as defined by ETSI GS NFV-TST001, are shown in :ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and diff --git a/docs/testing/user/userguide/03-architecture.rst b/docs/testing/user/userguide/03-architecture.rst index 62250d6a3..94081b0e1 100755 --- a/docs/testing/user/userguide/03-architecture.rst +++ b/docs/testing/user/userguide/03-architecture.rst @@ -1,23 +1,34 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 Huawei Technologies Co.,Ltd and others + License. + http://creativecommons.org/licenses/by/4.0 + (c) 2016 Huawei Technologies Co.,Ltd and others + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Architecture ============ Abstract -======== -This chapter describes the yardstick framework software architecture. We will -introduce it from Use-Case View, Logical View, Process View and Deployment +-------- + +This chapter describes the Yardstick framework software architecture. We will +introduce it from Use Case View, Logical View, Process View and Deployment View. More technical details will be introduced in this chapter. Overview -======== +-------- Architecture overview ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to run on a @@ -34,7 +45,8 @@ the test result will be shown with grafana. Concept -------- +^^^^^^^ + **Benchmark** - assess the relative performance of something **Benchmark** configuration file - describes a single test case in yaml format @@ -62,7 +74,7 @@ configuration file and evaluated by the runner. Runner types ------------- +^^^^^^^^^^^^ There exists several predefined runner types to choose between when designing a test scenario: @@ -129,7 +141,8 @@ Snippet of an Iteration runner configuration: Use-Case View -============= +------------- + Yardstick Use-Case View shows two kinds of users. One is the Tester who will do testing in cloud, the other is the User who is more concerned with test result and result analyses. @@ -158,7 +171,8 @@ on OPNFV testing dashboard which use MongoDB as backend. :alt: Yardstick Use-Case View Logical View -============ +------------ + Yardstick Logical View describes the most important classes, their organization, and the most important use-case realizations. @@ -195,7 +209,8 @@ finished. :alt: Yardstick framework architecture in Danube Process View (Test execution flow) -================================== +---------------------------------- + Yardstick process view shows how yardstick runs a test case. Below is the sequence graph about the test execution flow using heat context, and each object represents one module in yardstick: @@ -222,7 +237,8 @@ will call "Openstack" to undeploy the heat stack. Once the stack is undepoyed, the whole test ends. Deployment View -=============== +--------------- + Yardstick deployment view shows how the yardstick tool can be deployed into the underlying platform. Generally, yardstick tool is installed on JumpServer(see `07-installation` for detail installation steps), and JumpServer is @@ -235,7 +251,7 @@ result for better showing. :alt: Yardstick Deployment View Yardstick Directory structure -============================= +----------------------------- **yardstick/** - Yardstick main directory. diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index 3ba312ce7..2dff80ef9 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + + + Convention for heading levels in Yardstick documentation: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. .. Convention for heading levels in Yardstick documentation: @@ -18,7 +28,6 @@ Yardstick Installation ====================== - Yardstick supports installation by Docker or directly in Ubuntu. The installation procedure for Docker and direct installation are detailed in the sections below. @@ -139,7 +148,7 @@ in the following sections. Before that, access the Yardstick container:: and then configure Yardstick environments in the Yardstick container. Using the CLI command ``env prepare`` (first way) (**recommended**) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ In the Yardstick container, the Yardstick repository is located in the ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack @@ -171,10 +180,10 @@ terminal window and execute the following command:: Manually exporting the env variables and initializing OpenStack (second way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Export OpenStack environment variables -###################################### +'''''''''''''''''''''''''''''''''''''' Before running Yardstick it is necessary to export OpenStack environment variables:: @@ -200,7 +209,7 @@ A sample ``openrc`` file may look like this:: Manual creation of Yardstick flavor and guest images -#################################################### +'''''''''''''''''''''''''''''''''''''''''''''''''''' Before executing Yardstick test cases, make sure that Yardstick flavor and guest image are available in OpenStack. Detailed steps about creating the @@ -257,14 +266,14 @@ image. Add Cirros and Ubuntu images to OpenStack:: Automatic initialization of OpenStack (third way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++ Similar to the second way, the first step is also to `Export OpenStack environment variables`_. Then the following steps should be done. Automatic creation of Yardstick flavor and guest images -####################################################### +''''''''''''''''''''''''''''''''''''''''''''''''''''''' Yardstick has a script for automatically creating Yardstick flavor and building Yardstick guest images. This script is mainly used for CI and can be also used @@ -340,7 +349,6 @@ For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest images manually. - Uninstall Yardstick ^^^^^^^^^^^^^^^^^^^ @@ -655,11 +663,9 @@ Modify ``yardstick.conf`` to add the ``influxdb`` dispatcher:: Now Yardstick will store results in InfluxDB when you run a testcase. - Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**) --------------------------------------------------------- - Proxy Support ------------- diff --git a/docs/testing/user/userguide/06-yardstick-plugin.rst b/docs/testing/user/userguide/06-yardstick-plugin.rst index bc35e239d..a5d890b14 100644 --- a/docs/testing/user/userguide/06-yardstick-plugin.rst +++ b/docs/testing/user/userguide/06-yardstick-plugin.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Installing a plug-in into Yardstick =================================== Abstract -======== +-------- Yardstick provides a ``plugin`` CLI command to support integration with other OPNFV testing projects. Below is an example invocation of Yardstick plugin @@ -17,7 +27,7 @@ command and Storperf plug-in sample. Installing Storperf into Yardstick -================================== +---------------------------------- Storperf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/. @@ -31,7 +41,7 @@ In this introduction we will install Storperf on Jump Host. Step 0: Environment preparation -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Running Storperf on Jump Host Requirements: @@ -100,7 +110,7 @@ container. You may need to copy it to the root directory of the Storperf deployed host. Step 1: Plug-in configuration file preparation ----------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install a plug-in, first you need to prepare a plug-in configuration file in YAML format and store it in the "plugin" directory. The plugin configration @@ -125,7 +135,7 @@ Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment. Step 2: Plug-in install/remove scripts preparation --------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In ``yardstick/resource/scripts`` directory, there are two folders: an ``install`` folder and a ``remove`` folder. You need to store the plug-in @@ -139,15 +149,15 @@ For example, the install and remove scripts for Storperf are both named ``storperf.bash``. Step 3: Install and remove Storperf ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install Storperf, simply execute the following command:: # Install Storperf yardstick plugin install plugin/storperf.yaml -Removing Storperf from yardstick -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Removing Storperf from Yardstick +++++++++++++++++++++++++++++++++ To remove Storperf, simply execute the following command:: diff --git a/docs/testing/user/userguide/07-result-store-InfluxDB.rst b/docs/testing/user/userguide/07-result-store-InfluxDB.rst index cde931376..8a9196b1b 100644 --- a/docs/testing/user/userguide/07-result-store-InfluxDB.rst +++ b/docs/testing/user/userguide/07-result-store-InfluxDB.rst @@ -1,14 +1,23 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============================================== Store Other Project's Test Results in InfluxDB ============================================== Abstract -======== +-------- .. _Framework: https://wiki.opnfv.org/download/attachments/6827660/wiki.png?version=1&modificationDate=1470298075000&api=v2 @@ -21,7 +30,7 @@ into community's InfluxDB. The framework is shown in Framework_. :alt: Store Other Project's Test Results in InfluxDB Store Storperf Test Results into Community's InfluxDB -===================================================== +----------------------------------------------------- .. _Influxdb: https://git.opnfv.org/cgit/yardstick/tree/yardstick/dispatcher/influxdb.py .. _Mingjiang: mailto:limingjiang@huawei.com diff --git a/docs/testing/user/userguide/08-grafana.rst b/docs/testing/user/userguide/08-grafana.rst index 020a08a65..ebe9f570d 100644 --- a/docs/testing/user/userguide/08-grafana.rst +++ b/docs/testing/user/userguide/08-grafana.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) 2016 Huawei Technologies Co.,Ltd and others +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ================= Grafana dashboard ================= Abstract -======== +-------- This chapter describes the Yardstick grafana dashboard. The Yardstick grafana dashboard can be found here: http://testresults.opnfv.org/grafana/ @@ -21,14 +31,14 @@ dashboard can be found here: http://testresults.opnfv.org/grafana/ Public access -============= +------------- Yardstick provids a public account for accessing to the dashboard. The username and password are both set to ‘opnfv’. Testcase dashboard -================== +------------------ For each test case, there is a dedicated dashboard. Shown here is the dashboard of TC002. @@ -56,7 +66,7 @@ zoom out the chart. Administration access -===================== +--------------------- For a user with administration rights it is easy to update and save any dashboard configuration. Saved updates immediately take effect and become live. @@ -72,11 +82,11 @@ This may cause issues like: Any change made by administrator should be careful. -Add a dashboard into yardstick grafana -====================================== +Add a dashboard into Yardstick Grafana +-------------------------------------- Due to security concern, users that using the public opnfv account are not able -to edit the yardstick grafana directly.It takes a few more steps for a +to edit the yardstick grafana directly. It takes a few more steps for a non-yardstick user to add a custom dashboard into yardstick grafana. There are 6 steps to go. diff --git a/docs/testing/user/userguide/09-api.rst b/docs/testing/user/userguide/09-api.rst index 1a896699b..f227878ae 100644 --- a/docs/testing/user/userguide/09-api.rst +++ b/docs/testing/user/userguide/09-api.rst @@ -2,6 +2,15 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ===================== Yardstick Restful API @@ -9,16 +18,16 @@ Yardstick Restful API Abstract -======== +-------- Yardstick support restful API since Danube. Available API -============= +------------- /yardstick/env/action ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports: @@ -69,7 +78,7 @@ get the task result. /yardstick/asynctask --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the status of asynchronous tasks @@ -91,7 +100,7 @@ NOTE:: /yardstick/testcases --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to list all released Yardstick test cases. @@ -106,7 +115,7 @@ Example:: /yardstick/testcases/release/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick released test case. @@ -130,7 +139,7 @@ result. /yardstick/testcases/samples/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick sample test case. @@ -154,7 +163,7 @@ the result. /yardstick/testcases/<testcase_name>/docs ------------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to the documentation of a certain released test case. @@ -170,7 +179,7 @@ Example:: /yardstick/testsuites/action ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick test suite. @@ -194,7 +203,7 @@ result. /yardstick/tasks/<task_id>/log ------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the real time log of test case execution. @@ -209,7 +218,7 @@ Example:: /yardstick/results ------------------- +^^^^^^^^^^^^^^^^^^ Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use @@ -228,7 +237,7 @@ This API will return a list of test case result /api/v2/yardstick/openrcs -------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -282,7 +291,7 @@ Example:: /api/v2/yardstick/openrcs/<openrc_id> -------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -308,7 +317,7 @@ Example:: /api/v2/yardstick/pods ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -334,7 +343,7 @@ Example:: /api/v2/yardstick/pods/<pod_id> -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -358,7 +367,7 @@ Example:: /api/v2/yardstick/images ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -383,7 +392,7 @@ Example:: /api/v2/yardstick/images/<image_id> ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -407,7 +416,7 @@ Example:: /api/v2/yardstick/tasks ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -433,7 +442,7 @@ Example:: /api/v2/yardstick/tasks/<task_id> ---------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -518,7 +527,7 @@ Example:: /api/v2/yardstick/testcases ---------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick testcases. For Euphrates, it supports: @@ -553,7 +562,7 @@ Example:: /api/v2/yardstick/testcases/<case_name> ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: @@ -579,7 +588,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -617,7 +626,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -643,7 +652,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick test projects. For Euphrates, it supports: @@ -678,7 +687,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: @@ -704,7 +713,7 @@ Example:: /api/v2/yardstick/containers ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: @@ -744,7 +753,7 @@ Example:: /api/v2/yardstick/containers/<container_id> -------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: diff --git a/docs/testing/user/userguide/10-yardstick-user-interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst index b3056ec99..246e1b1df 100644 --- a/docs/testing/user/userguide/10-yardstick-user-interface.rst +++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst @@ -1,3 +1,17 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ======================== Yardstick User Interface ======================== @@ -18,7 +32,7 @@ The following layouts are available: Commands -======== +-------- To generate the compact HTML report, run:: @@ -30,7 +44,7 @@ To generate the dynamic HTML report, run:: Description -=========== +----------- 1. When the command is triggered, the relevant values for the provided task-id and testcase name are retrieved from the diff --git a/docs/testing/user/userguide/12-nsb-overview.rst b/docs/testing/user/userguide/12-nsb-overview.rst index c5e395ee6..70aba1e37 100644 --- a/docs/testing/user/userguide/12-nsb-overview.rst +++ b/docs/testing/user/userguide/12-nsb-overview.rst @@ -3,6 +3,16 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2017 Intel Corporation. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Network Services Benchmarking (NSB) =================================== @@ -19,22 +29,34 @@ Yardstick_ from Intel. Overview -------- -The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi -Characterization and benchmarking with repeatable and deterministic methods. - -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments - bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. +Network Services Benchmarking (:term:`NSB`) uses the :term:`Yardstick` +framework for performing :term:`VNF` and :term:`NFVI` characterisation in an +:term:`NFV` environment. + +For VNF characterisation, NSB will onboard a VNF, source and sink traffic to it +via traffic generators, and collect a variety of key performance indicators +(:term:`KPI`) during VNF execution. The stream of KPI data is stored in a +database, and it is visualized in a performance-visualization dashboard. + +For NFVI characterisation, a fixed test VNF, called :term:`PROX` is used. +PROX implements a suite of test cases and visualizes the output data of the +test suite. The PROX test cases implement various execution kernels found in +real-world VNFs, and the output of the test cases provides an indication of +the fitness of the infrastructure for running NFV services, in addition to +indicating potential performance optimizations for the NFVI. + +NSB extends the Yardstick framework to do VNF characterization in three +different execution environments - bare metal i.e. native Linux environment, +standalone virtual environment and managed virtualized environment (e.g. +OpenStack). It also brings in the capability to interact with external traffic +generators, both hardware and software based, for triggering and validating the +traffic according to user defined profiles. NSB extension includes: * Generic data models of Network Services, based on ETSI spec `ETSI GS NFV-TST 001`_ -* Standalone :term:`context` for VNF testing like SRIOV, OVS, OVS-DPDK, etc +* Standalone :term:`context` for VNF testing SRIOV, OVS, OVS-DPDK, etc * Generic VNF configuration models and metrics implemented with Python classes * Traffic generator features and traffic profiles @@ -62,7 +84,7 @@ NSB extension includes: * VNF KPIs e.g. packet_in, packet_drop, packet_fwd Architecture -============ +------------ The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using NFV infrastructure. @@ -97,14 +119,21 @@ Network Service framework performs the necessary test steps. It may involve: * Retrieve the KPI's provided by particular VNF Components of Network Service ------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. TODO: provide a list of components in this section and describe them in + later sub-sections + +.. Components are the methodology, TGs, framework extensions, KPI collection, + Testcases, SampleVNFs +.. Framework extentions include: VNF models, NSPerf Scenario, contexts * *Models for Network Service benchmarking*: The Network Service benchmarking requires the proper modelling approach. The NSB provides models using Python files and defining of NSDs and VNFDs. -The benchmark control application being a part of OPNFV yardstick can call -that python models to instantiate and configure the VNFs. Depending on +The benchmark control application being a part of OPNFV Yardstick can call +that Python models to instantiate and configure the VNFs. Depending on infrastructure type (bare-metal or fully virtualized) that calls could be made directly or using MANO system. @@ -147,8 +176,40 @@ for every combination of test case parameters: * RFC2544 throughput for various loss rate defined (1% is a default) +KPI Collection +^^^^^^^^^^^^^^ + +KPI collection is the process of sampling KPIs at multiple intervals to allow +for investigation into anomalies during runtime. Some KPI intervals are +adjustable. KPIs are collected from traffic generators and NFVI for the SUT. +There is already some reporting in NSB available, but NSB collects all KPIs for +analytics to process. + +Below is an example list of basic KPIs: +* Throughput +* Latency +* Packet delay variation +* Maximum establishment rate +* Maximum tear-down rate +* Maximum simultaneous number of sessions + +Of course, there can be many other KPIs that will be relevant for a specific +NFVI, but in most cases these KPIs are enough to give you a basic picture of +the SUT. NSB also uses :term:`collectd` in order to collect the KPIs. Currently +the following collectd plug-ins are enabled for NSB testcases: + +* Libvirt +* Interface stats +* OvS events +* vSwitch stats +* Huge Pages +* RAM +* CPU usage +* Intel® PMU +* Intel(r) RDT + Graphical Overview -================== +------------------ NSB Testing with Yardstick framework facilitate performance testing of various VNFs provided. @@ -174,13 +235,13 @@ VNFs provided. Figure 1: Network Service - 2 server configuration -VNFs supported for chracterization: ------------------------------------ +VNFs supported for chracterization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. CGNAPT - Carrier Grade Network Address and port Translation 2. vFW - Virtual Firewall 3. vACL - Access Control List -4. Prox - Packet pROcessing eXecution engine: +4. PROX - Packet pROcessing eXecution engine: * VNF can act as Drop, Basic Forwarding (no touch), L2 Forwarding (change MAC), GRE encap/decap, Load balance based on packet fields, Symmetric load balancing diff --git a/docs/testing/user/userguide/13-nsb-installation.rst b/docs/testing/user/userguide/13-nsb-installation.rst index 69f6a5a40..71ced43ea 100644 --- a/docs/testing/user/userguide/13-nsb-installation.rst +++ b/docs/testing/user/userguide/13-nsb-installation.rst @@ -8,12 +8,13 @@ ======= Heading 0 (reserved for the title in a document) ------- Heading 1 - ~~~~~~~ Heading 2 + ^^^^^^^ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. + ================ NSB Installation ================ @@ -21,13 +22,8 @@ NSB Installation .. _OVS-DPDK: http://docs.openvswitch.org/en/latest/intro/install/dpdk/ .. _devstack: https://docs.openstack.org/devstack/pike/> -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments viz., bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. +Abstract +-------- The steps needed to run Yardstick with NSB testing are: @@ -36,7 +32,6 @@ The steps needed to run Yardstick with NSB testing are: * Create/reference the test configuration yaml file. * Run the test case. - Prerequisites ------------- @@ -57,7 +52,7 @@ Several prerequisites are needed for Yardstick (VNF testing): * intel-cmt-cat Hardware & Software Ingredients -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SUT requirements: @@ -73,7 +68,6 @@ SUT requirements: Boot and BIOS settings: - ============= ================================================= Boot settings default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 @@ -220,13 +214,12 @@ Run Yardstick - Network Service Testcases ----------------------------------------- NS testing - using yardstick CLI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See :doc:`04-installation` Connect to the Yardstick container:: - docker exec -it yardstick /bin/bash If you're running ``heat`` testcases and ``nsb_setup.sh`` was not used:: @@ -245,7 +238,7 @@ Network Service Benchmarking - Bare-Metal ----------------------------------------- Bare-Metal Config pod.yaml describing Topology -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Bare-Metal 2-Node setup +++++++++++++++++++++++ @@ -350,11 +343,11 @@ topology and update all the required fields.:: if: "xe1" -Network Service Benchmarking - Standalone Virtualization --------------------------------------------------------- +Standalone Virtualization +------------------------- SR-IOV -~~~~~~ +^^^^^^ SR-IOV Pre-requisites +++++++++++++++++++++ @@ -538,8 +531,8 @@ SR-IOV Config host_sriov.yaml SR-IOV testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -582,12 +575,11 @@ Update "contexts" section gateway_ip: '152.16.100.20' - OVS-DPDK -~~~~~~~~ +^^^^^^^^ OVS-DPDK Pre-requisites -~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++ On Host, where VM is created: a) Create and configure a bridge named ``br-int`` for VM to connect to @@ -776,8 +768,8 @@ OVS-DPDK Config host_ovs.yaml ovs_dpdk testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -831,16 +823,16 @@ Update "contexts" section gateway_ip: '152.16.100.20' -Network Service Benchmarking - OpenStack with SR-IOV support ------------------------------------------------------------- +OpenStack with SR-IOV support +----------------------------- This section describes how to run a Sample VNF test case, using Heat context, with SR-IOV. It also covers how to install OpenStack in Ubuntu 16.04, using DevStack, with SR-IOV support. -Single node OpenStack setup with external TG -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Single node OpenStack with external TG +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -985,7 +977,6 @@ DevStack configuration file: Start the devstack installation on a host. - TG host configuration +++++++++++++++++++++ @@ -1056,7 +1047,6 @@ Controller/Compute pre-configuration Pre-configuration of the controller and compute hosts are the same as described in `Host pre-configuration`_ section. - DevStack configuration ++++++++++++++++++++++ @@ -1082,7 +1072,6 @@ DevStack configuration file for compute host: Start the devstack installation on the controller and compute hosts. - Run the sample vFW TC +++++++++++++++++++++ @@ -1148,7 +1137,7 @@ IxLoad ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml`` IxNetwork -~~~~~~~~~ +^^^^^^^^^ IxNetwork testcases use IxNetwork API Python Bindings module, which is installed as part of the requirements of the project. diff --git a/docs/testing/user/userguide/14-nsb-operation.rst b/docs/testing/user/userguide/14-nsb-operation.rst index 910985069..12e269187 100644 --- a/docs/testing/user/userguide/14-nsb-operation.rst +++ b/docs/testing/user/userguide/14-nsb-operation.rst @@ -2,6 +2,16 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2018 Intel Corporation. +.. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. Yardstick - NSB Testing - Operation =================================== |