diff options
Diffstat (limited to 'docs/testing')
| -rwxr-xr-x | docs/testing/user/userguide/01-introduction.rst | 20 | ||||
| -rw-r--r-- | docs/testing/user/userguide/02-methodology.rst | 22 | ||||
| -rwxr-xr-x | docs/testing/user/userguide/03-architecture.rst | 46 | ||||
| -rw-r--r-- | docs/testing/user/userguide/04-installation.rst | 32 | ||||
| -rw-r--r-- | docs/testing/user/userguide/06-yardstick-plugin.rst | 26 | ||||
| -rw-r--r-- | docs/testing/user/userguide/07-result-store-InfluxDB.rst | 19 | ||||
| -rw-r--r-- | docs/testing/user/userguide/08-grafana.rst | 24 | ||||
| -rw-r--r-- | docs/testing/user/userguide/09-api.rst | 63 | ||||
| -rw-r--r-- | docs/testing/user/userguide/10-yardstick-user-interface.rst | 18 | ||||
| -rw-r--r-- | docs/testing/user/userguide/12-nsb-overview.rst | 319 | ||||
| -rw-r--r-- | docs/testing/user/userguide/13-nsb-installation.rst | 364 | ||||
| -rw-r--r-- | docs/testing/user/userguide/14-nsb-operation.rst | 57 | ||||
| -rw-r--r-- | docs/testing/user/userguide/glossary.rst | 60 |
13 files changed, 623 insertions, 447 deletions
diff --git a/docs/testing/user/userguide/01-introduction.rst b/docs/testing/user/userguide/01-introduction.rst index 5fc2e8d0f..2a3ab4ea7 100755 --- a/docs/testing/user/userguide/01-introduction.rst +++ b/docs/testing/user/userguide/01-introduction.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Introduction @@ -32,7 +42,7 @@ independent. About This Document -=================== +------------------- This document consists of the following chapters: @@ -79,7 +89,7 @@ This document consists of the following chapters: cases. Contact Yardstick -================= +----------------- Feedback? `Contact us`_ diff --git a/docs/testing/user/userguide/02-methodology.rst b/docs/testing/user/userguide/02-methodology.rst index 34d271095..b1eee9781 100644 --- a/docs/testing/user/userguide/02-methodology.rst +++ b/docs/testing/user/userguide/02-methodology.rst @@ -1,20 +1,30 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. =========== Methodology =========== Abstract -======== +-------- This chapter describes the methodology implemented by the Yardstick project for verifying the :term:`NFVI` from the perspective of a :term:`VNF`. ETSI-NFV -======== +-------- .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf .. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2 @@ -53,7 +63,7 @@ The methodology includes five steps: .. seealso:: Yardsticktst_ for material on alignment ETSI TST001 and Yardstick. Metrics -======= +------- The metrics, as defined by ETSI GS NFV-TST001, are shown in :ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and diff --git a/docs/testing/user/userguide/03-architecture.rst b/docs/testing/user/userguide/03-architecture.rst index 62250d6a3..94081b0e1 100755 --- a/docs/testing/user/userguide/03-architecture.rst +++ b/docs/testing/user/userguide/03-architecture.rst @@ -1,23 +1,34 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 Huawei Technologies Co.,Ltd and others + License. + http://creativecommons.org/licenses/by/4.0 + (c) 2016 Huawei Technologies Co.,Ltd and others + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Architecture ============ Abstract -======== -This chapter describes the yardstick framework software architecture. We will -introduce it from Use-Case View, Logical View, Process View and Deployment +-------- + +This chapter describes the Yardstick framework software architecture. We will +introduce it from Use Case View, Logical View, Process View and Deployment View. More technical details will be introduced in this chapter. Overview -======== +-------- Architecture overview ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to run on a @@ -34,7 +45,8 @@ the test result will be shown with grafana. Concept -------- +^^^^^^^ + **Benchmark** - assess the relative performance of something **Benchmark** configuration file - describes a single test case in yaml format @@ -62,7 +74,7 @@ configuration file and evaluated by the runner. Runner types ------------- +^^^^^^^^^^^^ There exists several predefined runner types to choose between when designing a test scenario: @@ -129,7 +141,8 @@ Snippet of an Iteration runner configuration: Use-Case View -============= +------------- + Yardstick Use-Case View shows two kinds of users. One is the Tester who will do testing in cloud, the other is the User who is more concerned with test result and result analyses. @@ -158,7 +171,8 @@ on OPNFV testing dashboard which use MongoDB as backend. :alt: Yardstick Use-Case View Logical View -============ +------------ + Yardstick Logical View describes the most important classes, their organization, and the most important use-case realizations. @@ -195,7 +209,8 @@ finished. :alt: Yardstick framework architecture in Danube Process View (Test execution flow) -================================== +---------------------------------- + Yardstick process view shows how yardstick runs a test case. Below is the sequence graph about the test execution flow using heat context, and each object represents one module in yardstick: @@ -222,7 +237,8 @@ will call "Openstack" to undeploy the heat stack. Once the stack is undepoyed, the whole test ends. Deployment View -=============== +--------------- + Yardstick deployment view shows how the yardstick tool can be deployed into the underlying platform. Generally, yardstick tool is installed on JumpServer(see `07-installation` for detail installation steps), and JumpServer is @@ -235,7 +251,7 @@ result for better showing. :alt: Yardstick Deployment View Yardstick Directory structure -============================= +----------------------------- **yardstick/** - Yardstick main directory. diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index 3ba312ce7..2dff80ef9 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + + + Convention for heading levels in Yardstick documentation: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. .. Convention for heading levels in Yardstick documentation: @@ -18,7 +28,6 @@ Yardstick Installation ====================== - Yardstick supports installation by Docker or directly in Ubuntu. The installation procedure for Docker and direct installation are detailed in the sections below. @@ -139,7 +148,7 @@ in the following sections. Before that, access the Yardstick container:: and then configure Yardstick environments in the Yardstick container. Using the CLI command ``env prepare`` (first way) (**recommended**) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ In the Yardstick container, the Yardstick repository is located in the ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack @@ -171,10 +180,10 @@ terminal window and execute the following command:: Manually exporting the env variables and initializing OpenStack (second way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Export OpenStack environment variables -###################################### +'''''''''''''''''''''''''''''''''''''' Before running Yardstick it is necessary to export OpenStack environment variables:: @@ -200,7 +209,7 @@ A sample ``openrc`` file may look like this:: Manual creation of Yardstick flavor and guest images -#################################################### +'''''''''''''''''''''''''''''''''''''''''''''''''''' Before executing Yardstick test cases, make sure that Yardstick flavor and guest image are available in OpenStack. Detailed steps about creating the @@ -257,14 +266,14 @@ image. Add Cirros and Ubuntu images to OpenStack:: Automatic initialization of OpenStack (third way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++ Similar to the second way, the first step is also to `Export OpenStack environment variables`_. Then the following steps should be done. Automatic creation of Yardstick flavor and guest images -####################################################### +''''''''''''''''''''''''''''''''''''''''''''''''''''''' Yardstick has a script for automatically creating Yardstick flavor and building Yardstick guest images. This script is mainly used for CI and can be also used @@ -340,7 +349,6 @@ For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest images manually. - Uninstall Yardstick ^^^^^^^^^^^^^^^^^^^ @@ -655,11 +663,9 @@ Modify ``yardstick.conf`` to add the ``influxdb`` dispatcher:: Now Yardstick will store results in InfluxDB when you run a testcase. - Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**) --------------------------------------------------------- - Proxy Support ------------- diff --git a/docs/testing/user/userguide/06-yardstick-plugin.rst b/docs/testing/user/userguide/06-yardstick-plugin.rst index bc35e239d..a5d890b14 100644 --- a/docs/testing/user/userguide/06-yardstick-plugin.rst +++ b/docs/testing/user/userguide/06-yardstick-plugin.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Installing a plug-in into Yardstick =================================== Abstract -======== +-------- Yardstick provides a ``plugin`` CLI command to support integration with other OPNFV testing projects. Below is an example invocation of Yardstick plugin @@ -17,7 +27,7 @@ command and Storperf plug-in sample. Installing Storperf into Yardstick -================================== +---------------------------------- Storperf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/. @@ -31,7 +41,7 @@ In this introduction we will install Storperf on Jump Host. Step 0: Environment preparation -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Running Storperf on Jump Host Requirements: @@ -100,7 +110,7 @@ container. You may need to copy it to the root directory of the Storperf deployed host. Step 1: Plug-in configuration file preparation ----------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install a plug-in, first you need to prepare a plug-in configuration file in YAML format and store it in the "plugin" directory. The plugin configration @@ -125,7 +135,7 @@ Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment. Step 2: Plug-in install/remove scripts preparation --------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In ``yardstick/resource/scripts`` directory, there are two folders: an ``install`` folder and a ``remove`` folder. You need to store the plug-in @@ -139,15 +149,15 @@ For example, the install and remove scripts for Storperf are both named ``storperf.bash``. Step 3: Install and remove Storperf ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install Storperf, simply execute the following command:: # Install Storperf yardstick plugin install plugin/storperf.yaml -Removing Storperf from yardstick -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Removing Storperf from Yardstick +++++++++++++++++++++++++++++++++ To remove Storperf, simply execute the following command:: diff --git a/docs/testing/user/userguide/07-result-store-InfluxDB.rst b/docs/testing/user/userguide/07-result-store-InfluxDB.rst index cde931376..8a9196b1b 100644 --- a/docs/testing/user/userguide/07-result-store-InfluxDB.rst +++ b/docs/testing/user/userguide/07-result-store-InfluxDB.rst @@ -1,14 +1,23 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============================================== Store Other Project's Test Results in InfluxDB ============================================== Abstract -======== +-------- .. _Framework: https://wiki.opnfv.org/download/attachments/6827660/wiki.png?version=1&modificationDate=1470298075000&api=v2 @@ -21,7 +30,7 @@ into community's InfluxDB. The framework is shown in Framework_. :alt: Store Other Project's Test Results in InfluxDB Store Storperf Test Results into Community's InfluxDB -===================================================== +----------------------------------------------------- .. _Influxdb: https://git.opnfv.org/cgit/yardstick/tree/yardstick/dispatcher/influxdb.py .. _Mingjiang: mailto:limingjiang@huawei.com diff --git a/docs/testing/user/userguide/08-grafana.rst b/docs/testing/user/userguide/08-grafana.rst index 020a08a65..ebe9f570d 100644 --- a/docs/testing/user/userguide/08-grafana.rst +++ b/docs/testing/user/userguide/08-grafana.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) 2016 Huawei Technologies Co.,Ltd and others +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ================= Grafana dashboard ================= Abstract -======== +-------- This chapter describes the Yardstick grafana dashboard. The Yardstick grafana dashboard can be found here: http://testresults.opnfv.org/grafana/ @@ -21,14 +31,14 @@ dashboard can be found here: http://testresults.opnfv.org/grafana/ Public access -============= +------------- Yardstick provids a public account for accessing to the dashboard. The username and password are both set to ‘opnfv’. Testcase dashboard -================== +------------------ For each test case, there is a dedicated dashboard. Shown here is the dashboard of TC002. @@ -56,7 +66,7 @@ zoom out the chart. Administration access -===================== +--------------------- For a user with administration rights it is easy to update and save any dashboard configuration. Saved updates immediately take effect and become live. @@ -72,11 +82,11 @@ This may cause issues like: Any change made by administrator should be careful. -Add a dashboard into yardstick grafana -====================================== +Add a dashboard into Yardstick Grafana +-------------------------------------- Due to security concern, users that using the public opnfv account are not able -to edit the yardstick grafana directly.It takes a few more steps for a +to edit the yardstick grafana directly. It takes a few more steps for a non-yardstick user to add a custom dashboard into yardstick grafana. There are 6 steps to go. diff --git a/docs/testing/user/userguide/09-api.rst b/docs/testing/user/userguide/09-api.rst index 1a896699b..f227878ae 100644 --- a/docs/testing/user/userguide/09-api.rst +++ b/docs/testing/user/userguide/09-api.rst @@ -2,6 +2,15 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ===================== Yardstick Restful API @@ -9,16 +18,16 @@ Yardstick Restful API Abstract -======== +-------- Yardstick support restful API since Danube. Available API -============= +------------- /yardstick/env/action ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports: @@ -69,7 +78,7 @@ get the task result. /yardstick/asynctask --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the status of asynchronous tasks @@ -91,7 +100,7 @@ NOTE:: /yardstick/testcases --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to list all released Yardstick test cases. @@ -106,7 +115,7 @@ Example:: /yardstick/testcases/release/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick released test case. @@ -130,7 +139,7 @@ result. /yardstick/testcases/samples/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick sample test case. @@ -154,7 +163,7 @@ the result. /yardstick/testcases/<testcase_name>/docs ------------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to the documentation of a certain released test case. @@ -170,7 +179,7 @@ Example:: /yardstick/testsuites/action ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick test suite. @@ -194,7 +203,7 @@ result. /yardstick/tasks/<task_id>/log ------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the real time log of test case execution. @@ -209,7 +218,7 @@ Example:: /yardstick/results ------------------- +^^^^^^^^^^^^^^^^^^ Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use @@ -228,7 +237,7 @@ This API will return a list of test case result /api/v2/yardstick/openrcs -------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -282,7 +291,7 @@ Example:: /api/v2/yardstick/openrcs/<openrc_id> -------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -308,7 +317,7 @@ Example:: /api/v2/yardstick/pods ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -334,7 +343,7 @@ Example:: /api/v2/yardstick/pods/<pod_id> -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -358,7 +367,7 @@ Example:: /api/v2/yardstick/images ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -383,7 +392,7 @@ Example:: /api/v2/yardstick/images/<image_id> ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -407,7 +416,7 @@ Example:: /api/v2/yardstick/tasks ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -433,7 +442,7 @@ Example:: /api/v2/yardstick/tasks/<task_id> ---------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -518,7 +527,7 @@ Example:: /api/v2/yardstick/testcases ---------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick testcases. For Euphrates, it supports: @@ -553,7 +562,7 @@ Example:: /api/v2/yardstick/testcases/<case_name> ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: @@ -579,7 +588,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -617,7 +626,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -643,7 +652,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick test projects. For Euphrates, it supports: @@ -678,7 +687,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: @@ -704,7 +713,7 @@ Example:: /api/v2/yardstick/containers ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: @@ -744,7 +753,7 @@ Example:: /api/v2/yardstick/containers/<container_id> -------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: diff --git a/docs/testing/user/userguide/10-yardstick-user-interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst index b3056ec99..246e1b1df 100644 --- a/docs/testing/user/userguide/10-yardstick-user-interface.rst +++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst @@ -1,3 +1,17 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ======================== Yardstick User Interface ======================== @@ -18,7 +32,7 @@ The following layouts are available: Commands -======== +-------- To generate the compact HTML report, run:: @@ -30,7 +44,7 @@ To generate the dynamic HTML report, run:: Description -=========== +----------- 1. When the command is triggered, the relevant values for the provided task-id and testcase name are retrieved from the diff --git a/docs/testing/user/userguide/12-nsb-overview.rst b/docs/testing/user/userguide/12-nsb-overview.rst index 7b0d46804..70aba1e37 100644 --- a/docs/testing/user/userguide/12-nsb-overview.rst +++ b/docs/testing/user/userguide/12-nsb-overview.rst @@ -3,75 +3,88 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2017 Intel Corporation. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Network Services Benchmarking (NSB) =================================== -Abstract -======== - .. _Yardstick: https://wiki.opnfv.org/display/yardstick +.. _`ETSI GS NFV-TST001`: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_nfv-tst001v010101p.pdf + +Abstract +-------- This chapter provides an overview of the NSB, a contribution to OPNFV Yardstick_ from Intel. Overview -======== - -The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi -Characterization and benchmarking with repeatable and deterministic methods. - -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments - bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. +-------- + +Network Services Benchmarking (:term:`NSB`) uses the :term:`Yardstick` +framework for performing :term:`VNF` and :term:`NFVI` characterisation in an +:term:`NFV` environment. + +For VNF characterisation, NSB will onboard a VNF, source and sink traffic to it +via traffic generators, and collect a variety of key performance indicators +(:term:`KPI`) during VNF execution. The stream of KPI data is stored in a +database, and it is visualized in a performance-visualization dashboard. + +For NFVI characterisation, a fixed test VNF, called :term:`PROX` is used. +PROX implements a suite of test cases and visualizes the output data of the +test suite. The PROX test cases implement various execution kernels found in +real-world VNFs, and the output of the test cases provides an indication of +the fitness of the infrastructure for running NFV services, in addition to +indicating potential performance optimizations for the NFVI. + +NSB extends the Yardstick framework to do VNF characterization in three +different execution environments - bare metal i.e. native Linux environment, +standalone virtual environment and managed virtualized environment (e.g. +OpenStack). It also brings in the capability to interact with external traffic +generators, both hardware and software based, for triggering and validating the +traffic according to user defined profiles. NSB extension includes: - - Generic data models of Network Services, based on ETSI spec `ETSI GS NFV-TST 001 <http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_nfv-tst001v010101p.pdf>`_ - - - New Standalone context for VNF testing like SRIOV, OVS, OVS-DPDK etc - - - Generic VNF configuration models and metrics implemented with Python - classes - - - Traffic generator features and traffic profiles - - - L1-L3 state-less traffic profiles - - - L4-L7 state-full traffic profiles - - - Tunneling protocol / network overlay support - - - Test case samples - - - Ping +* Generic data models of Network Services, based on ETSI spec + `ETSI GS NFV-TST 001`_ +* Standalone :term:`context` for VNF testing SRIOV, OVS, OVS-DPDK, etc +* Generic VNF configuration models and metrics implemented with Python + classes +* Traffic generator features and traffic profiles - - Trex + * L1-L3 stateless traffic profiles + * L4-L7 state-full traffic profiles + * Tunneling protocol/network overlay support - - vPE,vCGNAT, vFirewall etc - ipv4 throughput, latency etc +* Test case samples - - Traffic generators like Trex, ab/nginx, ixia, iperf etc + * Ping + * Trex + * vPE, vCGNAT, vFirewall etc - ipv4 throughput, latency etc - - KPIs for a given use case: +* Traffic generators i.e. Trex, ab/nginx, ixia, iperf, etc +* KPIs for a given use case: - - System agent support for collecting NFVi KPI. This includes: + * System agent support for collecting NFVi KPI. This includes: - - CPU statistic + * CPU statistic + * Memory BW + * OVS-DPDK Stats - - Memory BW - - - OVS-DPDK Stats - - - Network KPIs, e.g., inpackets, outpackets, thoughput, latency etc - - - VNF KPIs, e.g., packet_in, packet_drop, packet_fwd etc + * Network KPIs e.g. inpackets, outpackets, thoughput, latency + * VNF KPIs e.g. packet_in, packet_drop, packet_fwd Architecture -============ +------------ The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using NFV infrastructure. @@ -83,124 +96,154 @@ performed network functionality. The part of the data model is a set of the configuration parameters, number of connection points used and flavor including core and memory amount. -The ETSI defines a Network Service as a set of configurable VNFs working in -some NFV Infrastructure connecting each other using Virtual Links available -through Connection Points. The ETSI MANO specification defines a set of -management entities called Network Service Descriptors (NSD) and -VNF Descriptors (VNFD) that define real Network Service. The picture below -makes an example how the real Network Operator use-case can map into ETSI -Network service definition - -Network Service framework performs the necessary test steps. It may involve - - - Interacting with traffic generator and providing the inputs on traffic - type / packet structure to generate the required traffic as per the - test case. Traffic profiles will be used for this. - - - Executing the commands required for the test procedure and analyses the - command output for confirming whether the command got executed correctly - or not. E.g. As per the test case, run the traffic for the given - time period / wait for the necessary time delay - - - Verify the test result. - - - Validate the traffic flow from SUT - - - Fetch the table / data from SUT and verify the value as per the test case - - - Upload the logs from SUT onto the Test Harness server - - - Read the KPI's provided by particular VNF +ETSI defines a Network Service as a set of configurable VNFs working in some +NFV Infrastructure connecting each other using Virtual Links available through +Connection Points. The ETSI MANO specification defines a set of management +entities called Network Service Descriptors (NSD) and VNF Descriptors (VNFD) +that define real Network Service. The picture below makes an example how the +real Network Operator use-case can map into ETSI Network service definition. + +Network Service framework performs the necessary test steps. It may involve: + +* Interacting with traffic generator and providing the inputs on traffic + type / packet structure to generate the required traffic as per the + test case. Traffic profiles will be used for this. +* Executing the commands required for the test procedure and analyses the + command output for confirming whether the command got executed correctly + or not e.g. as per the test case, run the traffic for the given + time period and wait for the necessary time delay. +* Verify the test result. +* Validate the traffic flow from SUT. +* Fetch the data from SUT and verify the value as per the test case. +* Upload the logs from SUT onto the Test Harness server +* Retrieve the KPI's provided by particular VNF Components of Network Service ------------------------------ - - * *Models for Network Service benchmarking*: The Network Service benchmarking - requires the proper modelling approach. The NSB provides models using Python - files and defining of NSDs and VNFDs. - - The benchmark control application being a part of OPNFV yardstick can call - that python models to instantiate and configure the VNFs. Depending on - infrastructure type (bare-metal or fully virtualized) that calls could be - made directly or using MANO system. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * *Traffic generators in NSB*: Any benchmark application requires a set of - traffic generator and traffic profiles defining the method in which traffic - is generated. +.. TODO: provide a list of components in this section and describe them in + later sub-sections - The Network Service benchmarking model extends the Network Service - definition with a set of Traffic Generators (TG) that are treated - same way as other VNFs being a part of benchmarked network service. - Same as other VNFs the traffic generator are instantiated and terminated. +.. Components are the methodology, TGs, framework extensions, KPI collection, + Testcases, SampleVNFs +.. Framework extentions include: VNF models, NSPerf Scenario, contexts - Every traffic generator has own configuration defined as a traffic profile - and a set of KPIs supported. The python models for TG is extended by - specific calls to listen and generate traffic. +* *Models for Network Service benchmarking*: The Network Service benchmarking + requires the proper modelling approach. The NSB provides models using Python + files and defining of NSDs and VNFDs. - * *The stateless TREX traffic generator*: The main traffic generator used as - Network Service stimulus is open source TREX tool. +The benchmark control application being a part of OPNFV Yardstick can call +that Python models to instantiate and configure the VNFs. Depending on +infrastructure type (bare-metal or fully virtualized) that calls could be +made directly or using MANO system. - The TREX tool can generate any kind of stateless traffic. +* *Traffic generators in NSB*: Any benchmark application requires a set of + traffic generator and traffic profiles defining the method in which traffic + is generated. - .. code-block:: console +The Network Service benchmarking model extends the Network Service +definition with a set of Traffic Generators (TG) that are treated +same way as other VNFs being a part of benchmarked network service. +Same as other VNFs the traffic generator are instantiated and terminated. - +--------+ +-------+ +--------+ - | | | | | | - | Trex | ---> | VNF | ---> | Trex | - | | | | | | - +--------+ +-------+ +--------+ +Every traffic generator has own configuration defined as a traffic profile +and a set of KPIs supported. The python models for TG is extended by +specific calls to listen and generate traffic. - Supported testcases scenarios: +* *The stateless TREX traffic generator*: The main traffic generator used as + Network Service stimulus is open source TREX tool. - - Correlated UDP traffic using TREX traffic generator and replay VNF. +The TREX tool can generate any kind of stateless traffic. - - using different IMIX configuration like pure voice, pure video traffic etc - - - using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows - - - Using different number of rules configured like 1 rule, 1K, 10K rules - - For UDP correlated traffic following Key Performance Indicators are collected - for every combination of test case parameters: +.. code-block:: console - - RFC2544 throughput for various loss rate defined (1% is a default) + +--------+ +-------+ +--------+ + | | | | | | + | Trex | ---> | VNF | ---> | Trex | + | | | | | | + +--------+ +-------+ +--------+ + +Supported testcases scenarios: + +* Correlated UDP traffic using TREX traffic generator and replay VNF. + + * using different IMIX configuration like pure voice, pure video traffic etc + * using different number IP flows e.g. 1, 1K, 16K, 64K, 256K, 1M flows + * Using different number of rules configured e.g. 1, 1K, 10K rules + +For UDP correlated traffic following Key Performance Indicators are collected +for every combination of test case parameters: + +* RFC2544 throughput for various loss rate defined (1% is a default) + +KPI Collection +^^^^^^^^^^^^^^ + +KPI collection is the process of sampling KPIs at multiple intervals to allow +for investigation into anomalies during runtime. Some KPI intervals are +adjustable. KPIs are collected from traffic generators and NFVI for the SUT. +There is already some reporting in NSB available, but NSB collects all KPIs for +analytics to process. + +Below is an example list of basic KPIs: +* Throughput +* Latency +* Packet delay variation +* Maximum establishment rate +* Maximum tear-down rate +* Maximum simultaneous number of sessions + +Of course, there can be many other KPIs that will be relevant for a specific +NFVI, but in most cases these KPIs are enough to give you a basic picture of +the SUT. NSB also uses :term:`collectd` in order to collect the KPIs. Currently +the following collectd plug-ins are enabled for NSB testcases: + +* Libvirt +* Interface stats +* OvS events +* vSwitch stats +* Huge Pages +* RAM +* CPU usage +* Intel® PMU +* Intel(r) RDT Graphical Overview -================== +------------------ -NSB Testing with yardstick framework facilitate performance testing of various +NSB Testing with Yardstick framework facilitate performance testing of various VNFs provided. .. code-block:: console +-----------+ - | | +-----------+ - | vPE | ->|TGen Port 0| - | TestCase | | +-----------+ - | | | - +-----------+ +------------------+ +-------+ | - | | -- API --> | VNF | <---> - +-----------+ | Yardstick | +-------+ | - | Test Case | --> | NSB Testing | | - +-----------+ | | | - | | | | - | +------------------+ | - +-----------+ | +-----------+ - | Traffic | ->|TGen Port 1| - | patterns | +-----------+ + | | +-------------+ + | vPE | -->| TGen Port 0 | + | TestCase | | +-------------+ + | | | + +-----------+ +---------------+ +-------+ | + | | ---> | VNF | <---> + +-----------+ | Yardstick | +-------+ | + | Test Case | --> | NSB Testing | | + +-----------+ | | | + | | | | + | +---------------+ | + +-----------+ | +-------------+ + | Traffic | -->| TGen Port 1 | + | patterns | +-------------+ +-----------+ Figure 1: Network Service - 2 server configuration -VNFs supported for chracterization: ------------------------------------ +VNFs supported for chracterization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. CGNAPT - Carrier Grade Network Address and port Translation 2. vFW - Virtual Firewall 3. vACL - Access Control List -4. Prox - Packet pROcessing eXecution engine: - - VNF can act as Drop, Basic Forwarding (no touch), - L2 Forwarding (change MAC), GRE encap/decap, Load balance based on - packet fields, Symmetric load balancing - - QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL +4. PROX - Packet pROcessing eXecution engine: + * VNF can act as Drop, Basic Forwarding (no touch), + L2 Forwarding (change MAC), GRE encap/decap, Load balance based on + packet fields, Symmetric load balancing + * QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL 5. UDP_Replay diff --git a/docs/testing/user/userguide/13-nsb-installation.rst b/docs/testing/user/userguide/13-nsb-installation.rst index 973d56628..71ced43ea 100644 --- a/docs/testing/user/userguide/13-nsb-installation.rst +++ b/docs/testing/user/userguide/13-nsb-installation.rst @@ -8,40 +8,35 @@ ======= Heading 0 (reserved for the title in a document) ------- Heading 1 - ~~~~~~~ Heading 2 + ^^^^^^^ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. -===================================== -Yardstick - NSB Testing -Installation -===================================== + +================ +NSB Installation +================ + +.. _OVS-DPDK: http://docs.openvswitch.org/en/latest/intro/install/dpdk/ +.. _devstack: https://docs.openstack.org/devstack/pike/> Abstract -------- -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments viz., bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. - The steps needed to run Yardstick with NSB testing are: * Install Yardstick (NSB Testing). -* Setup/Reference pod.yaml describing Test topology -* Create/Reference the test configuration yaml file. +* Setup/reference ``pod.yaml`` describing Test topology +* Create/reference the test configuration yaml file. * Run the test case. - Prerequisites ------------- -Refer chapter Yardstick Installation for more information on yardstick -prerequisites +Refer to :doc:`04-installation` for more information on Yardstick +prerequisites. Several prerequisites are needed for Yardstick (VNF testing): @@ -57,11 +52,10 @@ Several prerequisites are needed for Yardstick (VNF testing): * intel-cmt-cat Hardware & Software Ingredients -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SUT requirements: - ======= =================== Item Description ======= =================== @@ -74,7 +68,6 @@ SUT requirements: Boot and BIOS settings: - ============= ================================================= Boot settings default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 @@ -93,29 +86,25 @@ Boot and BIOS settings: Turbo Boost Disabled ============= ================================================= - - Install Yardstick (NSB Testing) ------------------------------- -Download the source code and install Yardstick from it +Download the source code and check out the latest stable branch:: .. code-block:: console git clone https://gerrit.opnfv.org/gerrit/yardstick - cd yardstick - # Switch to latest stable branch - # git checkout <tag or stable branch> - git checkout stable/euphrates + git checkout stable/gambia Configure the network proxy, either using the environment variables or setting -the global environment file: +the global environment file. + +* Set environment -.. code-block:: ini +.. code-block:: - cat /etc/environment http_proxy='http://proxy.company.com:port' https_proxy='http://proxy.company.com:port' @@ -124,14 +113,11 @@ the global environment file: export http_proxy='http://proxy.company.com:port' export https_proxy='http://proxy.company.com:port' -The last step is to modify the Yardstick installation inventory, used by -Ansible: - -.. code-block:: ini +Modify the Yardstick installation inventory, used by Ansible:: cat ./ansible/install-inventory.ini [jumphost] - localhost ansible_connection=local + localhost ansible_connection=local [yardstick-standalone] yardstick-standalone-node ansible_host=192.168.1.2 @@ -148,35 +134,29 @@ Ansible: .. note:: - SSH access without password needs to be configured for all your nodes defined in - ``install-inventory.ini`` file. - If you want to use password authentication you need to install sshpass - - .. code-block:: console + SSH access without password needs to be configured for all your nodes + defined in ``yardstick-install-inventory.ini`` file. + If you want to use password authentication you need to install ``sshpass``:: sudo -EH apt-get install sshpass -To execute an installation for a Bare-Metal or a Standalone context: - -.. code-block:: console +To execute an installation for a BareMetal or a Standalone context:: ./nsb_setup.sh -To execute an installation for an OpenStack context: - -.. code-block:: console +To execute an installation for an OpenStack context:: ./nsb_setup.sh <path to admin-openrc.sh> -Above command setup docker with latest yardstick code. To execute - -.. code-block:: console +The above commands will set up Docker with the latest Yardstick code. To +execute:: docker exec -it yardstick bash It will also automatically download all the packages needed for NSB Testing -setup. Refer chapter :doc:`04-installation` for more on docker +setup. Refer chapter :doc:`04-installation` for more on Docker + **Install Yardstick using Docker (recommended)** Another way to execute an installation for a Bare-Metal or a Standalone context @@ -201,24 +181,22 @@ System Topology Environment parameters and credentials -------------------------------------- -Config yardstick conf -~~~~~~~~~~~~~~~~~~~~~ +Configure yardstick.conf +^^^^^^^^^^^^^^^^^^^^^^^^ -If user did not run 'yardstick env influxdb' inside the container, which will -generate correct ``yardstick.conf``, then create the config file manually (run -inside the container): -:: +If you did not run ``yardstick env influxdb`` inside the container to generate + ``yardstick.conf``, then create the config file manually (run inside the +container):: cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf vi /etc/yardstick/yardstick.conf -Add trex_path, trex_client_lib and bin_path in 'nsb' section. - -:: +Add ``trex_path``, ``trex_client_lib`` and ``bin_path`` to the ``nsb`` +section:: [DEFAULT] debug = True - dispatcher = file, influxdb + dispatcher = influxdb [dispatcher_influxdb] timeout = 5 @@ -235,25 +213,32 @@ Add trex_path, trex_client_lib and bin_path in 'nsb' section. Run Yardstick - Network Service Testcases ----------------------------------------- - NS testing - using yardstick CLI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See :doc:`04-installation` -.. code-block:: console - +Connect to the Yardstick container:: docker exec -it yardstick /bin/bash - source /etc/yardstick/openstack.creds (only for heat TC if nsb_setup.sh was NOT used) - export EXTERNAL_NETWORK="<openstack public network>" (only for heat TC) + +If you're running ``heat`` testcases and ``nsb_setup.sh`` was not used:: + source /etc/yardstick/openstack.creds + +In addition to the above, you need to se the ``EXTERNAL_NETWORK`` for +OpenStack:: + + export EXTERNAL_NETWORK="<openstack public network>" + +Finally, you should be able to run the testcase:: + yardstick --debug task start yardstick/samples/vnf_samples/nsut/<vnf>/<test case> Network Service Benchmarking - Bare-Metal ----------------------------------------- Bare-Metal Config pod.yaml describing Topology -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Bare-Metal 2-Node setup +++++++++++++++++++++++ @@ -284,8 +269,8 @@ Bare-Metal 3-Node setup - Correlated Traffic Bare-Metal Config pod.yaml -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Before executing Yardstick test cases, make sure that pod.yaml reflects the +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Before executing Yardstick test cases, make sure that ``pod.yaml`` reflects the topology and update all the required fields.:: cp /etc/yardstick/nodes/pod.yaml.nsb.sample /etc/yardstick/nodes/pod.yaml @@ -358,22 +343,20 @@ topology and update all the required fields.:: if: "xe1" -Network Service Benchmarking - Standalone Virtualization --------------------------------------------------------- +Standalone Virtualization +------------------------- SR-IOV -~~~~~~ +^^^^^^ SR-IOV Pre-requisites +++++++++++++++++++++ On Host, where VM is created: - a) Create and configure a bridge named ``br-int`` for VM to connect to external network. - Currently this can be done using VXLAN tunnel. + a) Create and configure a bridge named ``br-int`` for VM to connect to + external network. Currently this can be done using VXLAN tunnel. - Execute the following on host, where VM is created: - - .. code-block:: console + Execute the following on host, where VM is created:: ip link add type vxlan remote <Jumphost IP> local <DUT IP> id <ID: 10> dstport 4789 brctl addbr br-int @@ -382,7 +365,7 @@ On Host, where VM is created: ip addr add <IP#1, like: 172.20.2.1/24> dev br-int ip link set dev br-int up - .. note:: May be needed to add extra rules to iptable to forward traffic. + .. note:: You may need to add extra rules to iptable to forward traffic. .. code-block:: console @@ -416,23 +399,24 @@ On Host, where VM is created: Yardstick has a tool for building this custom image with SampleVNF. It is necessary to have ``sudo`` rights to use this tool. - Also you may need to install several additional packages to use this tool, by - following the commands below:: + Also you may need to install several additional packages to use this tool, by + following the commands below:: - sudo apt-get update && sudo apt-get install -y qemu-utils kpartx + sudo apt-get update && sudo apt-get install -y qemu-utils kpartx - This image can be built using the following command in the directory where Yardstick is installed + This image can be built using the following command in the directory where + Yardstick is installed:: - .. code-block:: console + export YARD_IMG_ARCH='amd64' + sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers - export YARD_IMG_ARCH='amd64' - sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers + For instructions on generating a cloud image using Ansible, refer to + :doc:`04-installation`. - Please use ansible script to generate a cloud image refer to :doc:`04-installation` + for more details refer to chapter :doc:`04-installation` - for more details refer to chapter :doc:`04-installation` - - .. note:: VM should be build with static IP and should be accessible from yardstick host. + .. note:: VM should be build with static IP and be accessible from the + Yardstick host. SR-IOV Config pod.yaml describing Topology @@ -457,10 +441,10 @@ SR-IOV 2-Node setup +----------+ +-------------------------+ | | | ^ ^ | | | | | | | - | | (0)<----->(0) | ------ | | - | TG1 | | SUT | | - | | | | | - | | (n)<----->(n) |------------------ | + | | (0)<----->(0) | ------ SUT | | + | TG1 | | | | + | | (n)<----->(n) | ----------------- | + | | | | +----------+ +-------------------------+ trafficgen_1 host @@ -470,29 +454,29 @@ SR-IOV 3-Node setup - Correlated Traffic ++++++++++++++++++++++++++++++++++++++++ .. code-block:: console - +--------------------+ - | | - | | - | DUT | - | (VNF) | - | | - +--------------------+ - | VF NIC | | VF NIC | - +--------+ +--------+ - ^ ^ - | | - | | - +----------+ +-------------------------+ +--------------+ - | | | ^ ^ | | | - | | | | | | | | - | | (0)<----->(0) | ------ | | | TG2 | - | TG1 | | SUT | | | (UDP Replay) | - | | | | | | | - | | (n)<----->(n) | ------ | (n)<-->(n) | | - +----------+ +-------------------------+ +--------------+ - trafficgen_1 host trafficgen_2 - -Before executing Yardstick test cases, make sure that pod.yaml reflects the + +--------------------+ + | | + | | + | DUT | + | (VNF) | + | | + +--------------------+ + | VF NIC | | VF NIC | + +--------+ +--------+ + ^ ^ + | | + | | + +----------+ +---------------------+ +--------------+ + | | | ^ ^ | | | + | | | | | | | | + | | (0)<----->(0) |----- | | | TG2 | + | TG1 | | SUT | | | (UDP Replay) | + | | | | | | | + | | (n)<----->(n) | -----| (n)<-->(n) | | + +----------+ +---------------------+ +--------------+ + trafficgen_1 host trafficgen_2 + +Before executing Yardstick test cases, make sure that ``pod.yaml`` reflects the topology and update all the required fields. .. code-block:: console @@ -547,8 +531,8 @@ SR-IOV Config host_sriov.yaml SR-IOV testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -591,16 +575,15 @@ Update "contexts" section gateway_ip: '152.16.100.20' - OVS-DPDK -~~~~~~~~ +^^^^^^^^ OVS-DPDK Pre-requisites -~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++ On Host, where VM is created: - a) Create and configure a bridge named ``br-int`` for VM to connect to external network. - Currently this can be done using VXLAN tunnel. + a) Create and configure a bridge named ``br-int`` for VM to connect to + external network. Currently this can be done using VXLAN tunnel. Execute the following on host, where VM is created: @@ -647,26 +630,27 @@ On Host, where VM is created: Yardstick has a tool for building this custom image with SampleVNF. It is necessary to have ``sudo`` rights to use this tool. - Also you may need to install several additional packages to use this tool, by - following the commands below:: + You may need to install several additional packages to use this tool, by + following the commands below:: - sudo apt-get update && sudo apt-get install -y qemu-utils kpartx + sudo apt-get update && sudo apt-get install -y qemu-utils kpartx - This image can be built using the following command in the directory where Yardstick is installed:: + This image can be built using the following command in the directory where + Yardstick is installed:: - export YARD_IMG_ARCH='amd64' - sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers - sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh + export YARD_IMG_ARCH='amd64' + sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers + sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh - for more details refer to chapter :doc:`04-installation` + for more details refer to chapter :doc:`04-installation` - .. note:: VM should be build with static IP and should be accessible from yardstick host. + .. note:: VM should be build with static IP and should be accessible from + yardstick host. - c) OVS & DPDK version. - - OVS 2.7 and DPDK 16.11.1 above version is supported +3. OVS & DPDK version. + * OVS 2.7 and DPDK 16.11.1 above version is supported - d) Setup OVS/DPDK on host. - Please refer to below link on how to setup `OVS-DPDK <http://docs.openvswitch.org/en/latest/intro/install/dpdk/>`_ +4. Setup `OVS-DPDK`_ on host. OVS-DPDK Config pod.yaml describing Topology @@ -732,10 +716,8 @@ OVS-DPDK 3-Node setup - Correlated Traffic trafficgen_1 host trafficgen_2 -Before executing Yardstick test cases, make sure that pod.yaml reflects the -topology and update all the required fields. - -.. code-block:: console +Before executing Yardstick test cases, make sure that the ``pod.yaml`` reflects +the topology and update all the required fields:: cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml cp <yardstick>/etc/yardstick/nodes/standalone/host_ovs.yaml /etc/yardstick/nodes/standalone/host_ovs.yaml @@ -786,8 +768,8 @@ OVS-DPDK Config host_ovs.yaml ovs_dpdk testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -841,16 +823,16 @@ Update "contexts" section gateway_ip: '152.16.100.20' -Network Service Benchmarking - OpenStack with SR-IOV support ------------------------------------------------------------- +OpenStack with SR-IOV support +----------------------------- This section describes how to run a Sample VNF test case, using Heat context, with SR-IOV. It also covers how to install OpenStack in Ubuntu 16.04, using DevStack, with SR-IOV support. -Single node OpenStack setup with external TG -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Single node OpenStack with external TG +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -883,26 +865,22 @@ Single node OpenStack setup with external TG Host pre-configuration ++++++++++++++++++++++ -.. warning:: The following configuration requires sudo access to the system. Make - sure that your user have the access. +.. warning:: The following configuration requires sudo access to the system. + Make sure that your user have the access. -Enable the Intel VT-d or AMD-Vi extension in the BIOS. Some system manufacturers -disable this extension by default. +Enable the Intel VT-d or AMD-Vi extension in the BIOS. Some system +manufacturers disable this extension by default. Activate the Intel VT-d or AMD-Vi extension in the kernel by modifying the GRUB config file ``/etc/default/grub``. -For the Intel platform: - -.. code:: bash +For the Intel platform:: ... GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on" ... -For the AMD platform: - -.. code:: bash +For the AMD platform:: ... GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on" @@ -917,9 +895,7 @@ Update the grub configuration file and restart the system: sudo update-grub sudo reboot -Make sure the extension has been enabled: - -.. code:: bash +Make sure the extension has been enabled:: sudo journalctl -b 0 | grep -e IOMMU -e DMAR @@ -932,6 +908,8 @@ Make sure the extension has been enabled: Feb 06 14:50:14 hostname kernel: DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 +.. TODO: Refer to the yardstick installation guide for proxy set up + Setup system proxy (if needed). Add the following configuration into the ``/etc/environment`` file: @@ -954,13 +932,11 @@ Upgrade the system: sudo -EH apt-get upgrade sudo -EH apt-get dist-upgrade -Install dependencies needed for the DevStack +Install dependencies needed for DevStack .. code:: bash - sudo -EH apt-get install python - sudo -EH apt-get install python-dev - sudo -EH apt-get install python-pip + sudo -EH apt-get install python python-dev python-pip Setup SR-IOV ports on the host: @@ -983,10 +959,10 @@ Setup SR-IOV ports on the host: DevStack installation +++++++++++++++++++++ -Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ -documentation to install OpenStack on a host. Please note, that stable -``pike`` branch of devstack repo should be used during the installation. -The required `local.conf`` configuration file are described below. +If you want to try out NSB, but don't have OpenStack set-up, you can use +`Devstack`_ to install OpenStack on a host. Please note, that the +``stable/pike`` branch of devstack repo should be used during the installation. +The required ``local.conf`` configuration file are described below. DevStack configuration file: @@ -1001,15 +977,13 @@ DevStack configuration file: Start the devstack installation on a host. - TG host configuration +++++++++++++++++++++ -Yardstick automatically install and configure Trex traffic generator on TG +Yardstick automatically installs and configures Trex traffic generator on TG host based on provided POD file (see below). Anyway, it's recommended to check -the compatibility of the installed NIC on the TG server with software Trex using -the manual at https://trex-tgn.cisco.com/trex/doc/trex_manual.html. - +the compatibility of the installed NIC on the TG server with software Trex +using the `manual <https://trex-tgn.cisco.com/trex/doc/trex_manual.html>`_. Run the Sample VNF test case ++++++++++++++++++++++++++++ @@ -1018,7 +992,7 @@ There is an example of Sample VNF test case ready to be executed in an OpenStack environment with SR-IOV support: ``samples/vnf_samples/nsut/vfw/ tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``. -Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +Install Yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack context. Create pod file for TG in the yardstick repo folder located in the yardstick @@ -1071,16 +1045,14 @@ Controller/Compute pre-configuration ++++++++++++++++++++++++++++++++++++ Pre-configuration of the controller and compute hosts are the same as -described in `Host pre-configuration`_ section. Follow the steps in the section. - +described in `Host pre-configuration`_ section. DevStack configuration ++++++++++++++++++++++ -Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ -documentation to install OpenStack on a host. Please note, that stable -``pike`` branch of devstack repo should be used during the installation. -The required `local.conf`` configuration file are described below. +A reference ``local.conf`` for deploying OpenStack in a multi-host environment +using `Devstack`_ is shown in this section. The ``stable/pike`` branch of +devstack repo should be used during the installation. .. note:: Update the devstack configuration files by replacing angluar brackets with a short description inside. @@ -1100,17 +1072,17 @@ DevStack configuration file for compute host: Start the devstack installation on the controller and compute hosts. - Run the sample vFW TC +++++++++++++++++++++ -Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +Install Yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack context. -Run sample vFW RFC2544 SR-IOV TC (``samples/vnf_samples/nsut/vfw/ -tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) in the heat -context using steps described in `NS testing - using yardstick CLI`_ section -and the following yardtick command line arguments: +Run the sample vFW RFC2544 SR-IOV test case +(``samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) +in the heat context using steps described in +`NS testing - using yardstick CLI`_ section and the following Yardstick command +line arguments: .. code:: bash @@ -1118,8 +1090,8 @@ and the following yardtick command line arguments: samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml -Enabling other Traffic generator --------------------------------- +Enabling other Traffic generators +--------------------------------- IxLoad ~~~~~~ @@ -1138,14 +1110,16 @@ IxLoad .. code-block:: console - cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml + cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia \ + etc/yardstick/nodes/pod_ixia.yaml Config ``pod_ixia.yaml`` .. literalinclude:: code/pod_ixia.yaml :language: console - for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration + for sriov/ovs_dpdk pod files, please refer to `Standalone Virtualization`_ + for ovs-dpdk/sriov configuration 3. Start IxOS TCL Server (Install 'Ixia IxExplorer IxOS <version>') You will also need to configure the IxLoad machine to start the IXIA @@ -1155,7 +1129,7 @@ IxLoad * Go to: ``Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1`` or - ``"C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"`` + ``C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe`` 4. Create a folder ``Results`` in c:\ and share the folder on the network. @@ -1163,7 +1137,7 @@ IxLoad ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml`` IxNetwork -~~~~~~~~~ +^^^^^^^^^ IxNetwork testcases use IxNetwork API Python Bindings module, which is installed as part of the requirements of the project. @@ -1172,14 +1146,16 @@ installed as part of the requirements of the project. .. code-block:: console - cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml + cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia \ + etc/yardstick/nodes/pod_ixia.yaml - Config pod_ixia.yaml + Configure ``pod_ixia.yaml`` .. literalinclude:: code/pod_ixia.yaml :language: console - for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration + for sriov/ovs_dpdk pod files, please refer to above + `Standalone Virtualization`_ for ovs-dpdk/sriov configuration 2. Start IxNetwork TCL Server You will also need to configure the IxNetwork machine to start the IXIA diff --git a/docs/testing/user/userguide/14-nsb-operation.rst b/docs/testing/user/userguide/14-nsb-operation.rst index 72b1c4bbc..12e269187 100644 --- a/docs/testing/user/userguide/14-nsb-operation.rst +++ b/docs/testing/user/userguide/14-nsb-operation.rst @@ -2,6 +2,16 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2018 Intel Corporation. +.. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. Yardstick - NSB Testing - Operation =================================== @@ -89,9 +99,9 @@ Availability zone ^^^^^^^^^^^^^^^^^ The configuration of the availability zone is requred in cases where location -of exact compute host/group of compute hosts needs to be specified for SampleVNF -or traffic generator in the heat test case. If this is the case, please follow -the instructions below. +of exact compute host/group of compute hosts needs to be specified for +:term:`SampleVNF` or traffic generator in the heat test case. If this is the +case, please follow the instructions below. .. _`Create a host aggregate`: @@ -105,7 +115,8 @@ the instructions below. .. code-block:: bash # create host aggregate - openstack aggregate create --zone <AZ_NAME> --property availability_zone=<AZ_NAME> <AGG_NAME> + openstack aggregate create --zone <AZ_NAME> \ + --property availability_zone=<AZ_NAME> <AGG_NAME> # show available hosts openstack compute service list --service nova-compute # add selected host into the host aggregate @@ -136,8 +147,9 @@ the instructions below. networks: ... -There are two example of SampleVNF scale out test case which use the availability zone -feature to specify the exact location of scaled VNFs and traffic generators. +There are two example of SampleVNF scale out test case which use the +``availability zone`` feature to specify the exact location of scaled VNFs and +traffic generators. Those are: @@ -164,21 +176,19 @@ Those are: | 5 | agg1 | AZ_NAME_1 | +----+------+-------------------+ -2. If no host aggregates are configured, please use `steps above`__ to - configure them. +2. If no host aggregates are configured, please follow the instructions to + `Create a host aggregate`_ -__ `Create a host aggregate`_ - -3. Run the SampleVNF PROX scale-out test case, specifying the availability - zone of each VNF and traffic generator as a task arguments. +3. Run the SampleVNF PROX scale-out test case, specifying the + ``availability zone`` of each VNF and traffic generator as task arguments. .. note:: The ``az_0`` and ``az_1`` should be changed according to the host - aggregates created in the OpenStack. + aggregates created in the OpenStack. .. code-block:: console - yardstick -d task start\ + yardstick -d task start \ <repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml\ --task-args='{ "num_vnfs": 4, "availability_zone": { @@ -198,7 +208,7 @@ Collectd KPIs ------------- NSB can collect KPIs from collected. We have support for various plugins -enabled by the Barometer project. +enabled by the :term:`Barometer` project. The default yardstick-samplevnf has collectd installed. This allows for collecting KPIs from the VNF. @@ -208,12 +218,11 @@ We assume that collectd is not installed on the compute nodes. To collectd KPIs from the NFVi compute nodes: - * install_collectd on the compute nodes * create pod.yaml for the compute nodes * enable specific plugins depending on the vswitch and DPDK - example pod.yaml section for Compute node running collectd. + example ``pod.yaml`` section for Compute node running collectd. .. code-block:: yaml @@ -356,8 +365,8 @@ Scale-Out VNFs performance data with scale-out helps - * in capacity planning to meet the given network node requirements - * in comparison between different VNF vendor offerings + * capacity planning to meet the given network node requirements + * comparison between different VNF vendor offerings * better the scale-out index, provides the flexibility in meeting future capacity requirements @@ -488,11 +497,11 @@ Default values for OVS-DPDK: Sample test case file ^^^^^^^^^^^^^^^^^^^^^ - 1. Prepare SampleVNF image and copy it to ``flavor/images``. - 2. Prepare context files for TREX and SampleVNF under ``contexts/file``. - 3. Add bridge named ``br-int`` to the baremetal where SampleVNF image is deployed. - 4. Modify ``networks/phy_port`` accordingly to the baremetal setup. - 5. Run test from: +1. Prepare SampleVNF image and copy it to ``flavor/images``. +2. Prepare context files for TREX and SampleVNF under ``contexts/file``. +3. Add bridge named ``br-int`` to the baremetal where SampleVNF image is deployed. +4. Modify ``networks/phy_port`` accordingly to the baremetal setup. +5. Run test from: .. literalinclude:: /../samples/vnf_samples/nsut/acl/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml :language: yaml diff --git a/docs/testing/user/userguide/glossary.rst b/docs/testing/user/userguide/glossary.rst index 6a153943c..cef9b69a5 100644 --- a/docs/testing/user/userguide/glossary.rst +++ b/docs/testing/user/userguide/glossary.rst @@ -13,23 +13,54 @@ Glossary API Application Programming Interface + Barometer + OPNFV NFVi Service Assurance project. Barometer upstreams changes to + collectd, OpenStack, etc to improve features related to NFVi monitoring + and service assurance. + More info on: https://opnfv-barometer.readthedocs.io/en/latest/ + + collectd + collectd is a system statistics collection daemon. + More info on: https://collectd.org/ + + context + A context describes the environment in which a yardstick testcase will + be run. It can refer to a pre-provisioned environment, or an environment + that will be set up using OpenStack or Kubernetes. + Docker Docker provisions and manages containers. Yardstick and many other OPNFV projects are deployed in containers. Docker is required to launch the containerized versions of these projects. - DPI - Deep Packet Inspection - DPDK Data Plane Development Kit + DPI + Deep Packet Inspection + DSCP Differentiated Services Code Point + flavor + A specification of virtual resources used by OpenStack in the creation + of a VM instance. + + Grafana + A visualization tool, used in Yardstick to retrieve test data from + InfluxDB and display it. Grafana works by defining dashboards, which are + combinations of visualization panes (e.g. line charts and gauges) and + forms that assist the user in formulating SQL-like queries for InfluxDB. + More info on: https://grafana.com/ + IGMP Internet Group Management Protocol + InfluxDB + One of the Dispatchers supported by Yardstick, it allows test results to + be reported to a time-series database. + More info on: https://www.influxdata.com/ + IOPS Input/Output Operations Per Second A performance measurement used to benchmark storage devices. @@ -43,6 +74,9 @@ Glossary deployment, scaling and management of containerized applications. It is one of the contexts supported in Yardstick. + MPLS + Multiprotocol Label Switching + NFV Network Function Virtualization NFV is an initiative to take network services which were traditionally run @@ -56,6 +90,10 @@ Glossary NIC Network Interface Controller + NSB + Network Services Benchmarking. A subset of Yardstick features concerned + with NFVI and VNF characterization. + OpenStack OpenStack is a cloud operating system that controls pools of compute, storage, and networking resources. OpenStack is an open source project @@ -77,6 +115,18 @@ Glossary performance in Input/Output Operations Per Second (IOPS), throttling agreements, and performance expectations at peak load + runner + The part of a Yardstick testcase that determines how the test will be run + (e.g. for x iterations, y seconds or until state z is reached). The runner + also determines when the metrics are collected/reported. + + SampleVNF + OPNFV project providing a repository of reference VNFs. + More info on: https://opnfv-samplevnf.readthedocs.io/en/latest/ + + scenario + The part of a Yardstick testcase that describes each test step. + SLA Service Level Agreement An SLA is an agreement between a service provider and a customer to @@ -92,6 +142,10 @@ Glossary SUT System Under Test + testcase + A task in Yardstick; the yaml file that is read by Yardstick to + determine how to run a test. + ToS Type of Service |