diff options
Diffstat (limited to 'docs/testing')
17 files changed, 1540 insertions, 535 deletions
diff --git a/docs/testing/user/userguide/01-introduction.rst b/docs/testing/user/userguide/01-introduction.rst index 5fc2e8d0f..2a3ab4ea7 100755 --- a/docs/testing/user/userguide/01-introduction.rst +++ b/docs/testing/user/userguide/01-introduction.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Introduction @@ -32,7 +42,7 @@ independent. About This Document -=================== +------------------- This document consists of the following chapters: @@ -79,7 +89,7 @@ This document consists of the following chapters: cases. Contact Yardstick -================= +----------------- Feedback? `Contact us`_ diff --git a/docs/testing/user/userguide/02-methodology.rst b/docs/testing/user/userguide/02-methodology.rst index 34d271095..b1eee9781 100644 --- a/docs/testing/user/userguide/02-methodology.rst +++ b/docs/testing/user/userguide/02-methodology.rst @@ -1,20 +1,30 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB and others. + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. =========== Methodology =========== Abstract -======== +-------- This chapter describes the methodology implemented by the Yardstick project for verifying the :term:`NFVI` from the perspective of a :term:`VNF`. ETSI-NFV -======== +-------- .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf .. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2 @@ -53,7 +63,7 @@ The methodology includes five steps: .. seealso:: Yardsticktst_ for material on alignment ETSI TST001 and Yardstick. Metrics -======= +------- The metrics, as defined by ETSI GS NFV-TST001, are shown in :ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and diff --git a/docs/testing/user/userguide/03-architecture.rst b/docs/testing/user/userguide/03-architecture.rst index 62250d6a3..94081b0e1 100755 --- a/docs/testing/user/userguide/03-architecture.rst +++ b/docs/testing/user/userguide/03-architecture.rst @@ -1,23 +1,34 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 Huawei Technologies Co.,Ltd and others + License. + http://creativecommons.org/licenses/by/4.0 + (c) 2016 Huawei Technologies Co.,Ltd and others + + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============ Architecture ============ Abstract -======== -This chapter describes the yardstick framework software architecture. We will -introduce it from Use-Case View, Logical View, Process View and Deployment +-------- + +This chapter describes the Yardstick framework software architecture. We will +introduce it from Use Case View, Logical View, Process View and Deployment View. More technical details will be introduced in this chapter. Overview -======== +-------- Architecture overview ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Yardstick is mainly written in Python, and test configurations are made in YAML. Documentation is written in reStructuredText format, i.e. .rst files. Yardstick is inspired by Rally. Yardstick is intended to run on a @@ -34,7 +45,8 @@ the test result will be shown with grafana. Concept -------- +^^^^^^^ + **Benchmark** - assess the relative performance of something **Benchmark** configuration file - describes a single test case in yaml format @@ -62,7 +74,7 @@ configuration file and evaluated by the runner. Runner types ------------- +^^^^^^^^^^^^ There exists several predefined runner types to choose between when designing a test scenario: @@ -129,7 +141,8 @@ Snippet of an Iteration runner configuration: Use-Case View -============= +------------- + Yardstick Use-Case View shows two kinds of users. One is the Tester who will do testing in cloud, the other is the User who is more concerned with test result and result analyses. @@ -158,7 +171,8 @@ on OPNFV testing dashboard which use MongoDB as backend. :alt: Yardstick Use-Case View Logical View -============ +------------ + Yardstick Logical View describes the most important classes, their organization, and the most important use-case realizations. @@ -195,7 +209,8 @@ finished. :alt: Yardstick framework architecture in Danube Process View (Test execution flow) -================================== +---------------------------------- + Yardstick process view shows how yardstick runs a test case. Below is the sequence graph about the test execution flow using heat context, and each object represents one module in yardstick: @@ -222,7 +237,8 @@ will call "Openstack" to undeploy the heat stack. Once the stack is undepoyed, the whole test ends. Deployment View -=============== +--------------- + Yardstick deployment view shows how the yardstick tool can be deployed into the underlying platform. Generally, yardstick tool is installed on JumpServer(see `07-installation` for detail installation steps), and JumpServer is @@ -235,7 +251,7 @@ result for better showing. :alt: Yardstick Deployment View Yardstick Directory structure -============================= +----------------------------- **yardstick/** - Yardstick main directory. diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index 3ba312ce7..213821798 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -1,7 +1,17 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. + + + Convention for heading levels in Yardstick documentation: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. .. Convention for heading levels in Yardstick documentation: @@ -18,7 +28,6 @@ Yardstick Installation ====================== - Yardstick supports installation by Docker or directly in Ubuntu. The installation procedure for Docker and direct installation are detailed in the sections below. @@ -139,7 +148,7 @@ in the following sections. Before that, access the Yardstick container:: and then configure Yardstick environments in the Yardstick container. Using the CLI command ``env prepare`` (first way) (**recommended**) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ In the Yardstick container, the Yardstick repository is located in the ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack @@ -171,10 +180,10 @@ terminal window and execute the following command:: Manually exporting the env variables and initializing OpenStack (second way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Export OpenStack environment variables -###################################### +'''''''''''''''''''''''''''''''''''''' Before running Yardstick it is necessary to export OpenStack environment variables:: @@ -200,7 +209,7 @@ A sample ``openrc`` file may look like this:: Manual creation of Yardstick flavor and guest images -#################################################### +'''''''''''''''''''''''''''''''''''''''''''''''''''' Before executing Yardstick test cases, make sure that Yardstick flavor and guest image are available in OpenStack. Detailed steps about creating the @@ -257,14 +266,14 @@ image. Add Cirros and Ubuntu images to OpenStack:: Automatic initialization of OpenStack (third way) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++++++++++++++++++++++++++++ Similar to the second way, the first step is also to `Export OpenStack environment variables`_. Then the following steps should be done. Automatic creation of Yardstick flavor and guest images -####################################################### +''''''''''''''''''''''''''''''''''''''''''''''''''''''' Yardstick has a script for automatically creating Yardstick flavor and building Yardstick guest images. This script is mainly used for CI and can be also used @@ -340,7 +349,6 @@ For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is not available. You need to prepare OpenStack environment variables and create Yardstick flavor and guest images manually. - Uninstall Yardstick ^^^^^^^^^^^^^^^^^^^ @@ -455,112 +463,111 @@ These configuration files can be found in the ``samples`` directory. Default location for the output is ``/tmp/yardstick.out``. -Automatic installation of Yardstick using ansible -------------------------------------------------- +Automatic installation of Yardstick +----------------------------------- -Automatic installation can be used as an alternative to the manual. -Yardstick can be installed on the bare metal and to the container. Yardstick +Automatic installation can be used as an alternative to the manual by +providing parameters for ansible script ``install.yaml`` in a ``nsb_setup.sh`` +file. Yardstick can be installed on the bare metal and to the container. Yardstick container can be either pulled or built. Bare metal installation ^^^^^^^^^^^^^^^^^^^^^^^ -Use ansible script ``install.yaml`` to install Yardstick on Ubuntu server: +Modify ``nsb_setup.sh`` file ``install.yaml`` parameters to install Yardstick +on Ubuntu server: .. code-block:: console ansible-playbook -i install-inventory.ini install.yaml \ + -e IMAGE_PROPERTY='none' \ -e YARDSTICK_DIR=<path to Yardstick folder> .. note:: By default ``INSTALLATION_MODE`` is ``baremetal``. -.. note:: By default Ubuntu 16.04 is chosen (xenial). It can be changed to - Ubuntu 18.04 (bionic) by passing ``-e OS_RELEASE=bionic`` parameter. +.. note:: No modification in ``install-inventory.ini`` is needed for Yardstick + installation. .. note:: To install Yardstick in virtual environment pass parameter ``-e VIRTUAL_ENVIRONMENT=True``. -To build Yardstick NSB image pass ``IMG_PROPERTY=nsb`` as input parameter: - -.. code-block:: console - - ansible-playbook -i install-inventory.ini install.yaml \ - -e IMAGE_PROPERTY=nsb \ - -e YARDSTICK_DIR=<path to Yardstick folder> - -.. note:: In this ``INSTALLATION_MODE`` mode either Yardstick image or SampleVNF - images will be built. Image type is defined by parameter ``IMAGE_PROPERTY``. - By default Yardstick image will be built. - Container installation ^^^^^^^^^^^^^^^^^^^^^^ -Use ansible script ``install.yaml`` to pull or build Yardstick -container. To pull Yardstick image and start container run: +Modify ``install.yaml`` parameters in ``nsb_setup.sh`` file to pull or build +Yardstick container. To pull Yardstick image and start container run: .. code-block:: console ansible-playbook -i install-inventory.ini install.yaml \ - -e YARDSTICK_DIR=<path to Yardstick folder> \ + -e IMAGE_PROPERTY='none' \ -e INSTALLATION_MODE=container_pull -.. note:: In this ``INSTALLATION_MODE`` mode either Yardstick image or SampleVNF - images will be built. Image type is defined by variable ``IMG_PROPERTY`` in - file ``ansible/group_vars/all.yml``. By default Yardstick image will be - built. - -.. note:: Open question: How to know if Docker image is built on Ubuntu 16.04 and 18.04? - Do we need separate tag to be used? +.. note:: Yardstick docker image is available for both Ubuntu 16.04 and Ubuntu + 18.04. By default Ubuntu 16.04 based docker image is used. To use + Ubuntu 18.04 based docker image pass ``-i opnfv/yardstick-ubuntu-18.04`` + parameter to ``nsb_setup.sh``. -To build Yardstick image run: +To build Yardstick image modify Dockerfile as per comments in it and run: .. code-block:: console - ansible-playbook -i install-inventory.ini install.yaml \ - -e YARDSTICK_DIR=<path to Yardstick folder> \ - -e INSTALLATION_MODE=container + cd yardstick + docker build -f docker/Dockerfile -t opnfv/yardstick:<tag> . -.. note:: In this ``INSTALLATION_MODE`` mode neither Yardstick image nor SampleVNF - image will be built. +.. note:: Yardstick docker image based on Ubuntu 16.04 will be built. + Pass ``-f docker/Dockerfile_ubuntu18`` to build Yardstick docker image based + on Ubuntu 18.04. -.. note:: By default Ubuntu 16.04 is chosen (xenial). It can be changed to - Ubuntu 18.04 (bionic) by passing ``-e OS_RELEASE=bionic`` parameter. +.. note:: Add ``--build-arg http_proxy=http://<proxy_host>:<proxy_port>`` to + build docker image if server is behind the proxy. Parameters for ``install.yaml`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Description of the parameters used with ``install.yaml`` script +Description of the parameters used with ``install.yaml``: +-------------------------+-------------------------------------------------+ | Parameters | Detail | +=========================+=================================================+ - | -i install-inventory.ini| Installs package dependency to remote servers | - | | Mandatory parameter | - | | By default no remote servers are provided | - | | Needed packages will be installed on localhost | + | -i install-inventory.ini|| Installs package dependency to remote servers | + | || and localhost | + | || Mandatory parameter | + | || By default no remote servers are provided | +-------------------------+-------------------------------------------------+ - | -e YARDSTICK_DIR | Path to Yardstick folder | - | | Mandatory parameter | + | -e YARDSTICK_DIR || Path to Yardstick folder | + | || Mandatory parameter for Yardstick bare metal | + | || installation | +-------------------------+-------------------------------------------------+ - | -e INSTALLATION_MODE | baremetal: Yardstick is installed to the bare | - | | metal | - | | Default parameter | + | -e INSTALLATION_MODE || baremetal: Yardstick is installed to the bare | + | | metal | + | || Default parameter | | +-------------------------------------------------+ - | | container: Yardstick is installed in container | - | | Container is built from Dockerfile | + | || container: Yardstick is installed in container | + | || Container is built from Dockerfile | | +-------------------------------------------------+ - | | container_pull: Yardstick is installed in | - | | container | - | | Container is pulled from docker hub | + | || container_pull: Yardstick is installed in | + | || container | + | || Container is pulled from docker hub | + +-------------------------+-------------------------------------------------+ + | -e OS_RELEASE || xenial or bionic: Ubuntu version to be used for| + | || VM image (nsb or normal) | + | || Default is Ubuntu 16.04, xenial | +-------------------------+-------------------------------------------------+ - | -e OS_RELEASE | xenial or bionic: Ubuntu version to be used | - | | Default is Ubuntu 16.04 (xenial) | + | -e IMAGE_PROPERTY || nsb: Build Yardstick NSB VM image | + | || Used to run Yardstick NSB tests on sample VNF | + | || Default parameter | + | +-------------------------------------------------+ + | || normal: Build VM image to run ping test in | + | || OpenStack | + | +-------------------------------------------------+ + | || none: don't build a VM image. | +-------------------------+-------------------------------------------------+ - | -e IMAGE_PROPERTY | normal or nsb: Type of the VM image to be built | - | | Default image is Yardstick | + | -e VIRTUAL_ENVIRONMENT || False or True: Whether install in virtualenv | + | || Default is False | +-------------------------+-------------------------------------------------+ - | -e VIRTUAL_ENVIRONMENT | False or True: Whether install in virtualenv | - | | Default is False | + | -e YARD_IMAGE_ARCH || CPU architecture on servers | + | || Default is 'amd64' | +-------------------------+-------------------------------------------------+ @@ -655,11 +662,9 @@ Modify ``yardstick.conf`` to add the ``influxdb`` dispatcher:: Now Yardstick will store results in InfluxDB when you run a testcase. - Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**) --------------------------------------------------------- - Proxy Support ------------- diff --git a/docs/testing/user/userguide/06-yardstick-plugin.rst b/docs/testing/user/userguide/06-yardstick-plugin.rst index bc35e239d..a5d890b14 100644 --- a/docs/testing/user/userguide/06-yardstick-plugin.rst +++ b/docs/testing/user/userguide/06-yardstick-plugin.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Installing a plug-in into Yardstick =================================== Abstract -======== +-------- Yardstick provides a ``plugin`` CLI command to support integration with other OPNFV testing projects. Below is an example invocation of Yardstick plugin @@ -17,7 +27,7 @@ command and Storperf plug-in sample. Installing Storperf into Yardstick -================================== +---------------------------------- Storperf is delivered as a Docker container from https://hub.docker.com/r/opnfv/storperf/tags/. @@ -31,7 +41,7 @@ In this introduction we will install Storperf on Jump Host. Step 0: Environment preparation -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Running Storperf on Jump Host Requirements: @@ -100,7 +110,7 @@ container. You may need to copy it to the root directory of the Storperf deployed host. Step 1: Plug-in configuration file preparation ----------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install a plug-in, first you need to prepare a plug-in configuration file in YAML format and store it in the "plugin" directory. The plugin configration @@ -125,7 +135,7 @@ Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment. Step 2: Plug-in install/remove scripts preparation --------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In ``yardstick/resource/scripts`` directory, there are two folders: an ``install`` folder and a ``remove`` folder. You need to store the plug-in @@ -139,15 +149,15 @@ For example, the install and remove scripts for Storperf are both named ``storperf.bash``. Step 3: Install and remove Storperf ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To install Storperf, simply execute the following command:: # Install Storperf yardstick plugin install plugin/storperf.yaml -Removing Storperf from yardstick -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Removing Storperf from Yardstick +++++++++++++++++++++++++++++++++ To remove Storperf, simply execute the following command:: diff --git a/docs/testing/user/userguide/07-result-store-InfluxDB.rst b/docs/testing/user/userguide/07-result-store-InfluxDB.rst index cde931376..8a9196b1b 100644 --- a/docs/testing/user/userguide/07-result-store-InfluxDB.rst +++ b/docs/testing/user/userguide/07-result-store-InfluxDB.rst @@ -1,14 +1,23 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + License. + http://creativecommons.org/licenses/by/4.0 + (c) OPNFV, 2016 Huawei Technologies Co.,Ltd and others. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ============================================== Store Other Project's Test Results in InfluxDB ============================================== Abstract -======== +-------- .. _Framework: https://wiki.opnfv.org/download/attachments/6827660/wiki.png?version=1&modificationDate=1470298075000&api=v2 @@ -21,7 +30,7 @@ into community's InfluxDB. The framework is shown in Framework_. :alt: Store Other Project's Test Results in InfluxDB Store Storperf Test Results into Community's InfluxDB -===================================================== +----------------------------------------------------- .. _Influxdb: https://git.opnfv.org/cgit/yardstick/tree/yardstick/dispatcher/influxdb.py .. _Mingjiang: mailto:limingjiang@huawei.com diff --git a/docs/testing/user/userguide/08-grafana.rst b/docs/testing/user/userguide/08-grafana.rst index 020a08a65..ebe9f570d 100644 --- a/docs/testing/user/userguide/08-grafana.rst +++ b/docs/testing/user/userguide/08-grafana.rst @@ -3,13 +3,23 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) 2016 Huawei Technologies Co.,Ltd and others +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ================= Grafana dashboard ================= Abstract -======== +-------- This chapter describes the Yardstick grafana dashboard. The Yardstick grafana dashboard can be found here: http://testresults.opnfv.org/grafana/ @@ -21,14 +31,14 @@ dashboard can be found here: http://testresults.opnfv.org/grafana/ Public access -============= +------------- Yardstick provids a public account for accessing to the dashboard. The username and password are both set to ‘opnfv’. Testcase dashboard -================== +------------------ For each test case, there is a dedicated dashboard. Shown here is the dashboard of TC002. @@ -56,7 +66,7 @@ zoom out the chart. Administration access -===================== +--------------------- For a user with administration rights it is easy to update and save any dashboard configuration. Saved updates immediately take effect and become live. @@ -72,11 +82,11 @@ This may cause issues like: Any change made by administrator should be careful. -Add a dashboard into yardstick grafana -====================================== +Add a dashboard into Yardstick Grafana +-------------------------------------- Due to security concern, users that using the public opnfv account are not able -to edit the yardstick grafana directly.It takes a few more steps for a +to edit the yardstick grafana directly. It takes a few more steps for a non-yardstick user to add a custom dashboard into yardstick grafana. There are 6 steps to go. diff --git a/docs/testing/user/userguide/09-api.rst b/docs/testing/user/userguide/09-api.rst index 1a896699b..f227878ae 100644 --- a/docs/testing/user/userguide/09-api.rst +++ b/docs/testing/user/userguide/09-api.rst @@ -2,6 +2,15 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. ===================== Yardstick Restful API @@ -9,16 +18,16 @@ Yardstick Restful API Abstract -======== +-------- Yardstick support restful API since Danube. Available API -============= +------------- /yardstick/env/action ---------------------- +^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports: @@ -69,7 +78,7 @@ get the task result. /yardstick/asynctask --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the status of asynchronous tasks @@ -91,7 +100,7 @@ NOTE:: /yardstick/testcases --------------------- +^^^^^^^^^^^^^^^^^^^^ Description: This API is used to list all released Yardstick test cases. @@ -106,7 +115,7 @@ Example:: /yardstick/testcases/release/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick released test case. @@ -130,7 +139,7 @@ result. /yardstick/testcases/samples/action ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick sample test case. @@ -154,7 +163,7 @@ the result. /yardstick/testcases/<testcase_name>/docs ------------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to the documentation of a certain released test case. @@ -170,7 +179,7 @@ Example:: /yardstick/testsuites/action ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to run a Yardstick test suite. @@ -194,7 +203,7 @@ result. /yardstick/tasks/<task_id>/log ------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to get the real time log of test case execution. @@ -209,7 +218,7 @@ Example:: /yardstick/results ------------------- +^^^^^^^^^^^^^^^^^^ Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use @@ -228,7 +237,7 @@ This API will return a list of test case result /api/v2/yardstick/openrcs -------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -282,7 +291,7 @@ Example:: /api/v2/yardstick/openrcs/<openrc_id> -------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: @@ -308,7 +317,7 @@ Example:: /api/v2/yardstick/pods ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -334,7 +343,7 @@ Example:: /api/v2/yardstick/pods/<pod_id> -------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: @@ -358,7 +367,7 @@ Example:: /api/v2/yardstick/images ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -383,7 +392,7 @@ Example:: /api/v2/yardstick/images/<image_id> ------------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: @@ -407,7 +416,7 @@ Example:: /api/v2/yardstick/tasks ------------------------ +^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -433,7 +442,7 @@ Example:: /api/v2/yardstick/tasks/<task_id> ---------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: @@ -518,7 +527,7 @@ Example:: /api/v2/yardstick/testcases ---------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick testcases. For Euphrates, it supports: @@ -553,7 +562,7 @@ Example:: /api/v2/yardstick/testcases/<case_name> ---------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: @@ -579,7 +588,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -617,7 +626,7 @@ Example:: /api/v2/yardstick/testsuites ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: @@ -643,7 +652,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Yardstick test projects. For Euphrates, it supports: @@ -678,7 +687,7 @@ Example:: /api/v2/yardstick/projects --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: @@ -704,7 +713,7 @@ Example:: /api/v2/yardstick/containers ----------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: @@ -744,7 +753,7 @@ Example:: /api/v2/yardstick/containers/<container_id> -------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: diff --git a/docs/testing/user/userguide/10-yardstick-user-interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst index b3056ec99..246e1b1df 100644 --- a/docs/testing/user/userguide/10-yardstick-user-interface.rst +++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst @@ -1,3 +1,17 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + ======================== Yardstick User Interface ======================== @@ -18,7 +32,7 @@ The following layouts are available: Commands -======== +-------- To generate the compact HTML report, run:: @@ -30,7 +44,7 @@ To generate the dynamic HTML report, run:: Description -=========== +----------- 1. When the command is triggered, the relevant values for the provided task-id and testcase name are retrieved from the diff --git a/docs/testing/user/userguide/12-nsb-overview.rst b/docs/testing/user/userguide/12-nsb-overview.rst index 7b0d46804..012ea6c3d 100644 --- a/docs/testing/user/userguide/12-nsb-overview.rst +++ b/docs/testing/user/userguide/12-nsb-overview.rst @@ -3,75 +3,96 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2017 Intel Corporation. +.. Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + =================================== Network Services Benchmarking (NSB) =================================== -Abstract -======== - .. _Yardstick: https://wiki.opnfv.org/display/yardstick +.. _`ETSI GS NFV-TST001`: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_nfv-tst001v010101p.pdf + +Abstract +-------- This chapter provides an overview of the NSB, a contribution to OPNFV Yardstick_ from Intel. Overview -======== - -The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi -Characterization and benchmarking with repeatable and deterministic methods. - -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments - bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. +-------- + +Network Services Benchmarking (:term:`NSB`) uses the :term:`Yardstick` +framework for performing :term:`VNF` and :term:`NFVI` characterisation in an +:term:`NFV` environment. + +For VNF characterisation, NSB will onboard a VNF, source and sink traffic to it +via traffic generators, and collect a variety of key performance indicators +(:term:`KPI`) during VNF execution. The stream of KPI data is stored in a +database, and it is visualized in a performance-visualization dashboard. + +For NFVI characterisation, a fixed test VNF, called :term:`PROX` is used. +PROX implements a suite of test cases and visualizes the output data of the +test suite. The PROX test cases implement various execution kernels found in +real-world VNFs, and the output of the test cases provides an indication of +the fitness of the infrastructure for running NFV services, in addition to +indicating potential performance optimizations for the NFVI. + +NSB extends the Yardstick framework to do VNF characterization in three +different execution environments - bare metal i.e. native Linux environment, +standalone virtual environment and managed virtualized environment (e.g. +OpenStack). It also brings in the capability to interact with external traffic +generators, both hardware and software based, for triggering and validating the +traffic according to user defined profiles. NSB extension includes: - - Generic data models of Network Services, based on ETSI spec `ETSI GS NFV-TST 001 <http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_nfv-tst001v010101p.pdf>`_ - - - New Standalone context for VNF testing like SRIOV, OVS, OVS-DPDK etc - - - Generic VNF configuration models and metrics implemented with Python - classes - - - Traffic generator features and traffic profiles - - - L1-L3 state-less traffic profiles - - - L4-L7 state-full traffic profiles - - - Tunneling protocol / network overlay support - - - Test case samples +* Generic data models of Network Services, based on ETSI spec + `ETSI GS NFV-TST 001`_ +* Standalone :term:`context` for VNF testing SRIOV, OVS, OVS-DPDK, etc +* Generic VNF configuration models and metrics implemented with Python + classes +* Traffic generator features and traffic profiles - - Ping + * L1-L3 stateless traffic profiles + * L4-L7 state-full traffic profiles + * Tunneling protocol/network overlay support - - Trex +* Scenarios that handle NSB test cases execution - - vPE,vCGNAT, vFirewall etc - ipv4 throughput, latency etc + * NSPerf - scenario that handles generic NSB test case execution + (setup and init tg/vnf, trigger traffic on tg, collect kpi) + * NSPerf-RFC2544 - scenario that allows repeatable triggering of traffic on + traffic generators until test case acceptance criteria is met + (for example RFC2544 binary search) - - Traffic generators like Trex, ab/nginx, ixia, iperf etc +* Test case samples - - KPIs for a given use case: + * Ping + * Trex + * vPE, vCGNAT, vFirewall etc - ipv4 throughput, latency etc - - System agent support for collecting NFVi KPI. This includes: +* Traffic generators i.e. Trex, ab/nginx, ixia, iperf, etc +* KPIs for a given use case: - - CPU statistic + * System agent support for collecting NFVi KPI. This includes: - - Memory BW + * CPU statistic + * Memory BW + * OVS-DPDK Stats - - OVS-DPDK Stats - - - Network KPIs, e.g., inpackets, outpackets, thoughput, latency etc - - - VNF KPIs, e.g., packet_in, packet_drop, packet_fwd etc + * Network KPIs e.g. inpackets, outpackets, thoughput, latency + * VNF KPIs e.g. packet_in, packet_drop, packet_fwd Architecture -============ +------------ The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using NFV infrastructure. @@ -83,124 +104,154 @@ performed network functionality. The part of the data model is a set of the configuration parameters, number of connection points used and flavor including core and memory amount. -The ETSI defines a Network Service as a set of configurable VNFs working in -some NFV Infrastructure connecting each other using Virtual Links available -through Connection Points. The ETSI MANO specification defines a set of -management entities called Network Service Descriptors (NSD) and -VNF Descriptors (VNFD) that define real Network Service. The picture below -makes an example how the real Network Operator use-case can map into ETSI -Network service definition - -Network Service framework performs the necessary test steps. It may involve - - - Interacting with traffic generator and providing the inputs on traffic - type / packet structure to generate the required traffic as per the - test case. Traffic profiles will be used for this. - - - Executing the commands required for the test procedure and analyses the - command output for confirming whether the command got executed correctly - or not. E.g. As per the test case, run the traffic for the given - time period / wait for the necessary time delay - - - Verify the test result. - - - Validate the traffic flow from SUT - - - Fetch the table / data from SUT and verify the value as per the test case - - - Upload the logs from SUT onto the Test Harness server - - - Read the KPI's provided by particular VNF +ETSI defines a Network Service as a set of configurable VNFs working in some +NFV Infrastructure connecting each other using Virtual Links available through +Connection Points. The ETSI MANO specification defines a set of management +entities called Network Service Descriptors (NSD) and VNF Descriptors (VNFD) +that define real Network Service. The picture below makes an example how the +real Network Operator use-case can map into ETSI Network service definition. + +Network Service framework performs the necessary test steps. It may involve: + +* Interacting with traffic generator and providing the inputs on traffic + type / packet structure to generate the required traffic as per the + test case. Traffic profiles will be used for this. +* Executing the commands required for the test procedure and analyses the + command output for confirming whether the command got executed correctly + or not e.g. as per the test case, run the traffic for the given + time period and wait for the necessary time delay. +* Verify the test result. +* Validate the traffic flow from SUT. +* Fetch the data from SUT and verify the value as per the test case. +* Upload the logs from SUT onto the Test Harness server +* Retrieve the KPI's provided by particular VNF Components of Network Service ------------------------------ - - * *Models for Network Service benchmarking*: The Network Service benchmarking - requires the proper modelling approach. The NSB provides models using Python - files and defining of NSDs and VNFDs. - - The benchmark control application being a part of OPNFV yardstick can call - that python models to instantiate and configure the VNFs. Depending on - infrastructure type (bare-metal or fully virtualized) that calls could be - made directly or using MANO system. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - * *Traffic generators in NSB*: Any benchmark application requires a set of - traffic generator and traffic profiles defining the method in which traffic - is generated. +.. TODO: provide a list of components in this section and describe them in + later sub-sections - The Network Service benchmarking model extends the Network Service - definition with a set of Traffic Generators (TG) that are treated - same way as other VNFs being a part of benchmarked network service. - Same as other VNFs the traffic generator are instantiated and terminated. +.. Components are the methodology, TGs, framework extensions, KPI collection, + Testcases, SampleVNFs +.. Framework extentions include: VNF models, NSPerf Scenario, contexts - Every traffic generator has own configuration defined as a traffic profile - and a set of KPIs supported. The python models for TG is extended by - specific calls to listen and generate traffic. +* *Models for Network Service benchmarking*: The Network Service benchmarking + requires the proper modelling approach. The NSB provides models using Python + files and defining of NSDs and VNFDs. - * *The stateless TREX traffic generator*: The main traffic generator used as - Network Service stimulus is open source TREX tool. +The benchmark control application being a part of OPNFV Yardstick can call +that Python models to instantiate and configure the VNFs. Depending on +infrastructure type (bare-metal or fully virtualized) that calls could be +made directly or using MANO system. - The TREX tool can generate any kind of stateless traffic. +* *Traffic generators in NSB*: Any benchmark application requires a set of + traffic generator and traffic profiles defining the method in which traffic + is generated. - .. code-block:: console +The Network Service benchmarking model extends the Network Service +definition with a set of Traffic Generators (TG) that are treated +same way as other VNFs being a part of benchmarked network service. +Same as other VNFs the traffic generator are instantiated and terminated. - +--------+ +-------+ +--------+ - | | | | | | - | Trex | ---> | VNF | ---> | Trex | - | | | | | | - +--------+ +-------+ +--------+ +Every traffic generator has own configuration defined as a traffic profile +and a set of KPIs supported. The python models for TG is extended by +specific calls to listen and generate traffic. - Supported testcases scenarios: +* *The stateless TREX traffic generator*: The main traffic generator used as + Network Service stimulus is open source TREX tool. - - Correlated UDP traffic using TREX traffic generator and replay VNF. +The TREX tool can generate any kind of stateless traffic. - - using different IMIX configuration like pure voice, pure video traffic etc - - - using different number IP flows like 1 flow, 1K, 16K, 64K, 256K, 1M flows - - - Using different number of rules configured like 1 rule, 1K, 10K rules - - For UDP correlated traffic following Key Performance Indicators are collected - for every combination of test case parameters: +.. code-block:: console - - RFC2544 throughput for various loss rate defined (1% is a default) + +--------+ +-------+ +--------+ + | | | | | | + | Trex | ---> | VNF | ---> | Trex | + | | | | | | + +--------+ +-------+ +--------+ + +Supported testcases scenarios: + +* Correlated UDP traffic using TREX traffic generator and replay VNF. + + * using different IMIX configuration like pure voice, pure video traffic etc + * using different number IP flows e.g. 1, 1K, 16K, 64K, 256K, 1M flows + * Using different number of rules configured e.g. 1, 1K, 10K rules + +For UDP correlated traffic following Key Performance Indicators are collected +for every combination of test case parameters: + +* RFC2544 throughput for various loss rate defined (1% is a default) + +KPI Collection +^^^^^^^^^^^^^^ + +KPI collection is the process of sampling KPIs at multiple intervals to allow +for investigation into anomalies during runtime. Some KPI intervals are +adjustable. KPIs are collected from traffic generators and NFVI for the SUT. +There is already some reporting in NSB available, but NSB collects all KPIs for +analytics to process. + +Below is an example list of basic KPIs: +* Throughput +* Latency +* Packet delay variation +* Maximum establishment rate +* Maximum tear-down rate +* Maximum simultaneous number of sessions + +Of course, there can be many other KPIs that will be relevant for a specific +NFVI, but in most cases these KPIs are enough to give you a basic picture of +the SUT. NSB also uses :term:`collectd` in order to collect the KPIs. Currently +the following collectd plug-ins are enabled for NSB testcases: + +* Libvirt +* Interface stats +* OvS events +* vSwitch stats +* Huge Pages +* RAM +* CPU usage +* Intel® PMU +* Intel(r) RDT Graphical Overview -================== +------------------ -NSB Testing with yardstick framework facilitate performance testing of various +NSB Testing with Yardstick framework facilitate performance testing of various VNFs provided. .. code-block:: console +-----------+ - | | +-----------+ - | vPE | ->|TGen Port 0| - | TestCase | | +-----------+ - | | | - +-----------+ +------------------+ +-------+ | - | | -- API --> | VNF | <---> - +-----------+ | Yardstick | +-------+ | - | Test Case | --> | NSB Testing | | - +-----------+ | | | - | | | | - | +------------------+ | - +-----------+ | +-----------+ - | Traffic | ->|TGen Port 1| - | patterns | +-----------+ + | | +-------------+ + | vPE | -->| TGen Port 0 | + | TestCase | | +-------------+ + | | | + +-----------+ +---------------+ +-------+ | + | | ---> | VNF | <---> + +-----------+ | Yardstick | +-------+ | + | Test Case | --> | NSB Testing | | + +-----------+ | | | + | | | | + | +---------------+ | + +-----------+ | +-------------+ + | Traffic | -->| TGen Port 1 | + | patterns | +-------------+ +-----------+ Figure 1: Network Service - 2 server configuration -VNFs supported for chracterization: ------------------------------------ +VNFs supported for chracterization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. CGNAPT - Carrier Grade Network Address and port Translation 2. vFW - Virtual Firewall 3. vACL - Access Control List -4. Prox - Packet pROcessing eXecution engine: - - VNF can act as Drop, Basic Forwarding (no touch), - L2 Forwarding (change MAC), GRE encap/decap, Load balance based on - packet fields, Symmetric load balancing - - QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL +4. PROX - Packet pROcessing eXecution engine: + * VNF can act as Drop, Basic Forwarding (no touch), + L2 Forwarding (change MAC), GRE encap/decap, Load balance based on + packet fields, Symmetric load balancing + * QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL 5. UDP_Replay diff --git a/docs/testing/user/userguide/13-nsb-installation.rst b/docs/testing/user/userguide/13-nsb-installation.rst index 973d56628..694521d2b 100644 --- a/docs/testing/user/userguide/13-nsb-installation.rst +++ b/docs/testing/user/userguide/13-nsb-installation.rst @@ -8,40 +8,36 @@ ======= Heading 0 (reserved for the title in a document) ------- Heading 1 - ~~~~~~~ Heading 2 + ^^^^^^^ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. -===================================== -Yardstick - NSB Testing -Installation -===================================== + +================ +NSB Installation +================ + +.. _OVS-DPDK: http://docs.openvswitch.org/en/latest/intro/install/dpdk/ +.. _devstack: https://docs.openstack.org/devstack/pike/> +.. _OVS-DPDK-versions: http://docs.openvswitch.org/en/latest/faq/releases/ Abstract -------- -The Network Service Benchmarking (NSB) extends the yardstick framework to do -VNF characterization and benchmarking in three different execution -environments viz., bare metal i.e. native Linux environment, standalone virtual -environment and managed virtualized environment (e.g. Open stack etc.). -It also brings in the capability to interact with external traffic generators -both hardware & software based for triggering and validating the traffic -according to user defined profiles. - The steps needed to run Yardstick with NSB testing are: * Install Yardstick (NSB Testing). -* Setup/Reference pod.yaml describing Test topology -* Create/Reference the test configuration yaml file. +* Setup/reference ``pod.yaml`` describing Test topology. +* Create/reference the test configuration yaml file. * Run the test case. - Prerequisites ------------- -Refer chapter Yardstick Installation for more information on yardstick -prerequisites +Refer to :doc:`04-installation` for more information on Yardstick +prerequisites. Several prerequisites are needed for Yardstick (VNF testing): @@ -57,11 +53,10 @@ Several prerequisites are needed for Yardstick (VNF testing): * intel-cmt-cat Hardware & Software Ingredients -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SUT requirements: - ======= =================== Item Description ======= =================== @@ -74,7 +69,6 @@ SUT requirements: Boot and BIOS settings: - ============= ================================================= Boot settings default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 @@ -93,29 +87,29 @@ Boot and BIOS settings: Turbo Boost Disabled ============= ================================================= - - Install Yardstick (NSB Testing) ------------------------------- -Download the source code and install Yardstick from it +Yardstick with NSB can be installed using ``nsb_setup.sh``. +The ``nsb_setup.sh`` allows to: -.. code-block:: console +1. Install Yardstick in specified mode: bare metal or container. + Refer :doc:`04-installation`. +2. Install package dependencies on remote servers used as traffic generator or + sample VNF. Install DPDK, sample VNFs, TREX, collectd. + Add such servers to ``install-inventory.ini`` file to either + ``yardstick-standalone`` or ``yardstick-baremetal`` server groups. + It configures IOMMU, hugepages, open file limits, CPU isolation, etc. +3. Build VM image either nsb or normal. The nsb VM image is used to run + Yardstick sample VNF tests, like vFW, vACL, vCGNAPT, etc. + The normal VM image is used to run Yardstick ping tests in OpenStack context. +4. Add nsb or normal VM image to OpenStack together with OpenStack variables. - git clone https://gerrit.opnfv.org/gerrit/yardstick - - cd yardstick - - # Switch to latest stable branch - # git checkout <tag or stable branch> - git checkout stable/euphrates - -Configure the network proxy, either using the environment variables or setting -the global environment file: +Firstly, configure the network proxy, either using the environment variables or +setting the global environment file. -.. code-block:: ini +Set environment:: - cat /etc/environment http_proxy='http://proxy.company.com:port' https_proxy='http://proxy.company.com:port' @@ -124,64 +118,187 @@ the global environment file: export http_proxy='http://proxy.company.com:port' export https_proxy='http://proxy.company.com:port' -The last step is to modify the Yardstick installation inventory, used by -Ansible: +Download the source code and check out the latest stable branch -.. code-block:: ini +.. code-block:: console + + git clone https://gerrit.opnfv.org/gerrit/yardstick + cd yardstick + # Switch to latest stable branch + git checkout stable/gambia + +Modify the Yardstick installation inventory used by Ansible:: cat ./ansible/install-inventory.ini [jumphost] - localhost ansible_connection=local - - [yardstick-standalone] - yardstick-standalone-node ansible_host=192.168.1.2 - yardstick-standalone-node-2 ansible_host=192.168.1.3 + localhost ansible_connection=local # section below is only due backward compatibility. # it will be removed later [yardstick:children] jumphost + [yardstick-baremetal] + baremetal ansible_host=192.168.2.51 ansible_connection=ssh + + [yardstick-standalone] + standalone ansible_host=192.168.2.52 ansible_connection=ssh + [all:vars] - ansible_user=root - ansible_pass=root + # Uncomment credentials below if needed + ansible_user=root + ansible_ssh_pass=root + # ansible_ssh_private_key_file=/root/.ssh/id_rsa + # When IMG_PROPERTY is passed neither normal nor nsb set + # "path_to_vm=/path/to/image" to add it to OpenStack + # path_to_img=/tmp/workspace/yardstick-image.img + + # List of CPUs to be isolated (not used by default) + # Grub line will be extended with: + # "isolcpus=<ISOL_CPUS> nohz=on nohz_full=<ISOL_CPUS> rcu_nocbs=1<ISOL_CPUS>" + # ISOL_CPUS=2-27,30-55 # physical cpu's for all NUMA nodes, four cpu's reserved + +.. warning:: + + Before running ``nsb_setup.sh`` make sure python is installed on servers + added to ``yardstick-standalone`` or ``yardstick-baremetal`` groups. + +.. note:: + + SSH access without password needs to be configured for all your nodes + defined in ``install-inventory.ini`` file. + If you want to use password authentication you need to install ``sshpass``:: + + sudo -EH apt-get install sshpass + .. note:: - SSH access without password needs to be configured for all your nodes defined in + A VM image built by other means than Yardstick can be added to OpenStack. + Uncomment and set correct path to the VM image in the + ``install-inventory.ini`` file:: + + path_to_img=/tmp/workspace/yardstick-image.img + + +.. note:: + + CPU isolation can be applied to the remote servers, like: + ISOL_CPUS=2-27,30-55. Uncomment and modify accordingly in ``install-inventory.ini`` file. - If you want to use password authentication you need to install sshpass - .. code-block:: console +By default ``nsb_setup.sh`` pulls Yardstick image based on Ubuntu 16.04 from +docker hub and starts container, builds NSB VM image based on Ubuntu 16.04, +installs packages to the servers given in ``yardstick-standalone`` and +``yardstick-baremetal`` host groups. - sudo -EH apt-get install sshpass +To pull Yardstick built based on Ubuntu 18 run:: -To execute an installation for a Bare-Metal or a Standalone context: + ./nsb_setup.sh -i opnfv/yardstick-ubuntu-18.04:latest -.. code-block:: console +To change default behavior modify parameters for ``install.yaml`` in +``nsb_setup.sh`` file. - ./nsb_setup.sh +Refer chapter :doc:`04-installation` for more details on ``install.yaml`` +parameters. +To execute an installation for a **BareMetal** or a **Standalone context**:: -To execute an installation for an OpenStack context: + ./nsb_setup.sh -.. code-block:: console +To execute an installation for an **OpenStack** context:: ./nsb_setup.sh <path to admin-openrc.sh> -Above command setup docker with latest yardstick code. To execute +.. note:: + + Yardstick may not be operational after distributive linux kernel update if + it has been installed before. Run ``nsb_setup.sh`` again to resolve this. -.. code-block:: console +.. warning:: + + The Yardstick VM image (NSB or normal) cannot be built inside a VM. + +.. warning:: + + The ``nsb_setup.sh`` configures huge pages, CPU isolation, IOMMU on the grub. + Reboot of the servers from ``yardstick-standalone`` or + ``yardstick-baremetal`` groups in the file ``install-inventory.ini`` is + required to apply those changes. + +The above commands will set up Docker with the latest Yardstick code. To +execute:: docker exec -it yardstick bash +.. note:: + + It may be needed to configure tty in docker container to extend commandline + character length, for example: + + stty size rows 58 cols 234 + It will also automatically download all the packages needed for NSB Testing -setup. Refer chapter :doc:`04-installation` for more on docker +setup. Refer chapter :doc:`04-installation` for more on Docker. + **Install Yardstick using Docker (recommended)** -Another way to execute an installation for a Bare-Metal or a Standalone context -is to use ansible script ``install.yaml``. Refer chapter :doc:`04-installation` -for more details. +Bare Metal context example +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Let's assume there are three servers acting as TG, sample VNF DUT and jump host. + +Perform following steps to install NSB: + +1. Clone Yardstick repo to jump host. +2. Add TG and DUT servers to ``yardstick-baremetal`` group in + ``install-inventory.ini`` file to install NSB and dependencies. Install + python on servers. +3. Start deployment using docker image based on Ubuntu 16: + +.. code-block:: console + + ./nsb_setup.sh + +4. Reboot bare metal servers. +5. Enter to yardstick container and modify pod yaml file and run tests. + +Standalone context example for Ubuntu 18 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Let's assume there are three servers acting as TG, sample VNF DUT and jump host. +Ubuntu 18 is installed on all servers. + +Perform following steps to install NSB: + +1. Clone Yardstick repo to jump host. +2. Add TG server to ``yardstick-baremetal`` group in + ``install-inventory.ini`` file to install NSB and dependencies. + Add server where VM with sample VNF will be deployed to + ``yardstick-standalone`` group in ``install-inventory.ini`` file. + Target VM image named ``yardstick-nsb-image.img`` will be placed to + ``/var/lib/libvirt/images/``. + Install python on servers. +3. Modify ``nsb_setup.sh`` on jump host: + +.. code-block:: console + + ansible-playbook \ + -e IMAGE_PROPERTY='nsb' \ + -e OS_RELEASE='bionic' \ + -e INSTALLATION_MODE='container_pull' \ + -e YARD_IMAGE_ARCH='amd64' ${extra_args} \ + -i install-inventory.ini install.yaml + +4. Start deployment with Yardstick docker images based on Ubuntu 18: + +.. code-block:: console + + ./nsb_setup.sh -i opnfv/yardstick-ubuntu-18.04:latest -o <openrc_file> + +5. Reboot servers. +6. Enter to yardstick container and modify pod yaml file and run tests. + System Topology --------------- @@ -195,30 +312,28 @@ System Topology | | | | | | (1)<-----(1) | | +----------+ +----------+ - trafficgen_1 vnf + trafficgen_0 vnf Environment parameters and credentials -------------------------------------- -Config yardstick conf -~~~~~~~~~~~~~~~~~~~~~ +Configure yardstick.conf +^^^^^^^^^^^^^^^^^^^^^^^^ -If user did not run 'yardstick env influxdb' inside the container, which will -generate correct ``yardstick.conf``, then create the config file manually (run -inside the container): -:: +If you did not run ``yardstick env influxdb`` inside the container to generate +``yardstick.conf``, then create the config file manually (run inside the +container):: cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf vi /etc/yardstick/yardstick.conf -Add trex_path, trex_client_lib and bin_path in 'nsb' section. - -:: +Add ``trex_path``, ``trex_client_lib`` and ``bin_path`` to the ``nsb`` +section:: [DEFAULT] debug = True - dispatcher = file, influxdb + dispatcher = influxdb [dispatcher_influxdb] timeout = 5 @@ -235,25 +350,32 @@ Add trex_path, trex_client_lib and bin_path in 'nsb' section. Run Yardstick - Network Service Testcases ----------------------------------------- - NS testing - using yardstick CLI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See :doc:`04-installation` -.. code-block:: console - +Connect to the Yardstick container:: docker exec -it yardstick /bin/bash - source /etc/yardstick/openstack.creds (only for heat TC if nsb_setup.sh was NOT used) - export EXTERNAL_NETWORK="<openstack public network>" (only for heat TC) + +If you're running ``heat`` testcases and ``nsb_setup.sh`` was not used:: + source /etc/yardstick/openstack.creds + +In addition to the above, you need to se the ``EXTERNAL_NETWORK`` for +OpenStack:: + + export EXTERNAL_NETWORK="<openstack public network>" + +Finally, you should be able to run the testcase:: + yardstick --debug task start yardstick/samples/vnf_samples/nsut/<vnf>/<test case> Network Service Benchmarking - Bare-Metal ----------------------------------------- Bare-Metal Config pod.yaml describing Topology -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Bare-Metal 2-Node setup +++++++++++++++++++++++ @@ -266,7 +388,7 @@ Bare-Metal 2-Node setup | | | | | | (n)<-----(n) | | +----------+ +----------+ - trafficgen_1 vnf + trafficgen_0 vnf Bare-Metal 3-Node setup - Correlated Traffic ++++++++++++++++++++++++++++++++++++++++++++ @@ -280,12 +402,12 @@ Bare-Metal 3-Node setup - Correlated Traffic | | | | | | | | | |(1)<---->(0)| | +----------+ +----------+ +------------+ - trafficgen_1 vnf trafficgen_2 + trafficgen_0 vnf trafficgen_1 Bare-Metal Config pod.yaml -~~~~~~~~~~~~~~~~~~~~~~~~~~ -Before executing Yardstick test cases, make sure that pod.yaml reflects the +^^^^^^^^^^^^^^^^^^^^^^^^^^ +Before executing Yardstick test cases, make sure that ``pod.yaml`` reflects the topology and update all the required fields.:: cp /etc/yardstick/nodes/pod.yaml.nsb.sample /etc/yardstick/nodes/pod.yaml @@ -294,7 +416,7 @@ topology and update all the required fields.:: nodes: - - name: trafficgen_1 + name: trafficgen_0 role: TrafficGen ip: 1.1.1.1 user: root @@ -358,22 +480,20 @@ topology and update all the required fields.:: if: "xe1" -Network Service Benchmarking - Standalone Virtualization --------------------------------------------------------- +Standalone Virtualization +------------------------- SR-IOV -~~~~~~ +^^^^^^ SR-IOV Pre-requisites +++++++++++++++++++++ On Host, where VM is created: - a) Create and configure a bridge named ``br-int`` for VM to connect to external network. - Currently this can be done using VXLAN tunnel. - - Execute the following on host, where VM is created: + a) Create and configure a bridge named ``br-int`` for VM to connect to + external network. Currently this can be done using VXLAN tunnel. - .. code-block:: console + Execute the following on host, where VM is created:: ip link add type vxlan remote <Jumphost IP> local <DUT IP> id <ID: 10> dstport 4789 brctl addbr br-int @@ -382,7 +502,7 @@ On Host, where VM is created: ip addr add <IP#1, like: 172.20.2.1/24> dev br-int ip link set dev br-int up - .. note:: May be needed to add extra rules to iptable to forward traffic. + .. note:: You may need to add extra rules to iptable to forward traffic. .. code-block:: console @@ -405,7 +525,7 @@ On Host, where VM is created: .. code-block:: YAML servers: - vnf: + vnf_0: network_ports: mgmt: cidr: '1.1.1.7/24' @@ -416,23 +536,24 @@ On Host, where VM is created: Yardstick has a tool for building this custom image with SampleVNF. It is necessary to have ``sudo`` rights to use this tool. - Also you may need to install several additional packages to use this tool, by - following the commands below:: - - sudo apt-get update && sudo apt-get install -y qemu-utils kpartx + Also you may need to install several additional packages to use this tool, by + following the commands below:: - This image can be built using the following command in the directory where Yardstick is installed + sudo apt-get update && sudo apt-get install -y qemu-utils kpartx - .. code-block:: console + This image can be built using the following command in the directory where + Yardstick is installed:: - export YARD_IMG_ARCH='amd64' - sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers + export YARD_IMG_ARCH='amd64' + sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers - Please use ansible script to generate a cloud image refer to :doc:`04-installation` + For instructions on generating a cloud image using Ansible, refer to + :doc:`04-installation`. - for more details refer to chapter :doc:`04-installation` + for more details refer to chapter :doc:`04-installation` - .. note:: VM should be build with static IP and should be accessible from yardstick host. + .. note:: VM should be build with static IP and be accessible from the + Yardstick host. SR-IOV Config pod.yaml describing Topology @@ -457,12 +578,12 @@ SR-IOV 2-Node setup +----------+ +-------------------------+ | | | ^ ^ | | | | | | | - | | (0)<----->(0) | ------ | | - | TG1 | | SUT | | - | | | | | - | | (n)<----->(n) |------------------ | + | | (0)<----->(0) | ------ SUT | | + | TG1 | | | | + | | (n)<----->(n) | ----------------- | + | | | | +----------+ +-------------------------+ - trafficgen_1 host + trafficgen_0 host @@ -470,29 +591,29 @@ SR-IOV 3-Node setup - Correlated Traffic ++++++++++++++++++++++++++++++++++++++++ .. code-block:: console - +--------------------+ - | | - | | - | DUT | - | (VNF) | - | | - +--------------------+ - | VF NIC | | VF NIC | - +--------+ +--------+ - ^ ^ - | | - | | - +----------+ +-------------------------+ +--------------+ - | | | ^ ^ | | | - | | | | | | | | - | | (0)<----->(0) | ------ | | | TG2 | - | TG1 | | SUT | | | (UDP Replay) | - | | | | | | | - | | (n)<----->(n) | ------ | (n)<-->(n) | | - +----------+ +-------------------------+ +--------------+ - trafficgen_1 host trafficgen_2 - -Before executing Yardstick test cases, make sure that pod.yaml reflects the + +--------------------+ + | | + | | + | DUT | + | (VNF) | + | | + +--------------------+ + | VF NIC | | VF NIC | + +--------+ +--------+ + ^ ^ + | | + | | + +----------+ +---------------------+ +--------------+ + | | | ^ ^ | | | + | | | | | | | | + | | (0)<----->(0) |----- | | | TG2 | + | TG1 | | SUT | | | (UDP Replay) | + | | | | | | | + | | (n)<----->(n) | -----| (n)<-->(n) | | + +----------+ +---------------------+ +--------------+ + trafficgen_0 host trafficgen_1 + +Before executing Yardstick test cases, make sure that ``pod.yaml`` reflects the topology and update all the required fields. .. code-block:: console @@ -509,7 +630,7 @@ SR-IOV Config pod_trex.yaml nodes: - - name: trafficgen_1 + name: trafficgen_0 role: TrafficGen ip: 1.1.1.1 user: root @@ -547,8 +668,8 @@ SR-IOV Config host_sriov.yaml SR-IOV testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -570,7 +691,7 @@ Update "contexts" section user: "" # update VM username password: "" # update password servers: - vnf: + vnf_0: network_ports: mgmt: cidr: '1.1.1.61/24' # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask> @@ -591,16 +712,15 @@ Update "contexts" section gateway_ip: '152.16.100.20' - OVS-DPDK -~~~~~~~~ +^^^^^^^^ OVS-DPDK Pre-requisites -~~~~~~~~~~~~~~~~~~~~~~~ ++++++++++++++++++++++++ On Host, where VM is created: - a) Create and configure a bridge named ``br-int`` for VM to connect to external network. - Currently this can be done using VXLAN tunnel. + a) Create and configure a bridge named ``br-int`` for VM to connect to + external network. Currently this can be done using VXLAN tunnel. Execute the following on host, where VM is created: @@ -636,7 +756,7 @@ On Host, where VM is created: .. code-block:: YAML servers: - vnf: + vnf_0: network_ports: mgmt: cidr: '1.1.1.7/24' @@ -647,26 +767,27 @@ On Host, where VM is created: Yardstick has a tool for building this custom image with SampleVNF. It is necessary to have ``sudo`` rights to use this tool. - Also you may need to install several additional packages to use this tool, by - following the commands below:: + You may need to install several additional packages to use this tool, by + following the commands below:: - sudo apt-get update && sudo apt-get install -y qemu-utils kpartx + sudo apt-get update && sudo apt-get install -y qemu-utils kpartx - This image can be built using the following command in the directory where Yardstick is installed:: + This image can be built using the following command in the directory where + Yardstick is installed:: - export YARD_IMG_ARCH='amd64' - sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers - sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh + export YARD_IMG_ARCH='amd64' + sudo echo "Defaults env_keep += \'YARD_IMG_ARCH\'" >> /etc/sudoers + sudo tools/yardstick-img-dpdk-modify tools/ubuntu-server-cloudimg-samplevnf-modify.sh - for more details refer to chapter :doc:`04-installation` + for more details refer to chapter :doc:`04-installation` - .. note:: VM should be build with static IP and should be accessible from yardstick host. + .. note:: VM should be build with static IP and should be accessible from + yardstick host. - c) OVS & DPDK version. - - OVS 2.7 and DPDK 16.11.1 above version is supported +3. OVS & DPDK version. + * OVS 2.7 and DPDK 16.11.1 above version is supported - d) Setup OVS/DPDK on host. - Please refer to below link on how to setup `OVS-DPDK <http://docs.openvswitch.org/en/latest/intro/install/dpdk/>`_ +4. Setup `OVS-DPDK`_ on host. OVS-DPDK Config pod.yaml describing Topology @@ -699,7 +820,7 @@ OVS-DPDK 2-Node setup | | | (ovs-dpdk) | | | | (n)<----->(n) |------------------ | +----------+ +-------------------------+ - trafficgen_1 host + trafficgen_0 host OVS-DPDK 3-Node setup - Correlated Traffic @@ -729,13 +850,11 @@ OVS-DPDK 3-Node setup - Correlated Traffic | | | (ovs-dpdk) | | | | | | (n)<----->(n) | ------ |(n)<-->(n)| | +----------+ +-------------------------+ +------------+ - trafficgen_1 host trafficgen_2 + trafficgen_0 host trafficgen_1 -Before executing Yardstick test cases, make sure that pod.yaml reflects the -topology and update all the required fields. - -.. code-block:: console +Before executing Yardstick test cases, make sure that the ``pod.yaml`` reflects +the topology and update all the required fields:: cp <yardstick>/etc/yardstick/nodes/standalone/trex_bm.yaml.sample /etc/yardstick/nodes/standalone/pod_trex.yaml cp <yardstick>/etc/yardstick/nodes/standalone/host_ovs.yaml /etc/yardstick/nodes/standalone/host_ovs.yaml @@ -749,7 +868,7 @@ OVS-DPDK Config pod_trex.yaml nodes: - - name: trafficgen_1 + name: trafficgen_0 role: TrafficGen ip: 1.1.1.1 user: root @@ -786,8 +905,8 @@ OVS-DPDK Config host_ovs.yaml ovs_dpdk testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` -Update "contexts" section -''''''''''''''''''''''''' +Update contexts section +''''''''''''''''''''''' .. code-block:: YAML @@ -820,7 +939,7 @@ Update "contexts" section user: "" # update VM username password: "" # update password servers: - vnf: + vnf_0: network_ports: mgmt: cidr: '1.1.1.61/24' # Update VM IP address, if static, <ip>/<mask> or if dynamic, <start of ip>/<mask> @@ -840,17 +959,155 @@ Update "contexts" section cidr: '152.16.40.10/24' gateway_ip: '152.16.100.20' +OVS-DPDK configuration options +++++++++++++++++++++++++++++++ + +There are number of configuration options available for OVS-DPDK context in +test case. Mostly they are used for performance tuning. + +OVS-DPDK properties: +'''''''''''''''''''' + +OVS-DPDK properties example under *ovs_properties* section: + + .. code-block:: console + + ovs_properties: + version: + ovs: 2.8.1 + dpdk: 17.05.2 + pmd_threads: 4 + pmd_cpu_mask: "0x3c" + ram: + socket_0: 2048 + socket_1: 2048 + queues: 2 + vpath: "/usr/local" + max_idle: 30000 + lcore_mask: 0x02 + dpdk_pmd-rxq-affinity: + 0: "0:2,1:2" + 1: "0:2,1:2" + 2: "0:3,1:3" + 3: "0:3,1:3" + vhost_pmd-rxq-affinity: + 0: "0:3,1:3" + 1: "0:3,1:3" + 2: "0:4,1:4" + 3: "0:4,1:4" + +OVS-DPDK properties description: + + +-------------------------+-------------------------------------------------+ + | Parameters | Detail | + +=========================+=================================================+ + | version || Version of OVS and DPDK to be installed | + | || There is a relation between OVS and DPDK | + | | version which can be found at | + | | `OVS-DPDK-versions`_ | + | || By default OVS: 2.6.0, DPDK: 16.07.2 | + +-------------------------+-------------------------------------------------+ + | lcore_mask || Core bitmask used during DPDK initialization | + | | where the non-datapath OVS-DPDK threads such | + | | as handler and revalidator threads run | + +-------------------------+-------------------------------------------------+ + | pmd_cpu_mask || Core bitmask that sets which cores are used by | + | || OVS-DPDK for datapath packet processing | + +-------------------------+-------------------------------------------------+ + | pmd_threads || Number of PMD threads used by OVS-DPDK for | + | | datapath | + | || This core mask is evaluated in Yardstick | + | || It will be used if pmd_cpu_mask is not given | + | || Default is 2 | + +-------------------------+-------------------------------------------------+ + | ram || Amount of RAM to be used for each socket, MB | + | || Default is 2048 MB | + +-------------------------+-------------------------------------------------+ + | queues || Number of RX queues used for DPDK physical | + | | interface | + +-------------------------+-------------------------------------------------+ + | dpdk_pmd-rxq-affinity || RX queue assignment to PMD threads for DPDK | + | || e.g.: <port number> : <queue-id>:<core-id> | + +-------------------------+-------------------------------------------------+ + | vhost_pmd-rxq-affinity || RX queue assignment to PMD threads for vhost | + | || e.g.: <port number> : <queue-id>:<core-id> | + +-------------------------+-------------------------------------------------+ + | vpath || User path for openvswitch files | + | || Default is ``/usr/local`` | + +-------------------------+-------------------------------------------------+ + | max_idle || The maximum time that idle flows will remain | + | | cached in the datapath, ms | + +-------------------------+-------------------------------------------------+ + + +VM image properties +''''''''''''''''''' + +VM image properties example under *flavor* section: + + .. code-block:: console -Network Service Benchmarking - OpenStack with SR-IOV support ------------------------------------------------------------- + flavor: + images: <path> + ram: 8192 + extra_specs: + machine_type: 'pc-i440fx-xenial' + hw:cpu_sockets: 1 + hw:cpu_cores: 6 + hw:cpu_threads: 2 + hw_socket: 0 + cputune: | + <cputune> + <vcpupin vcpu="0" cpuset="7"/> + <vcpupin vcpu="1" cpuset="8"/> + ... + <vcpupin vcpu="11" cpuset="18"/> + <emulatorpin cpuset="11"/> + </cputune> + +VM image properties description: + + +-------------------------+-------------------------------------------------+ + | Parameters | Detail | + +=========================+=================================================+ + | images || Path to the VM image generated by | + | | ``nsb_setup.sh`` | + | || Default path is ``/var/lib/libvirt/images/`` | + | || Default file name ``yardstick-nsb-image.img`` | + | | or ``yardstick-image.img`` | + +-------------------------+-------------------------------------------------+ + | ram || Amount of RAM to be used for VM | + | || Default is 4096 MB | + +-------------------------+-------------------------------------------------+ + | hw:cpu_sockets || Number of sockets provided to the guest VM | + | || Default is 1 | + +-------------------------+-------------------------------------------------+ + | hw:cpu_cores || Number of cores provided to the guest VM | + | || Default is 2 | + +-------------------------+-------------------------------------------------+ + | hw:cpu_threads || Number of threads provided to the guest VM | + | || Default is 2 | + +-------------------------+-------------------------------------------------+ + | hw_socket || Generate vcpu cpuset from given HW socket | + | || Default is 0 | + +-------------------------+-------------------------------------------------+ + | cputune || Maps virtual cpu with logical cpu | + +-------------------------+-------------------------------------------------+ + | machine_type || Machine type to be emulated in VM | + | || Default is 'pc-i440fx-xenial' | + +-------------------------+-------------------------------------------------+ + + +OpenStack with SR-IOV support +----------------------------- This section describes how to run a Sample VNF test case, using Heat context, with SR-IOV. It also covers how to install OpenStack in Ubuntu 16.04, using DevStack, with SR-IOV support. -Single node OpenStack setup with external TG -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Single node OpenStack with external TG +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -877,32 +1134,28 @@ Single node OpenStack setup with external TG | | (PF1)<----->(PF1) +--------------------+ | | | | | +----------+ +----------------------------+ - trafficgen_1 host + trafficgen_0 host Host pre-configuration ++++++++++++++++++++++ -.. warning:: The following configuration requires sudo access to the system. Make - sure that your user have the access. +.. warning:: The following configuration requires sudo access to the system. + Make sure that your user have the access. -Enable the Intel VT-d or AMD-Vi extension in the BIOS. Some system manufacturers -disable this extension by default. +Enable the Intel VT-d or AMD-Vi extension in the BIOS. Some system +manufacturers disable this extension by default. Activate the Intel VT-d or AMD-Vi extension in the kernel by modifying the GRUB config file ``/etc/default/grub``. -For the Intel platform: - -.. code:: bash +For the Intel platform:: ... GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on" ... -For the AMD platform: - -.. code:: bash +For the AMD platform:: ... GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on" @@ -917,9 +1170,7 @@ Update the grub configuration file and restart the system: sudo update-grub sudo reboot -Make sure the extension has been enabled: - -.. code:: bash +Make sure the extension has been enabled:: sudo journalctl -b 0 | grep -e IOMMU -e DMAR @@ -932,6 +1183,8 @@ Make sure the extension has been enabled: Feb 06 14:50:14 hostname kernel: DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 +.. TODO: Refer to the yardstick installation guide for proxy set up + Setup system proxy (if needed). Add the following configuration into the ``/etc/environment`` file: @@ -954,13 +1207,11 @@ Upgrade the system: sudo -EH apt-get upgrade sudo -EH apt-get dist-upgrade -Install dependencies needed for the DevStack +Install dependencies needed for DevStack .. code:: bash - sudo -EH apt-get install python - sudo -EH apt-get install python-dev - sudo -EH apt-get install python-pip + sudo -EH apt-get install python python-dev python-pip Setup SR-IOV ports on the host: @@ -983,10 +1234,10 @@ Setup SR-IOV ports on the host: DevStack installation +++++++++++++++++++++ -Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ -documentation to install OpenStack on a host. Please note, that stable -``pike`` branch of devstack repo should be used during the installation. -The required `local.conf`` configuration file are described below. +If you want to try out NSB, but don't have OpenStack set-up, you can use +`Devstack`_ to install OpenStack on a host. Please note, that the +``stable/pike`` branch of devstack repo should be used during the installation. +The required ``local.conf`` configuration file are described below. DevStack configuration file: @@ -1001,15 +1252,13 @@ DevStack configuration file: Start the devstack installation on a host. - TG host configuration +++++++++++++++++++++ -Yardstick automatically install and configure Trex traffic generator on TG +Yardstick automatically installs and configures Trex traffic generator on TG host based on provided POD file (see below). Anyway, it's recommended to check -the compatibility of the installed NIC on the TG server with software Trex using -the manual at https://trex-tgn.cisco.com/trex/doc/trex_manual.html. - +the compatibility of the installed NIC on the TG server with software Trex +using the `manual <https://trex-tgn.cisco.com/trex/doc/trex_manual.html>`_. Run the Sample VNF test case ++++++++++++++++++++++++++++ @@ -1018,7 +1267,7 @@ There is an example of Sample VNF test case ready to be executed in an OpenStack environment with SR-IOV support: ``samples/vnf_samples/nsut/vfw/ tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``. -Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +Install Yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack context. Create pod file for TG in the yardstick repo folder located in the yardstick @@ -1037,7 +1286,7 @@ context using steps described in `NS testing - using yardstick CLI`_ section. Multi node OpenStack TG and VNF setup (two nodes) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -1048,7 +1297,7 @@ Multi node OpenStack TG and VNF setup (two nodes) | |sample-VNF VM | | | |sample-VNF VM | | | | | | | | | | | | TG | | | | DUT | | - | | trafficgen_1 | | | | (VNF) | | + | | trafficgen_0 | | | | (VNF) | | | | | | | | | | | +--------+ +--------+ | | +--------+ +--------+ | | | VF NIC | | VF NIC | | | | VF NIC | | VF NIC | | @@ -1071,16 +1320,14 @@ Controller/Compute pre-configuration ++++++++++++++++++++++++++++++++++++ Pre-configuration of the controller and compute hosts are the same as -described in `Host pre-configuration`_ section. Follow the steps in the section. - +described in `Host pre-configuration`_ section. DevStack configuration ++++++++++++++++++++++ -Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ -documentation to install OpenStack on a host. Please note, that stable -``pike`` branch of devstack repo should be used during the installation. -The required `local.conf`` configuration file are described below. +A reference ``local.conf`` for deploying OpenStack in a multi-host environment +using `Devstack`_ is shown in this section. The ``stable/pike`` branch of +devstack repo should be used during the installation. .. note:: Update the devstack configuration files by replacing angluar brackets with a short description inside. @@ -1100,17 +1347,17 @@ DevStack configuration file for compute host: Start the devstack installation on the controller and compute hosts. - Run the sample vFW TC +++++++++++++++++++++ -Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +Install Yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack context. -Run sample vFW RFC2544 SR-IOV TC (``samples/vnf_samples/nsut/vfw/ -tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) in the heat -context using steps described in `NS testing - using yardstick CLI`_ section -and the following yardtick command line arguments: +Run the sample vFW RFC2544 SR-IOV test case +(``samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) +in the heat context using steps described in +`NS testing - using yardstick CLI`_ section and the following Yardstick command +line arguments: .. code:: bash @@ -1118,11 +1365,11 @@ and the following yardtick command line arguments: samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml -Enabling other Traffic generator --------------------------------- +Enabling other Traffic generators +--------------------------------- IxLoad -~~~~~~ +^^^^^^ 1. Software needed: IxLoadAPI ``<IxLoadTclApi verson>Linux64.bin.tgz`` and ``<IxOS version>Linux64.bin.tar.gz`` (Download from ixia support site) @@ -1138,14 +1385,16 @@ IxLoad .. code-block:: console - cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml + cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia \ + etc/yardstick/nodes/pod_ixia.yaml Config ``pod_ixia.yaml`` .. literalinclude:: code/pod_ixia.yaml :language: console - for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration + for sriov/ovs_dpdk pod files, please refer to `Standalone Virtualization`_ + for ovs-dpdk/sriov configuration 3. Start IxOS TCL Server (Install 'Ixia IxExplorer IxOS <version>') You will also need to configure the IxLoad machine to start the IXIA @@ -1155,7 +1404,7 @@ IxLoad * Go to: ``Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1`` or - ``"C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"`` + ``C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe`` 4. Create a folder ``Results`` in c:\ and share the folder on the network. @@ -1163,7 +1412,7 @@ IxLoad ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml`` IxNetwork -~~~~~~~~~ +^^^^^^^^^ IxNetwork testcases use IxNetwork API Python Bindings module, which is installed as part of the requirements of the project. @@ -1172,14 +1421,16 @@ installed as part of the requirements of the project. .. code-block:: console - cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml + cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia \ + etc/yardstick/nodes/pod_ixia.yaml - Config pod_ixia.yaml + Configure ``pod_ixia.yaml`` .. literalinclude:: code/pod_ixia.yaml :language: console - for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration + for sriov/ovs_dpdk pod files, please refer to above + `Standalone Virtualization`_ for ovs-dpdk/sriov configuration 2. Start IxNetwork TCL Server You will also need to configure the IxNetwork machine to start the IXIA @@ -1221,9 +1472,9 @@ to be preinstalled and properly configured. ``PYTHONPATH`` environment variable. .. important:: - The current version of LsApi module has an issue with reading LD_LIBRARY_PATH. - For LsApi module to initialize correctly following lines (184-186) in - lsapi.py + The current version of LsApi module has an issue with reading LD_LIBRARY_PATH. + For LsApi module to initialize correctly following lines (184-186) in + lsapi.py .. code-block:: python diff --git a/docs/testing/user/userguide/14-nsb-operation.rst b/docs/testing/user/userguide/14-nsb-operation.rst index 72b1c4bbc..69ffb8a3b 100644 --- a/docs/testing/user/userguide/14-nsb-operation.rst +++ b/docs/testing/user/userguide/14-nsb-operation.rst @@ -2,6 +2,16 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2018 Intel Corporation. +.. + Convention for heading levels in Yardstick documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. Yardstick - NSB Testing - Operation =================================== @@ -89,9 +99,9 @@ Availability zone ^^^^^^^^^^^^^^^^^ The configuration of the availability zone is requred in cases where location -of exact compute host/group of compute hosts needs to be specified for SampleVNF -or traffic generator in the heat test case. If this is the case, please follow -the instructions below. +of exact compute host/group of compute hosts needs to be specified for +:term:`SampleVNF` or traffic generator in the heat test case. If this is the +case, please follow the instructions below. .. _`Create a host aggregate`: @@ -105,7 +115,8 @@ the instructions below. .. code-block:: bash # create host aggregate - openstack aggregate create --zone <AZ_NAME> --property availability_zone=<AZ_NAME> <AGG_NAME> + openstack aggregate create --zone <AZ_NAME> \ + --property availability_zone=<AZ_NAME> <AGG_NAME> # show available hosts openstack compute service list --service nova-compute # add selected host into the host aggregate @@ -125,7 +136,7 @@ the instructions below. image: yardstick-samplevnfs ... servers: - vnf__0: + vnf_0: ... availability_zone: <AZ_NAME> ... @@ -136,8 +147,9 @@ the instructions below. networks: ... -There are two example of SampleVNF scale out test case which use the availability zone -feature to specify the exact location of scaled VNFs and traffic generators. +There are two example of SampleVNF scale out test case which use the +``availability zone`` feature to specify the exact location of scaled VNFs and +traffic generators. Those are: @@ -164,21 +176,19 @@ Those are: | 5 | agg1 | AZ_NAME_1 | +----+------+-------------------+ -2. If no host aggregates are configured, please use `steps above`__ to - configure them. +2. If no host aggregates are configured, please follow the instructions to + `Create a host aggregate`_ -__ `Create a host aggregate`_ - -3. Run the SampleVNF PROX scale-out test case, specifying the availability - zone of each VNF and traffic generator as a task arguments. +3. Run the SampleVNF PROX scale-out test case, specifying the + ``availability zone`` of each VNF and traffic generator as task arguments. .. note:: The ``az_0`` and ``az_1`` should be changed according to the host - aggregates created in the OpenStack. + aggregates created in the OpenStack. .. code-block:: console - yardstick -d task start\ + yardstick -d task start \ <repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml\ --task-args='{ "num_vnfs": 4, "availability_zone": { @@ -198,7 +208,7 @@ Collectd KPIs ------------- NSB can collect KPIs from collected. We have support for various plugins -enabled by the Barometer project. +enabled by the :term:`Barometer` project. The default yardstick-samplevnf has collectd installed. This allows for collecting KPIs from the VNF. @@ -208,12 +218,11 @@ We assume that collectd is not installed on the compute nodes. To collectd KPIs from the NFVi compute nodes: - * install_collectd on the compute nodes * create pod.yaml for the compute nodes * enable specific plugins depending on the vswitch and DPDK - example pod.yaml section for Compute node running collectd. + example ``pod.yaml`` section for Compute node running collectd. .. code-block:: yaml @@ -323,8 +332,8 @@ Baremetal traffic_profile: ../../traffic_profiles/ipv4_throughput.yaml topology: vfw-tg-topology.yaml nodes: - tg__0: trafficgen_1.yardstick - vnf__0: vnf.yardstick + tg__0: trafficgen_0.yardstick + vnf__0: vnf_0.yardstick options: framesize: uplink: {64B: 100} @@ -356,8 +365,8 @@ Scale-Out VNFs performance data with scale-out helps - * in capacity planning to meet the given network node requirements - * in comparison between different VNF vendor offerings + * capacity planning to meet the given network node requirements + * comparison between different VNF vendor offerings * better the scale-out index, provides the flexibility in meeting future capacity requirements @@ -418,7 +427,7 @@ options section. scenarios: - type: NSPerf nodes: - tg__0: tg_0.yardstick + tg__0: trafficgen_0.yardstick options: tg_0: @@ -488,11 +497,11 @@ Default values for OVS-DPDK: Sample test case file ^^^^^^^^^^^^^^^^^^^^^ - 1. Prepare SampleVNF image and copy it to ``flavor/images``. - 2. Prepare context files for TREX and SampleVNF under ``contexts/file``. - 3. Add bridge named ``br-int`` to the baremetal where SampleVNF image is deployed. - 4. Modify ``networks/phy_port`` accordingly to the baremetal setup. - 5. Run test from: +1. Prepare SampleVNF image and copy it to ``flavor/images``. +2. Prepare context files for TREX and SampleVNF under ``contexts/file``. +3. Add bridge named ``br-int`` to the baremetal where SampleVNF image is deployed. +4. Modify ``networks/phy_port`` accordingly to the baremetal setup. +5. Run test from: .. literalinclude:: /../samples/vnf_samples/nsut/acl/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml :language: yaml @@ -631,3 +640,37 @@ A testcase can be started with the following command as an example: .. code-block:: bash yardstick task start /yardstick/samples/vnf_samples/nsut/vpe/tc_baremetal_rfc2544_ipv4_1flow_64B_ixia.yaml + +Preparing test run of vIPSEC test case +------------------------------------ + +Location of vIPSEC test cases: ``samples/vnf_samples/nsut/ipsec/``. + +Before running a specific vIPSEC test case using NSB, some dependencies have to be +preinstalled and properly configured. +- VPP + +.. code-block:: console + + export UBUNTU="xenial" + export RELEASE=".stable.1810" + sudo rm /etc/apt/sources.list.d/99fd.io.list + echo "deb [trusted=yes] https://nexus.fd.io/content/repositories/fd.io$RELEASE.ubuntu.$UBUNTU.main/ ./" | sudo tee -a /etc/apt/sources.list.d/99fd.io.list + sudo apt-get update + sudo apt-get install vpp vpp-lib vpp-plugin vpp-dbg vpp-dev vpp-api-java vpp-api-python vpp-api-lua + +- VAT templates + + VAT templates is required for the VPP API. + +.. code-block:: console + + mkdir -p /opt/nsb_bin/vpp/templates/ + echo 'exec trace add dpdk-input 50' > /opt/nsb_bin/vpp/templates/enable_dpdk_traces.vat + echo 'exec trace add vhost-user-input 50' > /opt/nsb_bin/vpp/templates/enable_vhost_user_traces.vat + echo 'exec trace add memif-input 50' > /opt/nsb_bin/vpp/templates/enable_memif_traces.vat + cat > /opt/nsb_bin/vpp/templates/dump_interfaces.vat << EOL + sw_interface_dump + dump_interface_table + quit + EOL diff --git a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst index 6c18c7d89..1a4bf32b5 100644 --- a/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst +++ b/docs/testing/user/userguide/nsb/nsb-list-of-tcs.rst @@ -36,3 +36,5 @@ NSB PROX Test Case Descriptions tc_vfw_rfc2544 tc_vfw_rfc2544_correlated tc_vfw_rfc3511 + tc_vpp_baremetal_crypto_ipsec + tc_vims_context_sipp diff --git a/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia.rst b/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia.rst new file mode 100644 index 000000000..ffe4f6c19 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia.rst @@ -0,0 +1,177 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2019 Intel Corporation. + +*************************************************************** +Yardstick Test Case Description: NSB vBNG RFC2544 QoS TEST CASE +*************************************************************** + ++-----------------------------------------------------------------------------+ +|NSB vBNG RFC2544 QoS base line test case without link congestion | +| | ++--------------+--------------------------------------------------------------+ +|test case id | tc_bng_pppoe_rfc2544_ixia_IMIX_scale_up | +| | | ++--------------+--------------------------------------------------------------+ +| metric | Network metrics: | +| | * TxThroughput | +| | * RxThroughput | +| | * TG packets in | +| | * TG packets out | +| | * Max Latency | +| | * Min Latency | +| | * Average Latency | +| | * Packets drop percentage | +| | | +| | PPPoE subscribers metrics: | +| | * Sessions up | +| | * Sessions down | +| | * Sessions Not Started | +| | * Sessions Total | +| | | +| | NOTE: the same network metrics list are collecting: | +| | * summary for all ports | +| | * per port | +| | * per priority flows summary on all ports | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test allows to measure performance of BNG network device| +| | according to RFC2544 testing methodology. Test case creates | +| | PPPoE subscriber connections to BNG, runs prioritized traffic| +| | on maximum throughput on all ports and collects network | +| | and PPPoE subscriber metrics. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | The BNG QoS RFC2544 test cases are listed below: | +| | | +| | * tc_bng_pppoe_rfc2544_ixia_IMIX_scale_up.yaml | +| | | +| | Mentioned test case is a template and number of ports in the | +| | setup could be passed using cli arguments, e.g: | +| | | +| | yardstick -d task start --task-args='{vports: 8}' <tc_yaml> | +| | | +| | By default, vports=2. | +| | | +| | Test duration: | +| | * set as 30sec; | +| | | +| | Traffic type: | +| | * IPv4; | +| | | +| | Packet sizes: | +| | * IMIX. The following default IMIX distribution is using: | +| | | +| | uplink: 70B - 33%, 940B - 33%, 1470B - 34% | +| | downlink: 68B - 3%, 932B - 1%, 1470B - 96% | +| | | +| | VLAN settings: | +| | * QinQ on access ports; | +| | * VLAN on core ports; | +| | | +| | Number of PPPoE subscribers: | +| | * 4000 per access port; | +| | * 1000 per SVLAN; | +| | | +| | Default ToS bits settings: | +| | * 0 - (000) Routine | +| | * 4 - (100) Flash Override | +| | * 7 - (111) Network Control. | +| | | +| | The above fields are the main options used for the test case | +| | and could be configured using cli options on test run or | +| | directly in test case yaml file. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | IXIA IxNetwork | +| | | +| | IXIA IxNetwork is using to emulates PPPoE sessions, generate | +| | L2-L3 traffic, analyze traffic flows and collect network | +| | metrics during test run. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Mentioned BNG QoS RFC2544 test case can be configured with | +| | different: | +| | | +| | * Number of PPPoE subscribers sessions; | +| | * Setup ports number; | +| | * IP Priority type; | +| | * Packet size; | +| | * Enable/disable BGP protocol on core ports; | +| | | +| | Default values exist. | +| | | ++--------------+--------------------------------------------------------------+ +|references | RFC2544 | +| | | ++--------------+--------------------------------------------------------------+ +| pre-test | 1. BNG is up and running and has configured: | +| conditions | * access ports with QinQ tagging; | +| | * core ports with configured IP addresses and VLAN; | +| | * PPPoE subscribers authorization settings (no auth or | +| | Radius server, PAP auth protocol); | +| | * QoS settings; | +| | | +| | 2. IxNetwork API server is running on specified in pod.yaml | +| | file TCL port; | +| | | +| | 3. BNG ports are connected to IXIA ports (IXIA uplink | +| | ports are connected to BNG access ports and IXIA | +| | downlink ports are connected to BNG core ports; | +| | | +| | 4. The pod.yaml file contains all necessary information | +| | (BNG access and core ports settings, core ports IP | +| | address, NICs, IxNetwork TCL port, IXIA uplink/downlink | +| | ports, etc). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Yardstick resolves the topology and connects to IxNetwork | +| | API server by TCL. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Test scenarios run, which performs the following steps: | +| | | +| | 1. Create access network topologies (this topologies are | +| | based on IXIA ports which are connected to BNG access | +| | ports); | +| | 2. Configure access network topologies with multiple device | +| | groups. Each device group represents single SVLAN with | +| | PPPoE subscribers sessions (number of created on port | +| | SVLANs and subscribers depends on specified if test case | +| | file options); | +| | 3. Create core network topologies (this topologies are | +| | based on IXIA ports which are connected to BNG core | +| | ports); | +| | 4. Configure core network topologies with single device | +| | group which represents one connection with configured | +| | VLAN and BGP protocol; | +| | 5. Establish PPPoE subscribers connections to BNG; | +| | 6. Create traffic flows between access and core ports | +| | (traffic flows are creating between access-core ports | +| | pairs, traffic is bi-directional); | +| | 7. Configure each traffic flow with specified in traffic | +| | profile options; | +| | 8. Run traffic with specified in test case file duration; | +| | 9. Collect network metrics after traffic was stopped; | +| | 10. In case drop percentage rate is higher than expected, | +| | reduce traffic line rate and repeat steps 7-10 again; | +| | 11. In case drop percentage rate is as expected or number | +| | of maximum iterations in step 10 achieved, disconnect | +| | PPPoE subscribers and stop traffic; | +| | 12. Stop test. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | During each iteration interval in the test run, all specified| +| | metrics are retrieved from IxNetwork and stored in the | +| | yardstick dispatcher. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The vBNG RFC2544 test case will achieve maximum traffic line | +| | rate with zero packet loss (or other non-zero allowed | +| | partial drop rate). | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia_8ports_1port_congested.rst b/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia_8ports_1port_congested.rst new file mode 100644 index 000000000..889ba2410 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_bng_pppoe_rfc2544_ixia_8ports_1port_congested.rst @@ -0,0 +1,179 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2019 Intel Corporation. + +*************************************************************** +Yardstick Test Case Description: NSB vBNG RFC2544 QoS TEST CASE +*************************************************************** + ++-----------------------------------------------------------------------------+ +|NSB vBNG RFC2544 QoS base line test case with link congestion | +| | ++--------------+--------------------------------------------------------------+ +|test case id | tc_bng_pppoe_rfc2544_ixia_8ports_1port_congested_IMIX | +| | | ++--------------+--------------------------------------------------------------+ +| metric | Network metrics: | +| | * TxThroughput | +| | * RxThroughput | +| | * TG packets in | +| | * TG packets out | +| | * Max Latency | +| | * Min Latency | +| | * Average Latency | +| | * Packets drop percentage | +| | | +| | PPPoE subscribers metrics: | +| | * Sessions up | +| | * Sessions down | +| | * Sessions Not Started | +| | * Sessions Total | +| | | +| | NOTE: the same network metrics list are collecting: | +| | * summary for all ports | +| | * per port | +| | * per priority flows summary on all ports | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test allows to measure performance of BNG network device| +| | according to RFC2544 testing methodology. Test case creates | +| | PPPoE subscribers connections to BNG, run prioritized traffic| +| | causing congestion of access port (port xe0) and collects | +| | network and PPPoE subscribers metrics. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | The BNG QoS RFC2544 test cases are listed below: | +| | | +| | * tc_bng_pppoe_rfc2544_ixia_8ports_1port_congested_IMIX.yaml | +| | | +| | Number of ports: | +| | * 8 ports | +| | | +| | Test duration: | +| | * set as 30sec; | +| | | +| | Traffic type: | +| | * IPv4; | +| | | +| | Packet sizes: | +| | * IMIX. The following default IMIX distribution is using: | +| | | +| | uplink: 70B - 33%, 940B - 33%, 1470B - 34% | +| | downlink: 68B - 3%, 932B - 1%, 1470B - 96% | +| | | +| | VLAN settings: | +| | * QinQ on access ports; | +| | * VLAN on core ports; | +| | | +| | Number of PPPoE subscribers: | +| | * 4000 per access port; | +| | * 1000 per SVLAN; | +| | | +| | Default ToS bits settings: | +| | * 0 - (000) Routine | +| | * 4 - (100) Flash Override | +| | * 7 - (111) Network Control. | +| | | +| | The above fields are the main options used for the test case | +| | and could be configured using cli options on test run or | +| | directly in test case yaml file. | +| | | +| | NOTE: that only parameter that can't be changed is ports | +| | number. To run the test with another number of ports | +| | traffic profile should be updated. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | IXIA IxNetwork | +| | | +| | IXIA IxNetwork is using to emulates PPPoE sessions, generate | +| | L2-L3 traffic, analyze traffic flows and collect network | +| | metrics during test run. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | Mentioned BNG QoS RFC2544 test cases can be configured with | +| | different: | +| | | +| | * Number of PPPoE subscribers sessions; | +| | * IP Priority type; | +| | * Packet size; | +| | * enable/disable BGP protocol on core ports; | +| | | +| | Default values exist. | +| | | ++--------------+--------------------------------------------------------------+ +|references | RFC2544 | +| | | ++--------------+--------------------------------------------------------------+ +| pre-test | 1. BNG is up and running and has configured: | +| conditions | * access ports with QinQ tagging; | +| | * core ports with configured IP addresses and VLAN; | +| | * PPPoE subscribers authorization settings (no auth or | +| | Radius server, PAP auth protocol); | +| | * QoS settings; | +| | | +| | 2. IxNetwork API server is running on specified in pod.yaml | +| | file TCL port; | +| | | +| | 3. BNG ports are connected to IXIA ports (IXIA uplink | +| | ports are connected to BNG access ports and IXIA | +| | downlink ports are connected to BNG core ports; | +| | | +| | 4. The pod.yaml file contains all necessary information | +| | (BNG access and core ports settings, core ports IP | +| | address, NICs, IxNetwork TCL port, IXIA uplink/downlink | +| | ports, etc). | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Yardstick resolve the topology and connects to IxNetwork | +| | API server by TCL. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Test scenarios run, which performs the following steps: | +| | | +| | 1. Create access network topologies (this topologies are | +| | based on IXIA ports which are connected to BNG access | +| | ports); | +| | 2. Configure access network topologies with multiple device | +| | groups. Each device group represents single SVLAN with | +| | PPPoE subscribers sessions (number of created on port | +| | SVLANs and subscribers depends on specified if test case | +| | file options); | +| | 3. Create core network topologies (this topologies are | +| | based on IXIA ports which are connected to BNG core | +| | ports); | +| | 4. Configure core network topologies with single device | +| | group which represents one connection with configured | +| | VLAN and BGP protocol; | +| | 5. Establish PPPoE subscribers connections to BNG; | +| | 6. Create traffic flows between access and core ports. | +| | While test covers case with access port congestion, | +| | flows between ports will be created in the following | +| | way: traffic from two core ports are going to one access | +| | port causing port congestion and traffic from other two | +| | core ports is splitting between remaining three access | +| | ports; | +| | 7. Configure each traffic flow with specified in traffic | +| | profile options; | +| | 8. Run traffic with specified in test case file duration; | +| | 9. Collect network metrics after traffic was stopped; | +| | 10. Measure drop percentage rate of different priority | +| | packets on congested port. Expected that all high and | +| | medium priority packets was forwarded and only low | +| | priority packets has drops. | +| | 11. Disconnect PPPoE subscribers and stop test. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | During test run, in the end of each iteration all specified | +| | in the document metrics are retrieved from IxNetwork and | +| | stored in the yardstick dispatcher. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | The test case is successful if all high and medium priority | +| | packets on congested port was forwarded and only low | +| | priority packets has drops. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/nsb/tc_vims_context_sipp.rst b/docs/testing/user/userguide/nsb/tc_vims_context_sipp.rst new file mode 100644 index 000000000..6df4ab880 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_vims_context_sipp.rst @@ -0,0 +1,96 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2019 Viosoft Corporation. + +********************************************** +Yardstick Test Case Description: NSB VIMS +********************************************** + ++-----------------------------------------------------------------------------+ +|NSB VIMS test for vIMS characterization | +| | ++--------------+--------------------------------------------------------------+ +|test case id | tc_vims_{context}_sipp | +| | | +| | * context = baremetal or heat; | +| | | ++--------------+--------------------------------------------------------------+ +|metric | * Successful registrations per second; | +| | * Total number of active registrations per server; | +| | * Successful de-registrations per second; | +| | * Successful session establishments per second; | +| | * Total number of active sessions per server; | +| | * Mean session setup time; | +| | * Successful re-registrations per second; | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The vIMS test handles registration rate, call rate, | +| | round trip delay, and message statistics of vIMS system. | +| | | +| | The vIMS test cases are implemented to run in baremetal | +| | and heat context default configuration. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | The vIMS test cases are listed below: | +| | | +| | * tc_vims_baremetal_sipp.yaml | +| | * tc_vims_heat_sipp.yaml | +| | | +| | Each test runs one time and collects all the KPIs. | +| | The configuration of vIMS and SIPp can be changed in each | +| | test. | ++--------------+--------------------------------------------------------------+ +|test tool | SIPp | +| | | +| | SIPp is an application that can simulate SIP scenarios, can | +| | generate RTP traffic and used for vIMS characterization. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | The SIPp test cases can be configured with different: | +| | | +| | * number of accounts; | +| | * the call per second (cps) of SIP test; | +| | * the holding time; | +| | * RTP configuratioin; | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | For Openstack test case, only vIMS is deployed by external | +|conditions | heat template, SIPp needs pod.yaml file with the necessary | +| | system and NIC information | +| | | +| | For Baremetal tests cases SIPp and vIMS must be installed in | +| | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | For Baremetal test: The TG and VNF are started on the hosts | +| | based on the pod file. | +| | | +| | For Heat test: One host VM for vIMS is booted, based on | +| | the test flavor. Another host for SIPp is booted as | +| | traffic generator, based on pod.yaml file | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Yardstick is connected with the vIMS and SIPp via ssh. | +| | The test will resolve the topology, instantiate the vIMS and | +| | SIPp and collect the KPIs/metrics. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | The SIPp will run scenario tests with parameters configured | +| | in test case files (tc_vims_baremetal_sipp.yaml and | +| | tc_vims_heat_sipp.yaml files). | +| | This is done until the KPIs of SIPp are within an acceptable | +| | threshold. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | In Baremetal test: The test quits the application. | +| | | +| | In Heat test: The host VM of vIMS is deleted on test | +| | completion. | ++--------------+--------------------------------------------------------------+ +|test verdict | The test case will collect the KPIs and plot on Grafana. | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/nsb/tc_vpp_baremetal_crypto_ipsec.rst b/docs/testing/user/userguide/nsb/tc_vpp_baremetal_crypto_ipsec.rst new file mode 100644 index 000000000..6a4a37697 --- /dev/null +++ b/docs/testing/user/userguide/nsb/tc_vpp_baremetal_crypto_ipsec.rst @@ -0,0 +1,113 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, 2019 Viosoft Corporation. + +*********************************************** +Yardstick Test Case Description: NSB VPP IPSEC +*********************************************** + ++------------------------------------------------------------------------------+ +|NSB VPP test for vIPSEC characterization | +| | ++--------------+---------------------------------------------------------------+ +|test case id | tc_baremetal_rfc2544_ipv4_{crypto_dev}_{crypto_alg} | +| | | +| | * crypto_dev = HW_cryptodev or SW_cryptodev; | +| | * crypto_alg = aes-gcm or cbc-sha1; | +| | | ++--------------+---------------------------------------------------------------+ +|metric | * Network Throughput NDR or PDR; | +| | * Connections Per Second (CPS); | +| | * Latency; | +| | * Number of tunnels; | +| | * TG Packets Out; | +| | * TG Packets In; | +| | * VNF Packets Out; | +| | * VNF Packets In; | +| | * Dropped packets; | +| | | ++--------------+---------------------------------------------------------------+ +|test purpose | IPv4 IPsec tunnel mode performance test: | +| | | +| | * Finds and reports throughput NDR (Non Drop Rate) with zero | +| | packet loss tolerance or throughput PDR (Partial Drop Rate) | +| | with non-zero packet loss tolerance (LT) expressed in | +| | number of packets transmitted. | +| | | +| | * The IPSEC test cases are implemented to run in baremetal | +| | | ++--------------+---------------------------------------------------------------+ +|configuration | The IPSEC test cases are listed below: | +| | | +| | * tc_baremetal_rfc2544_ipv4_hw_aesgcm_IMIX_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_hw_aesgcm_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_hw_cbcsha1_IMIX_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_hw_cbcsha1_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_sw_aesgcm_IMIX_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_sw_aesgcm_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_sw_cbcsha1_IMIX_trex.yaml | +| | * tc_baremetal_rfc2544_ipv4_sw_cbcsha1_trex.yaml | +| | | +| | Test duration is set as 500sec for each test. | +| | Packet size set as 64 bytes or higher. | +| | Number of tunnels set as 1 or higher. | +| | Number of connections set as 1 or higher | +| | These can be configured | +| | | ++--------------+---------------------------------------------------------------+ +|test tool | Vector Packet Processing (VPP) | +| | The VPP platform is an extensible framework that provides | +| | out-of-the-box production quality switch/router functionality.| +| | Its high performance, proven technology, its modularity and, | +| | flexibility and rich feature set | +| | | ++--------------+---------------------------------------------------------------+ +|applicability | This VPP IPSEC test cases can be configured with different: | +| | | +| | * packet sizes; | +| | * test durations; | +| | * tolerated loss; | +| | * crypto device type; | +| | * number of physical cores; | +| | * number of tunnels; | +| | * number of connections; | +| | * encryption algorithms - integrity algorithm; | +| | | +| | Default values exist. | +| | | ++--------------+---------------------------------------------------------------+ +|pre-test | For Baremetal tests cases VPP and DPDK must be installed in | +|conditions | the hosts where the test is executed. The pod.yaml file must | +| | have the necessary system and NIC information | +| | | ++--------------+---------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+---------------------------------------------------------------+ +|step 1 | For Baremetal test: The TG and VNF are started on the hosts | +| | based on the pod file. | +| | | ++--------------+---------------------------------------------------------------+ +|step 2 | Yardstick is connected with the TG and VNF by using ssh. | +| | The test will resolve the topology and instantiate the VNF | +| | and TG and collect the KPI's/metrics. | +| | | ++--------------+---------------------------------------------------------------+ +|step 3 | Test packets are generated by TG on links to DUTs. If the | +| | number of dropped packets is more than the tolerated loss | +| | the line rate or throughput is halved. This is done until | +| | the dropped packets are within an acceptable tolerated loss. | +| | | +| | The KPI is the number of packets per second for a packet size | +| | specified in the test case with an accepted minimal packet | +| | loss for the default configuration. | +| | | ++--------------+---------------------------------------------------------------+ +|step 4 | In Baremetal test: The test quits the application and unbind | +| | the DPDK ports. | +| | | ++--------------+---------------------------------------------------------------+ +|test verdict | The test case will achieve a Throughput with an accepted | +| | minimal tolerated packet loss. | ++--------------+---------------------------------------------------------------+
\ No newline at end of file |