aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing/developer
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/developer')
-rwxr-xr-xdocs/testing/developer/devguide/devguide.rst399
-rwxr-xr-xdocs/testing/developer/devguide/devguide_nsb_prox.rst644
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_1.pngbin0 -> 76923 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_2.pngbin0 -> 76976 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_3.pngbin0 -> 76762 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_4.pngbin0 -> 20013 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_5.pngbin0 -> 66525 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Grafana_6.pngbin0 -> 88051 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_BM_Script.pngbin76705 -> 94927 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_HEAT_Script.pngbin90040 -> 0 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_HEAT_Script1.pngbin0 -> 87627 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_HEAT_Script2.pngbin0 -> 85248 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.pngbin0 -> 60578 bytes
-rw-r--r--docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.pngbin0 -> 92664 bytes
-rw-r--r--docs/testing/developer/devguide/images/vPE_Diagram.pngbin0 -> 82640 bytes
-rw-r--r--docs/testing/developer/devguide/index.rst2
16 files changed, 766 insertions, 279 deletions
diff --git a/docs/testing/developer/devguide/devguide.rst b/docs/testing/developer/devguide/devguide.rst
index dade49b75..2065f6e0d 100755
--- a/docs/testing/developer/devguide/devguide.rst
+++ b/docs/testing/developer/devguide/devguide.rst
@@ -1,16 +1,42 @@
+..
+ Licensed under the Apache License, Version 2.0 (the "License"); you may
+ not use this file except in compliance with the License. You may obtain
+ a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ License for the specific language governing permissions and limitations
+ under the License.
+
+ Convention for heading levels in Yardstick documentation:
+
+ ======= Heading 0 (reserved for the title in a document)
+ ------- Heading 1
+ ~~~~~~~ Heading 2
+ +++++++ Heading 3
+ ''''''' Heading 4
+
+ Avoid deeper levels because they do not render well.
+
Introduction
-=============
+------------
-Yardstick is a project dealing with performance testing. Yardstick produces its own test cases but can also be considered as a framework to support feature project testing.
+Yardstick is a project dealing with performance testing. Yardstick produces
+its own test cases but can also be considered as a framework to support feature
+project testing.
-Yardstick developed a test API that can be used by any OPNFV project. Therefore there are many ways to contribute to Yardstick.
+Yardstick developed a test API that can be used by any OPNFV project. Therefore
+there are many ways to contribute to Yardstick.
You can:
* Develop new test cases
* Review codes
* Develop Yardstick API / framework
-* Develop Yardstick grafana dashboards and Yardstick reporting page
+* Develop Yardstick grafana dashboards and Yardstick reporting page
* Write Yardstick documentation
This developer guide describes how to interact with the Yardstick project.
@@ -19,28 +45,30 @@ part is a list of “How to” to help you to join the Yardstick family whatever
your field of interest is.
Where can I find some help to start?
---------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. _`user guide`: http://artifacts.opnfv.org/yardstick/danube/1.0/docs/stesting_user_userguide/index.html
+.. _`user guide`: https://artifacts.opnfv.org/yardstick/docs/testing_user_userguide/index.html
.. _`wiki page`: https://wiki.opnfv.org/display/yardstick/
This guide is made for you. You can have a look at the `user guide`_.
There are also references on documentation, video tutorials, tips in the
-project `wiki page`_. You can also directly contact us by mail with [Yardstick] prefix in the title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan #opnfv-yardstick.
+project `wiki page`_. You can also directly contact us by mail with
+``#yardstick`` or ``[yardstick]`` prefix in the subject at
+``opnfv-tech-discuss@lists.opnfv.org`` or on the IRC channel ``#opnfv-yardstick``.
Yardstick developer areas
-==========================
+-------------------------
Yardstick framework
---------------------
+~~~~~~~~~~~~~~~~~~~
-Yardstick can be considered as a framework. Yardstick is release as a docker
+Yardstick can be considered as a framework. Yardstick is released as a docker
file, including tools, scripts and a CLI to prepare the environement and run
-tests. It simplifies the integration of external test suites in CI pipeline
-and provide commodity tools to collect and display results.
+tests. It simplifies the integration of external test suites in CI pipelines
+and provides commodity tools to collect and display results.
-Since Danube, test categories also known as tiers have been created to group
+Since Danube, test categories (also known as tiers) have been created to group
similar tests, provide consistant sub-lists and at the end optimize test
duration for CI (see How To section).
@@ -56,44 +84,54 @@ The tiers are:
How Todos?
-===========
+----------
How Yardstick works?
----------------------
+~~~~~~~~~~~~~~~~~~~~
The installation and configuration of the Yardstick is described in the `user guide`_.
How to work with test cases?
-----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Sample Test cases
++++++++++++++++++
-**Sample Test cases**
+Yardstick provides many sample test cases which are located at ``samples`` directory of repo.
-Yardstick provides many sample test cases which are located at "samples" directory of repo.
+Sample test cases are designed with the following goals:
-Sample test cases are designed as following goals:
+1. Helping user better understand Yardstick features (including new feature and
+ new test capacity).
-1. Helping user better understand yardstick features(including new feature and new test capacity).
+2. Helping developer to debug a new feature and test case before it is
+ offically released.
-2. Helping developer to debug his new feature and test case before it is offical released.
+3. Helping other developers understand and verify the new patch before the
+ patch is merged.
-3. Helping other developers understand and verify the new patch before the patch merged.
+Developers should upload their sample test cases as well when they are
+uploading a new patch which is about the Yardstick new test case or new feature.
-So developers should upload your sample test case as well when they are trying to upload a new patch which is about the yardstick new test case or new feature.
+OPNFV Release Test cases
+++++++++++++++++++++++++
-**OPNFV Release Test cases**
+OPNFV Release test cases are located at ``yardstick/tests/opnfv/test_cases``.
+These test cases are run by OPNFV CI jobs, which means these test cases should
+be more mature than sample test cases.
+OPNFV scenario owners can select related test cases and add them into the test
+suites which represent their scenario.
-OPNFV Release test cases which are located at "tests/opnfv/test_cases" of repo.
-those test cases are runing by OPNFV CI jobs, It means those test cases should be more mature than sample test cases.
-OPNFV scenario owners can select related test cases and add them into the test suites which is represent the scenario.
-
-**Test case Description File**
+Test case Description File
+++++++++++++++++++++++++++
This section will introduce the meaning of the Test case description file.
-we will use ping.yaml as a example to show you how to understand the test case description file.
-In this Yaml file, you can easily find it consists of two sections. One is “Scenarios”, the other is “Context”.::
+we will use ping.yaml as a example to show you how to understand the test case
+description file.
+This ``yaml`` file consists of two sections. One is ``scenarios``, the other
+is ``context``.::
---
# Sample benchmark task config file
@@ -150,18 +188,32 @@ In this Yaml file, you can easily find it consists of two sections. One is “Sc
{% endif %}
-"Contexts" section is the description of pre-condition of testing. As ping.yaml shown, you can configure the image, flavor , name ,affinity and network of Test VM(servers), with this section, you will get a pre-condition env for Testing.
-Yardstick will automatic setup the stack which are described in this section.
-In fact, yardstick use convert this section to heat template and setup the VMs by heat-client (Meanwhile, yardstick can support to convert this section to Kubernetes template to setup containers).
-
-Two Test VMs(athena and ares) are configured by keyword "servers".
-"flavor" will determine how many vCPU, how much memory for test VMs.
-As "yardstick-flavor" is a basic flavor which will be automatically created when you run command "yardstick env prepare". "yardstick-flavor" is "1 vCPU 1G RAM,3G Disk".
-"image" is the image name of test VMs. if you use cirros.3.5.0, you need fill the username of this image into "user". the "policy" of placement of Test VMs have two values (affinity and availability).
-"availability" means anti-affinity. In "network" section, you can configure which provide network and physical_network you want Test VMs use.
-you may need to configure segmentation_id when your network is vlan.
-
-Moreover, you can configure your specific flavor as below, yardstick will setup the stack for you. ::
+The ``contexts`` section is the description of pre-condition of testing. As
+``ping.yaml`` shows, you can configure the image, flavor, name, affinity and
+network of Test VM (servers), with this section, you will get a pre-condition
+env for Testing.
+Yardstick will automatically setup the stack which are described in this
+section.
+Yardstick converts this section to heat template and sets up the VMs with
+heat-client (Yardstick can also support to convert this section to Kubernetes
+template to setup containers).
+
+In the examples above, two Test VMs (athena and ares) are configured by
+keyword ``servers``.
+``flavor`` will determine how many vCPU, how much memory for test VMs.
+As ``yardstick-flavor`` is a basic flavor which will be automatically created
+when you run command ``yardstick env prepare``. ``yardstick-flavor`` is
+``1 vCPU 1G RAM,3G Disk``.
+``image`` is the image name of test VMs. If you use ``cirros.3.5.0``, you need
+fill the username of this image into ``user``.
+The ``policy`` of placement of Test VMs have two values (``affinity`` and
+``availability``). ``availability`` means anti-affinity.
+In the ``network`` section, you can configure which ``provider`` network and
+``physical_network`` you want Test VMs to use.
+You may need to configure ``segmentation_id`` when your network is vlan.
+
+Moreover, you can configure your specific flavor as below, Yardstick will setup
+the stack for you. ::
flavor:
name: yardstick-new-flavor
@@ -170,7 +222,8 @@ Moreover, you can configure your specific flavor as below, yardstick will setup
disk: 2
-Besides default heat stack, yardstick also allow you to setup other two types stack. they are "Node" and "Kubernetes". ::
+Besides default ``Heat`` context, Yardstick also allows you to setup two other
+types of context. They are ``Node`` and ``Kubernetes``. ::
context:
type: Kubernetes
@@ -183,48 +236,64 @@ and ::
name: LF
+The ``scenarios`` section is the description of testing steps, you can
+orchestrate the complex testing step through scenarios.
-"Scenarios" section is the description of testing step, you can orchestrate the complex testing step through orchestrate scenarios.
+Each scenario will do one testing step.
+In one scenario, you can configure the type of scenario (operation), ``runner``
+type and ``sla`` of the scenario.
-Each scenario will do one testing step, In one scenario, you can configure the type of scenario(operation), runner type and SLA of the scenario.
+For TC002, We only have one step, which is Ping from host VM to target VM. In
+this step, we also have some detailed operations implemented (such as ssh to
+VM, ping from VM1 to VM2. Get the latency, verify the SLA, report the result).
-For TC002, We only have one step , that is Ping from host VM to target VM. In this step, we also have some detail operation implement ( such as ssh to VM, ping from VM1 to VM2. Get the latency, verify the SLA, report the result).
+If you want to get this implementation details implement, you can check with
+the scenario.py file. For Ping scenario, you can find it in Yardstick repo
+(``yardstick/yardstick/benchmark/scenarios/networking/ping.py``).
-If you want to get this detail implement , you can check with the scenario.py file. For Ping scenario, you can find it in yardstick repo ( yardstick / yardstick / benchmark / scenarios / networking / ping.py)
+After you select the type of scenario (such as Ping), you will select one type
+of ``runner``, there are 4 types of runner. ``Iteration`` and ``Duration`` are
+the most commonly used, and the default is ``Iteration``.
-after you select the type of scenario( such as Ping), you will select one type of runner, there are 4 types of runner. Usually, we use the "Iteration" and "Duration". and Default is "Iteration".
-For Iteration, you can specify the iteration number and interval of iteration. ::
+For ``Iteration``, you can specify the iteration number and interval of iteration. ::
runner:
type: Iteration
iterations: 10
interval: 1
-That means yardstick will iterate the 10 times of Ping test and the interval of each iteration is one second.
+That means Yardstick will repeat the Ping test 10 times and the interval of
+each iteration is one second.
-For Duration, you can specify the duration of this scenario and the interval of each ping test. ::
+For ``Duration``, you can specify the duration of this scenario and the
+interval of each ping test. ::
runner:
type: Duration
duration: 60
interval: 10
-That means yardstick will run the ping test as loop until the total time of this scenario reach the 60s and the interval of each loop is ten seconds.
+That means Yardstick will run the ping test as loop until the total time of
+this scenario reaches 60s and the interval of each loop is ten seconds.
-SLA is the criterion of this scenario. that depends on the scenario. different scenario can have different SLA metric.
+SLA is the criterion of this scenario. This depends on the scenario. Different
+scenarios can have different SLA metric.
-**How to write a new test case**
+How to write a new test case
+++++++++++++++++++++++++++++
-Yardstick already provide a library of testing step. that means yardstick provide lots of type scenario.
+Yardstick already provides a library of testing steps (i.e. different types of
+scenario).
-Basiclly, What you need to do is to orchestrate the scenario from the library.
-
-Here, We will show two cases. One is how to write a simple test case, the other is how to write a quite complex test case.
+Basically, what you need to do is to orchestrate the scenario from the library.
+Here, we will show two cases. One is how to write a simple test case, the other
+is how to write a quite complex test case.
Write a new simple test case
+''''''''''''''''''''''''''''
First, you can image a basic test case description as below.
@@ -314,7 +383,7 @@ First, you can image a basic test case description as below.
TODO
How can I contribute to Yardstick?
------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are already a contributor of any OPNFV project, you can contribute to
Yardstick. If you are totally new to OPNFV, you must first create your Linux
@@ -329,16 +398,17 @@ We distinguish 2 levels of contributors:
Yardstick commitors are promoted by the Yardstick contributors.
Gerrit & JIRA introduction
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++
.. _Gerrit: https://www.gerritcodereview.com/
-.. _`OPNFV Gerrit`: http://gerrit.opnfv.org/
+.. _`OPNFV Gerrit`: http://gerrit.opnfv.org/gerrit
.. _link: https://identity.linuxfoundation.org/
.. _JIRA: https://jira.opnfv.org/secure/Dashboard.jspa
OPNFV uses Gerrit_ for web based code review and repository management for the
Git Version Control System. You can access `OPNFV Gerrit`_. Please note that
-you need to have Linux Foundation ID in order to use OPNFV Gerrit. You can get one from this link_.
+you need to have Linux Foundation ID in order to use OPNFV Gerrit. You can get
+one from this link_.
OPNFV uses JIRA_ for issue management. An important principle of change
management is to have two-way trace-ability between issue management
@@ -350,14 +420,16 @@ If you want to contribute to Yardstick, you can pick a issue from Yardstick's
JIRA dashboard or you can create you own issue and submit it to JIRA.
Install Git and Git-reviews
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++++++++
Installing and configuring Git and Git-Review is necessary in order to submit
-code to Gerrit. The `Getting to the code <https://wiki.opnfv.org/display/DEV/Developer+Getting+Started>`_ page will provide you with some help for that.
+code to Gerrit. The
+`Getting to the code <https://wiki.opnfv.org/display/DEV/Developer+Getting+Started>`_
+page will provide you with some help for that.
Verify your patch locally before submitting
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++++++++++++++++++++++++
Once you finish a patch, you can submit it to Gerrit for code review. A
developer sends a new patch to Gerrit will trigger patch verify job on Jenkins
@@ -366,7 +438,8 @@ code coverage test. Before you submit your patch, it is recommended to run the
patch verification in your local environment first.
Open a terminal window and set the project's directory to the working
-directory using the ``cd`` command. Assume that ``YARDSTICK_REPO_DIR`` is the path to the Yardstick project folder on your computer::
+directory using the ``cd`` command. Assume that ``YARDSTICK_REPO_DIR`` is the
+path to the Yardstick project folder on your computer::
cd $YARDSTICK_REPO_DIR
@@ -376,8 +449,12 @@ Verify your patch::
It is used in CI but also by the CLI.
+For more details on ``tox`` and tests, please refer to the `Running tests`_
+and `working with tox`_ sections below, which describe the different available
+environments.
+
Submit the code with Git
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++
Tell Git which files you would like to take into account for the next commit.
This is called 'staging' the files, by placing them into the staging area,
@@ -408,7 +485,7 @@ Git repository::
JIRA: YARDSTICK-XXX
-.. _`this document`: http://chris.beams.io/posts/git-commit/
+.. _`this document`: https://chris.beams.io/posts/git-commit/
The message that is required for the commit should follow a specific set of
rules. This practice allows to standardize the description messages attached
@@ -417,7 +494,7 @@ to the commits, and eventually navigate among the latter more easily.
`This document`_ happened to be very clear and useful to get started with that.
Push the code to Gerrit for review
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++++++++++
Now that the code has been comitted into your local Git repository the
following step is to push it online to Gerrit for it to be reviewed. The
@@ -432,26 +509,27 @@ Yardstick committers and contributors to review your codes.
:width: 800px
:alt: Gerrit for code review
-You can find Yardstick people info `here <https://wiki.opnfv.org/display/yardstick/People>`_.
+You can find a list Yardstick people
+`here <https://wiki.opnfv.org/display/yardstick/Yardstick+People>`_, or use
+the ``yardstick-reviewers`` and ``yardstick-committers`` groups in gerrit.
Modify the code under review in Gerrit
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++++++++++++++
At the same time the code is being reviewed in Gerrit, you may need to edit it
to make some changes and then send it back for review. The following steps go
through the procedure.
Once you have modified/edited your code files under your IDE, you will have to
-stage them. The 'status' command is very helpful at this point as it provides
-an overview of Git's current state::
+stage them. The ``git status`` command is very helpful at this point as it
+provides an overview of Git's current state::
git status
-The output of the command provides us with the files that have been modified
-after the latest commit.
+This command lists the files that have been modified since the last commit.
You can now stage the files that have been modified as part of the Gerrit code
-review edition/modification/improvement using ``git add`` command. It is now
+review addition/modification/improvement using ``git add`` command. It is now
time to commit the newly modified files, but the objective here is not to
create a new commit, we simply want to inject the new changes into the
previous commit. You can achieve that with the '--amend' option on the
@@ -466,9 +544,172 @@ The final step consists in pushing the newly modified commit to Gerrit::
git review
+Backporting changes to stable branches
+--------------------------------------
+During the release cycle, when master and the ``stable/<release>`` branch have
+diverged, it may be necessary to backport (cherry-pick) changes top the
+``stable/<release>`` branch once they have merged to master.
+These changes should be identified by the committers reviewing the patch.
+Changes should be backported **as soon as possible** after merging of the
+original code.
+
+..note::
+ Besides the commit and review process below, the Jira tick must be updated to
+ add dual release versions and indicate that the change is to be backported.
+
+The process for backporting is as follows:
+
+* Committer A merges a change to master (process for normal changes).
+* Committer A cherry-picks the change to ``stable/<release>`` branch (if the
+ bug has been identified for backporting).
+* The original author should review the code and verify that it still works
+ (and give a ``+1``).
+* Committer B reviews the change, gives a ``+2`` and merges to
+ ``stable/<release>``.
+
+A backported change needs a ``+1`` and a ``+2`` from a committer who didn’t
+propose the change (i.e. minimum 3 people involved).
+
+Development guidelines
+----------------------
+This section provides guidelines and best practices for feature development
+and bug fixing in Yardstick.
+
+In general, bug fixes should be submitted as a single patch.
+
+When developing larger features, all commits on the local topic branch can be
+submitted together, by running ``git review`` on the tip of the branch. This
+creates a chain of related patches in gerrit.
+
+Each commit should contain one logical change and the author should aim for no
+more than 300 lines of code per commit. This helps to make the changes easier
+to review.
+
+Each feature should have the following:
+
+* Feature/bug fix code
+* Unit tests (both positive and negative)
+* Functional tests (optional)
+* Sample testcases (if applicable)
+* Documentation
+* Update to release notes
+
+Coding style
+~~~~~~~~~~~~
+.. _`OpenStack Style Guidelines`: https://docs.openstack.org/hacking/latest/user/hacking.html
+.. _`OPNFV coding guidelines`: https://wiki.opnfv.org/display/DEV/Contribution+Guidelines
+
+Please follow the `OpenStack Style Guidelines`_ for code contributions (the
+section on Internationalization (i18n) Strings is not applicable).
+
+When writing commit message, the `OPNFV coding guidelines`_ on git commit
+message style should also be used.
+
+Running tests
+~~~~~~~~~~~~~
+Once your patch has been submitted, a number of tests will be run by Jenkins
+CI to verify the patch. Before submitting your patch, you should run these
+tests locally. You can do this using ``tox``, which has a number of different
+test environments defined in ``tox.ini``.
+Calling ``tox`` without any additional arguments runs the default set of
+tests (unit tests, functional tests, coverage and pylint).
+
+If some tests are failing, you can save time and select test environments
+individually, by passing one or more of the following command-line options to
+``tox``:
+
+* ``-e py27``: Unit tests using Python 2.7
+* ``-e py3``: Unit tests using Python 3
+* ``-e pep8``: Linter and style checks on updated files
+* ``-e functional``: Functional tests using Python 2.7
+* ``-e functional-py3``: Functional tests using Python 3
+* ``-e coverage``: Code coverage checks
+
+.. note:: You need to stage your changes prior to running coverage for those
+ changes to be checked.
+
+In addition to the tests run by Jenkins (listed above), there are a number of
+other test environments defined.
+
+* ``-e pep8-full``: Linter and style checks are run on the whole repo (not
+ just on updated files)
+* ``-e os-requirements``: Check that the requirements are compatible with
+ OpenStack requirements.
+
+Working with tox
+++++++++++++++++
+.. _virtualenv: https://virtualenv.pypa.io/en/stable/
+
+``tox`` uses `virtualenv`_ to create isolated Python environments to run the
+tests in. The test environments are located at
+``.tox/<environment_name>`` e.g. ``.tox/py27``.
+
+If requirements are changed, you will need to recreate the tox test
+environment to make sure the new requirements are installed. This is done by
+passing the additional ``-r`` command-line option to ``tox``::
+
+ tox -r -e ...
+
+This can also be achieved by deleting the test environments manually before
+running ``tox``::
+
+ rm -rf .tox/<environment_name>
+ rm -rf .tox/py27
+
+Writing unit tests
+~~~~~~~~~~~~~~~~~~
+For each change submitted, a set of unit tests should be submitted, which
+should include both positive and negative testing.
+
+In order to help identify which tests are needed, follow the guidelines below.
+
+* In general, there should be a separate test for each branching point, return
+ value and input set.
+* Negative tests should be written to make sure exceptions are raised and/or
+ handled appropriately.
+
+The following convention should be used for naming tests::
+
+ test_<method_name>_<some_comment>
+
+The comment gives more information on the nature of the test, the side effect
+being checked, or the parameter being modified::
+
+ test_my_method_runtime_error
+ test_my_method_invalid_credentials
+ test_my_method_param1_none
+
+Mocking
++++++++
+The ``mock`` library is used for unit testing to stub out external libraries.
+
+The following conventions are used in Yardstick:
+
+* Use ``mock.patch.object`` instead of ``mock.patch``.
+
+* When naming mocked classes/functions, use ``mock_<class_and_function_name>``
+ e.g. ``mock_subprocess_call``
+
+* Avoid decorating classes with mocks. Apply the mocking in ``setUp()``::
+
+ @mock.patch.object(ssh, 'SSH')
+ class MyClassTestCase(unittest.TestCase):
+
+ should be::
+
+ class MyClassTestCase(unittest.TestCase):
+ def setUp(self):
+ self._mock_ssh = mock.patch.object(ssh, 'SSH')
+ self.mock_ssh = self._mock_ssh.start()
+
+ self.addCleanup(self._stop_mocks)
+
+ def _stop_mocks(self):
+ self._mock_ssh.stop()
Plugins
-==========
+-------
-For information about Yardstick plugins, refer to the chapter **Installing a plug-in into Yardstick** in the `user guide`_.
+For information about Yardstick plugins, refer to the chapter
+**Installing a plug-in into Yardstick** in the `user guide`_.
diff --git a/docs/testing/developer/devguide/devguide_nsb_prox.rst b/docs/testing/developer/devguide/devguide_nsb_prox.rst
index 22628413b..44a216b75 100755
--- a/docs/testing/developer/devguide/devguide_nsb_prox.rst
+++ b/docs/testing/developer/devguide/devguide_nsb_prox.rst
@@ -8,14 +8,17 @@ for Telco customers, investigate the impact of new Intel compute,
network and storage technologies, characterize performance, and develop
optimal system architectures and configurations.
+NSB PROX Supports Baremetal, Openstack and standalone configuration.
+
.. contents::
Prerequisites
=============
-In order to integrate PROX tests into NSB, the following prerequisites are required.
+In order to integrate PROX tests into NSB, the following prerequisites are
+required.
-.. _`dpdk wiki page`: http://dpdk.org/
+.. _`dpdk wiki page`: https://www.dpdk.org/
.. _`yardstick wiki page`: https://wiki.opnfv.org/display/yardstick/
.. _`Prox documentation`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation
.. _`openstack wiki page`: https://wiki.openstack.org/wiki/Main_Page
@@ -159,11 +162,13 @@ A NSB Prox test is composed of the following components :-
``tc_prox_heat_context_vpe-4.yaml``. This file describes the components
of the test, in the case of openstack the network description and
server descriptions, in the case of baremetal the hardware
- description location. It also contains the name of the Traffic Generator, the SUT config file
- and the traffic profile description, all described below. See nsb-test-description-label_
+ description location. It also contains the name of the Traffic Generator,
+ the SUT config file and the traffic profile description, all described below.
+ See `Test Description File`_
-* Traffic Profile file. Example ``prox_binsearch.yaml``. This describes the packet size, tolerated
- loss, initial line rate to start traffic at, test interval etc See nsb-traffic-profile-label_
+* Traffic Profile file. Example ``prox_binsearch.yaml``. This describes the
+ packet size, tolerated loss, initial line rate to start traffic at, test
+ interval etc See `Traffic Profile File`_
* Traffic Generator Config file. Usually called ``gen_<test>-<ports>.cfg``.
@@ -176,7 +181,7 @@ A NSB Prox test is composed of the following components :-
under test.
Example traffic generator config file ``gen_l2fwd-4.cfg``
- See nsb-traffic-generator-label_
+ See `Traffic Generator Config file`_
* SUT Config file. Usually called ``handle_<test>-<ports>.cfg``.
@@ -190,7 +195,7 @@ A NSB Prox test is composed of the following components :-
the ports to the Traffic Verifier tasks of the Traffic Generator.
Example traffic generator config file ``handle_l2fwd-4.cfg``
- See nsb-sut-generator-label_
+ See `SUT Config File`_
* NSB PROX Baremetal Configuration file. Usually called
``prox-baremetal-<ports>.yaml``
@@ -199,12 +204,12 @@ A NSB Prox test is composed of the following components :-
This is required for baremetal only. This describes hardware, NICs,
IP addresses, Network drivers, usernames and passwords.
- See baremetal-config-label_
+ See `Baremetal Configuration File`_
* Grafana Dashboard. Usually called
``Prox_<context>_<test>-<port>-<DateAndTime>.json`` where
- * <context> Is either ``BM`` or ``heat``
+ * <context> Is ``BM``,``heat``,``ovs_dpdk`` or ``sriov``
* <test> Is the a one or 2 word description of the test.
* <port> is the number of ports used express as ``2Port`` or ``4Port``
* <DateAndTime> is the Date and Time expressed as a string.
@@ -214,15 +219,15 @@ A NSB Prox test is composed of the following components :-
Other files may be required. These are test specific files and will be
covered later.
-.. _nsb-test-description-label:
-**Test Description File**
+Test Description File
+---------------------
-Here we will discuss the test description for both
-baremetal and openstack.
+Here we will discuss the test description for
+baremetal, openstack and standalone.
-*Test Description File for Baremetal*
--------------------------------------
+Test Description File for Baremetal
+-----------------------------------
This section will introduce the meaning of the Test case description
file. We will use ``tc_prox_baremetal_l2fwd-2.yaml`` as an example to
@@ -235,7 +240,8 @@ show you how to understand the test description file.
Now let's examine the components of the file in detail
1. ``traffic_profile`` - This specifies the traffic profile for the
- test. In this case ``prox_binsearch.yaml`` is used. See nsb-traffic-profile-label_
+ test. In this case ``prox_binsearch.yaml`` is used. See
+ `Traffic Profile File`_
2. ``topology`` - This is either ``prox-tg-topology-1.yaml`` or
``prox-tg-topology-2.yaml`` or ``prox-tg-topology-4.yaml``
@@ -244,10 +250,16 @@ Now let's examine the components of the file in detail
3. ``nodes`` - This names the Traffic Generator and the System
under Test. Does not need to change.
-4. ``prox_path`` - Location of the Prox executable on the traffic
+4. ``interface_speed_gbps`` - This is an optional parameter. If not present
+ the system defaults to 10Gbps. This defines the speed of the interfaces.
+
+5. ``collectd`` - (Optional) This specifies we want to collect NFVI statistics
+ like CPU Utilization,
+
+6. ``prox_path`` - Location of the Prox executable on the traffic
generator (Either baremetal or Openstack Virtual Machine)
-5. ``prox_config`` - This is the ``SUT Config File``.
+7. ``prox_config`` - This is the ``SUT Config File``.
In this case it is ``handle_l2fwd-2.cfg``
A number of additional parameters can be added. This example
@@ -256,6 +268,14 @@ Now let's examine the components of the file in detail
options:
interface_speed_gbps: 10
+ traffic_config:
+ tolerated_loss: 0.01
+ test_precision: 0.01
+ packet_sizes: [64]
+ duration: 30
+ lower_bound: 0.0
+ upper_bound: 100.0
+
vnf__0:
prox_path: /opt/nsb_bin/prox
prox_config: ``configs/handle_vpe-4.cfg``
@@ -269,53 +289,75 @@ Now let's examine the components of the file in detail
``configs/vpe_rules.lua`` : ````
prox_generate_parameter: True
- ``interface_speed_gbps`` - this specifies the speed of the interface
- in Gigabits Per Second. This is used to calculate pps(packets per second).
- If the interfaces are of different speeds, then this specifies the speed
- of the slowest interface. This parameter is optional. If omitted the
- interface speed defaults to 10Gbps.
+ ``interface_speed_gbps`` - this specifies the speed of the interface
+ in Gigabits Per Second. This is used to calculate pps(packets per second).
+ If the interfaces are of different speeds, then this specifies the speed
+ of the slowest interface. This parameter is optional. If omitted the
+ interface speed defaults to 10Gbps.
+
+ ``traffic_config`` - This allows the values here to override the values in
+ in the traffic_profile file. e.g. "prox_binsearch.yaml". Values provided
+ here override values provided in the "traffic_profile" section of the
+ traffic_profile file. Some, all or none of the values can be provided here.
- ``prox_files`` - this specified that a number of addition files
- need to be provided for the test to run correctly. This files
- could provide routing information,hashing information or a
- hashing algorithm and ip/mac information.
+ The values describes the packet size, tolerated loss, initial line rate
+ to start traffic at, test interval etc See `Traffic Profile File`_
- ``prox_generate_parameter`` - this specifies that the NSB application
- is required to provide information to the nsb Prox in the form
- of a file called ``parameters.lua``, which contains information
- retrieved from either the hardware or the openstack configuration.
+ ``prox_files`` - this specified that a number of addition files
+ need to be provided for the test to run correctly. This files
+ could provide routing information,hashing information or a
+ hashing algorithm and ip/mac information.
-6. ``prox_args`` - this specifies the command line arguments to start
+ ``prox_generate_parameter`` - this specifies that the NSB application
+ is required to provide information to the nsb Prox in the form
+ of a file called ``parameters.lua``, which contains information
+ retrieved from either the hardware or the openstack configuration.
+
+8. ``prox_args`` - this specifies the command line arguments to start
prox. See `prox command line`_.
-7. ``prox_config`` - This specifies the Traffic Generator config file.
+9. ``prox_config`` - This specifies the Traffic Generator config file.
+
+10. ``runner`` - This is set to ``ProxDuration`` - This specifies that the
+ test runs for a set duration. Other runner types are available
+ but it is recommend to use ``ProxDuration``. The following parameters
+ are supported
+
+ ``interval`` - (optional) - This specifies the sampling interval.
+ Default is 1 sec
+
+ ``sampled`` - (optional) - This specifies if sampling information is
+ required. Default ``no``
+
+ ``duration`` - This is the length of the test in seconds. Default
+ is 60 seconds.
+
+ ``confirmation`` - This specifies the number of confirmation retests to
+ be made before deciding to increase or decrease line speed. Default 0.
-8. ``runner`` - This is set to ``Duration`` - This specified that the
- test run for a set duration. Other runner types are available
- but it is recommend to use ``Duration``
+11. ``context`` - This is ``context`` for a 2 port Baremetal configuration.
-9. ``context`` - This is ``context`` for a 2 port Baremetal configuration.
If a 4 port configuration was required then file
``prox-baremetal-4.yaml`` would be used. This is the NSB Prox
baremetal configuration file.
-.. _nsb-traffic-profile-label:
+Traffic Profile File
+--------------------
-*Traffic Profile file*
-----------------------
-
-This describes the details of the traffic flow. In this case ``prox_binsearch.yaml`` is used.
+This describes the details of the traffic flow. In this case
+``prox_binsearch.yaml`` is used.
.. image:: images/PROX_Traffic_profile.png
:width: 800px
:alt: NSB PROX Traffic Profile
-1. ``name`` - The name of the traffic profile. This name should match the name specified in the
- ``traffic_profile`` field in the Test Description File.
+1. ``name`` - The name of the traffic profile. This name should match the
+ name specified in the ``traffic_profile`` field in the Test
+ Description File.
-2. ``traffic_type`` - This specifies the type of traffic pattern generated, This name matches
- class name of the traffic generator See::
+2. ``traffic_type`` - This specifies the type of traffic pattern generated,
+ This name matches class name of the traffic generator. See::
network_services/traffic_profile/prox_binsearch.py class ProxBinSearchProfile(ProxProfile)
@@ -326,21 +368,29 @@ This describes the details of the traffic flow. In this case ``prox_binsearch.ya
Custom traffic types can be created by creating a new traffic profile class.
-3. ``tolerated_loss`` - This specifies the percentage of packets that can be lost/dropped before
- we declare success or failure. Success is Transmitted-Packets from Traffic Generator is greater than or equal to
+3. ``tolerated_loss`` - This specifies the percentage of packets that
+ can be lost/dropped before
+ we declare success or failure. Success is Transmitted-Packets from
+ Traffic Generator is greater than or equal to
packets received by Traffic Generator plus tolerated loss.
-4. ``test_precision`` - This specifies the precision of the test results. For some tests the success criteria
- may never be achieved because the test precision may be greater than the successful throughput. For finer
- results increase the precision by making this value smaller.
+4. ``test_precision`` - This specifies the precision of the test
+ results. For some tests the success criteria may never be
+ achieved because the test precision may be greater than the
+ successful throughput. For finer results increase the precision
+ by making this value smaller.
-5. ``packet_sizes`` - This specifies the range of packets size this test is run for.
+5. ``packet_sizes`` - This specifies the range of packets size this
+ test is run for.
-6. ``duration`` - This specifies the sample duration that the test uses to check for success or failure.
+6. ``duration`` - This specifies the sample duration that the test
+ uses to check for success or failure.
-7. ``lower_bound`` - This specifies the test initial lower bound sample rate. On success this value is increased.
+7. ``lower_bound`` - This specifies the test initial lower bound sample rate.
+ On success this value is increased.
-8. ``upper_bound`` - This specifies the test initial upper bound sample rate. On success this value is decreased.
+8. ``upper_bound`` - This specifies the test initial upper bound sample rate.
+ On success this value is decreased.
Other traffic profiles exist eg prox_ACL.yaml which does not
compare what is received with what is transmitted. It just
@@ -365,20 +415,25 @@ See this prox_vpe.yaml as example::
lower_bound: 0.0
upper_bound: 100.0
-*Test Description File for Openstack*
--------------------------------------
+Test Description File for Openstack
+-----------------------------------
We will use ``tc_prox_heat_context_l2fwd-2.yaml`` as a example to show
you how to understand the test description file.
-.. image:: images/PROX_Test_HEAT_Script.png
- :width: 800px
- :alt: NSB PROX Test Description File
+ .. image:: images/PROX_Test_HEAT_Script1.png
+ :width: 800px
+ :alt: NSB PROX Test Description File - Part 1
+
+
+ .. image:: images/PROX_Test_HEAT_Script2.png
+ :width: 800px
+ :alt: NSB PROX Test Description File - Part 2
Now lets examine the components of the file in detail
-Sections 1 to 8 are exactly the same in Baremetal and in Heat. Section
-``9`` is replaced with sections A to F. Section 9 was for a baremetal
+Sections 1 to 9 are exactly the same in Baremetal and in Heat. Section
+``10`` is replaced with sections A to F. Section 10 was for a baremetal
configuration file. This has no place in a heat configuration.
A. ``image`` - yardstick-samplevnfs. This is the name of the image
@@ -418,21 +473,75 @@ F. ``networks`` - is composed of a management network labeled ``mgmt``
gateway_ip: 'null'
port_security_enabled: False
enable_dhcp: 'false'
- downlink_1:
+ uplink_1:
cidr: '10.0.4.0/24'
gateway_ip: 'null'
port_security_enabled: False
enable_dhcp: 'false'
- downlink_2:
+ downlink_1:
cidr: '10.0.5.0/24'
gateway_ip: 'null'
port_security_enabled: False
enable_dhcp: 'false'
-.. _nsb-traffic-generator-label:
+Test Description File for Standalone
+------------------------------------
+
+We will use ``tc_prox_ovs-dpdk_l2fwd-2.yaml`` as a example to show
+you how to understand the test description file.
+
+ .. image:: images/PROX_Test_ovs_dpdk_Script_1.png
+ :width: 800px
+ :alt: NSB PROX Test Standalone Description File - Part 1
-*Traffic Generator Config file*
--------------------------------
+ .. image:: images/PROX_Test_ovs_dpdk_Script_2.png
+ :width: 800px
+ :alt: NSB PROX Test Standalone Description File - Part 2
+
+Now lets examine the components of the file in detail
+
+Sections 1 to 9 are exactly the same in Baremetal and in Heat. Section
+``10`` is replaced with sections A to F. Section 10 was for a baremetal
+configuration file. This has no place in a heat configuration.
+
+A. ``file`` - Pod file for Baremetal Traffic Generator configuration:
+ IP Address, User/Password & Interfaces
+
+B. ``type`` - This defines the type of standalone configuration.
+ Possible values are ``StandaloneOvsDpdk`` or ``StandaloneSriov``
+
+C. ``file`` - Pod file for Standalone host configuration:
+ IP Address, User/Password & Interfaces
+
+D. ``vm_deploy`` - Deploy a new VM or use an existing VM
+
+E. ``ovs_properties`` - OVS Version, DPDK Version and configuration
+ to use.
+
+F. ``flavor``- NSB image generated when installing NSB using ansible-playbook::
+
+ ram- Configurable RAM for SUT VM
+ extra_specs
+ hw:cpu_sockets - Configurable number of Sockets for SUT VM
+ hw:cpu_cores - Configurable number of Cores for SUT VM
+ hw:cpu_threads- Configurable number of Threads for SUT VM
+
+G. ``mgmt`` - Management port of the SUT VM. Preconfig needed on TG & SUT host machines.
+ is the system under test.
+
+
+H. ``xe0`` - Upline Network port
+
+I. ``xe1`` - Downline Network port
+
+J. ``uplink_0`` - Uplink Phy port of the NIC on the host. This will be used to create
+ the Virtual Functions.
+
+K. ``downlink_0`` - Downlink Phy port of the NIC on the host. This will be used to
+ create the Virtual Functions.
+
+Traffic Generator Config file
+-----------------------------
This section will describe the traffic generator config file.
This is the same for both baremetal and heat. See this example
@@ -673,23 +782,29 @@ Now let's examine the components of the file in detail
physical core improves performance, however sometimes it
is optimal to move task to a separate core. This is
best decided by checking performance.
- c. ``mode=lat`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
- classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
- police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
- nat, decapnsh, encapnsh, gen, genl4 and lat. This task(0) per core(3) receives packets on port.
- d. ``rx port=p0`` - The port to receive packets on ``Port 0``. Core 4 will receive packets on ``Port 1``.
- e. ``lat pos=42`` - Describes where to put a 4-byte timestamp in the packet. Note that the packet length should
- be longer than ``lat pos`` + 4 bytes to avoid truncation of the timestamp. It defines where the timestamp is
- to be read from. Note that the SUT workload might cause the position of the timestamp to change
- (i.e. due to encapsulation).
-
-.. _nsb-sut-generator-label:
-
-*SUT Config file*
--------------------------------
+ c. ``mode=lat`` - Specifies the action carried out by this task on this
+ core.
+ Supported modes are: ``acl``, ``classify``, ``drop``, ``gredecap``,
+ ``greencap``, ``ipv6_decap``, ``ipv6_encap``, ``l2fwd``, ``lbnetwork``,
+ ``lbpos``, ``lbqinq``, ``nop``, ``police``, ``qinqdecapv4``,
+ ``qinqencapv4``, ``qos``, ``routing``, ``impair``, ``lb5tuple``,
+ ``mirror``, ``unmpls``, ``tagmpls``, ``nat``, ``decapnsh``, ``encapnsh``,
+ ``gen``, ``genl4`` and ``lat``. This task(0) per core(3) receives packets
+ on port.
+ d. ``rx port=p0`` - The port to receive packets on ``Port 0``. Core 4 will
+ receive packets on ``Port 1``.
+ e. ``lat pos=42`` - Describes where to put a 4-byte timestamp in the packet.
+ Note that the packet length should be longer than ``lat pos`` + 4 bytes
+ to avoid truncation of the timestamp. It defines where the timestamp is
+ to be read from. Note that the SUT workload might cause the position of
+ the timestamp to change (i.e. due to encapsulation).
+
+SUT Config File
+---------------
This section will describes the SUT(VNF) config file. This is the same for both
-baremetal and heat. See this example of ``handle_l2fwd_multiflow-2.cfg`` to explain the options.
+baremetal and heat. See this example of ``handle_l2fwd_multiflow-2.cfg`` to
+explain the options.
.. image:: images/PROX_Handle_2port_cfg.png
:width: 1400px
@@ -699,13 +814,15 @@ See `prox options`_ for details
Now let's examine the components of the file in detail
-1. ``[eal options]`` - same as the Generator config file. This specified the EAL (Environmental Abstraction Layer)
- options. These are default values and are not changed.
- See `dpdk wiki page`_.
+1. ``[eal options]`` - same as the Generator config file. This specified the
+ EAL (Environmental Abstraction Layer) options. These are default values and
+ are not changed. See `dpdk wiki page`_.
-2. ``[port 0]`` - This section describes the DPDK Port. The number following the keyword ``port`` usually refers to the DPDK Port Id. usually starting from ``0``.
- Because you can have multiple ports this entry usually repeated. Eg. For a 2 port setup ``[port0]`` and ``[port 1]`` and for a 4 port setup ``[port 0]``, ``[port 1]``,
- ``[port 2]`` and ``[port 3]``::
+2. ``[port 0]`` - This section describes the DPDK Port. The number following
+ the keyword ``port`` usually refers to the DPDK Port Id. usually starting
+ from ``0``. Because you can have multiple ports this entry usually
+ repeated. E.g. For a 2 port setup ``[port0]`` and ``[port 1]`` and for a 4
+ port setup ``[port 0]``, ``[port 1]``, ``[port 2]`` and ``[port 3]``::
[port 0]
name=if0
@@ -714,10 +831,14 @@ Now let's examine the components of the file in detail
tx desc=2048
promiscuous=yes
- a. In this example ``name =if0`` assigned the name ``if0`` to the port. Any name can be assigned to a port.
- b. ``mac=hardware`` sets the MAC address assigned by the hardware to data from this port.
- c. ``rx desc=2048`` sets the number of available descriptors to allocate for receive packets. This can be changed and can effect performance.
- d. ``tx desc=2048`` sets the number of available descriptors to allocate for transmit packets. This can be changed and can effect performance.
+ a. In this example ``name =if0`` assigned the name ``if0`` to the port. Any
+ name can be assigned to a port.
+ b. ``mac=hardware`` sets the MAC address assigned by the hardware to data
+ from this port.
+ c. ``rx desc=2048`` sets the number of available descriptors to allocate
+ for receive packets. This can be changed and can effect performance.
+ d. ``tx desc=2048`` sets the number of available descriptors to allocate
+ for transmit packets. This can be changed and can effect performance.
e. ``promiscuous=yes`` this enables promiscuous mode for this port.
3. ``[defaults]`` - Here default operations and settings can be over written.::
@@ -726,35 +847,46 @@ Now let's examine the components of the file in detail
mempool size=8K
memcache size=512
- a. In this example ``mempool size=8K`` the number of mbufs per task is altered. Altering this value could effect performance. See `prox options`_ for details.
- b. ``memcache size=512`` - number of mbufs cached per core, default is 256 this is the cache_size. Altering this value could effect performance.
+ a. In this example ``mempool size=8K`` the number of mbufs per task is
+ altered. Altering this value could effect performance. See
+ `prox options`_ for details.
+ b. ``memcache size=512`` - number of mbufs cached per core, default is 256
+ this is the cache_size. Altering this value could affect performance.
-4. ``[global]`` - Here application wide setting are supported. Things like application name, start time, duration and memory configurations can be set here.
+4. ``[global]`` - Here application wide setting are supported. Things like
+ application name, start time, duration and memory configurations can be set
+ here.
In this example.::
[global]
start time=5
name=Basic Gen
- a. ``start time=5`` Time is seconds after which average stats will be started.
+ a. ``start time=5`` Time is seconds after which average stats will be
+ started.
b. ``name=Handle L2FWD Multiflow (2x)`` Name of the configuration.
-5. ``[core 0]`` - This core is designated the master core. Every Prox application must have a master core. The master mode must be assigned to
+5. ``[core 0]`` - This core is designated the master core. Every Prox
+ application must have a master core. The master mode must be assigned to
exactly one task, running alone on one core.::
[core 0]
mode=master
-6. ``[core 1]`` - This describes the activity on core 1. Cores can be configured by means of a set of [core #] sections, where # represents either:
+6. ``[core 1]`` - This describes the activity on core 1. Cores can be
+ configured by means of a set of [core #] sections, where # represents
+ either:
- a. an absolute core number: e.g. on a 10-core, dual socket system with hyper-threading,
- cores are numbered from 0 to 39.
+ a. an absolute core number: e.g. on a 10-core, dual socket system with
+ hyper-threading, cores are numbered from 0 to 39.
- b. PROX allows a core to be identified by a core number, the letter 's', and a socket number.
- However NSB PROX is hardware agnostic (physical and virtual configurations are the same) it
- is advisable no to use physical core numbering.
+ b. PROX allows a core to be identified by a core number, the letter 's',
+ and a socket number. However NSB PROX is hardware agnostic (physical and
+ virtual configurations are the same) it is advisable no to use physical
+ core numbering.
- Each core can be assigned with a set of tasks, each running one of the implemented packet processing modes.::
+ Each core can be assigned with a set of tasks, each running one of the
+ implemented packet processing modes.::
[core 1]
name=none
@@ -765,20 +897,33 @@ Now let's examine the components of the file in detail
tx port=if1
a. ``name=none`` - No name assigned to the core.
- b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``. Task 1 can be defined later in this core or
- can be defined in another ``[core 1]`` section with ``task=1`` later in configuration file. Sometimes running
- multiple task related to the same packet on the same physical core improves performance, however sometimes it
- is optimal to move task to a separate core. This is best decided by checking performance.
- c. ``mode=l2fwd`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
- classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
- police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
- nat, decapnsh, encapnsh, gen, genl4 and lat. This code does ``l2fwd`` .. ie it does the L2FWD.
-
- d. ``dst mac=@@tester_mac1`` - The destination mac address of the packet will be set to the MAC address of ``Port 1`` of destination device. (The Traffic Generator/Verifier)
- e. ``rx port=if0`` - This specifies that the packets are received from ``Port 0`` called if0
- f. ``tx port=if1`` - This specifies that the packets are transmitted to ``Port 1`` called if1
-
- If this example we receive a packet on core on a port, carry out operation on the packet on the core and transmit it on on another port still using the same task on the same core.
+ b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``.
+ Task 1 can be defined later in this core or can be defined in another
+ ``[core 1]`` section with ``task=1`` later in configuration file.
+ Sometimes running multiple task related to the same packet on the same
+ physical core improves performance, however sometimes it is optimal to
+ move task to a separate core. This is best decided by checking
+ performance.
+ c. ``mode=l2fwd`` - Specifies the action carried out by this task on this
+ core. Supported modes are: ``acl``, ``classify``, ``drop``,
+ ``gredecap``, ``greencap``, ``ipv6_decap``, ``ipv6_encap``, ``l2fwd``,
+ ``lbnetwork``, ``lbpos``, ``lbqinq``, ``nop``, ``police``,
+ ``qinqdecapv4``, ``qinqencapv4``, ``qos``, ``routing``, ``impair``,
+ ``lb5tuple``, ``mirror``, ``unmpls``, ``tagmpls``, ``nat``,
+ ``decapnsh``, ``encapnsh``, ``gen``, ``genl4`` and ``lat``. This code
+ does ``l2fwd``. i.e. it does the L2FWD.
+
+ d. ``dst mac=@@tester_mac1`` - The destination mac address of the packet
+ will be set to the MAC address of ``Port 1`` of destination device.
+ (The Traffic Generator/Verifier)
+ e. ``rx port=if0`` - This specifies that the packets are received from
+ ``Port 0`` called if0
+ f. ``tx port=if1`` - This specifies that the packets are transmitted to
+ ``Port 1`` called if1
+
+ In this example we receive a packet on core on a port, carry out operation
+ on the packet on the core and transmit it on on another port still using
+ the same task on the same core.
On some implementation you may wish to use multiple tasks, like this.::
@@ -798,59 +943,76 @@ Now let's examine the components of the file in detail
tx port=if0
drop=no
- In this example you can see Core 1/Task 0 called ``rx_task`` receives the packet from if0 and perform the l2fwd. However instead of sending the packet to a
- port it sends it to a core see ``tx cores=1t1``. In this case it sends it to Core 1/Task 1.
+ In this example you can see Core 1/Task 0 called ``rx_task`` receives the
+ packet from if0 and perform the l2fwd. However instead of sending the
+ packet to a port it sends it to a core see ``tx cores=1t1``. In this case it
+ sends it to Core 1/Task 1.
- Core 1/Task 1 called ``l2fwd_if0``, receives the packet, not from a port but from the ring. See ``rx ring=yes``. It does not perform any operation on the packet See ``mode=none``
- and sends the packets to ``if0`` see ``tx port=if0``.
+ Core 1/Task 1 called ``l2fwd_if0``, receives the packet, not from a port but
+ from the ring. See ``rx ring=yes``. It does not perform any operation on the
+ packet See ``mode=none`` and sends the packets to ``if0`` see
+ ``tx port=if0``.
- It is also possible to implement more complex operations be chaining multiple operations in sequence and using rings to pass packets from one core to another.
+ It is also possible to implement more complex operations by chaining
+ multiple operations in sequence and using rings to pass packets from one
+ core to another.
- In thus example we show a Broadband Network Gateway (BNG) with Quality of Service (QoS). Communication from task to task is via rings.
+ In this example, we show a Broadband Network Gateway (BNG) with Quality of
+ Service (QoS). Communication from task to task is via rings.
.. image:: images/PROX_BNG_QOS.png
:width: 1000px
:alt: NSB PROX Config File for BNG_QOS
-*Baremetal Configuration file*
-------------------------------
-
-.. _baremetal-config-label:
+Baremetal Configuration File
+----------------------------
-This is required for baremetal testing. It describes the IP address of the various ports, the Network devices drivers and MAC addresses and the network
+This is required for baremetal testing. It describes the IP address of the
+various ports, the Network devices drivers and MAC addresses and the network
configuration.
-In this example we will describe a 2 port configuration. This file is the same for all 2 port NSB Prox tests on the same platforms/configuration.
+In this example we will describe a 2 port configuration. This file is the same
+for all 2 port NSB Prox tests on the same platforms/configuration.
.. image:: images/PROX_Baremetal_config.png
:width: 1000px
:alt: NSB PROX Yardstick Config
-Now lets describe the sections of the file.
-
- 1. ``TrafficGen`` - This section describes the Traffic Generator node of the test configuration. The name of the node ``trafficgen_1`` must match the node name
- in the ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
- can remain as default settings.
- 2. ``interfaces`` - This defines the DPDK interfaces on the Traffic Generator.
- 3. ``xe0`` is DPDK Port 0. ``lspci`` and `` ./dpdk-devbind.py -s`` can be used to provide the interface information. ``netmask`` and ``local_ip`` should not be changed
- 4. ``xe1`` is DPDK Port 1. If more than 2 ports are required then ``xe1`` section needs to be repeated and modified accordingly.
- 5. ``vnf`` - This section describes the SUT of the test configuration. The name of the node ``vnf`` must match the node name in the
- ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
- can remain as default settings
+Now let's describe the sections of the file.
+
+ 1. ``TrafficGen`` - This section describes the Traffic Generator node of the
+ test configuration. The name of the node ``trafficgen_1`` must match the
+ node name in the ``Test Description File for Baremetal`` mentioned
+ earlier. The password attribute of the test needs to be configured. All
+ other parameters can remain as default settings.
+ 2. ``interfaces`` - This defines the DPDK interfaces on the Traffic
+ Generator.
+ 3. ``xe0`` is DPDK Port 0. ``lspci`` and ``./dpdk-devbind.py -s`` can be used
+ to provide the interface information. ``netmask`` and ``local_ip`` should
+ not be changed
+ 4. ``xe1`` is DPDK Port 1. If more than 2 ports are required then ``xe1``
+ section needs to be repeated and modified accordingly.
+ 5. ``vnf`` - This section describes the SUT of the test configuration. The
+ name of the node ``vnf`` must match the node name in the
+ ``Test Description File for Baremetal`` mentioned earlier. The password
+ attribute of the test needs to be configured. All other parameters can
+ remain as default settings
6. ``interfaces`` - This defines the DPDK interfaces on the SUT
7. ``xe0`` - Same as 3 but for the ``SUT``.
8. ``xe1`` - Same as 4 but for the ``SUT`` also.
9. ``routing_table`` - All parameters should remain unchanged.
10. ``nd_route_tbl`` - All parameters should remain unchanged.
-*Grafana Dashboard*
--------------------
+Grafana Dashboard
+-----------------
-The grafana dashboard visually displays the results of the tests. The steps required to produce a grafana dashboard are described here.
+The grafana dashboard visually displays the results of the tests. The steps
+required to produce a grafana dashboard are described here.
.. _yardstick-config-label:
- a. Configure ``yardstick`` to use influxDB to store test results. See file ``/etc/yardstick/yardstick.conf``.
+ a. Configure ``yardstick`` to use influxDB to store test results. See file
+ ``/etc/yardstick/yardstick.conf``.
.. image:: images/PROX_Yardstick_config.png
:width: 1000px
@@ -859,10 +1021,12 @@ The grafana dashboard visually displays the results of the tests. The steps requ
1. Specify the dispatcher to use influxDB to store results.
2. "target = .. " - Specify location of influxDB to store results.
"db_name = yardstick" - name of database. Do not change
- "username = root" - username to use to store result. (Many tests are run as root)
+ "username = root" - username to use to store result. (Many tests are
+ run as root)
"password = ... " - Please set to root user password
- b. Deploy InfludDB & Grafana. See how to Deploy InfluxDB & Grafana. See `grafana deployment`_.
+ b. Deploy InfludDB & Grafana. See how to Deploy InfluxDB & Grafana. See
+ `grafana deployment`_.
c. Generate the test data. Run the tests as follows .::
yardstick --debug task start tc_prox_<context>_<test>-ports.yaml
@@ -879,7 +1043,8 @@ How to run NSB Prox Test on an baremetal environment
In order to run the NSB PROX test.
- 1. Install NSB on Traffic Generator node and Prox in SUT. See `NSB Installation`_
+ 1. Install NSB on Traffic Generator node and Prox in SUT. See
+ `NSB Installation`_
2. To enter container::
@@ -891,8 +1056,8 @@ In order to run the NSB PROX test.
cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox
- b. Install prox-baremetal-2.yam and prox-baremetal-4.yaml for that topology
- into this directory as per baremetal-config-label_
+ b. Install prox-baremetal-2.yam and prox-baremetal-4.yaml for that
+ topology into this directory as per `Baremetal Configuration File`_
c. Install and configure ``yardstick.conf`` ::
@@ -937,12 +1102,11 @@ Frequently Asked Questions
Here is a list of frequently asked questions.
-*NSB Prox does not work on Baremetal, How do I resolve this?*
--------------------------------------------------------------
+NSB Prox does not work on Baremetal, How do I resolve this?
+-----------------------------------------------------------
-If PROX NSB does not work on baremetal, problem is either in network configuration or test file.
-
-*Solution*
+If PROX NSB does not work on baremetal, problem is either in network
+configuration or test file.
1. Verify network configuration. Execute existing baremetal test.::
@@ -980,13 +1144,11 @@ If PROX NSB does not work on baremetal, problem is either in network configurati
See ``Link detected`` if ``yes`` .... Cable is good. If ``no`` you have an issue with your cable/port.
-2. If existing baremetal works then issue is with your test. Check the traffic generator gen_<test>-<ports>.cfg to ensure
- it is producing a valid packet.
-
-*How do I debug NSB Prox on Baremetal?*
----------------------------------------
+2. If existing baremetal works then issue is with your test. Check the traffic
+ generator gen_<test>-<ports>.cfg to ensure it is producing a valid packet.
-*Solution*
+How do I debug NSB Prox on Baremetal?
+-------------------------------------
1. Execute the test as follows::
@@ -1002,7 +1164,8 @@ If PROX NSB does not work on baremetal, problem is either in network configurati
cd
/opt/nsb_bin/prox -f /tmp/handle_<test>-<ports>.cfg
-4. Now let's examine the Generator Output. In this case the output of gen_l2fwd-4.cfg.
+4. Now let's examine the Generator Output. In this case the output of
+ ``gen_l2fwd-4.cfg``.
.. image:: images/PROX_Gen_GUI.png
:width: 1000px
@@ -1017,10 +1180,12 @@ If PROX NSB does not work on baremetal, problem is either in network configurati
It appears what is transmitted is received.
.. Caution::
- The number of packets MAY not exactly match because the ports are read in sequence.
+ The number of packets MAY not exactly match because the ports are read in
+ sequence.
.. Caution::
- What is transmitted on PORT X may not always be received on same port. Please check the Test scenario.
+ What is transmitted on PORT X may not always be received on same port.
+ Please check the Test scenario.
5. Now lets examine the SUT Output
@@ -1033,7 +1198,7 @@ If PROX NSB does not work on baremetal, problem is either in network configurati
1. What is received on 0 is transmitted on 1, received on 1 transmitted on 0,
received on 2 transmitted on 3 and received on 3 transmitted on 2.
2. No packets are Failed.
- 3. No Packets are discarded.
+ 3. No packets are discarded.
We can also dump the packets being received or transmitted via the following commands. ::
@@ -1049,26 +1214,24 @@ If PROX NSB does not work on baremetal, problem is either in network configurati
dump_tx 1 0 1
-*NSB Prox works on Baremetal but not in Openstack. How do I resolve this?*
---------------------------------------------------------------------------
+NSB Prox works on Baremetal but not in Openstack. How do I resolve this?
+------------------------------------------------------------------------
-NSB Prox on Baremetal is a lot more forgiving than NSB Prox on Openstack. A badly
-formed packed may still work with PROX on Baremetal. However on
+NSB Prox on Baremetal is a lot more forgiving than NSB Prox on Openstack. A
+badly formed packed may still work with PROX on Baremetal. However on
Openstack the packet must be correct and all fields of the header correct.
-Eg A packet with an invalid Protocol ID would still work in Baremetal
-but this packet would be rejected by openstack.
+E.g. A packet with an invalid Protocol ID would still work in Baremetal but
+this packet would be rejected by openstack.
-*Solution*
1. Check the validity of the packet.
2. Use a known good packet in your test
- 3. If using ``Random`` fields in the traffic generator, disable them and retry.
+ 3. If using ``Random`` fields in the traffic generator, disable them and
+ retry.
-*How do I debug NSB Prox on Openstack?*
----------------------------------------
-
-*Solution*
+How do I debug NSB Prox on Openstack?
+-------------------------------------
1. Execute the test as follows::
@@ -1080,7 +1243,8 @@ but this packet would be rejected by openstack.
3. Install openstack credentials.
- Depending on your openstack deployment, the location of these credentials may vary.
+ Depending on your openstack deployment, the location of these credentials
+ may vary.
On this platform I do this via::
scp root@10.237.222.55:/etc/kolla/admin-openrc.sh .
@@ -1096,8 +1260,8 @@ but this packet would be rejected by openstack.
b. Get the Floating IP of the Traffic Generator & SUT
- This generates a lot of information. Please not the floating IP of the VNF and
- the Traffic Generator.
+ This generates a lot of information. Please note the floating IP of the
+ VNF and the Traffic Generator.
.. image:: images/PROX_Openstack_stack_show_a.png
:width: 1000px
@@ -1138,10 +1302,8 @@ but this packet would be rejected by openstack.
Now continue as baremetal.
-*How do I resolve "Quota exceeded for resources"*
--------------------------------------------------
-
-*Solution*
+How do I resolve "Quota exceeded for resources"
+-----------------------------------------------
This usually occurs due to 2 reasons when executing an openstack test.
@@ -1175,16 +1337,15 @@ This usually occurs due to 2 reasons when executing an openstack test.
openstack quota set <resource>
-*Openstack Cli fails or hangs. How do I resolve this?*
-------------------------------------------------------
-
-*Solution*
+Openstack CLI fails or hangs. How do I resolve this?
+----------------------------------------------------
If it fails due to ::
Missing value auth-url required for auth plugin password
-Check your shell environment for Openstack variables. One of them should contain the authentication URL ::
+Check your shell environment for Openstack variables. One of them should
+contain the authentication URL ::
OS_AUTH_URL=``https://192.168.72.41:5000/v3``
@@ -1208,16 +1369,16 @@ Result ::
and visible.
-If the Openstack Cli appears to hang, then verify the proxys and no_proxy are set correctly.
-They should be similar to ::
+If the Openstack CLI appears to hang, then verify the proxys and ``no_proxy``
+are set correctly. They should be similar to ::
- FTP_PROXY="http://proxy.ir.intel.com:911/"
- HTTPS_PROXY="http://proxy.ir.intel.com:911/"
- HTTP_PROXY="http://proxy.ir.intel.com:911/"
+ FTP_PROXY="http://<your_proxy>:<port>/"
+ HTTPS_PROXY="http://<your_proxy>:<port>/"
+ HTTP_PROXY="http://<your_proxy>:<port>/"
NO_PROXY="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"
- ftp_proxy="http://proxy.ir.intel.com:911/"
- http_proxy="http://proxy.ir.intel.com:911/"
- https_proxy="http://proxy.ir.intel.com:911/"
+ ftp_proxy="http://<your_proxy>:<port>/"
+ http_proxy="http://<your_proxy>:<port>/"
+ https_proxy="http://<your_proxy>:<port>/"
no_proxy="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"
Where
@@ -1225,10 +1386,95 @@ Where
1) 10.237.222.55 = IP Address of deployment node
2) 10.237.223.80 = IP Address of Controller node
3) 10.237.222.134 = IP Address of Compute Node
- 4) ir.intel.com = local no proxy
+How to Understand the Grafana output?
+-------------------------------------
+
+ .. image:: images/PROX_Grafana_1.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_1
+
+ .. image:: images/PROX_Grafana_2.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_2
+
+ .. image:: images/PROX_Grafana_3.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_3
+
+ .. image:: images/PROX_Grafana_4.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_4
+
+ .. image:: images/PROX_Grafana_5.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_5
+
+ .. image:: images/PROX_Grafana_6.png
+ :width: 1000px
+ :alt: NSB PROX Grafana_6
+
+A. Test Parameters - Test interval, Duration, Tolerated Loss and Test Precision
+
+B. No. of packets send and received during test
+
+C. Generator Stats - Average Throughput per step (Step Duration is specified by
+ "Duration" field in A above)
+
+D. Packet size
+
+E. No. of packets sent by the generator per second per interface in millions
+ of packets per second.
+
+F. No. of packets recieved by the generator per second per interface in millions
+ of packets per second.
+
+G. No. of packets received by the SUT from the generator in millions of packets
+ per second.
+
+H. No. of packets sent by the the SUT to the generator in millions of packets
+ per second.
+
+I. No. of packets sent by the Generator to the SUT per step per interface
+ in millions of packets per second.
+
+J. No. of packets received by the Generator from the SUT per step per interface
+ in millions of packets per second.
+
+K. No. of packets sent and received by the generator and lost by the SUT that
+ meet the success criteria
+
+L. The change in the Percentage of Line Rate used over a test, The MAX and the
+ MIN should converge to within the interval specified as the
+ ``test-precision``.
+
+M. Packet size supported during test. If *N/A* appears in any field the
+ result has not been decided.
+
+N. The Theretical Maximum no. of packets per second that can be sent for this
+ packet size.
+
+O. No. of packets sent by the generator in MPPS
+
+P. No. of packets received by the generator in MPPS
+
+Q. No. of packets sent by SUT.
+
+R. No. of packets received by the SUT
+S. Total no. of dropped packets -- Packets sent but not received back by the
+ generator, these may be dropped by the SUT or the generator.
+T. The tolerated no. of dropped packets.
+U. Test throughput in Gbps
+V. Latencey per Port
+ * Va - Port XE0
+ * Vb - Port XE1
+ * Vc - Port XE0
+ * Vd - Port XE0
+W. CPU Utilization
+ * Wa - CPU Utilization of the Generator
+ * Wb - CPU Utilization of the SUT
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_1.png b/docs/testing/developer/devguide/images/PROX_Grafana_1.png
new file mode 100644
index 000000000..144000bb8
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_1.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_2.png b/docs/testing/developer/devguide/images/PROX_Grafana_2.png
new file mode 100644
index 000000000..af1ebb315
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_2.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_3.png b/docs/testing/developer/devguide/images/PROX_Grafana_3.png
new file mode 100644
index 000000000..a287670c9
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_3.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_4.png b/docs/testing/developer/devguide/images/PROX_Grafana_4.png
new file mode 100644
index 000000000..6752dc324
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_4.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_5.png b/docs/testing/developer/devguide/images/PROX_Grafana_5.png
new file mode 100644
index 000000000..45746d86b
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_5.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Grafana_6.png b/docs/testing/developer/devguide/images/PROX_Grafana_6.png
new file mode 100644
index 000000000..5bdbcf26a
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Grafana_6.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png b/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png
index 32530eb15..84e640a20 100644
--- a/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png
+++ b/docs/testing/developer/devguide/images/PROX_Test_BM_Script.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script.png b/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script.png
deleted file mode 100644
index 754973b4e..000000000
--- a/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script.png
+++ /dev/null
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script1.png b/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script1.png
new file mode 100644
index 000000000..bd375dba1
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script1.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script2.png b/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script2.png
new file mode 100644
index 000000000..99d9d24e6
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Test_HEAT_Script2.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png
new file mode 100644
index 000000000..73f6f2920
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_1.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png
new file mode 100644
index 000000000..ad7d22822
--- /dev/null
+++ b/docs/testing/developer/devguide/images/PROX_Test_ovs_dpdk_Script_2.png
Binary files differ
diff --git a/docs/testing/developer/devguide/images/vPE_Diagram.png b/docs/testing/developer/devguide/images/vPE_Diagram.png
new file mode 100644
index 000000000..c3985b7d9
--- /dev/null
+++ b/docs/testing/developer/devguide/images/vPE_Diagram.png
Binary files differ
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index 92a18f6ee..194099a27 100644
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -11,6 +11,6 @@ Yardstick Developer Guide
.. toctree::
:maxdepth: 4
- :numbered:
devguide
+ devguide_nsb_prox