Merge "Revert "Moved doc files to testing document structure"

1 files changed, 250 insertions, 0 deletions

diff --git a/docs/userguide/yardstick.rst b/docs/userguide/yardstick.rst
new file mode 100644
index 00000000..b5e5c72d
--- /dev/null
+++ b/docs/userguide/yardstick.rst
@@ -0,0 +1,250 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T and others.
+
+Execution of vswitchperf testcases by Yardstick
+-----------------------------------------------
+
+General
+^^^^^^^
+
+Yardstick is a generic framework for a test execution, which is used for
+validation of installation of OPNFV platform. In the future, Yardstick will
+support two options of vswitchperf testcase execution:
+
+- plugin mode, which will execute native vswitchperf testcases; Tests will
+  be executed natively by vsperf, and test results will be processed and
+  reported by yardstick.
+- traffic generator mode, which will run vswitchperf in **trafficgen**
+  mode only; Yardstick framework will be used to launch VNFs and to configure
+  flows to ensure, that traffic is properly routed. This mode will allow to
+  test OVS performance in real world scenarios.
+
+In Colorado release only the traffic generator mode is supported.
+
+Yardstick Installation
+^^^^^^^^^^^^^^^^^^^^^^
+
+In order to run Yardstick testcases, you will need to prepare your test
+environment. Please follow the `installation instructions
+<http://artifacts.opnfv.org/yardstick/docs/user_guides_framework/index.html>`__
+to install the yardstick.
+
+Please note, that yardstick uses OpenStack for execution of testcases.
+OpenStack must be installed with Heat and Neutron services. Otherwise
+vswitchperf testcases cannot be executed.
+
+VM image with vswitchperf
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A special VM image is required for execution of vswitchperf specific testcases
+by yardstick. It is possible to use a sample VM image available at OPNFV
+artifactory or to build customized image.
+
+Sample VM image with vswitchperf
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sample VM image is available at vswitchperf section of OPNFV artifactory
+for free download:
+
+.. code-block:: console
+
+    $ wget http://artifacts.opnfv.org/vswitchperf/vnf/vsperf-yardstick-image.qcow2
+
+This image can be used for execution of sample testcases with dummy traffic
+generator.
+
+**NOTE:** Traffic generators might require an installation of client software.
+This software is not included in the sample image and must be installed by user.
+
+**NOTE:** This image will be updated only in case, that new features related
+to yardstick integration will be added to the vswitchperf.
+
+Preparation of custom VM image
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In general, any Linux distribution supported by vswitchperf can be used as
+a base image for vswitchperf. One of the possibilities is to modify vloop-vnf
+image, which can be downloaded from `<http://artifacts.opnfv.org/vswitchperf.html/>`__
+(see :ref:`vloop-vnf`).
+
+Please follow the :ref:`vsperf-installation` to
+install vswitchperf inside vloop-vnf image. As vswitchperf will be run in
+trafficgen mode, it is possible to skip installation and compilation of OVS,
+QEMU and DPDK to keep image size smaller.
+
+In case, that selected traffic generator requires installation of additional
+client software, please follow appropriate documentation. For example in case
+of IXIA, you would need to install IxOS and IxNetowrk TCL API.
+
+VM image usage
+~~~~~~~~~~~~~~
+
+Image with vswitchperf must be uploaded into the glance service and
+vswitchperf specific flavor configured, e.g.:
+
+.. code-block:: console
+
+    $ glance --os-username admin --os-image-api-version 1 image-create --name \
+      vsperf --is-public true --disk-format qcow2 --container-format bare --file \
+      vsperf-yardstick-image.qcow2
+
+    $ nova --os-username admin flavor-create vsperf-flavor 100 2048 25 1
+
+Testcase execution
+^^^^^^^^^^^^^^^^^^
+
+After installation, yardstick is available as python package within yardstick
+specific virtual environment. It means, that yardstick environment must be
+enabled before the test execution, e.g.:
+
+.. code-block:: console
+
+   source ~/yardstick_venv/bin/activate
+
+
+Next step is configuration of OpenStack environment, e.g. in case of devstack:
+
+.. code-block:: console
+
+   source /opt/openstack/devstack/openrc
+   export EXTERNAL_NETWORK=public
+
+Vswitchperf testcases executable by yardstick are located at vswitchperf
+repository inside ``yardstick/tests`` directory. Example of their download
+and execution follows:
+
+.. code-block:: console
+
+   git clone https://gerrit.opnfv.org/gerrit/vswitchperf
+   cd vswitchperf
+
+   yardstick -d task start yardstick/tests/rfc2544_throughput_dummy.yaml
+
+**NOTE:** Optional argument ``-d`` shows debug output.
+
+Testcase customization
+^^^^^^^^^^^^^^^^^^^^^^
+
+Yardstick testcases are described by YAML files. vswitchperf specific testcases
+are part of the vswitchperf repository and their yaml files can be found at
+``yardstick/tests`` directory. For detailed description of yaml file structure,
+please see yardstick documentation and testcase samples. Only vswitchperf specific
+parts will be discussed here.
+
+Example of yaml file:
+
+.. code-block:: yaml
+
+    ...
+    scenarios:
+    -
+      type: Vsperf
+      options:
+        testname: 'p2p_rfc2544_throughput'
+        trafficgen_port1: 'eth1'
+        trafficgen_port2: 'eth3'
+        external_bridge: 'br-ex'
+        test_params: 'TRAFFICGEN_DURATION=30;TRAFFIC={'traffic_type':'rfc2544_throughput}'
+        conf_file: '~/vsperf-yardstick.conf'
+
+      host: vsperf.demo
+
+      runner:
+        type: Sequence
+        scenario_option_name: frame_size
+        sequence:
+        - 64
+        - 128
+        - 512
+        - 1024
+        - 1518
+      sla:
+        metrics: 'throughput_rx_fps'
+        throughput_rx_fps: 500000
+        action: monitor
+
+    context:
+    ...
+
+Section option
+~~~~~~~~~~~~~~
+
+Section **option** defines details of vswitchperf test scenario. Lot of options
+are identical to the vswitchperf parameters passed through ``--test-params``
+argument. Following options are supported:
+
+- **frame_size** - a packet size for which test should be executed;
+  Multiple packet sizes can be tested by modification of Sequence runner
+  section inside YAML definition. Default: '64'
+- **conf_file** - sets path to the vswitchperf configuration file, which will be
+  uploaded to VM; Default: '~/vsperf-yardstick.conf'
+- **setup_script** - sets path to the setup script, which will be executed
+  during setup and teardown phases
+- **trafficgen_port1** - specifies device name of 1st interface connected to
+  the trafficgen
+- **trafficgen_port2** - specifies device name of 2nd interface connected to
+  the trafficgen
+- **external_bridge** - specifies name of external bridge configured in OVS;
+  Default: 'br-ex'
+- **test_params** - specifies a string with a list of vsperf configuration
+  parameters, which will be passed to the ``--test-params`` CLI argument;
+  Parameters should be stated in the form of ``param=value`` and separated
+  by a semicolon. Configuration of traffic generator is driven by ``TRAFFIC``
+  dictionary, which can be also updated by values defined by ``test_params``.
+  Please check VSPERF documentation for details about available configuration
+  parameters and their data types.
+  In case that both **test_params** and **conf_file** are specified,
+  then values from **test_params** will override values defined
+  in the configuration file.
+
+In case that **trafficgen_port1** and/or **trafficgen_port2** are defined, then
+these interfaces will be inserted into the **external_bridge** of OVS. It is
+expected, that OVS runs at the same node, where the testcase is executed. In case
+of more complex OpenStack installation or a need of additional OVS configuration,
+**setup_script** can be used.
+
+**NOTE** It is essential to specify a configuration for selected traffic generator.
+In case, that standalone testcase is created, then traffic generator can be
+selected and configured directly in YAML file by **test_params**. On the other
+hand, if multiple testcases should be executed with the same traffic generator
+settings, then a customized configuration file should be prepared and its name
+passed by **conf_file** option.
+
+Section runner
+~~~~~~~~~~~~~~
+
+Yardstick supports several `runner types
+<http://artifacts.opnfv.org/yardstick/docs/userguide/architecture.html#runner-types>`__.
+In case of vswitchperf specific TCs, **Sequence** runner type can be used to
+execute the testcase for given list of frame sizes.
+
+
+Section sla
+~~~~~~~~~~~
+
+In case that sla section is not defined, then testcase will be always
+considered as successful. On the other hand, it is possible to define a set of
+test metrics and their minimal values to evaluate test success. Any numeric
+value, reported by vswitchperf inside CSV result file, can be used.
+Multiple metrics can be defined as a coma separated list of items. Minimal
+value must be set separately for each metric.
+
+e.g.:
+
+.. code-block:: yaml
+
+      sla:
+          metrics: 'throughput_rx_fps,throughput_rx_mbps'
+          throughput_rx_fps: 500000
+          throughput_rx_mbps: 1000
+
+In case that any of defined metrics will be lower than defined value, then
+testcase will be marked as failed. Based on ``action`` policy, yardstick
+will either stop test execution (value ``assert``) or it will run next test
+(value ``monitor``).
+
+**NOTE** The throughput SLA (or any other SLA) cannot be set to a meaningful
+value without knowledge of the server and networking environment, possibly
+including prior testing in that environment to establish a baseline SLA level
+under well-understood circumstances.