Merge "[docs] update the user guide for "yardstick-operation"" into stable/fraser

4 files changed, 367 insertions, 1 deletions

diff --git a/docs/testing/user/userguide/01-introduction.rst b/docs/testing/user/userguide/01-introduction.rst
index c0c0a0a2b..d846e759c 100755
--- a/docs/testing/user/userguide/01-introduction.rst
+++ b/docs/testing/user/userguide/01-introduction.rst
@@ -47,6 +47,9 @@ This document consists of the following chapters:
 
 * Chapter :doc:`04-installation` provides instructions to install *Yardstick*.
 
+* Chapter :doc:`05-operation` provides information on how to use *Yardstick*
+  to run and create testcases.
+
 * Chapter :doc:`06-yardstick-plugin` provides information on how to integrate
   other OPNFV testing projects into *Yardstick*.
 
diff --git a/docs/testing/user/userguide/05-operation.rst b/docs/testing/user/userguide/05-operation.rst
new file mode 100644
index 000000000..3da9fc509
--- /dev/null
+++ b/docs/testing/user/userguide/05-operation.rst
@@ -0,0 +1,362 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel, Ericsson AB, Huawei Technologies Co. Ltd and others.
+
+..
+      Convention for heading levels in Yardstick:
+      =======  Heading 0 (reserved for the title in a document)
+      -------  Heading 1
+      ^^^^^^^  Heading 2
+      +++++++  Heading 3
+      '''''''  Heading 4
+      Avoid deeper levels because they do not render well.
+
+===============
+Yardstick Usage
+===============
+
+Once you have yardstick installed, you can start using it to run testcases
+immediately, through the CLI. You can also define and run new testcases and
+test suites. This chapter details basic usage (running testcases), as well as
+more advanced usage (creating your own testcases).
+
+Yardstick common CLI
+--------------------
+
+List test cases
+^^^^^^^^^^^^^^^
+
+``yardstick testcase list``: This command line would list all test cases in
+Yardstick. It would show like below::
+
+   +---------------------------------------------------------------------------------------
+   | Testcase Name         | Description
+   +---------------------------------------------------------------------------------------
+   | opnfv_yardstick_tc001 | Measure network throughput using pktgen
+   | opnfv_yardstick_tc002 | measure network latency using ping
+   | opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio.
+   ...
+   +---------------------------------------------------------------------------------------
+
+
+Show a test case config file
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Take opnfv_yardstick_tc002 for an example. This test case measure network
+latency. You just need to type in ``yardstick testcase show
+opnfv_yardstick_tc002``, and the console would show the config yaml of this
+test case::
+
+   ---
+
+   schema: "yardstick:task:0.1"
+   description: >
+      Yardstick TC002 config file;
+      measure network latency using ping;
+
+   {% set image = image or "cirros-0.3.5" %}
+
+   {% set provider = provider or none %}
+   {% set physical_network = physical_network or 'physnet1' %}
+   {% set segmentation_id = segmentation_id or none %}
+   {% set packetsize = packetsize or 100 %}
+
+   scenarios:
+   {% for i in range(2) %}
+   -
+    type: Ping
+    options:
+      packetsize: {{packetsize}}
+    host: athena.demo
+    target: ares.demo
+
+    runner:
+      type: Duration
+      duration: 60
+      interval: 10
+
+    sla:
+      max_rtt: 10
+      action: monitor
+   {% endfor %}
+
+   context:
+    name: demo
+    image: {{image}}
+    flavor: yardstick-flavor
+    user: cirros
+
+    placement_groups:
+      pgrp1:
+        policy: "availability"
+
+    servers:
+      athena:
+        floating_ip: true
+        placement: "pgrp1"
+      ares:
+        placement: "pgrp1"
+
+    networks:
+      test:
+        cidr: '10.0.1.0/24'
+        {% if provider == "vlan" %}
+        provider: {{provider}}
+        physical_network: {{physical_network}}
+          {% if segmentation_id %}
+        segmentation_id: {{segmentation_id}}
+          {% endif %}
+        {% endif %}
+
+
+Run a Yardstick test case
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want run a test case, then you need to use ``yardstick task start
+<test_case_path>`` this command support some parameters as below:
+
+   +---------------------+--------------------------------------------------+
+   | Parameters          | Detail                                           |
+   +=====================+==================================================+
+   | -d                  | show debug log of yardstick running              |
+   |                     |                                                  |
+   +---------------------+--------------------------------------------------+
+   | --task-args         | If you want to customize test case parameters,   |
+   |                     | use "--task-args" to pass the value. The format  |
+   |                     | is a json string with parameter key-value pair.  |
+   |                     |                                                  |
+   +---------------------+--------------------------------------------------+
+   | --task-args-file    | If you want to use yardstick                     |
+   |                     | env prepare command(or                           |
+   |                     | related API) to load the                         |
+   +---------------------+--------------------------------------------------+
+   | --parse-only        |                                                  |
+   |                     |                                                  |
+   |                     |                                                  |
+   +---------------------+--------------------------------------------------+
+   | --output-file \     | Specify where to output the log. if not pass,    |
+   | OUTPUT_FILE_PATH    | the default value is                             |
+   |                     | "/tmp/yardstick/yardstick.log"                   |
+   |                     |                                                  |
+   +---------------------+--------------------------------------------------+
+   | --suite \           | run a test suite, TEST_SUITE_PATH specify where  |
+   | TEST_SUITE_PATH     | the test suite locates                           |
+   |                     |                                                  |
+   +---------------------+--------------------------------------------------+
+
+
+Run Yardstick in a local environment
+------------------------------------
+
+We also have a guide about `How to run Yardstick in a local environment`_.
+This work is contributed by Tapio Tallgren.
+
+Create a new testcase for Yardstick
+-----------------------------------
+
+As a user, you may want to define a new testcase in addition to the ones
+already available in Yardstick. This section will show you how to do this.
+
+Each testcase consists of two sections:
+
+* ``scenarios`` describes what will be done by the test
+* ``context`` describes the environment in which the test will be run.
+
+Defining the testcase scenarios
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+TODO
+
+Defining the testcase context(s)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each testcase consists of one or more contexts, which describe the environment
+in which the testcase will be run.
+Current available contexts are:
+
+* ``Dummy``: this is a no-op context, and is used when there is no environment
+  to set up e.g. when testing whether OpenStack services are available
+* ``Node``: this context is used to perform operations on baremetal servers
+* ``Heat``: uses OpenStack to provision the required hosts, networks, etc.
+* ``Kubernetes``: uses Kubernetes to provision the resources required for the
+  test.
+
+Regardless of the context type, the ``context`` section of the testcase will
+consist of the following::
+
+   context:
+     name: demo
+     type: Dummy|Node|Heat|Kubernetes
+
+The content of the ``context`` section will vary based on the context type.
+
+Dummy Context
++++++++++++++
+
+No additional information is required for the Dummy context::
+
+  context:
+    name: my_context
+    type: Dummy
+
+Node Context
+++++++++++++
+
+TODO
+
+Heat Context
+++++++++++++
+
+In addition to ``name`` and ``type``, a Heat context requires the following
+arguments:
+
+* ``image``: the image to be used to boot VMs
+* ``flavor``: the flavor to be used for VMs in the context
+* ``user``: the username for connecting into the VMs
+* ``networks``: The networks to be created, networks are identified by name
+
+  * ``name``: network name (required)
+  * (TODO) Any optional attributes
+
+* ``servers``: The servers to be created
+
+  * ``name``: server name
+  * (TODO) Any optional attributes
+
+In addition to the required arguments, the following optional arguments can be
+passed to the Heat context:
+
+* ``placement_groups``:
+
+  * ``name``: the name of the placement group to be created
+  * ``policy``: either ``affinity`` or ``availability``
+* ``server_groups``:
+
+  * ``name``: the name of the server group
+  * ``policy``: either ``affinity`` or ``anti-affinity``
+
+Combining these elements together, a sample Heat context config looks like:
+
+.. literalinclude::
+   ../../../../yardstick/tests/integration/dummy-scenario-heat-context.yaml
+   :start-after: ---
+
+Using exisiting HOT Templates
+'''''''''''''''''''''''''''''
+
+TODO
+
+Kubernetes Context
+++++++++++++++++++
+
+TODO
+
+Using multiple contexts in a testcase
++++++++++++++++++++++++++++++++++++++
+
+When using multiple contexts in a testcase, the ``context`` section is replaced
+by a ``contexts`` section, and each context is separated with a ``-`` line::
+
+  contexts:
+  -
+    name: context1
+    type: Heat
+    ...
+  -
+    name: context2
+    type: Node
+    ...
+
+
+Reusing a context
++++++++++++++++++
+
+Typically, a context is torn down after a testcase is run, however, the user
+may wish to keep an context intact after a testcase is complete.
+
+.. note::
+  This feature has been implemented for the Heat context only
+
+To keep or reuse a context, the ``flags`` option must be specified:
+
+* ``no_setup``: skip the deploy stage, and fetch the details of a deployed
+   context/Heat stack.
+* ``no_teardown``: skip the undeploy stage, thus keeping the stack intact for
+   the next test
+
+If either of these ``flags`` are ``True``, the context information must still
+be given. By default, these flags are disabled::
+
+  context:
+    name: mycontext
+    type: Heat
+    flags:
+      no_setup: True
+      no_teardown: True
+    ...
+
+Create a test suite for Yardstick
+---------------------------------
+
+A test suite in Yardstick is a .yaml file which includes one or more test
+cases. Yardstick is able to support running test suite task, so you can
+customize your own test suite and run it in one task.
+
+``tests/opnfv/test_suites`` is the folder where Yardstick puts CI test suite.
+A typical test suite is like below (the ``fuel_test_suite.yaml`` example)::
+
+   ---
+   # Fuel integration test task suite
+
+   schema: "yardstick:suite:0.1"
+
+   name: "fuel_test_suite"
+   test_cases_dir: "samples/"
+   test_cases:
+   -
+    file_name: ping.yaml
+   -
+    file_name: iperf3.yaml
+
+As you can see, there are two test cases in the ``fuel_test_suite.yaml``. The
+``schema`` and the ``name`` must be specified. The test cases should be listed
+via the tag ``test_cases`` and their relative path is also marked via the tag
+``test_cases_dir``.
+
+Yardstick test suite also supports constraints and task args for each test
+case. Here is another sample (the ``os-nosdn-nofeature-ha.yaml`` example) to
+show this, which is digested from one big test suite::
+
+   ---
+
+   schema: "yardstick:suite:0.1"
+
+   name: "os-nosdn-nofeature-ha"
+   test_cases_dir: "tests/opnfv/test_cases/"
+   test_cases:
+   -
+     file_name: opnfv_yardstick_tc002.yaml
+   -
+     file_name: opnfv_yardstick_tc005.yaml
+   -
+     file_name: opnfv_yardstick_tc043.yaml
+        constraint:
+           installer: compass
+           pod: huawei-pod1
+        task_args:
+           huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
+           "host": "node4.LF","target": "node5.LF"}'
+
+As you can see in test case ``opnfv_yardstick_tc043.yaml``, there are two
+tags, ``constraint`` and ``task_args``. ``constraint`` is to specify which
+installer or pod it can be run in the CI environment. ``task_args`` is to
+specify the task arguments for each pod.
+
+All in all, to create a test suite in Yardstick, you just need to create a
+yaml file and add test cases, constraint or task arguments if necessary.
+
+References
+----------
+
+.. _`How to run Yardstick in a local environment`: https://wiki.opnfv.org/display/yardstick/How+to+run+Yardstick+in+a+local+environment
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
index 57c6cf594..b936e723d 100644
--- a/docs/testing/user/userguide/index.rst
+++ b/docs/testing/user/userguide/index.rst
@@ -17,6 +17,7 @@ Yardstick User Guide
    02-methodology
    03-architecture
    04-installation
+   05-operation
    06-yardstick-plugin
    07-result-store-InfluxDB
    08-grafana
diff --git a/yardstick/tests/integration/dummy-scenario-heat-context.yaml b/yardstick/tests/integration/dummy-scenario-heat-context.yaml
index ce4b744e8..7c980b412 100644
--- a/yardstick/tests/integration/dummy-scenario-heat-context.yaml
+++ b/yardstick/tests/integration/dummy-scenario-heat-context.yaml
@@ -7,7 +7,7 @@
 # http://www.apache.org/licenses/LICENSE-2.0
 ##############################################################################
 ---
-# Sample benchmark task config file
+# Sample Heat context config with Dummy context
 
 schema: "yardstick:task:0.1"