Merge "Add documentation for dovetail tool"

6 files changed, 338 insertions, 2 deletions

diff --git a/docs/installationprocedure/feature.configuation.rst b/docs/installationprocedure/feature.configuation.rst
new file mode 100644
index 00000000..0bc4ae33
--- /dev/null
+++ b/docs/installationprocedure/feature.configuation.rst
@@ -0,0 +1,45 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+Testcase Template Syntax
+=========================
+
+The testcases certification are defined in ``dovetail/dovetail/testcase``.
+Take testcase ``ipv6.tc001.yml`` as an example.
+
+Basic template syntax
+---------------------
+
+::
+
+  dovetail.ipv6.tc001:
+    name: dovetail.ipv6.tc001
+    objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
+    scripts:
+      type: functest
+      testcase: tempest_smoke_serial
+      sub_testcase_list:
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
+
+The testcase needs at least three keys named as 'name', 'objective' and 'scripts'. The whole
+dovetail project can just recognize testcases by the 'name' section, which here is
+'dovetail.ipv6.tc001'. The 'objective' describes what exactly this testcase does. In the
+'scripts' part, there are some subsections such as 'type', 'testcase' and 'sub_testcase_list'.
+Dovetail now supports two kinds of types, functest and yardstick. If you define the type as
+functest, then you need to give the functest testcase that you want to use. If the type is
+yardstick, then a yardstick testcase is needed. The 'sub_testcase_list' lists the sub_testcases
+that you put a high value on. Even though the whole testcase faild, we still think it passed
+when all the sub_testcases pass. The sub_testcase_list is just available for functest. When
+for yardstick testcase, retain 'sub_testcase_list' section and keep the content empty.
+
+::
+
+  sub_testcase_list:
+    -
+
+This is the definition of the dovetail testcases. They can just be tested by adding into
+scenarios such as ``cert/basic.yml``.
diff --git a/docs/installationprocedure/index.rst b/docs/installationprocedure/index.rst
new file mode 100644
index 00000000..9a755800
--- /dev/null
+++ b/docs/installationprocedure/index.rst
@@ -0,0 +1,14 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+==================
+Dovetail Overview
+==================
+
+.. toctree::
+   :maxdepth: 2
+
+   installation.instruction
+   feature.configuation
diff --git a/docs/installationprocedure/installation.instruction.rst b/docs/installationprocedure/installation.instruction.rst
new file mode 100644
index 00000000..aa1b436a
--- /dev/null
+++ b/docs/installationprocedure/installation.instruction.rst
@@ -0,0 +1,142 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+=====================
+Dovetail Installation
+=====================
+
+Abstract
+========
+
+Dovetail currently supports installation on Ubuntu 14.04 or by using a docker image.
+Detailed steps about installing dovetail can be found below.
+
+To use dovetail you should have access to an OpenStack environment,
+with at least Nova, Neutron, Glance, Keystone and Heat installed.
+
+
+Run dovetail on local machine
+=============================
+
+The steps needed to run dovetail on Ubuntu are:
+
+1. Download source code and prepare environment.
+2. Create certification configuration file.
+3. Run certification.
+
+
+Download dovetail and prepare environment
+-----------------------------------------
+
+The source code of dovetail can be got from Gerrit:
+
+::
+
+  git clone https://gerrit.opnfv.org/gerrit/dovetail
+
+After that you need to prepare the environment which means install some tools,
+modules and anything else dovetail needs. A file named ``prepare_env.py`` can
+do all of these for you.
+
+::
+
+  cd dovetail/dovetail/
+  python prepare_env.py
+
+Now the environment should be prepared for running the certification.
+
+Besides, you may want to install it in a python virtualenv, that is beyond the scope
+of this installation instruction.
+
+Create certification configuration file
+---------------------------------------
+
+You can wirte a certification configuration file with the format of yaml,
+which can be defined to meet your own requirements. The yaml file should
+be located in the folder ``dovetail/dovetail/cert``. There provide a sample
+file named ``basic.yml``.
+
+::
+
+  certification_basic:
+    name: certification_basic
+    testcase_list:
+      - dovetail.ipv6.tc001
+
+As you can see, the configuration file combines the testcases you want to run
+together, and all of them should already been defined in ``dovetail/dovetail/testcase``.
+The name of the configuration file should be certification_%s, where %s (here is basic)
+can be used to run this certification.
+
+
+Run the certification
+---------------------
+
+When you get the dovetail source code, prepare the environment and define the
+certification configuration file, you can then run the certification. Take
+certification_basic as the example. You can run it using the file ``run.py`` with
+basic (scenario name) as its argument.
+
+::
+
+  python run.py --scenario basic
+
+If you did not give an argument, it will be default set as basic.
+
+
+Run dovetail on Docker container
+================================
+
+steps:
+1. pull image
+2. run container
+3. config
+4. run testcase
+
+pull image
+----------
+
+Get the latest docker image from docker hub.
+
+::
+ sudo docker pull opnfv/dovetail:latest
+
+run container
+-------------
+
+Start a container from the image that you just pulled.
+
+::
+ sudo docker run -it --privileged=true -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results
+ -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:latest "/bin/bash"
+
+config
+------
+
+For now, we need only change value of ``INSTALLER_TYPE`` and ``INSTALLER_IP``
+of the variable ``envs`` to your own environment.
+
+::
+ sudo docker exec -it $(CONTAINER_ID) "/bin/bash"
+ vi /home/opnfv/dovetail/dovetail/conf/functest_config.yml
+ vi /home/opnfv/dovetail/dovetail/conf/yardstick_config.yml
+
+run testcase
+------------
+
+In docker, you just run following cmds to execute the testcase.
+
+::
+ cd /home/opnfv/dovetail/dovetail
+ python run.py
+
+
+
+
+Output
+======
+
+The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``.
+The report of certification is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``.
diff --git a/docs/userguide/01-introduction.rst b/docs/userguide/01-introduction.rst
index f3d57be8..dbe33d5d 100644
--- a/docs/userguide/01-introduction.rst
+++ b/docs/userguide/01-introduction.rst
@@ -2,7 +2,7 @@
 Dovetail Introduction
 ======================
 
-**Welcome to Dovetail' documentation !**
+**Welcome to Dovetail's documentation !**
 
 .. _Dovetail: https://wiki.opnfv.org/dovetail
 
@@ -11,7 +11,7 @@ Dovetail_ is an OPNFV Project.
 This project intends to define and provide a set of OPNFV related validation
 criteria that will provide input for the evaluation of the use of OPNFV trademarks.
 The dovetail project will be executed with the guidance and oversight of the Compliance
-and Certification committee and work to secure the goals of the C&C committee for each release
+and Certification committee and work to secure the goals of the C&C committee for each release.
 
 This project intends to incrementally define qualification criteria
 that establish the foundations of how we are able to measure the ability
diff --git a/docs/userguide/04-dovetail_config_template.rst b/docs/userguide/04-dovetail_config_template.rst
new file mode 100644
index 00000000..fce99d21
--- /dev/null
+++ b/docs/userguide/04-dovetail_config_template.rst
@@ -0,0 +1,134 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+Config Template Syntax
+======================
+
+Dovetail uses Functest/Yardstick Docker container to run its testcases. So you need to give
+configurations for building the container and the commands it needs to do. In dovetail,
+all of these are defined in config yaml files ``dovetail/dovetail/conf/functest_config.yml``
+and ``dovetail/dovetail/conf/yardstick_config.yml``.
+
+Functest template syntax
+------------------------
+
+For example, you can define your ``functest_config.yml`` as:
+
+::
+
+  functest:
+    image_name: opnfv/functest
+    docker_tag: latest
+    envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
+           -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+           -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
+    opts: '-id --privileged=true'
+    pre_condition:
+      cmds:
+        - 'echo test for precondition'
+    testcase:
+      cmds:
+        - 'python /home/opnfv/repos/functest/ci/prepare_env.py start'
+        - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r'
+    post_condition:
+      cmds:
+        - ''
+    result:
+      dir: '/home/opnfv/functest/results'
+      store_type: 'file'
+      file_path: 'tempest/tempest.log'
+      db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
+
+First, you need to give the image that you want to use for building functest/yardstick container.
+Besides, there also need some envirnment parameters such as ``INSTALLER_TYPE`` and ``INSTALLER_IP``
+and the options for you container. Then the functest/yardstick container can be build with your
+settings.
+
+Second, there need three kinds of commands, ``pre_condition``, ``testcase`` and ``post_condition``.
+If you want to do some cleanups or preparations, the commands can be put into ``pre_condition``
+section orderly. All commands in this section will just be executed once in the begining.
+The ``testcase`` section does the main jobs of the testing. All functest testcases will use the
+container to execute these commands one by one. After finishing that, the test is accomplished
+and the results are stored in files or uploaded to database. The ``post_condition`` section
+does some work such as clean Docker images or something else after all testcases finished.
+All commands in this section will just execute once.
+
+Besides, there need a ``result`` section and it gives the directory of the functest/yardstick
+results. The ``store_type`` should be the same with the cmds in ``testcase``. That means if the
+test results are stored in files, then store_type need to be file and the file_path is also
+needed. If the test results are uploaded to database, then a db_url is needed for acquiring the results.
+
+Yardstick template syntax
+-------------------------
+
+The framework of ``yardstick_config.yml`` is almost the same as ``functest_config.yml``.
+
+::
+
+  yardstick:
+    image_name: opnfv/yardstick
+    docker_tag: latest
+    envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
+           -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+           -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal
+           -e EXTERNAL_NETWORK=ext-net'
+    opts: '-id --privileged=true'
+    pre_condition:
+      cmds:
+        - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
+           source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup'
+        - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
+           cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh'
+    testcase:
+      cmds:
+        - 'mkdir -p /home/opnfv/yardstick/results/'
+        - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh &&
+           yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml
+           --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &>
+           /home/opnfv/yardstick/results/yardstick.log'
+    post_condition:
+      cmds:
+        - ''
+    result:
+      dir: '/home/opnfv/yardstick/results'
+      store_type: 'file'
+      file_path: 'yardstick.log'
+      db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
+
+The main differences between ``yardstick_config.yml`` and ``functest_config.yml`` are the commands.
+
+Jinja2 template syntax
+----------------------
+
+Note that you can use jinja2 template for your parameters such as ``{{script_testcase}}``. The
+parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``:
+
+::
+
+  parameters:
+  - name: testcase
+    path: '("name",)'
+  - name: script_testcase
+    path: '("scripts", "testcase")'
+
+Here ``path`` is the path in testcase config files that you can find the value of parameters. Take
+``script_testcase`` as the example. For testcase dovetail.ipv6.tc001:
+
+::
+
+  dovetail.ipv6.tc001:
+    name: dovetail.ipv6.tc001
+    objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
+    scripts:
+      type: functest
+      testcase: tempest_smoke_serial
+      sub_testcase_list:
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
+        - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
+
+The path ("scripts", "testcase") means 'testcase' subsection of 'scripts' section. So follow
+the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` that is
+'tempest_smoke_serial'.
diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst
index 03246e7b..0dcf55ce 100644
--- a/docs/userguide/index.rst
+++ b/docs/userguide/index.rst
@@ -14,3 +14,4 @@ Dovetail - user guide
    ./01-introduction.rst
    ./02-certification_criteria.rst
    ./03-certification_progress.rst
+   ./04-dovetail_config_template.rst