summaryrefslogtreecommitdiffstats
path: root/docs/dovetailtool
diff options
context:
space:
mode:
authorMatthewLi <matthew.lijun@huawei.com>2017-01-14 04:08:13 -0500
committerMatthewLi <matthew.lijun@huawei.com>2017-01-19 03:38:07 -0500
commit9dee0a4b210d9c682464f7cf8052b803139b1b95 (patch)
treedc123734fc8951d10858ef40bbbc054f32e8c917 /docs/dovetailtool
parentc2401cd1bf3898aebeef08ca699072b1dca683f0 (diff)
dovetail tool userguide document
JIRA: DOVETAIL-182 Change-Id: Icbfa347dda3840520484d10b9d539da9c8f1d31c Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
Diffstat (limited to 'docs/dovetailtool')
-rw-r--r--docs/dovetailtool/dovetail.tool.cli.rst165
-rw-r--r--docs/dovetailtool/dovetail.tool.configuration.rst35
-rw-r--r--docs/dovetailtool/dovetail.tool.installation.rst90
-rw-r--r--docs/dovetailtool/dovetail.tool.template.rst (renamed from docs/dovetailtool/dovetail.tool.configtemplate.rst)55
-rw-r--r--docs/dovetailtool/index.rst3
5 files changed, 192 insertions, 156 deletions
diff --git a/docs/dovetailtool/dovetail.tool.cli.rst b/docs/dovetailtool/dovetail.tool.cli.rst
index bdcf46e4..f13bc289 100644
--- a/docs/dovetailtool/dovetail.tool.cli.rst
+++ b/docs/dovetailtool/dovetail.tool.cli.rst
@@ -3,21 +3,80 @@
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
-Command Line Interface
-======================
+================================
+Dovetail Command-line Interface
+================================
-Dovetail supports modifying some parameters at the run-time by using the command
-line interface (CLI). The parameters can be defined through a config file by
-developers easily and then be used at the run-time.
-The CLI now fits all three kinds of running: directly running the python script,
-running after setup and running within Docker containers.
+Command Line Introduction
+==========================
-Define CLI with config file
----------------------------
+The Dovetail command-line interface provides a method for interacting with
+Dovetail from the console. For help on the ``dovetail`` command, enter:
-For easy to be modified, Dovetail provides ``dovetail/dovetail/conf/cmd_config.yml``
-to define CLI automatically.
+::
+
+ dovetail --help
+
+**dovetail** optional arguments:
+
+::
+
+ --version
+ show program's version number and exit
+ list <testsuite_name>
+ list the testsuite details
+ show <testcase_name>
+ show the testcase details
+ run <arguments>
+ run the testcases
+
+For **dovetail list**, if the ``<testsuite_name>`` is omitted,
+all the testsuites defined under ``/dovetail/compliance`` directory
+and related testcases will be listed, otherwise,
+the test area and test cases defined in ``<testsuite_name>`` are listed.
+
+For **dovetail show**, the ``<testcase_name>`` is required, the contents defined
+in ``<testcase_name>.yml`` is shown.
+
+For **dovetail run**, by running ``dovetail run --help``, the ``dovetail run``
+usage is shown as:
+
++------------------------+-----------------------------------------------------+
+|Options | |
++========================+=====================================================+
+| -t, --SUT_TYPE |Installer type of the system under test (SUT). |
++------------------------+-----------------------------------------------------+
+| --creds |Openstack credential file location |
++------------------------+-----------------------------------------------------+
+| -i, --SUT_IP |IP of the system under test (SUT). |
++------------------------+-----------------------------------------------------+
+| -d, --debug |True for showing debug log on screen. |
++------------------------+-----------------------------------------------------+
+| -f, --func_tag |Overwrite tag for functest docker container (e.g. |
+| |stable or latest) |
++------------------------+-----------------------------------------------------+
+| -y, --yard_tag |Overwrite tag for yardstick docker container (e.g. |
+| |stable or latest) |
++------------------------+-----------------------------------------------------+
+| --testarea |compliance testarea within testsuite |
++------------------------+-----------------------------------------------------+
+| --testsuite |compliance testsuite. |
++------------------------+-----------------------------------------------------+
+| -h, --help |Show this message and exit. |
++------------------------+-----------------------------------------------------+
+
+
+If no arguments are given, the default testsuite will be performed, i.e., the ``compliance_set``
+testsuite with default configurations.
+
+For more information about **dovetail** command-line interface, please refer to the wiki page [3]_
+
+Parameters definition with config file
+======================================
+
+The default **dovetail run** parameters can be modified in
+``dovetail/dovetail/conf/cmd_config.yml``, which is shown as:
::
@@ -71,40 +130,14 @@ to define CLI automatically.
default: 'full'
help: 'compliance testarea within testsuite'
-Dovetail uses click module in python to parse parameters defined in the above
-config file. The basic config file shown above contains two subsections:
-**arguments** and **options** corresponding to two types of parameters in click.
-
-Add options
-+++++++++++
-
-Just as the name suggested, option parameters can either be given or not by users
-after adding into CLI.
-
-Then how to add an option for developers?
-
-For each option, it at least needs a key **flags** to give its name. Customarily,
-each option has two names, full name and short name, and they are begin with '--'
-and '-' respectively. All other keys should be consistent with click's keys.
-
-Take option **scenario** as the example. Its full name is '--scenario', and its
-short name is '-s'. Actually full name is necessary but short name is optional.
-The full name '--scenario' should be the same with the block's name **scenario**.
-**default** section gives the default value of this option if it doesn't given
-by users. Without the **default** section, it will be set None. **help** section
-offers its help message that will be shown when excute -h/--help command. For
-more information about click, please refer to: http://click.pocoo.org/5/
-
-Add arguments
-+++++++++++++
+The Click module is used to parse parameters defined in the above config file,
+two subsections are included in this file, ``arguments`` and ``options``,
+which corresponds to two types of parameters in Click.
-Arguments must given orderly by users once they are defined in the config file.
-The Dovetail tool doesn't need any argument parameters currently. However, here
-just give a simple example for its format.
-
-Arguments also need subsection **flags** to give its name. Each argument can just
-have one name, and the name should be the same with the key of this section. Other
-keys should also be consistent with the click module.
+Arguments and Options
++++++++++++++++++++++
+Only ``options`` is used currently, which means parameters can be given (or not) without
+sequence restriction.
Config and control
++++++++++++++++++
@@ -131,50 +164,10 @@ the configs will be changed into
-e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
-e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
-The config options/arguments can be added or deleted just by modifying
+The config options/arguments can be added or deleted by modifying
``cmd_config.yml`` rather than changing the source code. However, for control
command, besides adding it into ``cmd_config.yml``, some other operations about
the source code are also needed.
-Run with CLI
-------------
-
-For users, they can use CLI to input their own envs at the run-time instead of
-modifying the config files of functest or yardstick. So Dovetail can supports
-different environments more flexible with CLI. Dovetail now can be run with three
-methods, directly running ``run.py`` script, running after setup and running
-in Docker containers. The uses of CLI are almost the same for these three methods
-and here take the first one as the example.
-
-All parameters offered by Dovetail can be listed by using help option ``--help``.
-
-::
-
- root@90256c4efd05:~/dovetail/dovetail$ python run.py --help
- Usage: run.py [OPTIONS]
-
- Dovetail compliance test entry!
-
- Options:
- -t, --SUT_TYPE TEXT Installer type of the system under test (SUT).
- -f, --func_tag TEXT Overwrite tag for functest docker container (e.g.
- stable or latest)
- -i, --SUT_IP TEXT IP of the system under test (SUT).
- -y, --yard_tag TEXT Overwrite tag for yardstick docker container (e.g.
- stable or latest)
- -d, --DEBUG TEXT DEBUG for showing debug log.
- --testarea TEXT compliance testarea within testsuite
- --testsuite TEXT compliance testsuite.
- -h, --help Show this message and exit.
-
-All options listed can be used to input special environment values at the run-time.
-For example:
-
-::
-
- python run.py --SUT_TYPE compass -y stable
-There is no need to give all these options. If it is not given by CLI, it will
-be set with the system's environment value. If it is not included in system's
-environment variables, it will be set with the default value in functest/yardstick
-config file.
+. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line
diff --git a/docs/dovetailtool/dovetail.tool.configuration.rst b/docs/dovetailtool/dovetail.tool.configuration.rst
deleted file mode 100644
index 8e97e73c..00000000
--- a/docs/dovetailtool/dovetail.tool.configuration.rst
+++ /dev/null
@@ -1,35 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International
-.. License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
-
-=========================
-Testcase Template Syntax
-=========================
-
-The testcases used for compliance and certification are defined in the ``dovetail/testcase`` directory,
-which are defined in yaml format. Take the testcase ``ipv6.tc001.yml`` as an example, it is shown as:
-
-::
-
- dovetail.ipv6.tc001:
- name: dovetail.ipv6.tc001
- objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
- scripts:
- type: functest
- testcase: tempest_smoke_serial
- sub_testcase_list:
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
- - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
-
-- At least three sections named 'name', 'objective', 'scripts' must be included
-- Section 'name' distinguishes different test cases used for compliance and certification,
- and should start with ``dovetail.``
-- Section 'objective' describes what this testcase does
-- Section 'scripts' has subsections such as 'type', 'testcase' and 'sub_testcase_list'
-- Two kinds of 'type' is supported by now, functest and yardstick
-- For functest, the 'testcase' represents the testcases in slicing/tier,
- 'sub_testcase_list' represents the testcases in this slicing compliance and certification will use.
- For yardstick, since it is not sliced by now, 'sub_testcase_list' is not needed, only to edit the 'testcase' part
- such as ``yardstick_tc027``
diff --git a/docs/dovetailtool/dovetail.tool.installation.rst b/docs/dovetailtool/dovetail.tool.installation.rst
index 73a66cd1..5cd511ff 100644
--- a/docs/dovetailtool/dovetail.tool.installation.rst
+++ b/docs/dovetailtool/dovetail.tool.installation.rst
@@ -19,13 +19,30 @@ running on the SUT (System Under Test):
::
- SUT_TYPE, SUT type, e.g., apex, compass, fuel, joid, etc
- SUT_IP, SUT external network IP, e.g., 192.168.200.2
- NODE_NAME, this can be shown in the test result for users to see which pod the dovetail tool runs
- DEPLOY_SCENARIO, deployment scenario, e.g., os-nosdn-nofeature-ha
- BUILD_TAG, this can be shown in the test result for users to identify logs
- CI_DEBUG, true for debug information printed and false for not printed
- DEPLOY_TYPE, baremetal or virtual
+ SUT_TYPE
+ SUT type, e.g., apex, compass, fuel, joid, etc
+ SUT_IP
+ SUT external network IP, e.g., 192.168.200.2
+ NODE_NAME
+ this can be shown in the test result for users to see which pod the dovetail tool runs
+ DEPLOY_SCENARIO
+ deployment scenario, e.g., os-nosdn-nofeature-ha
+ BUILD_TAG
+ this can be shown in the test result for users to identify logs
+ CI_DEBUG
+ true for debug information printed and false for not printed
+ DEPLOY_TYPE
+ baremetal or virtual
+
+The above configuration can be achieved by
+
+- modifying the environment variables in files which live under ``/dovetail/conf/`` directory
+- set and use Linux environment variables using ``export`` command
+- set and use these variables when using ``dovetail run`` command line, for details see the
+ `Dovetail Command-line Interface`_ section
+- enable the OpenStack credential file, which can be achieved by using
+ ``dovetail run --creds </path/creds>``
+
Dovetail tool installation on local Linux host environment
##########################################################
@@ -114,33 +131,33 @@ After environment preparation is complete and test cases added, the Dovetail too
::
- python run.py --testsuite compliance_set
+ dovetail run --testsuite compliance_set
-The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced with the test cases yaml file.
-If not argument is given, the compliance_set testsuite will be run as the default.
+The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced
+with the testsuite yaml file name which want to be run.
+If no argument is given, the compliance_set testsuite will be run as the default.
Moreover, the testcases in given testarea can be run with ``testarea`` command line argument, such as
testarea ``ipv6`` in ``compliance_set``
::
- python run.py --testsuite compliance_set --testarea ipv6
+ dovetail run --testsuite compliance_set --testarea ipv6
Dovetail provides some sets, ``debug``, ``proposed_tests`` and ``compliance_set``,
``debug`` is used for locally and Continuous Integration(CI) developing purpose,
which provides typical testcase examples, feel free to edit it when develops locally, such as
only to run a testcase which only takes minutes. ``proposed_tests`` is the testcase
-candidate which mainly comes from the wiki link
-https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases.
+candidate which mainly comes from the wiki link [1]_.
``compliance_set`` is used for compliance. Moreover, dovetail tool can be easily
extended to support more complicated compliance requirements,
such as feature set based or scenario based compliance.
-If you want to run ``debug``, just run with
+If you want to run the ``debug`` testsuite, just run with
::
- python run.py --testsuite debug
+ dovetail run --testsuite debug
Running Dovetail in a Docker container
########################################
@@ -165,7 +182,8 @@ Docker image.
::
- sudo docker build -t <your_new_image_name> -f <your_Dockerfile> .
+ cd {dovetail_path}/dovetail/docker
+ docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master .
Dovetail Docker container creation
----------------------------------
@@ -178,13 +196,15 @@ Next, create the ``dovetail-docker-env`` file to define the environment paramete
DEPLOY_SCENARIO=ha-nosdn
CI_DEBUG=true
+or if an OpenStack credential file is provided.
+
Then to instantiate the Dovetail Docker container, execute::
sudo docker run --privileged=true --rm -t \
- --env-file dovetail-docker-env \
+ --env-file dovetail-docker-env OR </path/creds> \
-v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \
-v /var/run/docker.sock:/var/run/docker.sock \
- --name <Dovetail_Container_Name> \
+ --name <Dovetail_Container_Name> (optional) \
opnfv/dovetail:<Tag> /bin/bash
To attach dovetail container and Running test cases
@@ -192,16 +212,27 @@ To attach dovetail container and Running test cases
Before connecting to the container, you can check the container status by running ::
- docker ps -a
+ sudo docker ps -a
Attach to the container by starting it and obtaining a bash prompt with ::
- docker exec -it <Dovetail_Container_Name> bash
+ sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash
+
+Inside the container the following commands can be executed to trigger the testing ::
+
+ dovetail run --testsuite compliance_set
-Inside the container the following commands can be executed to trigger the testcases ::
+Offline Support
+################
- cd /home/opnfv/dovetail/dovetail
- python run.py --testsuite compliance_set
+There are some SUTs that are isolated from the public internet,
+so offline support is needed. The idea is to provide all of the packages of dovetail
+release in http://artifacts.opnfv.org, then the user can download and transfer to their inner
+development environment.
+
+The packages are shown in [2]_
+
+TO DO: to introduce more when it is mature enough.
Results Output
###############
@@ -212,11 +243,10 @@ The compliance report is stored in ``/home/opnfv/dovetail/results/dovetail_repor
Dovetail Version and Release
############################
-Dovetail version tag is shown in ``setup.cfg``, which will also shown in the
-``dovetail report``. At the time of version release, just to set the version value in
-``setup.cfg``.
+Dovetail version information is defined in ``setup.cfg``.
+At the time of release, it is the dovetail team's responsibility to set
+the ``version`` value in ``setup.cfg``.
+
-# TO DO: (which should be discussed)
-1)how to pubish version, such as both the online and offline package in some website
-or somewhere.
-2)provide version download address, userguide, etc.
+.. [1] https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases.
+.. [2] http://artifacts.opnfv.org/dovetail.html.
diff --git a/docs/dovetailtool/dovetail.tool.configtemplate.rst b/docs/dovetailtool/dovetail.tool.template.rst
index 9c0748a9..1a483dd9 100644
--- a/docs/dovetailtool/dovetail.tool.configtemplate.rst
+++ b/docs/dovetailtool/dovetail.tool.template.rst
@@ -3,9 +3,58 @@
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
-======================
+==================
+Template Syntax
+==================
+
+Testcase Template Syntax
+=========================
+
+The testcases used for compliance and certification are defined in the
+``dovetail/testcase`` directory, which are written in yaml format.
+Take the testcase ``ipv6.tc001.yml`` as an example. It is shown as:
+
+::
+
+ ---
+ dovetail.ipv6.tc001:
+ name: dovetail.ipv6.tc001
+ objective: Bulk creation and deletion of IPv6 networks, ports and subnets
+ validate:
+ type: functest
+ testcase: tempest_smoke_serial
+ pre_condition:
+ - 'echo test for precondition in testcase'
+ cmds:
+ - 'functest env prepare'
+ - 'functest testcase run {{validate_testcase}}'
+ post_condition:
+ - 'echo test for precondition in testcase'
+ report:
+ sub_testcase_list:
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
+
+
+- At least three sections named ``name``, ``objective``, ``validate``,
+ ``report`` must be included
+- Section ``name`` distinguishes different test cases used for compliance,
+ it is composed of 3 parts, ``dovetail.``, belongs to which test area,
+ and the serial number
+- Section ``objective`` briefly describes what this testcase does
+- Section ``validate`` defines the scripts and configurations for the
+ validation of the test case. ``type`` defines which method is used to validate,
+ 3 ways, i.e., functest, yardstick and shell is supported currently.
+ ``testcase`` represents the testcases in slicing/tier.
+- Section ``report`` defines the sub_testcases to be run.
+ For ``yardstick``, since it is not sliced by now,
+ ``sub_testcase_list`` is not needed, only to edit the ``testcase`` part in
+ section ``validate``, such as ``yardstick_tc027``
+
+
Config Template Syntax
-======================
+=======================
For Dovetail tool, the config files are located in ``dovetail/dovetail/conf``, which are written
in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files
@@ -15,7 +64,7 @@ respectively.
Functest config template syntax
-------------------------------
-An example functest configuration is shown as follows:
+An example of ``functest`` configuration is shown as follows:
::
diff --git a/docs/dovetailtool/index.rst b/docs/dovetailtool/index.rst
index 676c72db..b095704a 100644
--- a/docs/dovetailtool/index.rst
+++ b/docs/dovetailtool/index.rst
@@ -12,6 +12,5 @@ Dovetail Overview
dovetail.tool.overview.rst
dovetail.tool.installation.rst
- dovetail.tool.configuration.rst
- dovetail.tool.configtemplate.rst
+ dovetail.tool.template.rst
dovetail.tool.cli.rst