summaryrefslogtreecommitdiffstats
path: root/docs/testing/developer
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing/developer')
-rw-r--r--docs/testing/developer/testframework/dovetail.tool.cli.rst173
-rw-r--r--docs/testing/developer/testframework/dovetail.tool.installation.rst253
-rw-r--r--docs/testing/developer/testframework/dovetail.tool.overview.rst27
-rw-r--r--docs/testing/developer/testframework/dovetail.tool.template.rst174
-rw-r--r--docs/testing/developer/testframework/index.rst16
5 files changed, 643 insertions, 0 deletions
diff --git a/docs/testing/developer/testframework/dovetail.tool.cli.rst b/docs/testing/developer/testframework/dovetail.tool.cli.rst
new file mode 100644
index 00000000..f13bc289
--- /dev/null
+++ b/docs/testing/developer/testframework/dovetail.tool.cli.rst
@@ -0,0 +1,173 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+================================
+Dovetail Command-line Interface
+================================
+
+
+Command Line Introduction
+==========================
+
+The Dovetail command-line interface provides a method for interacting with
+Dovetail from the console. For help on the ``dovetail`` command, enter:
+
+::
+
+ dovetail --help
+
+**dovetail** optional arguments:
+
+::
+
+ --version
+ show program's version number and exit
+ list <testsuite_name>
+ list the testsuite details
+ show <testcase_name>
+ show the testcase details
+ run <arguments>
+ run the testcases
+
+For **dovetail list**, if the ``<testsuite_name>`` is omitted,
+all the testsuites defined under ``/dovetail/compliance`` directory
+and related testcases will be listed, otherwise,
+the test area and test cases defined in ``<testsuite_name>`` are listed.
+
+For **dovetail show**, the ``<testcase_name>`` is required, the contents defined
+in ``<testcase_name>.yml`` is shown.
+
+For **dovetail run**, by running ``dovetail run --help``, the ``dovetail run``
+usage is shown as:
+
++------------------------+-----------------------------------------------------+
+|Options | |
++========================+=====================================================+
+| -t, --SUT_TYPE |Installer type of the system under test (SUT). |
++------------------------+-----------------------------------------------------+
+| --creds |Openstack credential file location |
++------------------------+-----------------------------------------------------+
+| -i, --SUT_IP |IP of the system under test (SUT). |
++------------------------+-----------------------------------------------------+
+| -d, --debug |True for showing debug log on screen. |
++------------------------+-----------------------------------------------------+
+| -f, --func_tag |Overwrite tag for functest docker container (e.g. |
+| |stable or latest) |
++------------------------+-----------------------------------------------------+
+| -y, --yard_tag |Overwrite tag for yardstick docker container (e.g. |
+| |stable or latest) |
++------------------------+-----------------------------------------------------+
+| --testarea |compliance testarea within testsuite |
++------------------------+-----------------------------------------------------+
+| --testsuite |compliance testsuite. |
++------------------------+-----------------------------------------------------+
+| -h, --help |Show this message and exit. |
++------------------------+-----------------------------------------------------+
+
+
+If no arguments are given, the default testsuite will be performed, i.e., the ``compliance_set``
+testsuite with default configurations.
+
+For more information about **dovetail** command-line interface, please refer to the wiki page [3]_
+
+Parameters definition with config file
+======================================
+
+The default **dovetail run** parameters can be modified in
+``dovetail/dovetail/conf/cmd_config.yml``, which is shown as:
+
+::
+
+ cli:
+ arguments:
+ config:
+ # This is a simple example of arguments.
+ # Dovetail has no need of this kind of parameters currently.
+ # The arguments must be given orderly at the run-time.
+ #
+ # docker_tag:
+ # flags: 'docker_tag'
+ # path:
+ # - 'functest/docker_tag'
+ # - 'yardstick/docker_tag'
+ control:
+
+ options:
+ config:
+ SUT_TYPE:
+ flags:
+ - '--SUT_TYPE'
+ - '-t'
+ path:
+ - 'functest/envs'
+ - 'yardstick/envs'
+ help: 'Installer type of the system under test (SUT).'
+ yard_tag:
+ flags:
+ - '--yard_tag'
+ - '-y'
+ path:
+ - 'yardstick/docker_tag'
+ help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)'
+ func_tag:
+ flags:
+ - '--func_tag'
+ - '-f'
+ path:
+ - 'functest/docker_tag'
+ help: 'Overwrite tag for functest docker container (e.g. stable or latest)'
+ control:
+ testsuite:
+ flags:
+ - '--testsuite'
+ default: 'compliance_set'
+ help: 'compliance testsuite.'
+ testarea:
+ flags:
+ - '--testarea'
+ default: 'full'
+ help: 'compliance testarea within testsuite'
+
+The Click module is used to parse parameters defined in the above config file,
+two subsections are included in this file, ``arguments`` and ``options``,
+which corresponds to two types of parameters in Click.
+
+Arguments and Options
++++++++++++++++++++++
+Only ``options`` is used currently, which means parameters can be given (or not) without
+sequence restriction.
+
+Config and control
+++++++++++++++++++
+
+All options/arguments are divided into two parts: **config** and **control**.
+The config ones are used for updating functest or yardstick config files according
+to the **path** given. For example, functest's config file is
+``dovetail/dovetail/conf/functest_config.yml``, following is a simple example:
+
+::
+
+ docker_tag: latest
+ envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
+ -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+ -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
+
+If running with the command ``python run.py --SUT_TYPE fuel -f stable``, then
+the configs will be changed into
+
+::
+
+ docker_tag: stable
+ envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2
+ -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+ -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
+
+The config options/arguments can be added or deleted by modifying
+``cmd_config.yml`` rather than changing the source code. However, for control
+command, besides adding it into ``cmd_config.yml``, some other operations about
+the source code are also needed.
+
+
+. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line
diff --git a/docs/testing/developer/testframework/dovetail.tool.installation.rst b/docs/testing/developer/testframework/dovetail.tool.installation.rst
new file mode 100644
index 00000000..9d3bc39f
--- /dev/null
+++ b/docs/testing/developer/testframework/dovetail.tool.installation.rst
@@ -0,0 +1,253 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+===========================
+Dovetail Tool Installation
+===========================
+
+Abstract
+########
+
+The Dovetail tool supports installation either directly on the Linux host, or within a Docker container.
+The detailed installation procedure on the Linux host or via the Docker container are shown
+in the following sections.
+
+There is a need to config the following parameters before Dovetail tool
+running on the SUT (System Under Test):
+
+::
+
+ SUT_TYPE
+ SUT type, e.g., apex, compass, fuel, joid, etc
+ SUT_IP
+ SUT external network IP, e.g., 192.168.200.2
+ NODE_NAME
+ this can be shown in the test result for users to see which pod the dovetail tool runs
+ DEPLOY_SCENARIO
+ deployment scenario, e.g., os-nosdn-nofeature-ha
+ BUILD_TAG
+ this can be shown in the test result for users to identify logs
+ CI_DEBUG
+ true for debug information printed and false for not printed
+ DEPLOY_TYPE
+ baremetal or virtual
+
+The above configuration can be achieved by
+
+- modifying the environment variables in files which live under ``/dovetail/conf/`` directory
+- set and use Linux environment variables using ``export`` command
+- set and use these variables when using ``dovetail run`` command line, for details see the
+ `Dovetail Command-line Interface`_ section
+- enable the OpenStack credential file, which can be achieved by using
+ ``dovetail run --creds </path/creds>``
+
+
+Dovetail tool installation on local Linux host environment
+##########################################################
+
+In order to install Dovetail into a local Linux host environment, the following steps should
+be performed:
+
+Downloading Dovetail tool
+--------------------------
+
+Source code downloading:
+
+::
+
+ git clone https://git.opnfv.org/dovetail
+
+Environment preparation
+-----------------------
+
+Dovetail configuration:
+
+::
+
+ cd {dovetail_path}/dovetail/conf
+ vim functest_config.yml
+ vim yardstick_config.yml
+
+To modify the ``envs`` parameters, e.g., SUT_TYPE, SUT_IP, NODE_NAME,
+DEPLOY_SCENARIO, BUILD_TAG, CI_DEBUG, DEPLOY_TYPE, in the yml files.
+
+System dependencies and project configuration information are listed in
+``requirements.txt``, ``setup.py`` and ``setup.cfg``, in Ubuntu environment,
+to run
+
+::
+
+ apt-get -y install python-pip
+ pip install -e .
+
+in ``centos`` and ``redhat`` environment, to run
+
+::
+
+ yum -y install python-pip
+ pip install -e .
+
+in ``fedora`` environment, to run
+
+
+::
+
+ dnf -y install python-pip
+ pip install -e .
+
+then the setup requirements are installed and **dovetail** commmand-line can be used,
+by using ``dovetail --help``, **dovetail** command-line details are shown.
+
+The tool ``tox`` is used, which configuration information is defined in
+``tox.ini``, unitttest, flake8-checking, coverage check, etc, can be done
+by running:
+
+::
+
+ pip install tox
+ tox
+
+Compliance and certification test cases
+----------------------------------------
+
+The compliance and certification test cases can be defined under the ``/dovetail/compliance``
+directory, which is defined in yaml format.
+A sample file named ``compliance_set.yml`` is provided as follows:
+
+::
+
+ compliance_set:
+ name: compliance_set
+ testcases_list:
+ - dovetail.ipv6.tc001
+
+The testcase listed here is dovetail.ipv6.tc001, defined within ``dovetail/testcase``.
+
+Running Dovetail tool
+---------------------
+
+After environment preparation is complete and test cases added, the Dovetail tool can be run with
+
+::
+
+ dovetail run --testsuite compliance_set
+
+The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced
+with the testsuite yaml file name which want to be run.
+If no argument is given, the compliance_set testsuite will be run as the default.
+
+Moreover, the testcases in given testarea can be run with ``testarea`` command line argument, such as
+testarea ``ipv6`` in ``compliance_set``
+
+::
+
+ dovetail run --testsuite compliance_set --testarea ipv6
+
+Dovetail provides some sets, ``debug``, ``proposed_tests`` and ``compliance_set``,
+``debug`` is used for locally and Continuous Integration(CI) developing purpose,
+which provides typical testcase examples, feel free to edit it when develops locally, such as
+only to run a testcase which only takes minutes. ``proposed_tests`` is the testcase
+candidate which mainly comes from the wiki link [1]_.
+``compliance_set`` is used for compliance. Moreover, dovetail tool can be easily
+extended to support more complicated compliance requirements,
+such as feature set based or scenario based compliance.
+
+If you want to run the ``debug`` testsuite, just run with
+
+::
+
+ dovetail run --testsuite debug
+
+Running Dovetail in a Docker container
+########################################
+
+The Dovetail tool can be run in a Docker container by utilizing the following steps:
+
+Pull Dovetail Docker image from public Dockerhub
+------------------------------------------------
+
+::
+
+ sudo docker pull opnfv/dovetail:<Tag>
+
+<Tag> here is the version, 'latest' is used for the master branch.
+
+(Optional) Create Docker image from Dockerfile
+-----------------------------------------------
+Instead of an official Docker image, you can build your own Docker image from
+Dockerfile(s) located under the ``docker/`` directory. The ``Dockerfile``
+is based on Ubuntu and the ``Dockerfile.centos7`` file is for a CentOS based
+Docker image.
+
+::
+
+ cd {dovetail_path}/dovetail/docker
+ docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master .
+
+Dovetail Docker container creation
+----------------------------------
+
+Next, create the ``dovetail-docker-env`` file to define the environment parameters ::
+
+ INSTALLER_TYPE=compass
+ INSTALLER_IP=192.168.200.2
+ DEPLOY_TYPE=baremetal
+ DEPLOY_SCENARIO=ha-nosdn
+ CI_DEBUG=true
+
+or if an OpenStack credential file is provided.
+
+Then to instantiate the Dovetail Docker container, execute::
+
+ sudo docker run --privileged=true --rm -t \
+ --env-file dovetail-docker-env OR </path/creds> \
+ -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ --name <Dovetail_Container_Name> (optional) \
+ opnfv/dovetail:<Tag> /bin/bash
+
+To attach dovetail container and Running test cases
+----------------------------------------------------
+
+Before connecting to the container, you can check the container status by running ::
+
+ sudo docker ps -a
+
+Attach to the container by starting it and obtaining a bash prompt with ::
+
+ sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash
+
+Inside the container the following commands can be executed to trigger the testing ::
+
+ dovetail run --testsuite compliance_set
+
+Offline Support
+################
+
+There are some SUTs that are isolated from the public internet,
+so offline support is needed. The idea is to provide all of the packages of dovetail
+release in http://artifacts.opnfv.org, then the user can download and transfer to their inner
+development environment.
+
+The packages are shown in [2]_
+
+TO DO: to introduce more when it is mature enough.
+
+Results Output
+###############
+
+The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``.
+The compliance report is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``.
+
+Dovetail Version and Release
+############################
+
+Dovetail version information is defined in ``setup.cfg``.
+At the time of release, it is the dovetail team's responsibility to set
+the ``version`` value in ``setup.cfg``.
+
+
+.. [1] https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases.
+.. [2] http://artifacts.opnfv.org/dovetail.html.
diff --git a/docs/testing/developer/testframework/dovetail.tool.overview.rst b/docs/testing/developer/testframework/dovetail.tool.overview.rst
new file mode 100644
index 00000000..c4b7fa22
--- /dev/null
+++ b/docs/testing/developer/testframework/dovetail.tool.overview.rst
@@ -0,0 +1,27 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+==============
+Dovetail Tool
+==============
+
+What is Dovetail tool
+#####################
+
+A toolset for providing an evaluation of a deployed cloud environment for OPNFV compliance.
+
+Overview
+########
+
+Dovetail tool intends on being THE source of tools to provide evaluation of the use of OPNFV trademarks.
+
+Dovetail tool provides users with a wrapper which leverage other test projects, such as Functest,
+Yardstick, etc, that helps to provide evaluation of the use of OPNFV trademarks.
+It does so by validating any NFV(Network Function Virtualization) implementation against the test cases in Dovetail.
+
+Get involved!
+#############
+- Wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Home
+- Git Repo: https://git.opnfv.org/dovetail
diff --git a/docs/testing/developer/testframework/dovetail.tool.template.rst b/docs/testing/developer/testframework/dovetail.tool.template.rst
new file mode 100644
index 00000000..1a483dd9
--- /dev/null
+++ b/docs/testing/developer/testframework/dovetail.tool.template.rst
@@ -0,0 +1,174 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+==================
+Template Syntax
+==================
+
+Testcase Template Syntax
+=========================
+
+The testcases used for compliance and certification are defined in the
+``dovetail/testcase`` directory, which are written in yaml format.
+Take the testcase ``ipv6.tc001.yml`` as an example. It is shown as:
+
+::
+
+ ---
+ dovetail.ipv6.tc001:
+ name: dovetail.ipv6.tc001
+ objective: Bulk creation and deletion of IPv6 networks, ports and subnets
+ validate:
+ type: functest
+ testcase: tempest_smoke_serial
+ pre_condition:
+ - 'echo test for precondition in testcase'
+ cmds:
+ - 'functest env prepare'
+ - 'functest testcase run {{validate_testcase}}'
+ post_condition:
+ - 'echo test for precondition in testcase'
+ report:
+ sub_testcase_list:
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
+
+
+- At least three sections named ``name``, ``objective``, ``validate``,
+ ``report`` must be included
+- Section ``name`` distinguishes different test cases used for compliance,
+ it is composed of 3 parts, ``dovetail.``, belongs to which test area,
+ and the serial number
+- Section ``objective`` briefly describes what this testcase does
+- Section ``validate`` defines the scripts and configurations for the
+ validation of the test case. ``type`` defines which method is used to validate,
+ 3 ways, i.e., functest, yardstick and shell is supported currently.
+ ``testcase`` represents the testcases in slicing/tier.
+- Section ``report`` defines the sub_testcases to be run.
+ For ``yardstick``, since it is not sliced by now,
+ ``sub_testcase_list`` is not needed, only to edit the ``testcase`` part in
+ section ``validate``, such as ``yardstick_tc027``
+
+
+Config Template Syntax
+=======================
+
+For Dovetail tool, the config files are located in ``dovetail/dovetail/conf``, which are written
+in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files
+should be configured as follows, within the ``functest_config.yml`` and ``yardstick_config.yml`` files,
+respectively.
+
+Functest config template syntax
+-------------------------------
+
+An example of ``functest`` configuration is shown as follows:
+
+::
+
+ functest:
+ image_name: opnfv/functest
+ docker_tag: latest
+ envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
+ -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+ -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
+ opts: '-id --privileged=true'
+ pre_condition:
+ cmds:
+ - 'echo test for precondition'
+ testcase:
+ cmds:
+ - 'python /home/opnfv/repos/functest/ci/prepare_env.py start'
+ - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r'
+ post_condition:
+ cmds:
+ - ''
+ result:
+ dir: '/home/opnfv/functest/results'
+ store_type: 'file'
+ file_path: 'tempest/tempest.log'
+ db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
+
+- ``image_name`` and ``docker_tag`` sections define the docker image pulled from the public dockerhub.
+- ``envs`` should be correctly configed according to the SUT(System Under Test).
+- ``pre_condition`` represents some cleanups or preparations.
+ ``testcase`` represents the testcases running cmds.
+ ``post_condition`` represents some cleanups needed after all testcases finished.
+- ``result`` section gives the directory of the dovetail tool test result.
+ ``db_url`` gives the database URL of the dovetail results to be stored.
+
+Yardstick config template syntax
+---------------------------------
+
+The configuration in ``yardstick_config.yml`` is similiar to ``functest_config.yml``,
+and an example is shown as follows:
+
+::
+
+ yardstick:
+ image_name: opnfv/yardstick
+ docker_tag: latest
+ envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
+ -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
+ -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal
+ -e EXTERNAL_NETWORK=ext-net'
+ opts: '-id --privileged=true'
+ pre_condition:
+ cmds:
+ - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
+ source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup'
+ - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
+ cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh'
+ testcase:
+ cmds:
+ - 'mkdir -p /home/opnfv/yardstick/results/'
+ - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh &&
+ yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml
+ --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &>
+ /home/opnfv/yardstick/results/yardstick.log'
+ post_condition:
+ cmds:
+ - ''
+ result:
+ dir: '/home/opnfv/yardstick/results'
+ store_type: 'file'
+ file_path: 'yardstick.log'
+ db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'
+
+The main differences between ``yardstick_config.yml`` and ``functest_config.yml``
+are the ``cmds`` subsection.
+
+Jinja2 template syntax
+----------------------
+
+Jinja2 module can be used to config the ``{{script_testcase}}``. The
+parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``:
+
+::
+
+ parameters:
+ - name: testcase
+ path: '("name",)'
+ - name: script_testcase
+ path: '("scripts", "testcase")'
+
+Here ``path`` is the path defined in the testcase configuration files.
+Take ``script_testcase`` as an example. For testcase ``dovetail.ipv6.tc001``:
+
+::
+
+ dovetail.ipv6.tc001:
+ name: dovetail.ipv6.tc001
+ objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
+ scripts:
+ type: functest
+ testcase: tempest_smoke_serial
+ sub_testcase_list:
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
+ - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet
+
+The path ("scripts", "testcase") means 'testcase' is the subsection of 'scripts' section. From above,
+by following the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` is 'tempest_smoke_serial'.
diff --git a/docs/testing/developer/testframework/index.rst b/docs/testing/developer/testframework/index.rst
new file mode 100644
index 00000000..b095704a
--- /dev/null
+++ b/docs/testing/developer/testframework/index.rst
@@ -0,0 +1,16 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Technologies Co.,Ltd and others.
+
+==================
+Dovetail Overview
+==================
+
+.. toctree::
+ :maxdepth: 2
+
+ dovetail.tool.overview.rst
+ dovetail.tool.installation.rst
+ dovetail.tool.template.rst
+ dovetail.tool.cli.rst