diff options
Diffstat (limited to 'docs/testing/developer')
5 files changed, 643 insertions, 0 deletions
diff --git a/docs/testing/developer/testframework/dovetail.tool.cli.rst b/docs/testing/developer/testframework/dovetail.tool.cli.rst new file mode 100644 index 00000000..f13bc289 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.cli.rst @@ -0,0 +1,173 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================================ +Dovetail Command-line Interface +================================ + + +Command Line Introduction +========================== + +The Dovetail command-line interface provides a method for interacting with +Dovetail from the console. For help on the ``dovetail`` command, enter: + +:: + + dovetail --help + +**dovetail** optional arguments: + +:: + + --version + show program's version number and exit + list <testsuite_name> + list the testsuite details + show <testcase_name> + show the testcase details + run <arguments> + run the testcases + +For **dovetail list**, if the ``<testsuite_name>`` is omitted, +all the testsuites defined under ``/dovetail/compliance`` directory +and related testcases will be listed, otherwise, +the test area and test cases defined in ``<testsuite_name>`` are listed. + +For **dovetail show**, the ``<testcase_name>`` is required, the contents defined +in ``<testcase_name>.yml`` is shown. + +For **dovetail run**, by running ``dovetail run --help``, the ``dovetail run`` +usage is shown as: + ++------------------------+-----------------------------------------------------+ +|Options | | ++========================+=====================================================+ +| -t, --SUT_TYPE |Installer type of the system under test (SUT). | ++------------------------+-----------------------------------------------------+ +| --creds |Openstack credential file location | ++------------------------+-----------------------------------------------------+ +| -i, --SUT_IP |IP of the system under test (SUT). | ++------------------------+-----------------------------------------------------+ +| -d, --debug |True for showing debug log on screen. | ++------------------------+-----------------------------------------------------+ +| -f, --func_tag |Overwrite tag for functest docker container (e.g. | +| |stable or latest) | ++------------------------+-----------------------------------------------------+ +| -y, --yard_tag |Overwrite tag for yardstick docker container (e.g. | +| |stable or latest) | ++------------------------+-----------------------------------------------------+ +| --testarea |compliance testarea within testsuite | ++------------------------+-----------------------------------------------------+ +| --testsuite |compliance testsuite. | ++------------------------+-----------------------------------------------------+ +| -h, --help |Show this message and exit. | ++------------------------+-----------------------------------------------------+ + + +If no arguments are given, the default testsuite will be performed, i.e., the ``compliance_set`` +testsuite with default configurations. + +For more information about **dovetail** command-line interface, please refer to the wiki page [3]_ + +Parameters definition with config file +====================================== + +The default **dovetail run** parameters can be modified in +``dovetail/dovetail/conf/cmd_config.yml``, which is shown as: + +:: + + cli: + arguments: + config: + # This is a simple example of arguments. + # Dovetail has no need of this kind of parameters currently. + # The arguments must be given orderly at the run-time. + # + # docker_tag: + # flags: 'docker_tag' + # path: + # - 'functest/docker_tag' + # - 'yardstick/docker_tag' + control: + + options: + config: + SUT_TYPE: + flags: + - '--SUT_TYPE' + - '-t' + path: + - 'functest/envs' + - 'yardstick/envs' + help: 'Installer type of the system under test (SUT).' + yard_tag: + flags: + - '--yard_tag' + - '-y' + path: + - 'yardstick/docker_tag' + help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)' + func_tag: + flags: + - '--func_tag' + - '-f' + path: + - 'functest/docker_tag' + help: 'Overwrite tag for functest docker container (e.g. stable or latest)' + control: + testsuite: + flags: + - '--testsuite' + default: 'compliance_set' + help: 'compliance testsuite.' + testarea: + flags: + - '--testarea' + default: 'full' + help: 'compliance testarea within testsuite' + +The Click module is used to parse parameters defined in the above config file, +two subsections are included in this file, ``arguments`` and ``options``, +which corresponds to two types of parameters in Click. + +Arguments and Options ++++++++++++++++++++++ +Only ``options`` is used currently, which means parameters can be given (or not) without +sequence restriction. + +Config and control +++++++++++++++++++ + +All options/arguments are divided into two parts: **config** and **control**. +The config ones are used for updating functest or yardstick config files according +to the **path** given. For example, functest's config file is +``dovetail/dovetail/conf/functest_config.yml``, following is a simple example: + +:: + + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + +If running with the command ``python run.py --SUT_TYPE fuel -f stable``, then +the configs will be changed into + +:: + + docker_tag: stable + envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + +The config options/arguments can be added or deleted by modifying +``cmd_config.yml`` rather than changing the source code. However, for control +command, besides adding it into ``cmd_config.yml``, some other operations about +the source code are also needed. + + +. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line diff --git a/docs/testing/developer/testframework/dovetail.tool.installation.rst b/docs/testing/developer/testframework/dovetail.tool.installation.rst new file mode 100644 index 00000000..9d3bc39f --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.installation.rst @@ -0,0 +1,253 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +=========================== +Dovetail Tool Installation +=========================== + +Abstract +######## + +The Dovetail tool supports installation either directly on the Linux host, or within a Docker container. +The detailed installation procedure on the Linux host or via the Docker container are shown +in the following sections. + +There is a need to config the following parameters before Dovetail tool +running on the SUT (System Under Test): + +:: + + SUT_TYPE + SUT type, e.g., apex, compass, fuel, joid, etc + SUT_IP + SUT external network IP, e.g., 192.168.200.2 + NODE_NAME + this can be shown in the test result for users to see which pod the dovetail tool runs + DEPLOY_SCENARIO + deployment scenario, e.g., os-nosdn-nofeature-ha + BUILD_TAG + this can be shown in the test result for users to identify logs + CI_DEBUG + true for debug information printed and false for not printed + DEPLOY_TYPE + baremetal or virtual + +The above configuration can be achieved by + +- modifying the environment variables in files which live under ``/dovetail/conf/`` directory +- set and use Linux environment variables using ``export`` command +- set and use these variables when using ``dovetail run`` command line, for details see the + `Dovetail Command-line Interface`_ section +- enable the OpenStack credential file, which can be achieved by using + ``dovetail run --creds </path/creds>`` + + +Dovetail tool installation on local Linux host environment +########################################################## + +In order to install Dovetail into a local Linux host environment, the following steps should +be performed: + +Downloading Dovetail tool +-------------------------- + +Source code downloading: + +:: + + git clone https://git.opnfv.org/dovetail + +Environment preparation +----------------------- + +Dovetail configuration: + +:: + + cd {dovetail_path}/dovetail/conf + vim functest_config.yml + vim yardstick_config.yml + +To modify the ``envs`` parameters, e.g., SUT_TYPE, SUT_IP, NODE_NAME, +DEPLOY_SCENARIO, BUILD_TAG, CI_DEBUG, DEPLOY_TYPE, in the yml files. + +System dependencies and project configuration information are listed in +``requirements.txt``, ``setup.py`` and ``setup.cfg``, in Ubuntu environment, +to run + +:: + + apt-get -y install python-pip + pip install -e . + +in ``centos`` and ``redhat`` environment, to run + +:: + + yum -y install python-pip + pip install -e . + +in ``fedora`` environment, to run + + +:: + + dnf -y install python-pip + pip install -e . + +then the setup requirements are installed and **dovetail** commmand-line can be used, +by using ``dovetail --help``, **dovetail** command-line details are shown. + +The tool ``tox`` is used, which configuration information is defined in +``tox.ini``, unitttest, flake8-checking, coverage check, etc, can be done +by running: + +:: + + pip install tox + tox + +Compliance and certification test cases +---------------------------------------- + +The compliance and certification test cases can be defined under the ``/dovetail/compliance`` +directory, which is defined in yaml format. +A sample file named ``compliance_set.yml`` is provided as follows: + +:: + + compliance_set: + name: compliance_set + testcases_list: + - dovetail.ipv6.tc001 + +The testcase listed here is dovetail.ipv6.tc001, defined within ``dovetail/testcase``. + +Running Dovetail tool +--------------------- + +After environment preparation is complete and test cases added, the Dovetail tool can be run with + +:: + + dovetail run --testsuite compliance_set + +The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced +with the testsuite yaml file name which want to be run. +If no argument is given, the compliance_set testsuite will be run as the default. + +Moreover, the testcases in given testarea can be run with ``testarea`` command line argument, such as +testarea ``ipv6`` in ``compliance_set`` + +:: + + dovetail run --testsuite compliance_set --testarea ipv6 + +Dovetail provides some sets, ``debug``, ``proposed_tests`` and ``compliance_set``, +``debug`` is used for locally and Continuous Integration(CI) developing purpose, +which provides typical testcase examples, feel free to edit it when develops locally, such as +only to run a testcase which only takes minutes. ``proposed_tests`` is the testcase +candidate which mainly comes from the wiki link [1]_. +``compliance_set`` is used for compliance. Moreover, dovetail tool can be easily +extended to support more complicated compliance requirements, +such as feature set based or scenario based compliance. + +If you want to run the ``debug`` testsuite, just run with + +:: + + dovetail run --testsuite debug + +Running Dovetail in a Docker container +######################################## + +The Dovetail tool can be run in a Docker container by utilizing the following steps: + +Pull Dovetail Docker image from public Dockerhub +------------------------------------------------ + +:: + + sudo docker pull opnfv/dovetail:<Tag> + +<Tag> here is the version, 'latest' is used for the master branch. + +(Optional) Create Docker image from Dockerfile +----------------------------------------------- +Instead of an official Docker image, you can build your own Docker image from +Dockerfile(s) located under the ``docker/`` directory. The ``Dockerfile`` +is based on Ubuntu and the ``Dockerfile.centos7`` file is for a CentOS based +Docker image. + +:: + + cd {dovetail_path}/dovetail/docker + docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master . + +Dovetail Docker container creation +---------------------------------- + +Next, create the ``dovetail-docker-env`` file to define the environment parameters :: + + INSTALLER_TYPE=compass + INSTALLER_IP=192.168.200.2 + DEPLOY_TYPE=baremetal + DEPLOY_SCENARIO=ha-nosdn + CI_DEBUG=true + +or if an OpenStack credential file is provided. + +Then to instantiate the Dovetail Docker container, execute:: + + sudo docker run --privileged=true --rm -t \ + --env-file dovetail-docker-env OR </path/creds> \ + -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \ + -v /var/run/docker.sock:/var/run/docker.sock \ + --name <Dovetail_Container_Name> (optional) \ + opnfv/dovetail:<Tag> /bin/bash + +To attach dovetail container and Running test cases +---------------------------------------------------- + +Before connecting to the container, you can check the container status by running :: + + sudo docker ps -a + +Attach to the container by starting it and obtaining a bash prompt with :: + + sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash + +Inside the container the following commands can be executed to trigger the testing :: + + dovetail run --testsuite compliance_set + +Offline Support +################ + +There are some SUTs that are isolated from the public internet, +so offline support is needed. The idea is to provide all of the packages of dovetail +release in http://artifacts.opnfv.org, then the user can download and transfer to their inner +development environment. + +The packages are shown in [2]_ + +TO DO: to introduce more when it is mature enough. + +Results Output +############### + +The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``. +The compliance report is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``. + +Dovetail Version and Release +############################ + +Dovetail version information is defined in ``setup.cfg``. +At the time of release, it is the dovetail team's responsibility to set +the ``version`` value in ``setup.cfg``. + + +.. [1] https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. +.. [2] http://artifacts.opnfv.org/dovetail.html. diff --git a/docs/testing/developer/testframework/dovetail.tool.overview.rst b/docs/testing/developer/testframework/dovetail.tool.overview.rst new file mode 100644 index 00000000..c4b7fa22 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.overview.rst @@ -0,0 +1,27 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +============== +Dovetail Tool +============== + +What is Dovetail tool +##################### + +A toolset for providing an evaluation of a deployed cloud environment for OPNFV compliance. + +Overview +######## + +Dovetail tool intends on being THE source of tools to provide evaluation of the use of OPNFV trademarks. + +Dovetail tool provides users with a wrapper which leverage other test projects, such as Functest, +Yardstick, etc, that helps to provide evaluation of the use of OPNFV trademarks. +It does so by validating any NFV(Network Function Virtualization) implementation against the test cases in Dovetail. + +Get involved! +############# +- Wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Home +- Git Repo: https://git.opnfv.org/dovetail diff --git a/docs/testing/developer/testframework/dovetail.tool.template.rst b/docs/testing/developer/testframework/dovetail.tool.template.rst new file mode 100644 index 00000000..1a483dd9 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.template.rst @@ -0,0 +1,174 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================== +Template Syntax +================== + +Testcase Template Syntax +========================= + +The testcases used for compliance and certification are defined in the +``dovetail/testcase`` directory, which are written in yaml format. +Take the testcase ``ipv6.tc001.yml`` as an example. It is shown as: + +:: + + --- + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: Bulk creation and deletion of IPv6 networks, ports and subnets + validate: + type: functest + testcase: tempest_smoke_serial + pre_condition: + - 'echo test for precondition in testcase' + cmds: + - 'functest env prepare' + - 'functest testcase run {{validate_testcase}}' + post_condition: + - 'echo test for precondition in testcase' + report: + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + + +- At least three sections named ``name``, ``objective``, ``validate``, + ``report`` must be included +- Section ``name`` distinguishes different test cases used for compliance, + it is composed of 3 parts, ``dovetail.``, belongs to which test area, + and the serial number +- Section ``objective`` briefly describes what this testcase does +- Section ``validate`` defines the scripts and configurations for the + validation of the test case. ``type`` defines which method is used to validate, + 3 ways, i.e., functest, yardstick and shell is supported currently. + ``testcase`` represents the testcases in slicing/tier. +- Section ``report`` defines the sub_testcases to be run. + For ``yardstick``, since it is not sliced by now, + ``sub_testcase_list`` is not needed, only to edit the ``testcase`` part in + section ``validate``, such as ``yardstick_tc027`` + + +Config Template Syntax +======================= + +For Dovetail tool, the config files are located in ``dovetail/dovetail/conf``, which are written +in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files +should be configured as follows, within the ``functest_config.yml`` and ``yardstick_config.yml`` files, +respectively. + +Functest config template syntax +------------------------------- + +An example of ``functest`` configuration is shown as follows: + +:: + + functest: + image_name: opnfv/functest + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'echo test for precondition' + testcase: + cmds: + - 'python /home/opnfv/repos/functest/ci/prepare_env.py start' + - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/functest/results' + store_type: 'file' + file_path: 'tempest/tempest.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +- ``image_name`` and ``docker_tag`` sections define the docker image pulled from the public dockerhub. +- ``envs`` should be correctly configed according to the SUT(System Under Test). +- ``pre_condition`` represents some cleanups or preparations. + ``testcase`` represents the testcases running cmds. + ``post_condition`` represents some cleanups needed after all testcases finished. +- ``result`` section gives the directory of the dovetail tool test result. + ``db_url`` gives the database URL of the dovetail results to be stored. + +Yardstick config template syntax +--------------------------------- + +The configuration in ``yardstick_config.yml`` is similiar to ``functest_config.yml``, +and an example is shown as follows: + +:: + + yardstick: + image_name: opnfv/yardstick + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal + -e EXTERNAL_NETWORK=ext-net' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup' + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh' + testcase: + cmds: + - 'mkdir -p /home/opnfv/yardstick/results/' + - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh && + yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml + --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &> + /home/opnfv/yardstick/results/yardstick.log' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/yardstick/results' + store_type: 'file' + file_path: 'yardstick.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +The main differences between ``yardstick_config.yml`` and ``functest_config.yml`` +are the ``cmds`` subsection. + +Jinja2 template syntax +---------------------- + +Jinja2 module can be used to config the ``{{script_testcase}}``. The +parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``: + +:: + + parameters: + - name: testcase + path: '("name",)' + - name: script_testcase + path: '("scripts", "testcase")' + +Here ``path`` is the path defined in the testcase configuration files. +Take ``script_testcase`` as an example. For testcase ``dovetail.ipv6.tc001``: + +:: + + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation + scripts: + type: functest + testcase: tempest_smoke_serial + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + +The path ("scripts", "testcase") means 'testcase' is the subsection of 'scripts' section. From above, +by following the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` is 'tempest_smoke_serial'. diff --git a/docs/testing/developer/testframework/index.rst b/docs/testing/developer/testframework/index.rst new file mode 100644 index 00000000..b095704a --- /dev/null +++ b/docs/testing/developer/testframework/index.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================== +Dovetail Overview +================== + +.. toctree:: + :maxdepth: 2 + + dovetail.tool.overview.rst + dovetail.tool.installation.rst + dovetail.tool.template.rst + dovetail.tool.cli.rst |