diff options
Diffstat (limited to 'docs/testing')
5 files changed, 0 insertions, 643 deletions
diff --git a/docs/testing/developer/testframework/dovetail.tool.cli.rst b/docs/testing/developer/testframework/dovetail.tool.cli.rst deleted file mode 100644 index f13bc289..00000000 --- a/docs/testing/developer/testframework/dovetail.tool.cli.rst +++ /dev/null @@ -1,173 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. - -================================ -Dovetail Command-line Interface -================================ - - -Command Line Introduction -========================== - -The Dovetail command-line interface provides a method for interacting with -Dovetail from the console. For help on the ``dovetail`` command, enter: - -:: - - dovetail --help - -**dovetail** optional arguments: - -:: - - --version - show program's version number and exit - list <testsuite_name> - list the testsuite details - show <testcase_name> - show the testcase details - run <arguments> - run the testcases - -For **dovetail list**, if the ``<testsuite_name>`` is omitted, -all the testsuites defined under ``/dovetail/compliance`` directory -and related testcases will be listed, otherwise, -the test area and test cases defined in ``<testsuite_name>`` are listed. - -For **dovetail show**, the ``<testcase_name>`` is required, the contents defined -in ``<testcase_name>.yml`` is shown. - -For **dovetail run**, by running ``dovetail run --help``, the ``dovetail run`` -usage is shown as: - -+------------------------+-----------------------------------------------------+ -|Options | | -+========================+=====================================================+ -| -t, --SUT_TYPE |Installer type of the system under test (SUT). | -+------------------------+-----------------------------------------------------+ -| --creds |Openstack credential file location | -+------------------------+-----------------------------------------------------+ -| -i, --SUT_IP |IP of the system under test (SUT). | -+------------------------+-----------------------------------------------------+ -| -d, --debug |True for showing debug log on screen. | -+------------------------+-----------------------------------------------------+ -| -f, --func_tag |Overwrite tag for functest docker container (e.g. | -| |stable or latest) | -+------------------------+-----------------------------------------------------+ -| -y, --yard_tag |Overwrite tag for yardstick docker container (e.g. | -| |stable or latest) | -+------------------------+-----------------------------------------------------+ -| --testarea |compliance testarea within testsuite | -+------------------------+-----------------------------------------------------+ -| --testsuite |compliance testsuite. | -+------------------------+-----------------------------------------------------+ -| -h, --help |Show this message and exit. | -+------------------------+-----------------------------------------------------+ - - -If no arguments are given, the default testsuite will be performed, i.e., the ``compliance_set`` -testsuite with default configurations. - -For more information about **dovetail** command-line interface, please refer to the wiki page [3]_ - -Parameters definition with config file -====================================== - -The default **dovetail run** parameters can be modified in -``dovetail/dovetail/conf/cmd_config.yml``, which is shown as: - -:: - - cli: - arguments: - config: - # This is a simple example of arguments. - # Dovetail has no need of this kind of parameters currently. - # The arguments must be given orderly at the run-time. - # - # docker_tag: - # flags: 'docker_tag' - # path: - # - 'functest/docker_tag' - # - 'yardstick/docker_tag' - control: - - options: - config: - SUT_TYPE: - flags: - - '--SUT_TYPE' - - '-t' - path: - - 'functest/envs' - - 'yardstick/envs' - help: 'Installer type of the system under test (SUT).' - yard_tag: - flags: - - '--yard_tag' - - '-y' - path: - - 'yardstick/docker_tag' - help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)' - func_tag: - flags: - - '--func_tag' - - '-f' - path: - - 'functest/docker_tag' - help: 'Overwrite tag for functest docker container (e.g. stable or latest)' - control: - testsuite: - flags: - - '--testsuite' - default: 'compliance_set' - help: 'compliance testsuite.' - testarea: - flags: - - '--testarea' - default: 'full' - help: 'compliance testarea within testsuite' - -The Click module is used to parse parameters defined in the above config file, -two subsections are included in this file, ``arguments`` and ``options``, -which corresponds to two types of parameters in Click. - -Arguments and Options -+++++++++++++++++++++ -Only ``options`` is used currently, which means parameters can be given (or not) without -sequence restriction. - -Config and control -++++++++++++++++++ - -All options/arguments are divided into two parts: **config** and **control**. -The config ones are used for updating functest or yardstick config files according -to the **path** given. For example, functest's config file is -``dovetail/dovetail/conf/functest_config.yml``, following is a simple example: - -:: - - docker_tag: latest - envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 - -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn - -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' - -If running with the command ``python run.py --SUT_TYPE fuel -f stable``, then -the configs will be changed into - -:: - - docker_tag: stable - envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2 - -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn - -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' - -The config options/arguments can be added or deleted by modifying -``cmd_config.yml`` rather than changing the source code. However, for control -command, besides adding it into ``cmd_config.yml``, some other operations about -the source code are also needed. - - -. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line diff --git a/docs/testing/developer/testframework/dovetail.tool.installation.rst b/docs/testing/developer/testframework/dovetail.tool.installation.rst deleted file mode 100644 index 9d3bc39f..00000000 --- a/docs/testing/developer/testframework/dovetail.tool.installation.rst +++ /dev/null @@ -1,253 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. - -=========================== -Dovetail Tool Installation -=========================== - -Abstract -######## - -The Dovetail tool supports installation either directly on the Linux host, or within a Docker container. -The detailed installation procedure on the Linux host or via the Docker container are shown -in the following sections. - -There is a need to config the following parameters before Dovetail tool -running on the SUT (System Under Test): - -:: - - SUT_TYPE - SUT type, e.g., apex, compass, fuel, joid, etc - SUT_IP - SUT external network IP, e.g., 192.168.200.2 - NODE_NAME - this can be shown in the test result for users to see which pod the dovetail tool runs - DEPLOY_SCENARIO - deployment scenario, e.g., os-nosdn-nofeature-ha - BUILD_TAG - this can be shown in the test result for users to identify logs - CI_DEBUG - true for debug information printed and false for not printed - DEPLOY_TYPE - baremetal or virtual - -The above configuration can be achieved by - -- modifying the environment variables in files which live under ``/dovetail/conf/`` directory -- set and use Linux environment variables using ``export`` command -- set and use these variables when using ``dovetail run`` command line, for details see the - `Dovetail Command-line Interface`_ section -- enable the OpenStack credential file, which can be achieved by using - ``dovetail run --creds </path/creds>`` - - -Dovetail tool installation on local Linux host environment -########################################################## - -In order to install Dovetail into a local Linux host environment, the following steps should -be performed: - -Downloading Dovetail tool --------------------------- - -Source code downloading: - -:: - - git clone https://git.opnfv.org/dovetail - -Environment preparation ------------------------ - -Dovetail configuration: - -:: - - cd {dovetail_path}/dovetail/conf - vim functest_config.yml - vim yardstick_config.yml - -To modify the ``envs`` parameters, e.g., SUT_TYPE, SUT_IP, NODE_NAME, -DEPLOY_SCENARIO, BUILD_TAG, CI_DEBUG, DEPLOY_TYPE, in the yml files. - -System dependencies and project configuration information are listed in -``requirements.txt``, ``setup.py`` and ``setup.cfg``, in Ubuntu environment, -to run - -:: - - apt-get -y install python-pip - pip install -e . - -in ``centos`` and ``redhat`` environment, to run - -:: - - yum -y install python-pip - pip install -e . - -in ``fedora`` environment, to run - - -:: - - dnf -y install python-pip - pip install -e . - -then the setup requirements are installed and **dovetail** commmand-line can be used, -by using ``dovetail --help``, **dovetail** command-line details are shown. - -The tool ``tox`` is used, which configuration information is defined in -``tox.ini``, unitttest, flake8-checking, coverage check, etc, can be done -by running: - -:: - - pip install tox - tox - -Compliance and certification test cases ----------------------------------------- - -The compliance and certification test cases can be defined under the ``/dovetail/compliance`` -directory, which is defined in yaml format. -A sample file named ``compliance_set.yml`` is provided as follows: - -:: - - compliance_set: - name: compliance_set - testcases_list: - - dovetail.ipv6.tc001 - -The testcase listed here is dovetail.ipv6.tc001, defined within ``dovetail/testcase``. - -Running Dovetail tool ---------------------- - -After environment preparation is complete and test cases added, the Dovetail tool can be run with - -:: - - dovetail run --testsuite compliance_set - -The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced -with the testsuite yaml file name which want to be run. -If no argument is given, the compliance_set testsuite will be run as the default. - -Moreover, the testcases in given testarea can be run with ``testarea`` command line argument, such as -testarea ``ipv6`` in ``compliance_set`` - -:: - - dovetail run --testsuite compliance_set --testarea ipv6 - -Dovetail provides some sets, ``debug``, ``proposed_tests`` and ``compliance_set``, -``debug`` is used for locally and Continuous Integration(CI) developing purpose, -which provides typical testcase examples, feel free to edit it when develops locally, such as -only to run a testcase which only takes minutes. ``proposed_tests`` is the testcase -candidate which mainly comes from the wiki link [1]_. -``compliance_set`` is used for compliance. Moreover, dovetail tool can be easily -extended to support more complicated compliance requirements, -such as feature set based or scenario based compliance. - -If you want to run the ``debug`` testsuite, just run with - -:: - - dovetail run --testsuite debug - -Running Dovetail in a Docker container -######################################## - -The Dovetail tool can be run in a Docker container by utilizing the following steps: - -Pull Dovetail Docker image from public Dockerhub ------------------------------------------------- - -:: - - sudo docker pull opnfv/dovetail:<Tag> - -<Tag> here is the version, 'latest' is used for the master branch. - -(Optional) Create Docker image from Dockerfile ------------------------------------------------ -Instead of an official Docker image, you can build your own Docker image from -Dockerfile(s) located under the ``docker/`` directory. The ``Dockerfile`` -is based on Ubuntu and the ``Dockerfile.centos7`` file is for a CentOS based -Docker image. - -:: - - cd {dovetail_path}/dovetail/docker - docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master . - -Dovetail Docker container creation ----------------------------------- - -Next, create the ``dovetail-docker-env`` file to define the environment parameters :: - - INSTALLER_TYPE=compass - INSTALLER_IP=192.168.200.2 - DEPLOY_TYPE=baremetal - DEPLOY_SCENARIO=ha-nosdn - CI_DEBUG=true - -or if an OpenStack credential file is provided. - -Then to instantiate the Dovetail Docker container, execute:: - - sudo docker run --privileged=true --rm -t \ - --env-file dovetail-docker-env OR </path/creds> \ - -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \ - -v /var/run/docker.sock:/var/run/docker.sock \ - --name <Dovetail_Container_Name> (optional) \ - opnfv/dovetail:<Tag> /bin/bash - -To attach dovetail container and Running test cases ----------------------------------------------------- - -Before connecting to the container, you can check the container status by running :: - - sudo docker ps -a - -Attach to the container by starting it and obtaining a bash prompt with :: - - sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash - -Inside the container the following commands can be executed to trigger the testing :: - - dovetail run --testsuite compliance_set - -Offline Support -################ - -There are some SUTs that are isolated from the public internet, -so offline support is needed. The idea is to provide all of the packages of dovetail -release in http://artifacts.opnfv.org, then the user can download and transfer to their inner -development environment. - -The packages are shown in [2]_ - -TO DO: to introduce more when it is mature enough. - -Results Output -############### - -The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``. -The compliance report is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``. - -Dovetail Version and Release -############################ - -Dovetail version information is defined in ``setup.cfg``. -At the time of release, it is the dovetail team's responsibility to set -the ``version`` value in ``setup.cfg``. - - -.. [1] https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. -.. [2] http://artifacts.opnfv.org/dovetail.html. diff --git a/docs/testing/developer/testframework/dovetail.tool.overview.rst b/docs/testing/developer/testframework/dovetail.tool.overview.rst deleted file mode 100644 index c4b7fa22..00000000 --- a/docs/testing/developer/testframework/dovetail.tool.overview.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. - -============== -Dovetail Tool -============== - -What is Dovetail tool -##################### - -A toolset for providing an evaluation of a deployed cloud environment for OPNFV compliance. - -Overview -######## - -Dovetail tool intends on being THE source of tools to provide evaluation of the use of OPNFV trademarks. - -Dovetail tool provides users with a wrapper which leverage other test projects, such as Functest, -Yardstick, etc, that helps to provide evaluation of the use of OPNFV trademarks. -It does so by validating any NFV(Network Function Virtualization) implementation against the test cases in Dovetail. - -Get involved! -############# -- Wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Home -- Git Repo: https://git.opnfv.org/dovetail diff --git a/docs/testing/developer/testframework/dovetail.tool.template.rst b/docs/testing/developer/testframework/dovetail.tool.template.rst deleted file mode 100644 index 1a483dd9..00000000 --- a/docs/testing/developer/testframework/dovetail.tool.template.rst +++ /dev/null @@ -1,174 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. - -================== -Template Syntax -================== - -Testcase Template Syntax -========================= - -The testcases used for compliance and certification are defined in the -``dovetail/testcase`` directory, which are written in yaml format. -Take the testcase ``ipv6.tc001.yml`` as an example. It is shown as: - -:: - - --- - dovetail.ipv6.tc001: - name: dovetail.ipv6.tc001 - objective: Bulk creation and deletion of IPv6 networks, ports and subnets - validate: - type: functest - testcase: tempest_smoke_serial - pre_condition: - - 'echo test for precondition in testcase' - cmds: - - 'functest env prepare' - - 'functest testcase run {{validate_testcase}}' - post_condition: - - 'echo test for precondition in testcase' - report: - sub_testcase_list: - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet - - -- At least three sections named ``name``, ``objective``, ``validate``, - ``report`` must be included -- Section ``name`` distinguishes different test cases used for compliance, - it is composed of 3 parts, ``dovetail.``, belongs to which test area, - and the serial number -- Section ``objective`` briefly describes what this testcase does -- Section ``validate`` defines the scripts and configurations for the - validation of the test case. ``type`` defines which method is used to validate, - 3 ways, i.e., functest, yardstick and shell is supported currently. - ``testcase`` represents the testcases in slicing/tier. -- Section ``report`` defines the sub_testcases to be run. - For ``yardstick``, since it is not sliced by now, - ``sub_testcase_list`` is not needed, only to edit the ``testcase`` part in - section ``validate``, such as ``yardstick_tc027`` - - -Config Template Syntax -======================= - -For Dovetail tool, the config files are located in ``dovetail/dovetail/conf``, which are written -in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files -should be configured as follows, within the ``functest_config.yml`` and ``yardstick_config.yml`` files, -respectively. - -Functest config template syntax -------------------------------- - -An example of ``functest`` configuration is shown as follows: - -:: - - functest: - image_name: opnfv/functest - docker_tag: latest - envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 - -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn - -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' - opts: '-id --privileged=true' - pre_condition: - cmds: - - 'echo test for precondition' - testcase: - cmds: - - 'python /home/opnfv/repos/functest/ci/prepare_env.py start' - - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r' - post_condition: - cmds: - - '' - result: - dir: '/home/opnfv/functest/results' - store_type: 'file' - file_path: 'tempest/tempest.log' - db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' - -- ``image_name`` and ``docker_tag`` sections define the docker image pulled from the public dockerhub. -- ``envs`` should be correctly configed according to the SUT(System Under Test). -- ``pre_condition`` represents some cleanups or preparations. - ``testcase`` represents the testcases running cmds. - ``post_condition`` represents some cleanups needed after all testcases finished. -- ``result`` section gives the directory of the dovetail tool test result. - ``db_url`` gives the database URL of the dovetail results to be stored. - -Yardstick config template syntax ---------------------------------- - -The configuration in ``yardstick_config.yml`` is similiar to ``functest_config.yml``, -and an example is shown as follows: - -:: - - yardstick: - image_name: opnfv/yardstick - docker_tag: latest - envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 - -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn - -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal - -e EXTERNAL_NETWORK=ext-net' - opts: '-id --privileged=true' - pre_condition: - cmds: - - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && - source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup' - - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && - cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh' - testcase: - cmds: - - 'mkdir -p /home/opnfv/yardstick/results/' - - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh && - yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml - --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &> - /home/opnfv/yardstick/results/yardstick.log' - post_condition: - cmds: - - '' - result: - dir: '/home/opnfv/yardstick/results' - store_type: 'file' - file_path: 'yardstick.log' - db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' - -The main differences between ``yardstick_config.yml`` and ``functest_config.yml`` -are the ``cmds`` subsection. - -Jinja2 template syntax ----------------------- - -Jinja2 module can be used to config the ``{{script_testcase}}``. The -parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``: - -:: - - parameters: - - name: testcase - path: '("name",)' - - name: script_testcase - path: '("scripts", "testcase")' - -Here ``path`` is the path defined in the testcase configuration files. -Take ``script_testcase`` as an example. For testcase ``dovetail.ipv6.tc001``: - -:: - - dovetail.ipv6.tc001: - name: dovetail.ipv6.tc001 - objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation - scripts: - type: functest - testcase: tempest_smoke_serial - sub_testcase_list: - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port - - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet - -The path ("scripts", "testcase") means 'testcase' is the subsection of 'scripts' section. From above, -by following the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` is 'tempest_smoke_serial'. diff --git a/docs/testing/developer/testframework/index.rst b/docs/testing/developer/testframework/index.rst deleted file mode 100644 index b095704a..00000000 --- a/docs/testing/developer/testframework/index.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. - -================== -Dovetail Overview -================== - -.. toctree:: - :maxdepth: 2 - - dovetail.tool.overview.rst - dovetail.tool.installation.rst - dovetail.tool.template.rst - dovetail.tool.cli.rst |