diff options
| author |   ChristopherPrice <christopher.price@ericsson.com> | 2017-04-26 15:25:55 +0200 |
|---|---|---|
| committer | Christopher Price <christopher.price@ericsson.com> | 2017-04-27 10:04:23 +0000 |
| commit | b8722fae5e61ca71a43f3d69d1d093906b140bbe (patch) | |
| tree | 40584e9f72f3b38cd98f3064df0d6b5a476dfa7a /docs/testing | |
| parent | acee332cda9fab1315ee1147ab12bc4003f05990 (diff) | |
Setting the document directory structure according to the Plugfest outcome.
Implemented a directory and file structure that supports the requirements defined
in the plugfest etherpad: https://etherpad.opnfv.org/p/cvp_tooling_sketch
Change-Id: Icb7d2db4b0132f2039c680883e173ff5d7fc0120
Signed-off-by: ChristopherPrice <christopher.price@ericsson.com>
Diffstat (limited to 'docs/testing')
23 files changed, 1598 insertions, 13 deletions
diff --git a/docs/testing/developer/testframework/dovetail.tool.cli.rst b/docs/testing/developer/testframework/dovetail.tool.cli.rst new file mode 100644 index 00000000..f13bc289 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.cli.rst @@ -0,0 +1,173 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================================ +Dovetail Command-line Interface +================================ + + +Command Line Introduction +========================== + +The Dovetail command-line interface provides a method for interacting with +Dovetail from the console. For help on the ``dovetail`` command, enter: + +:: + + dovetail --help + +**dovetail** optional arguments: + +:: + + --version + show program's version number and exit + list <testsuite_name> + list the testsuite details + show <testcase_name> + show the testcase details + run <arguments> + run the testcases + +For **dovetail list**, if the ``<testsuite_name>`` is omitted, +all the testsuites defined under ``/dovetail/compliance`` directory +and related testcases will be listed, otherwise, +the test area and test cases defined in ``<testsuite_name>`` are listed. + +For **dovetail show**, the ``<testcase_name>`` is required, the contents defined +in ``<testcase_name>.yml`` is shown. + +For **dovetail run**, by running ``dovetail run --help``, the ``dovetail run`` +usage is shown as: + ++------------------------+-----------------------------------------------------+ +|Options | | ++========================+=====================================================+ +| -t, --SUT_TYPE |Installer type of the system under test (SUT). | ++------------------------+-----------------------------------------------------+ +| --creds |Openstack credential file location | ++------------------------+-----------------------------------------------------+ +| -i, --SUT_IP |IP of the system under test (SUT). | ++------------------------+-----------------------------------------------------+ +| -d, --debug |True for showing debug log on screen. | ++------------------------+-----------------------------------------------------+ +| -f, --func_tag |Overwrite tag for functest docker container (e.g. | +| |stable or latest) | ++------------------------+-----------------------------------------------------+ +| -y, --yard_tag |Overwrite tag for yardstick docker container (e.g. | +| |stable or latest) | ++------------------------+-----------------------------------------------------+ +| --testarea |compliance testarea within testsuite | ++------------------------+-----------------------------------------------------+ +| --testsuite |compliance testsuite. | ++------------------------+-----------------------------------------------------+ +| -h, --help |Show this message and exit. | ++------------------------+-----------------------------------------------------+ + + +If no arguments are given, the default testsuite will be performed, i.e., the ``compliance_set`` +testsuite with default configurations. + +For more information about **dovetail** command-line interface, please refer to the wiki page [3]_ + +Parameters definition with config file +====================================== + +The default **dovetail run** parameters can be modified in +``dovetail/dovetail/conf/cmd_config.yml``, which is shown as: + +:: + + cli: + arguments: + config: + # This is a simple example of arguments. + # Dovetail has no need of this kind of parameters currently. + # The arguments must be given orderly at the run-time. + # + # docker_tag: + # flags: 'docker_tag' + # path: + # - 'functest/docker_tag' + # - 'yardstick/docker_tag' + control: + + options: + config: + SUT_TYPE: + flags: + - '--SUT_TYPE' + - '-t' + path: + - 'functest/envs' + - 'yardstick/envs' + help: 'Installer type of the system under test (SUT).' + yard_tag: + flags: + - '--yard_tag' + - '-y' + path: + - 'yardstick/docker_tag' + help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)' + func_tag: + flags: + - '--func_tag' + - '-f' + path: + - 'functest/docker_tag' + help: 'Overwrite tag for functest docker container (e.g. stable or latest)' + control: + testsuite: + flags: + - '--testsuite' + default: 'compliance_set' + help: 'compliance testsuite.' + testarea: + flags: + - '--testarea' + default: 'full' + help: 'compliance testarea within testsuite' + +The Click module is used to parse parameters defined in the above config file, +two subsections are included in this file, ``arguments`` and ``options``, +which corresponds to two types of parameters in Click. + +Arguments and Options ++++++++++++++++++++++ +Only ``options`` is used currently, which means parameters can be given (or not) without +sequence restriction. + +Config and control +++++++++++++++++++ + +All options/arguments are divided into two parts: **config** and **control**. +The config ones are used for updating functest or yardstick config files according +to the **path** given. For example, functest's config file is +``dovetail/dovetail/conf/functest_config.yml``, following is a simple example: + +:: + + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + +If running with the command ``python run.py --SUT_TYPE fuel -f stable``, then +the configs will be changed into + +:: + + docker_tag: stable + envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + +The config options/arguments can be added or deleted by modifying +``cmd_config.yml`` rather than changing the source code. However, for control +command, besides adding it into ``cmd_config.yml``, some other operations about +the source code are also needed. + + +. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line diff --git a/docs/testing/developer/testframework/dovetail.tool.installation.rst b/docs/testing/developer/testframework/dovetail.tool.installation.rst new file mode 100644 index 00000000..9d3bc39f --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.installation.rst @@ -0,0 +1,253 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +=========================== +Dovetail Tool Installation +=========================== + +Abstract +######## + +The Dovetail tool supports installation either directly on the Linux host, or within a Docker container. +The detailed installation procedure on the Linux host or via the Docker container are shown +in the following sections. + +There is a need to config the following parameters before Dovetail tool +running on the SUT (System Under Test): + +:: + + SUT_TYPE + SUT type, e.g., apex, compass, fuel, joid, etc + SUT_IP + SUT external network IP, e.g., 192.168.200.2 + NODE_NAME + this can be shown in the test result for users to see which pod the dovetail tool runs + DEPLOY_SCENARIO + deployment scenario, e.g., os-nosdn-nofeature-ha + BUILD_TAG + this can be shown in the test result for users to identify logs + CI_DEBUG + true for debug information printed and false for not printed + DEPLOY_TYPE + baremetal or virtual + +The above configuration can be achieved by + +- modifying the environment variables in files which live under ``/dovetail/conf/`` directory +- set and use Linux environment variables using ``export`` command +- set and use these variables when using ``dovetail run`` command line, for details see the + `Dovetail Command-line Interface`_ section +- enable the OpenStack credential file, which can be achieved by using + ``dovetail run --creds </path/creds>`` + + +Dovetail tool installation on local Linux host environment +########################################################## + +In order to install Dovetail into a local Linux host environment, the following steps should +be performed: + +Downloading Dovetail tool +-------------------------- + +Source code downloading: + +:: + + git clone https://git.opnfv.org/dovetail + +Environment preparation +----------------------- + +Dovetail configuration: + +:: + + cd {dovetail_path}/dovetail/conf + vim functest_config.yml + vim yardstick_config.yml + +To modify the ``envs`` parameters, e.g., SUT_TYPE, SUT_IP, NODE_NAME, +DEPLOY_SCENARIO, BUILD_TAG, CI_DEBUG, DEPLOY_TYPE, in the yml files. + +System dependencies and project configuration information are listed in +``requirements.txt``, ``setup.py`` and ``setup.cfg``, in Ubuntu environment, +to run + +:: + + apt-get -y install python-pip + pip install -e . + +in ``centos`` and ``redhat`` environment, to run + +:: + + yum -y install python-pip + pip install -e . + +in ``fedora`` environment, to run + + +:: + + dnf -y install python-pip + pip install -e . + +then the setup requirements are installed and **dovetail** commmand-line can be used, +by using ``dovetail --help``, **dovetail** command-line details are shown. + +The tool ``tox`` is used, which configuration information is defined in +``tox.ini``, unitttest, flake8-checking, coverage check, etc, can be done +by running: + +:: + + pip install tox + tox + +Compliance and certification test cases +---------------------------------------- + +The compliance and certification test cases can be defined under the ``/dovetail/compliance`` +directory, which is defined in yaml format. +A sample file named ``compliance_set.yml`` is provided as follows: + +:: + + compliance_set: + name: compliance_set + testcases_list: + - dovetail.ipv6.tc001 + +The testcase listed here is dovetail.ipv6.tc001, defined within ``dovetail/testcase``. + +Running Dovetail tool +--------------------- + +After environment preparation is complete and test cases added, the Dovetail tool can be run with + +:: + + dovetail run --testsuite compliance_set + +The value ``compliance_set`` passed to the ``testsuite`` flag can be replaced +with the testsuite yaml file name which want to be run. +If no argument is given, the compliance_set testsuite will be run as the default. + +Moreover, the testcases in given testarea can be run with ``testarea`` command line argument, such as +testarea ``ipv6`` in ``compliance_set`` + +:: + + dovetail run --testsuite compliance_set --testarea ipv6 + +Dovetail provides some sets, ``debug``, ``proposed_tests`` and ``compliance_set``, +``debug`` is used for locally and Continuous Integration(CI) developing purpose, +which provides typical testcase examples, feel free to edit it when develops locally, such as +only to run a testcase which only takes minutes. ``proposed_tests`` is the testcase +candidate which mainly comes from the wiki link [1]_. +``compliance_set`` is used for compliance. Moreover, dovetail tool can be easily +extended to support more complicated compliance requirements, +such as feature set based or scenario based compliance. + +If you want to run the ``debug`` testsuite, just run with + +:: + + dovetail run --testsuite debug + +Running Dovetail in a Docker container +######################################## + +The Dovetail tool can be run in a Docker container by utilizing the following steps: + +Pull Dovetail Docker image from public Dockerhub +------------------------------------------------ + +:: + + sudo docker pull opnfv/dovetail:<Tag> + +<Tag> here is the version, 'latest' is used for the master branch. + +(Optional) Create Docker image from Dockerfile +----------------------------------------------- +Instead of an official Docker image, you can build your own Docker image from +Dockerfile(s) located under the ``docker/`` directory. The ``Dockerfile`` +is based on Ubuntu and the ``Dockerfile.centos7`` file is for a CentOS based +Docker image. + +:: + + cd {dovetail_path}/dovetail/docker + docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master . + +Dovetail Docker container creation +---------------------------------- + +Next, create the ``dovetail-docker-env`` file to define the environment parameters :: + + INSTALLER_TYPE=compass + INSTALLER_IP=192.168.200.2 + DEPLOY_TYPE=baremetal + DEPLOY_SCENARIO=ha-nosdn + CI_DEBUG=true + +or if an OpenStack credential file is provided. + +Then to instantiate the Dovetail Docker container, execute:: + + sudo docker run --privileged=true --rm -t \ + --env-file dovetail-docker-env OR </path/creds> \ + -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \ + -v /var/run/docker.sock:/var/run/docker.sock \ + --name <Dovetail_Container_Name> (optional) \ + opnfv/dovetail:<Tag> /bin/bash + +To attach dovetail container and Running test cases +---------------------------------------------------- + +Before connecting to the container, you can check the container status by running :: + + sudo docker ps -a + +Attach to the container by starting it and obtaining a bash prompt with :: + + sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash + +Inside the container the following commands can be executed to trigger the testing :: + + dovetail run --testsuite compliance_set + +Offline Support +################ + +There are some SUTs that are isolated from the public internet, +so offline support is needed. The idea is to provide all of the packages of dovetail +release in http://artifacts.opnfv.org, then the user can download and transfer to their inner +development environment. + +The packages are shown in [2]_ + +TO DO: to introduce more when it is mature enough. + +Results Output +############### + +The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``. +The compliance report is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``. + +Dovetail Version and Release +############################ + +Dovetail version information is defined in ``setup.cfg``. +At the time of release, it is the dovetail team's responsibility to set +the ``version`` value in ``setup.cfg``. + + +.. [1] https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. +.. [2] http://artifacts.opnfv.org/dovetail.html. diff --git a/docs/testing/developer/testframework/dovetail.tool.overview.rst b/docs/testing/developer/testframework/dovetail.tool.overview.rst new file mode 100644 index 00000000..c4b7fa22 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.overview.rst @@ -0,0 +1,27 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +============== +Dovetail Tool +============== + +What is Dovetail tool +##################### + +A toolset for providing an evaluation of a deployed cloud environment for OPNFV compliance. + +Overview +######## + +Dovetail tool intends on being THE source of tools to provide evaluation of the use of OPNFV trademarks. + +Dovetail tool provides users with a wrapper which leverage other test projects, such as Functest, +Yardstick, etc, that helps to provide evaluation of the use of OPNFV trademarks. +It does so by validating any NFV(Network Function Virtualization) implementation against the test cases in Dovetail. + +Get involved! +############# +- Wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Home +- Git Repo: https://git.opnfv.org/dovetail diff --git a/docs/testing/developer/testframework/dovetail.tool.template.rst b/docs/testing/developer/testframework/dovetail.tool.template.rst new file mode 100644 index 00000000..1a483dd9 --- /dev/null +++ b/docs/testing/developer/testframework/dovetail.tool.template.rst @@ -0,0 +1,174 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================== +Template Syntax +================== + +Testcase Template Syntax +========================= + +The testcases used for compliance and certification are defined in the +``dovetail/testcase`` directory, which are written in yaml format. +Take the testcase ``ipv6.tc001.yml`` as an example. It is shown as: + +:: + + --- + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: Bulk creation and deletion of IPv6 networks, ports and subnets + validate: + type: functest + testcase: tempest_smoke_serial + pre_condition: + - 'echo test for precondition in testcase' + cmds: + - 'functest env prepare' + - 'functest testcase run {{validate_testcase}}' + post_condition: + - 'echo test for precondition in testcase' + report: + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + + +- At least three sections named ``name``, ``objective``, ``validate``, + ``report`` must be included +- Section ``name`` distinguishes different test cases used for compliance, + it is composed of 3 parts, ``dovetail.``, belongs to which test area, + and the serial number +- Section ``objective`` briefly describes what this testcase does +- Section ``validate`` defines the scripts and configurations for the + validation of the test case. ``type`` defines which method is used to validate, + 3 ways, i.e., functest, yardstick and shell is supported currently. + ``testcase`` represents the testcases in slicing/tier. +- Section ``report`` defines the sub_testcases to be run. + For ``yardstick``, since it is not sliced by now, + ``sub_testcase_list`` is not needed, only to edit the ``testcase`` part in + section ``validate``, such as ``yardstick_tc027`` + + +Config Template Syntax +======================= + +For Dovetail tool, the config files are located in ``dovetail/dovetail/conf``, which are written +in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files +should be configured as follows, within the ``functest_config.yml`` and ``yardstick_config.yml`` files, +respectively. + +Functest config template syntax +------------------------------- + +An example of ``functest`` configuration is shown as follows: + +:: + + functest: + image_name: opnfv/functest + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'echo test for precondition' + testcase: + cmds: + - 'python /home/opnfv/repos/functest/ci/prepare_env.py start' + - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/functest/results' + store_type: 'file' + file_path: 'tempest/tempest.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +- ``image_name`` and ``docker_tag`` sections define the docker image pulled from the public dockerhub. +- ``envs`` should be correctly configed according to the SUT(System Under Test). +- ``pre_condition`` represents some cleanups or preparations. + ``testcase`` represents the testcases running cmds. + ``post_condition`` represents some cleanups needed after all testcases finished. +- ``result`` section gives the directory of the dovetail tool test result. + ``db_url`` gives the database URL of the dovetail results to be stored. + +Yardstick config template syntax +--------------------------------- + +The configuration in ``yardstick_config.yml`` is similiar to ``functest_config.yml``, +and an example is shown as follows: + +:: + + yardstick: + image_name: opnfv/yardstick + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal + -e EXTERNAL_NETWORK=ext-net' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup' + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh' + testcase: + cmds: + - 'mkdir -p /home/opnfv/yardstick/results/' + - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh && + yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml + --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &> + /home/opnfv/yardstick/results/yardstick.log' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/yardstick/results' + store_type: 'file' + file_path: 'yardstick.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +The main differences between ``yardstick_config.yml`` and ``functest_config.yml`` +are the ``cmds`` subsection. + +Jinja2 template syntax +---------------------- + +Jinja2 module can be used to config the ``{{script_testcase}}``. The +parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``: + +:: + + parameters: + - name: testcase + path: '("name",)' + - name: script_testcase + path: '("scripts", "testcase")' + +Here ``path`` is the path defined in the testcase configuration files. +Take ``script_testcase`` as an example. For testcase ``dovetail.ipv6.tc001``: + +:: + + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation + scripts: + type: functest + testcase: tempest_smoke_serial + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + +The path ("scripts", "testcase") means 'testcase' is the subsection of 'scripts' section. From above, +by following the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` is 'tempest_smoke_serial'. diff --git a/docs/testing/developer/testframework/index.rst b/docs/testing/developer/testframework/index.rst new file mode 100644 index 00000000..b095704a --- /dev/null +++ b/docs/testing/developer/testframework/index.rst @@ -0,0 +1,16 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================== +Dovetail Overview +================== + +.. toctree:: + :maxdepth: 2 + + dovetail.tool.overview.rst + dovetail.tool.installation.rst + dovetail.tool.template.rst + dovetail.tool.cli.rst diff --git a/docs/testing/user/certificationworkflow/ApplicationForm.rst b/docs/testing/user/certificationworkflow/ApplicationForm.rst new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/user/certificationworkflow/ApplicationForm.rst diff --git a/docs/testing/user/certificationworkflow/index.rst b/docs/testing/user/certificationworkflow/index.rst new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/user/certificationworkflow/index.rst diff --git a/docs/testing/user/testspecification/faultmanagement/index.rst b/docs/testing/user/testspecification/faultmanagement/index.rst new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/user/testspecification/faultmanagement/index.rst diff --git a/docs/testing/user/testspecification/highavailability/index.rst b/docs/testing/user/testspecification/highavailability/index.rst new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/docs/testing/user/testspecification/highavailability/index.rst diff --git a/docs/testing/user/testspecification/index.rst b/docs/testing/user/testspecification/index.rst index 4d70ea1d..6d2bedf8 100644 --- a/docs/testing/user/testspecification/index.rst +++ b/docs/testing/user/testspecification/index.rst @@ -9,20 +9,24 @@ Compliance and Verification program test specification .. toctree:: :maxdepth: 2 -Version history -=============== +Introduction +============ -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2017-03-15 | 0.0.0 | | Blank | -| | | | | -+------------+----------+------------------+----------------------------------+ +The OPNFV CVP provides a series or test areas aimed to evaluate the operation of an +NFV system in accordance with carrier networking needs. Each test area conatins a number +of associated test cases which are descrobed in detail in the associated test specification. +All tests in the CVP are required to fulfill a specific set of criteria in order that the +CVP is able to provide a fair assesment of the system under test. Test requierements are +described in the 'Test Case Requierements'_ document. -Introduction -============ +All tests areas addressed in the CVP are covered in the following test specification documents. + + ./faultmanagement/index.rst + ./highavailability/index.rst + ./vimoperationscompute/index.rst + ./vimoperationsidentity/index.rst + ./vimoperationsimage/index.rst + ./vimoperationsnetwork/index.rst + ./vimoperationsvolume/index.rst -Write useful things about the test cases here. Break each section/area out into a seperate -RST file for manageablity and tracking. diff --git a/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst b/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst new file mode 100644 index 00000000..9e403472 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/designspecification.rst @@ -0,0 +1,133 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Christopher Price (Ericsson AB) and others + +============================== +IPv6 test design specification +============================== + +This document outlines the approach and method for testing IPv6 in the OPNFV compliance test +suite. Providing a brief outline of the features to be tested, the methodology for testing, +schema's and criteria. + +Features to be tested +===================== + +The IPv6 compliance test plan outlines the method for testing IPv6 compliance to the OPNFV +platform behaviours and features of IPv6 enabled VNFi platforms. The specific features to +be tested by the IPv6 compliance test suite is outlined in the following table. + +.. table:: + :class: longtable + ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Features / Requirements |Tests available | Test Cases | ++===========================================================+===================+====================================================================+ +|All topologies work in a multi-tenant environment |No | | +| | | | +| | | | +| | | | +| | | | +| | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|IPv6 VM to VM only |No | | +| | | | +| | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|IPv6 external L2 VLAN directly attached to a VM |No | | +| | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|IPv6 subnet routed via L3 agent to an external IPv6 network|No | | +| | | | +|1. Both VLAN and overlay (e.g. GRE, VXLAN) subnet attached | | | +| to VMs; | | | +|2. Must be able to support multiple L3 agents for a given | | | +| external network to support scaling (neutron scheduler | | | +| to assign vRouters to the L3 agents) | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Ability for a NIC to support both IPv4 and IPv6 (dual |No | | +|stack) address. | | | +| | | | +|1. VM with a single interface associated with a network, | | | +| which is then associated with two subnets. | | | +|2. VM with two different interfaces associated with two | | | +| different networks and two different subnets. | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Support IPv6 Address assignment modes. |No | | +| | | | +|1. SLAAC | | | +|2. DHCPv6 Stateless | | | +|3. DHCPv6 Stateful | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Ability to create a port on an IPv6 DHCPv6 Stateful subnet |No | | +|and assign a specific IPv6 address to the port and have it | | | +|taken out of the DHCP address pool. | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Full support for IPv6 matching (i.e., IPv6, ICMPv6, TCP, |No | | +|UDP) in security groups. Ability to control and manage all | | | +|IPv6 security group capabilities via Neutron/Nova API (REST| | | +|and CLI) as well as via Horizon. | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|During network/subnet/router create, there should be an |No | | +|option to allow user to specify the type of address | | | +|management they would like. This includes all options | | | +|including those low priority if implemented (e.g., toggle | | | +|on/off router and address prefix advertisements); It must | | | +|be supported via Neutron API (REST and CLI) as well as via | | | +|Horizon | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Security groups anti-spoofing: Prevent VM from using a |No | | +|source IPv6/MAC address which is not assigned to the VM | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Protect tenant and provider network from rogue RAs |No | | +| | | | +| | | | +| | | | +| | | | +| | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Support the ability to assign multiple IPv6 addresses to |No | | +|an interface; both for Neutron router interfaces and VM | | | +|interfaces. | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Ability for a VM to support a mix of multiple IPv4 and IPv6|No | | +|networks, including multiples of the same type. | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|Support for IPv6 Prefix Delegation. |No | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|IPv6 First-Hop Security, IPv6 ND spoofing |No | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ +|IPv6 support in Neutron Layer3 High Availability |No | | +|(keepalived+VRRP). | | | ++-----------------------------------------------------------+-------------------+--------------------------------------------------------------------+ + + +Test approach for IPv6 +====================== + +The most common approach for testing IPv6 capabilities in the test suite is through interaction with the SUT control plane. +In this instance the test framework will exercise the NBI provided by the VIM to configure and leverage IPv6 related features +in the platform, instantiate workloads, and invoke behaviours in the platform. The suite may also interact directly with the +data plane to exercise platform capabilities and further invoke helper functions on the platform for the same purpose. + +Test result analysis +-------------------- + +All functional tests in the IPv6 test suite will provide a pass/fail result on completion of the test. In addition test logs +and relevant additional information will be provided as part of the test log, available on test suite completion. + +Some tests in the compliance suite measure such metrics as latency and performance. At this time these tests are intended to +provide a feature based pass/fail metric not related to system performance. +These tests may however provide detailed results of performance and latency in the 'test report'_ document. + +Test identification +=================== + +TBD: WE need to identify the test naming scheme we will use in DoveTail in order that we can cross reference to the test +projects and maintain our suite effectively. This naming scheme needs to be externally relevant to non-OPNFV consumers and as +such some consideration is required on the selection. + +Pass Fail Criteria +================== + +This section requires some further work with the test teams to identify how and where we generate, store and provide results. diff --git a/docs/testing/user/testspecification/old_files/ipv6/index.rst b/docs/testing/user/testspecification/old_files/ipv6/index.rst new file mode 100644 index 00000000..a806d644 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/index.rst @@ -0,0 +1,19 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +******************************* +OPNFV IPv6 Compliance Test Plan +******************************* + +.. toctree:: + :maxdepth: 2 + + ./testplan.rst + ./testprocedure.rst + ./testspecification.rst + ./designspecification.rst + ./ipv6.tc001.specification.rst + ./ipv6.tc026.specification.rst + ./ipv6_all_testcases.rst + diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst new file mode 100644 index 00000000..5afb2095 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc001.specification.rst @@ -0,0 +1,59 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +================================================================================================== +Dovetail IPv6 tc001 specification - Bulk Creation and Deletion of IPv6 Networks, Ports and Subnets +================================================================================================== + + ++-----------------------+----------------------------------------------------------------------------------------------------+ +|test case name |Bulk creation and deletion of IPv6 networks, ports and subnets | +| | | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|id |dovetail.ipv6.tc001 | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|objective |To verify that platform is able to create/delete networks, ports and subnets in bulk operation | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|test items |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network | +| |{idempotent_id('d4f9024d-1e28-4fc1-a6b1-25dbc6fa11e2')} | +| |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port | +| |{idempotent_id('48037ff2-e889-4c3b-b86a-8e3f34d2d060')} | +| |tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet | +| |{idempotent_id('8936533b-c0aa-4f29-8e53-6cc873aec489')} | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|environmental | | +|requirements & | environment can be deployed on bare metal of virtualized infrastructure | +|preconditions | deployment can be HA or non-HA | +| | | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|scenario dependencies | NA | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|procedural |Step 1: create/delete network: | +|requirements | create 2 networks in one request | +| | asserting that the networks are found in the list after creation | +| | | +| |Step 2: create/delete subnet: | +| | create 2 subnets in one request | +| | asserting that the subnets are found in the list after creation | +| | | +| |Step 3: create/delete port: | +| | create 2 ports in one request | +| | asserting that the ports are found in the list after creation | +| | | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|input specifications |The parameters needed to execute Neutron network APIs. | +| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|output specifications |The responses after executing Network network APIs. | +| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|pass/fail criteria |If normal response code 200 is returned, the test passes. | +| |Otherwise, the test fails with various error codes. | +| |Refer to Neutron Networking API v2.0 `[1]`_ `[2]`_ | ++-----------------------+----------------------------------------------------------------------------------------------------+ +|test report |TBD | ++-----------------------+----------------------------------------------------------------------------------------------------+ + +.. _`[1]`: http://developer.openstack.org/api-ref/networking/v2/ +.. _`[2]`: http://wiki.openstack.org/wiki/Neutron/APIv2-specification diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst new file mode 100644 index 00000000..e7fd82e7 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/ipv6.tc026.specification.rst @@ -0,0 +1,54 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +============================================================== +Dovetail IPv6 tc026 specification - Service VM as IPv6 vRouter +============================================================== + + ++-----------------------+--------------------------------------------------------------------------+ +|test case name |Service VM as IPv6 vRouter | +| | | ++-----------------------+--------------------------------------------------------------------------+ +|id |dovetail.ipv6.tc026 | ++-----------------------+--------------------------------------------------------------------------+ +|objective |IPv6 connnectivity, service VM as IPv6 vRouter | ++-----------------------+--------------------------------------------------------------------------+ +|modules under test |neutron, nova, etc | ++-----------------------+--------------------------------------------------------------------------+ +|dependent test project |yardstick | ++-----------------------+--------------------------------------------------------------------------+ +|test items |yardstick_tc027 | ++-----------------------+--------------------------------------------------------------------------+ +|environmental | OpenStack-only environment | +|requirements & | environment can be deplyed on bare metal of virtualized infrastructure | +|preconditions | deployment can be HA or non-HA | +| | test case image needs to be installed into Glance with ping6 included | ++-----------------------+--------------------------------------------------------------------------+ +|scenario dependencies | nosdn | ++-----------------------+--------------------------------------------------------------------------+ +|procedural |step 1: to setup IPv6 testing environment | +|requirements | 1.1 disable security group | +| | 1.2 create (ipv6, ipv4) router, network and subnet | +| | 1.3 create vRouter, VM1, VM2 | +| |step 2: to run ping6 to verify IPv6 connectivity | +| | 2.1 ssh to VM1 | +| | 2.2 ping6 to ipv6 router from VM1 | +| | 2.3 get the result and store the logs | +| |step 3: to teardown IPv6 testing environment | +| | 3.1 delete vRouter, VM1, VM2 | +| | 3.2 delete (ipv6, ipv4) router, network and subnet | +| | 3.3 enable security group | ++-----------------------+--------------------------------------------------------------------------+ +|input specifications |packetsize: 56 | +| |ping_count: 5 | +| | | ++-----------------------+--------------------------------------------------------------------------+ +|output specifications |output includes max_rtt, min_rtt, average_rtt | ++-----------------------+--------------------------------------------------------------------------+ +|pass/fail criteria |ping6 connectivity success, no SLA | ++-----------------------+--------------------------------------------------------------------------+ +|test report | dovetail dashboard DB here | ++-----------------------+--------------------------------------------------------------------------+ + diff --git a/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst b/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst new file mode 100644 index 00000000..462219a8 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/ipv6_all_testcases.rst @@ -0,0 +1,249 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +================================================== +IPv6 Compliance Testing Methodology and Test Cases +================================================== + +IPv6 Compliance Testing focuses on overlay IPv6 capabilities, i.e. to validate that +IPv6 capability is supported in tenant networks, subnets and routers. Both Tempest API +testing and Tempest Scenario testing are reused as much as we can in IPv6 Compliance +Testing. In addition, Yardstick Test Case 027 is also used to validate a specific use case +of using a Service VM as an IPv6 vRouter. + +IPv6 Compliance Testing test cases are described as follows: + +--------------------------------------------------------------- +Test Case 1: Create and Delete an IPv6 Network, Port and Subnet +--------------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + +----------------------------------------------------------------- +Test Case 2: Create, Update and Delete an IPv6 Network and Subnet +----------------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_networks.NetworksIpV6Test.test_create_update_delete_network_subnet + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_create_update_delete_network_subnet + +---------------------------------------------- +Test Case 3: Check External Network Visibility +---------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_networks.NetworksIpV6Test.test_external_network_visibility + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_external_network_visibility + +------------------------------------------------------- +Test Case 4: List IPv6 Networks and Subnets of a Tenant +------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_networks.NetworksIpV6Test.test_list_networks + tempest.api.network.test_networks.NetworksIpV6Test.test_list_subnets + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_networks + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_list_subnets + +----------------------------------------------------------- +Test Case 5: Show Information of an IPv6 Network and Subnet +----------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_networks.NetworksIpV6Test.test_show_network + tempest.api.network.test_networks.NetworksIpV6Test.test_show_subnet + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_network + tempest.api.network.test_networks.NetworksIpV6TestAttrs.test_show_subnet + +------------------------------------------------------------ +Test Case 6: Create an IPv6 Port in Allowed Allocation Pools +------------------------------------------------------------ + +.. code-block:: bash + + tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools + +-------------------------------------------------------- +Test Case 7: Create an IPv6 Port without Security Groups +-------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups + +--------------------------------------------------- +Test Case 8: Create, Update and Delete an IPv6 Port +--------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_update_delete_port + +---------------------------------------- +Test Case 9: List IPv6 Ports of a Tenant +---------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_ports.PortsIpV6TestJSON.test_list_ports + +---------------------------------------------- +Test Case 10: Show Information of an IPv6 Port +---------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_ports.PortsIpV6TestJSON.test_show_port + +-------------------------------------------------------- +Test Case 11: Add Multiple Interfaces for an IPv6 Router +-------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces + +------------------------------------------------------------------ +Test Case 12: Add and Remove an IPv6 Router Interface with port_id +------------------------------------------------------------------ + +.. code-block:: bash + + tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id + +-------------------------------------------------------------------- +Test Case 13: Add and Remove an IPv6 Router Interface with subnet_id +-------------------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id + +------------------------------------------------------------------ +Test Case 14: Create, Update, Delete, List and Show an IPv6 Router +------------------------------------------------------------------ + +.. code-block:: bash + + tempest.api.network.test_routers.RoutersIpV6Test.test_create_show_list_update_delete_router + +-------------------------------------------------------------------------- +Test Case 15: Create, Update, Delete, List and Show an IPv6 Security Group +-------------------------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_list_update_show_delete_security_group + +---------------------------------------------------------- +Test Case 16: Create, Delete and Show Security Group Rules +---------------------------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_security_groups.SecGroupIPv6Test.test_create_show_delete_security_group_rule + +-------------------------------------- +Test Case 17: List All Security Groups +-------------------------------------- + +.. code-block:: bash + + tempest.api.network.test_security_groups.SecGroupIPv6Test.test_list_security_groups + +-------------------------------------------------------- +Test Case 18: IPv6 Address Assignment - DHCPv6 Stateless +-------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os + +-------------------------------------------------------------------- +Test Case 19: IPv6 Address Assignment - Dual Stack, DHCPv6 Stateless +-------------------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os + +--------------------------------------------------------------------------- +Test Case 20: IPv6 Address Assignment - Multiple Prefixes, DHCPv6 Stateless +--------------------------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_dhcpv6_stateless + +--------------------------------------------------------------------------------------- +Test Case 21: IPv6 Address Assignment - Dual Stack, Multiple Prefixes, DHCPv6 Stateless +--------------------------------------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless + +--------------------------------------------- +Test Case 22: IPv6 Address Assignment - SLAAC +--------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os + +--------------------------------------------------------- +Test Case 23: IPv6 Address Assignment - Dual Stack, SLAAC +--------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_slaac_from_os + +---------------------------------------------------------------- +Test Case 24: IPv6 Address Assignment - Multiple Prefixes, SLAAC +---------------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac + +---------------------------------------------------------------------------- +Test Case 25: IPv6 Address Assignment - Dual Stack, Multiple Prefixes, SLAAC +---------------------------------------------------------------------------- + +.. code-block:: bash + + tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac + +------------------------------------------- +Test Case 26: Service VM as an IPv6 vRouter +------------------------------------------- + +.. code-block:: bash + + # Refer to Yardstick Test Case 027 + # Instruction: http://artifacts.opnfv.org/ipv6/docs/configurationguide/index.html + # Step 1: Set up Service VM as an IPv6 vRouter + # 1.1: Install OPNFV and Preparation + # 1.2: Disable Security Groups in OpenStack ML2 Setup + # 1.3: Create IPv4 and IPv6 Neutron routers, networks and subnets + # 1.4: Boot vRouter VM, and Guest VM1 and Guest VM2 + # Step 2: Verify IPv6 Connectivity + # 2.1: ssh to Guest VM1 + # 2.2: Ping6 from Guest VM1 to Guest VM2 + # 2.3: Ping6 from Guest VM1 to vRouter VM + # 2.4: Ping6 from Guest VM1 to Neutron IPv6 Router Namespace + # Step 3: Tear down Setup + # 3.1: Delete Guest VM1, Guest VM2 and vRouter VM + # 3.2: Delete IPv4 and IPv6 Neutron routers, networks and subnets + # 3.3: Enable Security Groups + diff --git a/docs/testing/user/testspecification/old_files/ipv6/testplan.rst b/docs/testing/user/testspecification/old_files/ipv6/testplan.rst new file mode 100644 index 00000000..3470e7a6 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/testplan.rst @@ -0,0 +1,34 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +=============================== +OPNFV IPv6 Compliance Test Plan +=============================== + +Introduction +============ + +The IPv6 compliance test plan outlines the method for testing IPv6 Tenant Network feature +compliance with the OPNFV platform. + +Scope +----- + +This test, and other tests in the test suite, are designed to verify an entire SUT, +and not any individual component of the system. + +Test suite scope and procedures +=============================== + +The IPv6 compliance test suite will evaluate the ability for a SUT to support IPv6 +Tenant Network features and functionality provided by OPNFV platform. + +Please refer to the complete list of the test cases for details. + +Test suite execution +==================== + +Please refer to each test case for specific setup and execution procedure. + +.._[1]: http://www.opnfv.org diff --git a/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst b/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst new file mode 100644 index 00000000..2119ed61 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/testprocedure.rst @@ -0,0 +1,9 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Christopher Price (Ericsson AB) and others + +=================== +IPv6 test procedure +=================== + +Draft to be patched this week, someone feel free to work on this in parallel. diff --git a/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst b/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst new file mode 100644 index 00000000..6f7caba8 --- /dev/null +++ b/docs/testing/user/testspecification/old_files/ipv6/testspecification.rst @@ -0,0 +1,54 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Christopher Price (Ericsson AB) and others + +=============================================== +Test specification - Service VM as IPv6 vRouter +=============================================== + +Draft to be worked on, this represents the YardStick test but I would suggest we need to break +this into a set of tests which provide more details per action with boundary validation. + +Test Item +========= + +TBD -> IPv6 Ping... + +Identify the items or features to be tested by this test case. The item description and +definition can be referenced from any one of several sources, depending on the level of the +test case specification. It may be a good idea to reference the source documents as well. + +Environmental requirements +========================== + +TBD + +Preconditions and procedural requirements +========================================= + +TBD + +.. <Start> +.. this section may be iterated over for a set of simillar test cases that would be run as one. + +Input Specifications +==================== + +TBD + +Output Specifications +===================== + +TBD + +.. <End> + +Test Reporting +============== + +The test report for this test case will be generated with links to relevant data sources. +This section can be updated once we have a template for the report in place. + +http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc027 + + diff --git a/docs/testing/user/testspecification/vimoperationscompute/index.rst b/docs/testing/user/testspecification/vimoperationscompute/index.rst new file mode 100644 index 00000000..f8dc5870 --- /dev/null +++ b/docs/testing/user/testspecification/vimoperationscompute/index.rst @@ -0,0 +1,92 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +========================================= +VIM compute operations test specification +========================================= + +.. toctree:: + :maxdepth: 2 + +Each test case requires documentation according to: +* Use case specification +* Test preconditions +* Basic test flow execution descriptor +* Post conditions and pass fail criteria + +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers_with_detail +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_host_name_is_same_as_server_name +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers_with_detail +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_created_server_vcpus +tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_get_instance_action +tempest.api.compute.servers.test_instance_actions.InstanceActionsTestJSON.test_list_instance_actions +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_flavor +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_image +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_name +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_filter_by_server_status +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_detailed_limit_results +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_active_status +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_flavor +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_image +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_limit +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_name +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_server_status +tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filtered_by_name_wildcard +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_changes_since_future_date +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_changes_since_invalid_date +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits_greater_than_actual_count +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits_pass_negative_value +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_limits_pass_string +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_non_existing_flavor +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_non_existing_image +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_by_non_existing_server_name +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_detail_server_is_deleted +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_status_non_existing +tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON.test_list_servers_with_a_deleted_server +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_lock_unlock_server +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server +tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_delete_server_metadata_item +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_get_server_metadata_item +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_list_server_metadata +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_set_server_metadata_item +tempest.api.compute.servers.test_server_metadata.ServerMetadataTestJSON.test_update_server_metadata +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_server_with_admin_password +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_specify_keypair +tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_access_server_address +tempest.api.compute.servers.test_servers.ServersTestJSON.test_update_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_numeric_server_name +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_metadata_exceeds_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_server_name_length_exceeds_256 +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_flavor +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_image +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_create_with_invalid_network_uuid +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_id_exceeding_length_limit +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_delete_server_pass_negative_id +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_get_non_existent_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_invalid_ip_v6_address +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_deleted_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_non_existent_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_rebuild_reboot_deleted_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_server_name_blank +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_stop_non_existent_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_name_of_non_existent_server +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_name_length_exceeds_256 +tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_update_server_set_empty_name +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_default_quotas +tempest.api.compute.test_quotas.QuotasTestJSON.test_get_quotas +tempest.api.compute.test_versions.TestVersions.test_list_api_versions +tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume +tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_list_get_volume_attachments diff --git a/docs/testing/user/testspecification/vimoperationsidentity/index.rst b/docs/testing/user/testspecification/vimoperationsidentity/index.rst new file mode 100644 index 00000000..8ab4596f --- /dev/null +++ b/docs/testing/user/testspecification/vimoperationsidentity/index.rst @@ -0,0 +1,24 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +========================================== +VIM identity operations test specification +========================================== + +.. toctree:: + :maxdepth: 2 + +Each test case requires documentation according to: +* Use case specification +* Test preconditions +* Basic test flow execution descriptor +* Post conditions and pass fail criteria + +tempest.api.identity.v3.TestApiDiscovery.test_api_media_types +tempest.api.identity.v3.TestApiDiscovery.test_api_version_resources +tempest.api.identity.v3.TestApiDiscovery.test_api_version_statuses +tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_media_types +tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources +tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_statuses +tempest.api.identity.v3.test_tokens.TokensV3Test.test_create_token diff --git a/docs/testing/user/testspecification/vimoperationsimage/index.rst b/docs/testing/user/testspecification/vimoperationsimage/index.rst new file mode 100644 index 00000000..932c7382 --- /dev/null +++ b/docs/testing/user/testspecification/vimoperationsimage/index.rst @@ -0,0 +1,48 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +======================================= +VIM image operations test specification +======================================= + +.. toctree:: + :maxdepth: 2 + +Each test case requires documentation according to: +* Use case specification +* Test preconditions +* Basic test flow execution descriptor +* Post conditions and pass fail criteria + +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_delete_image +tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image +tempest.api.image.v2.test_images.ListImagesTest.test_get_image_schema +tempest.api.image.v2.test_images.ListImagesTest.test_get_images_schema +tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_container_format +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_disk_format +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_limit +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_min_max_size +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_size +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_status +tempest.api.image.v2.test_images.ListImagesTest.test_list_images_param_visibility +tempest.api.image.v2.test_images.ListImagesTest.test_list_no_params +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_image_schema +tempest.api.image.v2.test_images.ListUserImagesTest.test_get_images_schema +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_container_format +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_disk_format +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_limit +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_min_max_size +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_size +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_status +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_images_param_visibility +tempest.api.image.v2.test_images.ListUserImagesTest.test_list_no_params +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_image_null_id +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_delete_non_existing_image +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_delete_deleted_image +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_image_null_id +tempest.api.image.v2.test_images_negative.ImagesNegativeTest.test_get_non_existent_image +tempest.api.image.v2.test_images_tags.ImagesTagsTest.test_update_delete_tags_for_image +tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_delete_non_existing_tag +tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest.test_update_tags_for_non_existing_image diff --git a/docs/testing/user/testspecification/vimoperationsnetwork/index.rst b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst new file mode 100644 index 00000000..252513ee --- /dev/null +++ b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst @@ -0,0 +1,81 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +========================================= +VIM network operations test specification +========================================= + +.. toctree:: + :maxdepth: 2 + +Each test case requires documentation according to: +* Use case specification +* Test preconditions +* Basic test flow execution descriptor +* Post conditions and pass fail criteria + +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_all_attributes +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_allocation_pools +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_dhcp_enabled +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_gw +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_gw_and_allocation_pools +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_with_host_routes_and_dns_nameservers +tempest.api.network.test_networks.NetworksTest.test_create_delete_subnet_without_gateway +tempest.api.network.test_networks.NetworksTest.test_create_update_delete_network_subnet +tempest.api.network.test_networks.NetworksTest.test_delete_network_with_subnet +tempest.api.network.test_networks.NetworksTest.test_list_networks +tempest.api.network.test_networks.NetworksTest.test_list_networks_fields +tempest.api.network.test_networks.NetworksTest.test_list_subnets +tempest.api.network.test_networks.NetworksTest.test_list_subnets_fields +tempest.api.network.test_networks.NetworksTest.test_show_network +tempest.api.network.test_networks.NetworksTest.test_show_network_fields +tempest.api.network.test_networks.NetworksTest.test_show_subnet +tempest.api.network.test_networks.NetworksTest.test_show_subnet_fields +tempest.api.network.test_networks.NetworksTest.test_update_subnet_gw_dns_host_routes_dhcp +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_all_attributes +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_allocation_pools +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_dhcp_enabled +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_gw +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_gw_and_allocation_pools +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_with_host_routes_and_dns_nameservers +tempest.api.network.test_networks.NetworksTestJSON.test_create_delete_subnet_without_gateway +tempest.api.network.test_networks.NetworksTestJSON.test_create_update_delete_network_subnet +tempest.api.network.test_networks.NetworksTestJSON.test_delete_network_with_subnet +tempest.api.network.test_networks.NetworksTestJSON.test_list_networks +tempest.api.network.test_networks.NetworksTestJSON.test_list_networks_fields +tempest.api.network.test_networks.NetworksTestJSON.test_list_subnets +tempest.api.network.test_networks.NetworksTestJSON.test_list_subnets_fields +tempest.api.network.test_networks.NetworksTestJSON.test_show_network +tempest.api.network.test_networks.NetworksTestJSON.test_show_network_fields +tempest.api.network.test_networks.NetworksTestJSON.test_show_subnet +tempest.api.network.test_networks.NetworksTestJSON.test_show_subnet_fields +tempest.api.network.test_networks.NetworksTestJSON.test_update_subnet_gw_dns_host_routes_dhcp +tempest.api.network.test_ports.PortsTestJSON.test_create_bulk_port +tempest.api.network.test_ports.PortsTestJSON.test_create_port_in_allowed_allocation_pools +tempest.api.network.test_ports.PortsTestJSON.test_create_update_delete_port +tempest.api.network.test_ports.PortsTestJSON.test_list_ports +tempest.api.network.test_ports.PortsTestJSON.test_list_ports_fields +tempest.api.network.test_ports.PortsTestJSON.test_show_port +tempest.api.network.test_ports.PortsTestJSON.test_show_port_fields +tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_security_group_and_extra_attributes +tempest.api.network.test_ports.PortsTestJSON.test_update_port_with_two_security_groups_and_extra_attributes +tempest.api.network.test_security_groups.SecGroupTest.test_create_list_update_show_delete_security_group +tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_additional_args +tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_icmp_type_code +tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_protocol_integer_value +tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_remote_group_id +tempest.api.network.test_security_groups.SecGroupTest.test_create_security_group_rule_with_remote_ip_prefix +tempest.api.network.test_security_groups.SecGroupTest.test_create_show_delete_security_group_rule +tempest.api.network.test_security_groups.SecGroupTest.test_list_security_groups +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_additional_default_security_group_fails +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_duplicate_security_group_rule_fails +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_bad_ethertype +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_bad_protocol +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_bad_remote_ip_prefix +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_invalid_ports +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_non_existent_remote_groupid +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_create_security_group_rule_with_non_existent_security_group +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_delete_non_existent_security_group +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group +tempest.api.network.test_security_groups_negative.NegativeSecGroupTest.test_show_non_existent_security_group_rule diff --git a/docs/testing/user/testspecification/vimoperationsvolume/index.rst b/docs/testing/user/testspecification/vimoperationsvolume/index.rst new file mode 100644 index 00000000..ce039a5b --- /dev/null +++ b/docs/testing/user/testspecification/vimoperationsvolume/index.rst @@ -0,0 +1,82 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +======================================== +VIM volume operations test specification +======================================== + +.. toctree:: + :maxdepth: 2 + +Each test case requires documentation according to: +* Use case specification +* Test preconditions +* Basic test flow execution descriptor +* Post conditions and pass fail criteria + +tempest.api.volume.test_availability_zone.AvailabilityZoneV2TestJSON.test_get_availability_zone_list +tempest.api.volume.test_extensions.ExtensionsV2TestJSON.test_list_extensions +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_create_get_delete_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_crud_snapshot_metadata +tempest.api.volume.test_snapshot_metadata.SnapshotV2MetadataTestJSON.test_update_snapshot_metadata_item +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_create_get_delete_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_crud_volume_metadata +tempest.api.volume.test_volume_metadata.VolumesV2MetadataTest.test_update_volume_metadata_item +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_attach_detach_volume_to_instance +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_get_volume_attachment +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_reserve_unreserve_volume +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_bootable +tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest.test_volume_readonly_update +tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete +tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_as_clone +tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete_from_image +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_by_name +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_by_name +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_with_detail_param_display_name_and_status +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_with_detail_param_metadata +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_with_details +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_with_param_metadata +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_by_status +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_availability_zone +tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON.test_volumes_list_details_by_status +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_attach_volumes_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_invalid_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_snapshot_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_volume_type +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_out_passing_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_negative +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_size_zero +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_without_passing_size +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_delete_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_detach_volumes_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_get_volume_without_passing_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_detail_with_nonexistent_name +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_invalid_status +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_list_volumes_with_nonexistent_name +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_negative_volume_status +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_reserve_volume_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_unreserve_volume_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_empty_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_invalid_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_update_volume_with_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_delete_nonexistent_volume_id +tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_volume_get_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshot_create_get_list_update_delete +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_details_with_params +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_snapshots_list_with_params +tempest.api.volume.test_volumes_snapshots.VolumesV2SnapshotTestJSON.test_volume_from_snapshot +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_details_with_params +tempest.api.volume.test_volumes_snapshots_list.VolumesV2SnapshotListTestJSON.test_snapshots_list_with_params +tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_with_nonexistent_volume_id +tempest.api.volume.test_volumes_snapshots_negative.VolumesV2SnapshotNegativeTestJSON.test_create_snapshot_without_passing_volume_id +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_pagination +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_details_with_multiple_params +tempest.api.volume.v2.test_volumes_list.VolumesV2ListTestJSON.test_volume_list_pagination |