diff options
Diffstat (limited to 'docs')
6 files changed, 103 insertions, 88 deletions
diff --git a/docs/testing/user/testspecification/vimoperationscompute/index.rst b/docs/testing/user/testspecification/vimoperationscompute/index.rst index 42a8e70e..b6bb852f 100644 --- a/docs/testing/user/testspecification/vimoperationscompute/index.rst +++ b/docs/testing/user/testspecification/vimoperationscompute/index.rst @@ -32,7 +32,7 @@ References - https://wiki.openstack.org/wiki/Governance/InteropWG -- Defcore test cases +- OpenStack Interoperability guidelines (version 2016.08) - https://github.com/openstack/interop/blob/master/2016.08.json diff --git a/docs/testing/user/testspecification/vimoperationsidentity/index.rst b/docs/testing/user/testspecification/vimoperationsidentity/index.rst index 1a65999e..284ffcbc 100644 --- a/docs/testing/user/testspecification/vimoperationsidentity/index.rst +++ b/docs/testing/user/testspecification/vimoperationsidentity/index.rst @@ -20,7 +20,7 @@ the Identity API. References ================ -- Defcore test cases +- OpenStack interoperability guidelines (version 2016.08) - https://github.com/openstack/interop/blob/master/2016.08.json @@ -28,13 +28,10 @@ References - https://www.openstack.org/brand/interop/ -- Defcore test cases listed using the Rest API from RefStack project +- OpenStack interoperability test cases excluding object storage - https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false -- Refstack client - - - https://github.com/openstack/refstack-client Definitions and abbreviations ============================= @@ -59,14 +56,10 @@ is able to run independently, i.e. irrelevant of the state created by a previous Dependency Description ====================== -The VIM identity operation test area is a part of the DefCore tempest test -cases run by refstack, then inherited by dovetail. Refstack-client is a command line -utility that allows you to execute Tempest test runs based on configurations you specify. -It is the official tool to run Defcore test cases, -which focuses on testing interoperability between OpenStack clouds. - -For Danube based dovetail release, the approved Defcore 2016.08 guidelines is adopted, -which is valid for Kilo, Liberty, Mitaka and Newton releases of Openstack. +The VIM identity operations test cases are a part of the OpenStack +interoperability tempest test cases. For Danube based dovetail release, the +OpenStack interoperability guidelines (version 2016.08) is adopted, which is +valid for Kilo, Liberty, Mitaka and Newton releases of Openstack. Test Descriptions ================= diff --git a/docs/testing/user/testspecification/vimoperationsimage/index.rst b/docs/testing/user/testspecification/vimoperationsimage/index.rst index 8e5f08ce..7ebc30c9 100644 --- a/docs/testing/user/testspecification/vimoperationsimage/index.rst +++ b/docs/testing/user/testspecification/vimoperationsimage/index.rst @@ -21,7 +21,7 @@ image creation, image list, image update and image deletion capabilities using G References ========== -- Defcore test cases +- OpenStack Interoperability guidelines (version 2016.08) - https://github.com/openstack/interop/blob/master/2016.08.json diff --git a/docs/testing/user/testspecification/vimoperationsnetwork/index.rst b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst index 47c08ef2..d1d9dce3 100644 --- a/docs/testing/user/testspecification/vimoperationsnetwork/index.rst +++ b/docs/testing/user/testspecification/vimoperationsnetwork/index.rst @@ -21,7 +21,7 @@ basic CRUD operations on L2 networks, L2 network ports and security groups. References ========== -- Defcore test cases +- OpenStack Interoperability guidelines (version 2016.08) - https://github.com/openstack/interop/blob/master/2016.08.json diff --git a/docs/testing/user/testspecification/vimoperationsvolume/index.rst b/docs/testing/user/testspecification/vimoperationsvolume/index.rst index 9a76c37c..9d5e275f 100644 --- a/docs/testing/user/testspecification/vimoperationsvolume/index.rst +++ b/docs/testing/user/testspecification/vimoperationsvolume/index.rst @@ -34,7 +34,7 @@ References - https://wiki.openstack.org/wiki/Governance/InteropWG -- Defcore test cases +- OpenStack Interoperability guidelines (version 2016.08) - https://github.com/openstack/interop/blob/master/2016.08.json diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst index 3a9c6dc4..d73ee206 100644 --- a/docs/testing/user/userguide/testing_guide.rst +++ b/docs/testing/user/userguide/testing_guide.rst @@ -184,20 +184,6 @@ Export all these variables into environment by, $ source ${DOVETAIL_HOME}/pre_config/env_config.sh -If OpenStack client is installed, you can validate that the OpenStack environmental -settings are correct by, - -.. code-block:: bash - - $ openstack service list - -If the SUT uses https but have no cacert, you can validate the OpenStack environmental -settings by, - -.. code-block:: bash - - $ openstack --insecure service list - If your SUT uses hosts file to translate hostnames into the IP of OS_AUTH_URL, then you need to provide these hosts info in file ``$DOVETAIL_HOME/pre_config/hosts.yaml``. @@ -297,8 +283,8 @@ and cirros images. $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/pre_config $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P ${DOVETAIL_HOME}/pre_config - $ sudo docker pull opnfv/dovetail:cvp.0.6.0 - cvp.0.6.0: Pulling from opnfv/dovetail + $ sudo docker pull opnfv/dovetail:cvp.0.7.0 + cvp.0.7.0: Pulling from opnfv/dovetail 30d541b48fc0: Pull complete 8ecd7f80d390: Pull complete 46ec9927bb81: Pull complete @@ -311,9 +297,9 @@ and cirros images. bf7c644692de: Pull complete cdc345e3f363: Pull complete Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee - Status: Downloaded newer image for opnfv/dovetail:cvp.0.6.0 + Status: Downloaded newer image for opnfv/dovetail:cvp.0.7.0 -An example of the <tag> is *cvp.0.6.0*. +An example of the <tag> is *cvp.0.7.0*. If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the dependent images that Dovetail uses, to a host that is online. The reason that you need @@ -323,10 +309,12 @@ offline, then all these dependencies will also need to be manually copied. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.6.0 + $ sudo docker pull opnfv/dovetail:cvp.0.7.0 $ sudo docker pull opnfv/functest:cvp.0.5.0 $ sudo docker pull opnfv/yardstick:danube.3.2 $ sudo docker pull opnfv/bottlenecks:cvp.0.4.0 + $ sudo docker pull opnfv/testapi:cvp.0.3.0 + $ sudo docker pull mongo:3.2.1 $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P {ANY_DIR} @@ -337,8 +325,9 @@ At the online host, save images. .. code-block:: bash - $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.6.0 opnfv/functest:cvp.0.5.0 \ - opnfv/yardstick:danube.3.2 opnfv/bottlenecks:cvp.0.4.0 + $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.7.0 \ + opnfv/functest:cvp.0.5.0 opnfv/yardstick:danube.3.2 \ + opnfv/bottlenecks:cvp.0.4.0 opnfv/testapi:cvp.0.3.0 mongo:3.2.1 Copy dovetail.tar file to the Test Host, and then load the images on the Test Host. @@ -355,10 +344,12 @@ Now check to see that all Docker images have been pulled or loaded properly. $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest cvp.0.5.0 9eaeaea5f203 8 days ago 1.53GB - opnfv/dovetail cvp.0.6.0 5d25b289451c 8 days ago 516MB - opnfv/yardstick danube.3.2 574596b6ea12 8 days ago 1.2GB - opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB + opnfv/functest cvp.0.5.0 e2b286547478 6 weeks ago 1.26 GB + opnfv/dovetail cvp.0.7.0 5d25b289451c 8 days ago 516MB + opnfv/yardstick danube.3.2 df830d5c2cb2 6 weeks ago 1.21 GB + opnfv/bottlenecks cvp.0.4.0 00450688bcae 7 weeks ago 622 MB + opnfv/testapi cvp.0.3.0 05c6d5ebce6c 2 months ago 448 MB + mongo 3.2.1 7e350b877a9a 19 months ago 317 MB Regardless of whether you pulled down the Dovetail image directly online, or loaded from a static image tar file, you are ready to run Dovetail. @@ -374,6 +365,31 @@ a static image tar file, you are ready to run Dovetail. The ``-e`` options set the env variables in the container and the ``-v`` options map files in the host to files in the container. +Build Local DB and Testapi Service +---------------------------------- + +It needs to build local DB and testapi service for storing and reporting results +to CVP web portal. There is a script in Dovetail container for building local DB. +All the following commands should be executed in Dovetail container. +The DB will use port 27017 and the testapi will use port 8000. They can be reset by + +.. code-block:: bash + + $ export mongodb_port=<new_DB_port> + $ export testapi_port=<new_testapi_port> + +Then you can run the script in Dovetail container to build local DB and testapi service. +You need to provide the IP of the Test Host when running this script. + +.. code-block:: bash + + $ cd /home/opnfv/dovetail/dovetail/utils/local_db/ + $ ./launch_db.sh <test_host_ip> + +After that, you can check the DB and testapi service with your browser. The url of +the results is ``http://<test_host_ip>:<testapi_port>/api/v1/results``. If you +can access this url successfully, it means the DB and testapi service are OK now. + Running the CVP Test Suite ---------------------------- @@ -381,57 +397,70 @@ Now you should be in the Dovetail container's prompt and ready to execute test suites. The Dovetail client CLI allows the tester to specify which test suite to run. +You can refer to `Dovetail CLI wiki page <https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line>`_ +for the details of the CLI. By default the results are stored in a local file ``$DOVETAIL_HOME/results``. .. code-block:: bash $ dovetail run --testsuite <test-suite-name> -If you want to see more DEBUG logs, ``--debug`` should be added. +Multiple test suites may be available. For the purpose of running +CVP test suite, the test suite name follows the following format, +``CVP_<major>_<minor>_<patch>`` +For example, CVP_1_0_0. .. code-block:: bash - $ dovetail run --testsuite <test-suite-name> --debug + $ dovetail run --testsuite CVP_1_0_0 -If the Test Host is offline, ``--offline`` should be added to support running with -local resources. +If you are not running the entire test suite, you can choose to run an +individual test area instead. The test area can be a single test area or +mandatory/optional test areas. The mandatory test areas are "osinterop", "vping" +and "ha". The optional test areas are "ipv6", "sdnvpn" and "tempest". .. code-block:: bash - $ dovetail run --testsuite <test-suite-name> --offline + $ dovetail run --testsuite CVP_1_0_0 --testarea mandatory -Multiple test suites may be available. For the purpose of running -CVP test suite, the test suite name follows the following format, -``CVP_<major>_<minor>_<patch>`` -For example, CVP_1_0_0. +You need to push the results to local DB if you want to store the results +or report the results to CVP. .. code-block:: bash - $ dovetail run --testsuite CVP_1_0_0 + $ dovetail run --testsuite CVP_1_0_0 --report http://<test_host_ip>:<testapi_port>/api/v1/results -If you are not running the entire test suite, you can choose to run an -individual test area instead. +If the Test Host is offline, ``--offline`` should be added to support running with +local resources. .. code-block:: bash - $ dovetail run --testsuite CVP_1_0_0 --testarea ipv6 + $ dovetail run --testsuite CVP_1_0_0 --offline --report http://<test_host_ip>:<testapi_port>/api/v1/results Until the official test suite is approved and released, you can use the *proposed_tests* for your trial runs, like this. .. code-block:: bash - $ dovetail run --testsuite proposed_tests --testarea ipv6 - 2017-05-23 05:01:49,488 - run - INFO - ================================================ - 2017-05-23 05:01:49,488 - run - INFO - Dovetail compliance: proposed_tests! - 2017-05-23 05:01:49,488 - run - INFO - ================================================ - 2017-05-23 05:01:49,488 - run - INFO - Build tag: daily-master-4bdde6b8-afa6-40bb-8fc9-5d568d74c8d7 - 2017-05-23 05:01:49,536 - run - INFO - - 2017-05-23 05:01:49,710 - run - INFO - >>[testcase]: dovetail.ipv6.tc001 - 2017-05-23 05:08:22,532 - run - INFO - Results have been stored with file /home/dovetail/results/functest_results.txt. - 2017-05-23 05:08:22,538 - run - INFO - >>[testcase]: dovetail.ipv6.tc002 + $ dovetail run --testsuite proposed_tests --testarea mandatory --report http://192.168.135.2:8000/api/v1/results + 2017-09-29 07:00:55,718 - run - INFO - ================================================ + 2017-09-29 07:00:55,718 - run - INFO - Dovetail compliance: proposed_tests! + 2017-09-29 07:00:55,718 - run - INFO - ================================================ + 2017-09-29 07:00:55,719 - run - INFO - Build tag: daily-master-f0795af6-a4e3-11e7-acc5-0242ac110004 + 2017-09-29 07:00:55,956 - run - INFO - >>[testcase]: dovetail.osinterop.tc001 + 2017-09-29 07:15:19,514 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:15:19,514 - run - INFO - >>[testcase]: dovetail.vping.tc001 + 2017-09-29 07:17:24,095 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:17:24,095 - run - INFO - >>[testcase]: dovetail.vping.tc002 + 2017-09-29 07:20:42,434 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:20:42,434 - run - INFO - >>[testcase]: dovetail.ha.tc001 ... +After that there will be a tar file including the result file and all the log files, +for example ``${DOVETAIL_HOME}/logs_29_07_22.tar.gz``. +The file is named according to the current time. In this case, it was generated at 07:22 29th. +This tar file is used to upload to CVP. + Special Configuration for Running HA Test Cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -452,7 +481,7 @@ There is a sample file for users. # This must be controller. role: Controller - # This is the install IP of a controller node. + # This is the install IP of a controller node, which is the haproxy primary node ip: xx.xx.xx.xx # User name of this node. This user must have sudo privileges. @@ -479,15 +508,14 @@ Users need to create file ``$DOVETAIL_HOME/pre_config/id_rsa`` to store the priv key_filename: /root/.ssh/id_rsa -Special Configuration for Running defcore, IPv6 and tempest Test Cases -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Configuration for Running Tempest Test Cases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -All test cases in test area defcore, ipv6 and tempest are picked up from Tempest. -For all these test cases, they need some configurations. Most of them -are done by Dovetail. However, there are still some configurations needed to be -given by users who know the SUT better. All these additional configurations should -be in file ``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. If this file is not given, -some test cases will be skipped or failed. +The test cases in the test areas `osinterop` (OpenStack Interoperability tests), +`ipv6` and `tempest` are based on Tempest. A SUT-specific configuration of +Tempest is required in order to run those test cases successfully. The +corresponding SUT-specific configuration options should be supplied in the file +``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. Create and edit file ``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. Here is an example of what this file should contain. @@ -502,13 +530,6 @@ Here is an example of what this file should contain. # Expected device name when a volume is attached to an instance. volume_device_name: vdb - # Lower version of the test target microversion range. - min_microversion: 2.2 - - # Upper version of the test target microversion range. - max_microversion: latest - - Making Sense of CVP Test Results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -529,9 +550,9 @@ When a tester is performing trial runs, Dovetail stores results in a local file * Additional log files may be of interests: refstack.log, dovetail_ha_tcXXX.out ... - * Example: defcore test case example + * Example: OpenStack Interoperability test cases - * Can see the log details in ``defcore_logs/dovetail.defcore.tc001.log``, + * Can see the log details in ``osinterop_logs/dovetail.osinterop.tc001.log``, which has the passed, skipped and failed test cases results. * The skipped test cases have the reason for the users to see why these test cases skipped. @@ -562,26 +583,27 @@ When a tester is performing trial runs, Dovetail stores results in a local file * web site url - * Wait for the complement of LF, test community, etc. + * https://cvp.opnfv.org * Sign in / Sign up * You need to sign in you account, then you can upload results, and check your private results. - CVP is now using openstack id as account provider, but will soon support Linux Foundation ID - as well. + CVP is now using Linux Foundation ID as account provider. - * If you already have a openstack id, you can sign in directly with your id. + * If you already have a Linux Foundation ID, you can sign in directly with your id. - * If you do not have a openstack id, you can sign up a new one on the sign up page. + * If you do not have a Linux Foundation ID, you can sign up a new one on the sign up page. * If you do not sign in, you can only check the community results. * My results - * This page lists all results uploaded by you after you signed in, + * This page lists all results uploaded by you after you signed in. * You can also upload your results on this page. + * The generated results tar file is located at ``${DOVETAIL_HOME}/``, named as ``logs_29_07_22.tar.gz``. + * There is a *choose file* button, once you click it, you can choose your reuslt file in your harddisk then click the *upload* button, and you will see a results id once your uploading succeed. |