diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/user/userguide/testing_guide.rst | 147 |
1 files changed, 85 insertions, 62 deletions
diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst index d88d5715..d73ee206 100644 --- a/docs/testing/user/userguide/testing_guide.rst +++ b/docs/testing/user/userguide/testing_guide.rst @@ -184,20 +184,6 @@ Export all these variables into environment by, $ source ${DOVETAIL_HOME}/pre_config/env_config.sh -If OpenStack client is installed, you can validate that the OpenStack environmental -settings are correct by, - -.. code-block:: bash - - $ openstack service list - -If the SUT uses https but have no cacert, you can validate the OpenStack environmental -settings by, - -.. code-block:: bash - - $ openstack --insecure service list - If your SUT uses hosts file to translate hostnames into the IP of OS_AUTH_URL, then you need to provide these hosts info in file ``$DOVETAIL_HOME/pre_config/hosts.yaml``. @@ -297,8 +283,8 @@ and cirros images. $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/pre_config $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P ${DOVETAIL_HOME}/pre_config - $ sudo docker pull opnfv/dovetail:cvp.0.6.0 - cvp.0.6.0: Pulling from opnfv/dovetail + $ sudo docker pull opnfv/dovetail:cvp.0.7.0 + cvp.0.7.0: Pulling from opnfv/dovetail 30d541b48fc0: Pull complete 8ecd7f80d390: Pull complete 46ec9927bb81: Pull complete @@ -311,9 +297,9 @@ and cirros images. bf7c644692de: Pull complete cdc345e3f363: Pull complete Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee - Status: Downloaded newer image for opnfv/dovetail:cvp.0.6.0 + Status: Downloaded newer image for opnfv/dovetail:cvp.0.7.0 -An example of the <tag> is *cvp.0.6.0*. +An example of the <tag> is *cvp.0.7.0*. If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the dependent images that Dovetail uses, to a host that is online. The reason that you need @@ -323,10 +309,12 @@ offline, then all these dependencies will also need to be manually copied. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.6.0 + $ sudo docker pull opnfv/dovetail:cvp.0.7.0 $ sudo docker pull opnfv/functest:cvp.0.5.0 $ sudo docker pull opnfv/yardstick:danube.3.2 $ sudo docker pull opnfv/bottlenecks:cvp.0.4.0 + $ sudo docker pull opnfv/testapi:cvp.0.3.0 + $ sudo docker pull mongo:3.2.1 $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P {ANY_DIR} @@ -337,8 +325,9 @@ At the online host, save images. .. code-block:: bash - $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.6.0 opnfv/functest:cvp.0.5.0 \ - opnfv/yardstick:danube.3.2 opnfv/bottlenecks:cvp.0.4.0 + $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.7.0 \ + opnfv/functest:cvp.0.5.0 opnfv/yardstick:danube.3.2 \ + opnfv/bottlenecks:cvp.0.4.0 opnfv/testapi:cvp.0.3.0 mongo:3.2.1 Copy dovetail.tar file to the Test Host, and then load the images on the Test Host. @@ -355,10 +344,12 @@ Now check to see that all Docker images have been pulled or loaded properly. $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest cvp.0.5.0 9eaeaea5f203 8 days ago 1.53GB - opnfv/dovetail cvp.0.6.0 5d25b289451c 8 days ago 516MB - opnfv/yardstick danube.3.2 574596b6ea12 8 days ago 1.2GB - opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB + opnfv/functest cvp.0.5.0 e2b286547478 6 weeks ago 1.26 GB + opnfv/dovetail cvp.0.7.0 5d25b289451c 8 days ago 516MB + opnfv/yardstick danube.3.2 df830d5c2cb2 6 weeks ago 1.21 GB + opnfv/bottlenecks cvp.0.4.0 00450688bcae 7 weeks ago 622 MB + opnfv/testapi cvp.0.3.0 05c6d5ebce6c 2 months ago 448 MB + mongo 3.2.1 7e350b877a9a 19 months ago 317 MB Regardless of whether you pulled down the Dovetail image directly online, or loaded from a static image tar file, you are ready to run Dovetail. @@ -374,6 +365,31 @@ a static image tar file, you are ready to run Dovetail. The ``-e`` options set the env variables in the container and the ``-v`` options map files in the host to files in the container. +Build Local DB and Testapi Service +---------------------------------- + +It needs to build local DB and testapi service for storing and reporting results +to CVP web portal. There is a script in Dovetail container for building local DB. +All the following commands should be executed in Dovetail container. +The DB will use port 27017 and the testapi will use port 8000. They can be reset by + +.. code-block:: bash + + $ export mongodb_port=<new_DB_port> + $ export testapi_port=<new_testapi_port> + +Then you can run the script in Dovetail container to build local DB and testapi service. +You need to provide the IP of the Test Host when running this script. + +.. code-block:: bash + + $ cd /home/opnfv/dovetail/dovetail/utils/local_db/ + $ ./launch_db.sh <test_host_ip> + +After that, you can check the DB and testapi service with your browser. The url of +the results is ``http://<test_host_ip>:<testapi_port>/api/v1/results``. If you +can access this url successfully, it means the DB and testapi service are OK now. + Running the CVP Test Suite ---------------------------- @@ -381,57 +397,70 @@ Now you should be in the Dovetail container's prompt and ready to execute test suites. The Dovetail client CLI allows the tester to specify which test suite to run. +You can refer to `Dovetail CLI wiki page <https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line>`_ +for the details of the CLI. By default the results are stored in a local file ``$DOVETAIL_HOME/results``. .. code-block:: bash $ dovetail run --testsuite <test-suite-name> -If you want to see more DEBUG logs, ``--debug`` should be added. +Multiple test suites may be available. For the purpose of running +CVP test suite, the test suite name follows the following format, +``CVP_<major>_<minor>_<patch>`` +For example, CVP_1_0_0. .. code-block:: bash - $ dovetail run --testsuite <test-suite-name> --debug + $ dovetail run --testsuite CVP_1_0_0 -If the Test Host is offline, ``--offline`` should be added to support running with -local resources. +If you are not running the entire test suite, you can choose to run an +individual test area instead. The test area can be a single test area or +mandatory/optional test areas. The mandatory test areas are "osinterop", "vping" +and "ha". The optional test areas are "ipv6", "sdnvpn" and "tempest". .. code-block:: bash - $ dovetail run --testsuite <test-suite-name> --offline + $ dovetail run --testsuite CVP_1_0_0 --testarea mandatory -Multiple test suites may be available. For the purpose of running -CVP test suite, the test suite name follows the following format, -``CVP_<major>_<minor>_<patch>`` -For example, CVP_1_0_0. +You need to push the results to local DB if you want to store the results +or report the results to CVP. .. code-block:: bash - $ dovetail run --testsuite CVP_1_0_0 + $ dovetail run --testsuite CVP_1_0_0 --report http://<test_host_ip>:<testapi_port>/api/v1/results -If you are not running the entire test suite, you can choose to run an -individual test area instead. +If the Test Host is offline, ``--offline`` should be added to support running with +local resources. .. code-block:: bash - $ dovetail run --testsuite CVP_1_0_0 --testarea ipv6 + $ dovetail run --testsuite CVP_1_0_0 --offline --report http://<test_host_ip>:<testapi_port>/api/v1/results Until the official test suite is approved and released, you can use the *proposed_tests* for your trial runs, like this. .. code-block:: bash - $ dovetail run --testsuite proposed_tests --testarea ipv6 - 2017-05-23 05:01:49,488 - run - INFO - ================================================ - 2017-05-23 05:01:49,488 - run - INFO - Dovetail compliance: proposed_tests! - 2017-05-23 05:01:49,488 - run - INFO - ================================================ - 2017-05-23 05:01:49,488 - run - INFO - Build tag: daily-master-4bdde6b8-afa6-40bb-8fc9-5d568d74c8d7 - 2017-05-23 05:01:49,536 - run - INFO - - 2017-05-23 05:01:49,710 - run - INFO - >>[testcase]: dovetail.ipv6.tc001 - 2017-05-23 05:08:22,532 - run - INFO - Results have been stored with file /home/dovetail/results/functest_results.txt. - 2017-05-23 05:08:22,538 - run - INFO - >>[testcase]: dovetail.ipv6.tc002 + $ dovetail run --testsuite proposed_tests --testarea mandatory --report http://192.168.135.2:8000/api/v1/results + 2017-09-29 07:00:55,718 - run - INFO - ================================================ + 2017-09-29 07:00:55,718 - run - INFO - Dovetail compliance: proposed_tests! + 2017-09-29 07:00:55,718 - run - INFO - ================================================ + 2017-09-29 07:00:55,719 - run - INFO - Build tag: daily-master-f0795af6-a4e3-11e7-acc5-0242ac110004 + 2017-09-29 07:00:55,956 - run - INFO - >>[testcase]: dovetail.osinterop.tc001 + 2017-09-29 07:15:19,514 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:15:19,514 - run - INFO - >>[testcase]: dovetail.vping.tc001 + 2017-09-29 07:17:24,095 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:17:24,095 - run - INFO - >>[testcase]: dovetail.vping.tc002 + 2017-09-29 07:20:42,434 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. + 2017-09-29 07:20:42,434 - run - INFO - >>[testcase]: dovetail.ha.tc001 ... +After that there will be a tar file including the result file and all the log files, +for example ``${DOVETAIL_HOME}/logs_29_07_22.tar.gz``. +The file is named according to the current time. In this case, it was generated at 07:22 29th. +This tar file is used to upload to CVP. + Special Configuration for Running HA Test Cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -452,7 +481,7 @@ There is a sample file for users. # This must be controller. role: Controller - # This is the install IP of a controller node. + # This is the install IP of a controller node, which is the haproxy primary node ip: xx.xx.xx.xx # User name of this node. This user must have sudo privileges. @@ -482,7 +511,7 @@ Users need to create file ``$DOVETAIL_HOME/pre_config/id_rsa`` to store the priv Configuration for Running Tempest Test Cases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The test cases in the test areas `interop` (OpenStack Interoperability tests), +The test cases in the test areas `osinterop` (OpenStack Interoperability tests), `ipv6` and `tempest` are based on Tempest. A SUT-specific configuration of Tempest is required in order to run those test cases successfully. The corresponding SUT-specific configuration options should be supplied in the file @@ -501,13 +530,6 @@ Here is an example of what this file should contain. # Expected device name when a volume is attached to an instance. volume_device_name: vdb - # Lower version of the test target microversion range. - min_microversion: 2.2 - - # Upper version of the test target microversion range. - max_microversion: latest - - Making Sense of CVP Test Results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -561,26 +583,27 @@ When a tester is performing trial runs, Dovetail stores results in a local file * web site url - * Wait for the complement of LF, test community, etc. + * https://cvp.opnfv.org * Sign in / Sign up * You need to sign in you account, then you can upload results, and check your private results. - CVP is now using openstack id as account provider, but will soon support Linux Foundation ID - as well. + CVP is now using Linux Foundation ID as account provider. - * If you already have a openstack id, you can sign in directly with your id. + * If you already have a Linux Foundation ID, you can sign in directly with your id. - * If you do not have a openstack id, you can sign up a new one on the sign up page. + * If you do not have a Linux Foundation ID, you can sign up a new one on the sign up page. * If you do not sign in, you can only check the community results. * My results - * This page lists all results uploaded by you after you signed in, + * This page lists all results uploaded by you after you signed in. * You can also upload your results on this page. + * The generated results tar file is located at ``${DOVETAIL_HOME}/``, named as ``logs_29_07_22.tar.gz``. + * There is a *choose file* button, once you click it, you can choose your reuslt file in your harddisk then click the *upload* button, and you will see a results id once your uploading succeed. |