diff options
Diffstat (limited to 'docs/testing/user/userguide/testing_guide.rst')
| -rw-r--r-- | docs/testing/user/userguide/testing_guide.rst | 145 |
1 files changed, 118 insertions, 27 deletions
diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst index 08fd8acf..3a9c6dc4 100644 --- a/docs/testing/user/userguide/testing_guide.rst +++ b/docs/testing/user/userguide/testing_guide.rst @@ -131,8 +131,8 @@ credentials and privileges to execute all test operations. If the SUT uses terms somewhat differently from the standard Openstack naming, you will need to adjust this file accordingly. -In our example, we will use the file '${DOVETAIL_HOME}/pre_config/env_config.sh'. Create and edit -the file so that all parameters are set correctly to match your SUT. Here is an example of what +Create and edit the file ``${DOVETAIL_HOME}/pre_config/env_config.sh`` so that +all parameters are set correctly to match your SUT. Here is an example of what this file should contain. .. code-block:: bash @@ -169,6 +169,15 @@ this file should contain. # Home directory for dovetail that you have created before. export DOVETAIL_HOME=/home/dovetail + # Special environment parameters for https. + # If using https + cacert, the path of cacert file should be provided. + # The cacert file should be put at $DOVETAIL_HOME/pre_config. + export OS_CACERT=$DOVETAIL_HOME/pre_config/cacert.pem + + # If using https + no cacert, should add OS_INSECURE environment parameter. + export OS_INSECURE=True + + Export all these variables into environment by, .. code-block:: bash @@ -182,6 +191,29 @@ settings are correct by, $ openstack service list +If the SUT uses https but have no cacert, you can validate the OpenStack environmental +settings by, + +.. code-block:: bash + + $ openstack --insecure service list + +If your SUT uses hosts file to translate hostnames into the IP +of OS_AUTH_URL, then you need to provide these hosts info in file +``$DOVETAIL_HOME/pre_config/hosts.yaml``. + +Create and edit file ``$DOVETAIL_HOME/pre_config/hosts.yaml``. Here is an example of +what this file should contain. + +.. code-block:: bash + + $ cat ${DOVETAIL_HOME}/pre_config/hosts.yaml + + --- + hosts_info: + - image.xx.xx.xx.com:172.xxx.xxx.xxx + - compute.xx.xx.xx.com:172.xxx.xxx.xxx + - <Hostname>:<IPaddress> Installing Prerequisite on the Test Host ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -257,12 +289,16 @@ The Dovetail project maintains a Docker image that has Dovetail test tools prein This Docker image is tagged with versions. Before pulling the Dovetail image, check the OPNFV's CVP web page first to determine the right tag for CVP testing. -If the Test Host is online, you can directly pull. +If the Test Host is online, you can directly pull Dovetail Docker image and download ubuntu +and cirros images. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.5.0 - cvp.0.5.0: Pulling from opnfv/dovetail + $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/pre_config + $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P ${DOVETAIL_HOME}/pre_config + + $ sudo docker pull opnfv/dovetail:cvp.0.6.0 + cvp.0.6.0: Pulling from opnfv/dovetail 30d541b48fc0: Pull complete 8ecd7f80d390: Pull complete 46ec9927bb81: Pull complete @@ -275,9 +311,9 @@ If the Test Host is online, you can directly pull. bf7c644692de: Pull complete cdc345e3f363: Pull complete Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee - Status: Downloaded newer image for opnfv/dovetail:cvp.0.5.0 + Status: Downloaded newer image for opnfv/dovetail:cvp.0.6.0 -An example of the <tag> is *cvp.0.5.0*. +An example of the <tag> is *cvp.0.6.0*. If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the dependent images that Dovetail uses, to a host that is online. The reason that you need @@ -287,11 +323,12 @@ offline, then all these dependencies will also need to be manually copied. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.5.0 + $ sudo docker pull opnfv/dovetail:cvp.0.6.0 $ sudo docker pull opnfv/functest:cvp.0.5.0 $ sudo docker pull opnfv/yardstick:danube.3.2 $ sudo docker pull opnfv/bottlenecks:cvp.0.4.0 $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} + $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P {ANY_DIR} Once all these images are pulled, save the images, copy to the Test Host, and then load the Dovetail and all dependent images at the Test Host. @@ -300,7 +337,7 @@ At the online host, save images. .. code-block:: bash - $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.5.0 opnfv/functest:cvp.0.5.0 \ + $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.6.0 opnfv/functest:cvp.0.5.0 \ opnfv/yardstick:danube.3.2 opnfv/bottlenecks:cvp.0.4.0 Copy dovetail.tar file to the Test Host, and then load the images on the Test Host. @@ -309,16 +346,17 @@ Copy dovetail.tar file to the Test Host, and then load the images on the Test Ho $ sudo docker load --input dovetail.tar -Copy sdnvpn test area image ubuntu-16.04-server-cloudimg-amd64-disk1.img to ${DOVETAIL_HOME}/pre_config/. +Copy sdnvpn test area image ubuntu-16.04-server-cloudimg-amd64-disk1.img to ``${DOVETAIL_HOME}/pre_config/``. +Copy cirros image cirros-0.3.5-x86_64-disk.img to ``${DOVETAIL_HOME}/pre_config/``. -Now check to see that the Dovetail image has been pulled or loaded properly. +Now check to see that all Docker images have been pulled or loaded properly. .. code-block:: bash $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE opnfv/functest cvp.0.5.0 9eaeaea5f203 8 days ago 1.53GB - opnfv/dovetail cvp.0.5.0 5d25b289451c 8 days ago 516MB + opnfv/dovetail cvp.0.6.0 5d25b289451c 8 days ago 516MB opnfv/yardstick danube.3.2 574596b6ea12 8 days ago 1.2GB opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB @@ -343,13 +381,25 @@ Now you should be in the Dovetail container's prompt and ready to execute test suites. The Dovetail client CLI allows the tester to specify which test suite to run. -By default the results are stored in a local file -``$DOVETAIL_HOME/results``. +By default the results are stored in a local file ``$DOVETAIL_HOME/results``. .. code-block:: bash $ dovetail run --testsuite <test-suite-name> +If you want to see more DEBUG logs, ``--debug`` should be added. + +.. code-block:: bash + + $ dovetail run --testsuite <test-suite-name> --debug + +If the Test Host is offline, ``--offline`` should be added to support running with +local resources. + +.. code-block:: bash + + $ dovetail run --testsuite <test-suite-name> --offline + Multiple test suites may be available. For the purpose of running CVP test suite, the test suite name follows the following format, ``CVP_<major>_<minor>_<patch>`` @@ -387,8 +437,8 @@ Special Configuration for Running HA Test Cases HA test cases need to know the info of a controller node of the OpenStack. It should include the node's name, role, ip, as well as the user and key_filename -or password to login the node. Users should create file ${DOVETAIL_HOME}/pre_config/pod.yaml -to store the info. +or password to login the node. Users should create file +``${DOVETAIL_HOME}/pre_config/pod.yaml`` to store the info. There is a sample file for users. @@ -412,10 +462,12 @@ There is a sample file for users. password: root Besides the 'password', user could also provide 'key_filename' to login the node. -Users need to create file $DOVETAIL_HOME/pre_config/id_rsa to store the private key. +Users need to create file ``$DOVETAIL_HOME/pre_config/id_rsa`` to store the private key. .. code-block:: bash + nodes: + - name: node1 role: Controller ip: 10.1.0.50 @@ -427,6 +479,36 @@ Users need to create file $DOVETAIL_HOME/pre_config/id_rsa to store the private key_filename: /root/.ssh/id_rsa +Special Configuration for Running defcore, IPv6 and tempest Test Cases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All test cases in test area defcore, ipv6 and tempest are picked up from Tempest. +For all these test cases, they need some configurations. Most of them +are done by Dovetail. However, there are still some configurations needed to be +given by users who know the SUT better. All these additional configurations should +be in file ``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. If this file is not given, +some test cases will be skipped or failed. + +Create and edit file ``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. +Here is an example of what this file should contain. + +.. code-block:: bash + + compute: + # The minimum number of compute nodes expected. + # This should be no less than 2 and no larger than the compute nodes the SUT actually has. + min_compute_nodes: 2 + + # Expected device name when a volume is attached to an instance. + volume_device_name: vdb + + # Lower version of the test target microversion range. + min_microversion: 2.2 + + # Upper version of the test target microversion range. + max_microversion: latest + + Making Sense of CVP Test Results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -447,23 +529,32 @@ When a tester is performing trial runs, Dovetail stores results in a local file * Additional log files may be of interests: refstack.log, dovetail_ha_tcXXX.out ... - * Example: Openstack refstack test case example + * Example: defcore test case example + + * Can see the log details in ``defcore_logs/dovetail.defcore.tc001.log``, + which has the passed, skipped and failed test cases results. + + * The skipped test cases have the reason for the users to see why these test cases skipped. + + * The failed test cases have rich debug information for the users to see why these test cases fail. + + * Example: vping test case example - * Can see the log details in refstack.log, which has the passed/skipped/failed - test cases result, the failed test cases have rich debug information for the - users to see why this test case fails. + * Its log is stored in dovetail.log. - * Example: OPNFV Functest test case example + * Its result is stored in functest_results.txt. - * For Functest tool, its log is stored in functest.log + * Example: ha test case example - * For each test case result in Functest, the logs are stored in functest_results.txt. + * Its log is stored in dovetail.log. - * Example: OPNFV Yardstick test case example + * Its result is stored in dovetail_ha_tcXXX.out. - * For Yardstick tool, its log is stored in yardstick.log + * Example: ipv6, sdnvpn and tempest test cases examples - * For each test case result in Yardstick, the logs are stored in dovetail_ha_tcXXX.out, respectively. + * Can see the log details in ``ipv6_logs/dovetail.ipv6.tcXXX.log``, + ``sdnvpn_logs/dovetail.sdnvpn.tcXXX.log`` and ``tempest_logs/dovetail.tempest.tcXXX.log``, + respectively. They all have the passed, skipped and failed test cases results. #. OPNFV web interface CVP will host a web site to collect test results. Users can upload their results to this web site, |