summaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
authorhongbo tian <hongbo.tianhongbo@huawei.com>2017-09-30 07:49:41 +0000
committerGerrit Code Review <gerrit@opnfv.org>2017-09-30 07:49:41 +0000
commit494e5cd6de10007826e8dd882a296d6ae0aca752 (patch)
tree940693214b88b2fcb7cad5458de9d86ae31717d7 /docs/testing
parent043ef2dad58c3aef0ea91cf3063dc9deb6f40bd9 (diff)
parent638e9391416ab407f7db733596480038785dfa4e (diff)
Merge "Update user guide with the local DB operations"
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/user/userguide/testing_guide.rst147
1 files changed, 85 insertions, 62 deletions
diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst
index d88d5715..d73ee206 100644
--- a/docs/testing/user/userguide/testing_guide.rst
+++ b/docs/testing/user/userguide/testing_guide.rst
@@ -184,20 +184,6 @@ Export all these variables into environment by,
$ source ${DOVETAIL_HOME}/pre_config/env_config.sh
-If OpenStack client is installed, you can validate that the OpenStack environmental
-settings are correct by,
-
-.. code-block:: bash
-
- $ openstack service list
-
-If the SUT uses https but have no cacert, you can validate the OpenStack environmental
-settings by,
-
-.. code-block:: bash
-
- $ openstack --insecure service list
-
If your SUT uses hosts file to translate hostnames into the IP
of OS_AUTH_URL, then you need to provide these hosts info in file
``$DOVETAIL_HOME/pre_config/hosts.yaml``.
@@ -297,8 +283,8 @@ and cirros images.
$ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/pre_config
$ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P ${DOVETAIL_HOME}/pre_config
- $ sudo docker pull opnfv/dovetail:cvp.0.6.0
- cvp.0.6.0: Pulling from opnfv/dovetail
+ $ sudo docker pull opnfv/dovetail:cvp.0.7.0
+ cvp.0.7.0: Pulling from opnfv/dovetail
30d541b48fc0: Pull complete
8ecd7f80d390: Pull complete
46ec9927bb81: Pull complete
@@ -311,9 +297,9 @@ and cirros images.
bf7c644692de: Pull complete
cdc345e3f363: Pull complete
Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee
- Status: Downloaded newer image for opnfv/dovetail:cvp.0.6.0
+ Status: Downloaded newer image for opnfv/dovetail:cvp.0.7.0
-An example of the <tag> is *cvp.0.6.0*.
+An example of the <tag> is *cvp.0.7.0*.
If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the
dependent images that Dovetail uses, to a host that is online. The reason that you need
@@ -323,10 +309,12 @@ offline, then all these dependencies will also need to be manually copied.
.. code-block:: bash
- $ sudo docker pull opnfv/dovetail:cvp.0.6.0
+ $ sudo docker pull opnfv/dovetail:cvp.0.7.0
$ sudo docker pull opnfv/functest:cvp.0.5.0
$ sudo docker pull opnfv/yardstick:danube.3.2
$ sudo docker pull opnfv/bottlenecks:cvp.0.4.0
+ $ sudo docker pull opnfv/testapi:cvp.0.3.0
+ $ sudo docker pull mongo:3.2.1
$ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR}
$ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P {ANY_DIR}
@@ -337,8 +325,9 @@ At the online host, save images.
.. code-block:: bash
- $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.6.0 opnfv/functest:cvp.0.5.0 \
- opnfv/yardstick:danube.3.2 opnfv/bottlenecks:cvp.0.4.0
+ $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.7.0 \
+ opnfv/functest:cvp.0.5.0 opnfv/yardstick:danube.3.2 \
+ opnfv/bottlenecks:cvp.0.4.0 opnfv/testapi:cvp.0.3.0 mongo:3.2.1
Copy dovetail.tar file to the Test Host, and then load the images on the Test Host.
@@ -355,10 +344,12 @@ Now check to see that all Docker images have been pulled or loaded properly.
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
- opnfv/functest cvp.0.5.0 9eaeaea5f203 8 days ago 1.53GB
- opnfv/dovetail cvp.0.6.0 5d25b289451c 8 days ago 516MB
- opnfv/yardstick danube.3.2 574596b6ea12 8 days ago 1.2GB
- opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB
+ opnfv/functest cvp.0.5.0 e2b286547478 6 weeks ago 1.26 GB
+ opnfv/dovetail cvp.0.7.0 5d25b289451c 8 days ago 516MB
+ opnfv/yardstick danube.3.2 df830d5c2cb2 6 weeks ago 1.21 GB
+ opnfv/bottlenecks cvp.0.4.0 00450688bcae 7 weeks ago 622 MB
+ opnfv/testapi cvp.0.3.0 05c6d5ebce6c 2 months ago 448 MB
+ mongo 3.2.1 7e350b877a9a 19 months ago 317 MB
Regardless of whether you pulled down the Dovetail image directly online, or loaded from
a static image tar file, you are ready to run Dovetail.
@@ -374,6 +365,31 @@ a static image tar file, you are ready to run Dovetail.
The ``-e`` options set the env variables in the container and the ``-v`` options map files
in the host to files in the container.
+Build Local DB and Testapi Service
+----------------------------------
+
+It needs to build local DB and testapi service for storing and reporting results
+to CVP web portal. There is a script in Dovetail container for building local DB.
+All the following commands should be executed in Dovetail container.
+The DB will use port 27017 and the testapi will use port 8000. They can be reset by
+
+.. code-block:: bash
+
+ $ export mongodb_port=<new_DB_port>
+ $ export testapi_port=<new_testapi_port>
+
+Then you can run the script in Dovetail container to build local DB and testapi service.
+You need to provide the IP of the Test Host when running this script.
+
+.. code-block:: bash
+
+ $ cd /home/opnfv/dovetail/dovetail/utils/local_db/
+ $ ./launch_db.sh <test_host_ip>
+
+After that, you can check the DB and testapi service with your browser. The url of
+the results is ``http://<test_host_ip>:<testapi_port>/api/v1/results``. If you
+can access this url successfully, it means the DB and testapi service are OK now.
+
Running the CVP Test Suite
----------------------------
@@ -381,57 +397,70 @@ Now you should be in the Dovetail container's prompt and ready to execute
test suites.
The Dovetail client CLI allows the tester to specify which test suite to run.
+You can refer to `Dovetail CLI wiki page <https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line>`_
+for the details of the CLI.
By default the results are stored in a local file ``$DOVETAIL_HOME/results``.
.. code-block:: bash
$ dovetail run --testsuite <test-suite-name>
-If you want to see more DEBUG logs, ``--debug`` should be added.
+Multiple test suites may be available. For the purpose of running
+CVP test suite, the test suite name follows the following format,
+``CVP_<major>_<minor>_<patch>``
+For example, CVP_1_0_0.
.. code-block:: bash
- $ dovetail run --testsuite <test-suite-name> --debug
+ $ dovetail run --testsuite CVP_1_0_0
-If the Test Host is offline, ``--offline`` should be added to support running with
-local resources.
+If you are not running the entire test suite, you can choose to run an
+individual test area instead. The test area can be a single test area or
+mandatory/optional test areas. The mandatory test areas are "osinterop", "vping"
+and "ha". The optional test areas are "ipv6", "sdnvpn" and "tempest".
.. code-block:: bash
- $ dovetail run --testsuite <test-suite-name> --offline
+ $ dovetail run --testsuite CVP_1_0_0 --testarea mandatory
-Multiple test suites may be available. For the purpose of running
-CVP test suite, the test suite name follows the following format,
-``CVP_<major>_<minor>_<patch>``
-For example, CVP_1_0_0.
+You need to push the results to local DB if you want to store the results
+or report the results to CVP.
.. code-block:: bash
- $ dovetail run --testsuite CVP_1_0_0
+ $ dovetail run --testsuite CVP_1_0_0 --report http://<test_host_ip>:<testapi_port>/api/v1/results
-If you are not running the entire test suite, you can choose to run an
-individual test area instead.
+If the Test Host is offline, ``--offline`` should be added to support running with
+local resources.
.. code-block:: bash
- $ dovetail run --testsuite CVP_1_0_0 --testarea ipv6
+ $ dovetail run --testsuite CVP_1_0_0 --offline --report http://<test_host_ip>:<testapi_port>/api/v1/results
Until the official test suite is approved and released, you can use
the *proposed_tests* for your trial runs, like this.
.. code-block:: bash
- $ dovetail run --testsuite proposed_tests --testarea ipv6
- 2017-05-23 05:01:49,488 - run - INFO - ================================================
- 2017-05-23 05:01:49,488 - run - INFO - Dovetail compliance: proposed_tests!
- 2017-05-23 05:01:49,488 - run - INFO - ================================================
- 2017-05-23 05:01:49,488 - run - INFO - Build tag: daily-master-4bdde6b8-afa6-40bb-8fc9-5d568d74c8d7
- 2017-05-23 05:01:49,536 - run - INFO -
- 2017-05-23 05:01:49,710 - run - INFO - >>[testcase]: dovetail.ipv6.tc001
- 2017-05-23 05:08:22,532 - run - INFO - Results have been stored with file /home/dovetail/results/functest_results.txt.
- 2017-05-23 05:08:22,538 - run - INFO - >>[testcase]: dovetail.ipv6.tc002
+ $ dovetail run --testsuite proposed_tests --testarea mandatory --report http://192.168.135.2:8000/api/v1/results
+ 2017-09-29 07:00:55,718 - run - INFO - ================================================
+ 2017-09-29 07:00:55,718 - run - INFO - Dovetail compliance: proposed_tests!
+ 2017-09-29 07:00:55,718 - run - INFO - ================================================
+ 2017-09-29 07:00:55,719 - run - INFO - Build tag: daily-master-f0795af6-a4e3-11e7-acc5-0242ac110004
+ 2017-09-29 07:00:55,956 - run - INFO - >>[testcase]: dovetail.osinterop.tc001
+ 2017-09-29 07:15:19,514 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json.
+ 2017-09-29 07:15:19,514 - run - INFO - >>[testcase]: dovetail.vping.tc001
+ 2017-09-29 07:17:24,095 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json.
+ 2017-09-29 07:17:24,095 - run - INFO - >>[testcase]: dovetail.vping.tc002
+ 2017-09-29 07:20:42,434 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json.
+ 2017-09-29 07:20:42,434 - run - INFO - >>[testcase]: dovetail.ha.tc001
...
+After that there will be a tar file including the result file and all the log files,
+for example ``${DOVETAIL_HOME}/logs_29_07_22.tar.gz``.
+The file is named according to the current time. In this case, it was generated at 07:22 29th.
+This tar file is used to upload to CVP.
+
Special Configuration for Running HA Test Cases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -452,7 +481,7 @@ There is a sample file for users.
# This must be controller.
role: Controller
- # This is the install IP of a controller node.
+ # This is the install IP of a controller node, which is the haproxy primary node
ip: xx.xx.xx.xx
# User name of this node. This user must have sudo privileges.
@@ -482,7 +511,7 @@ Users need to create file ``$DOVETAIL_HOME/pre_config/id_rsa`` to store the priv
Configuration for Running Tempest Test Cases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The test cases in the test areas `interop` (OpenStack Interoperability tests),
+The test cases in the test areas `osinterop` (OpenStack Interoperability tests),
`ipv6` and `tempest` are based on Tempest. A SUT-specific configuration of
Tempest is required in order to run those test cases successfully. The
corresponding SUT-specific configuration options should be supplied in the file
@@ -501,13 +530,6 @@ Here is an example of what this file should contain.
# Expected device name when a volume is attached to an instance.
volume_device_name: vdb
- # Lower version of the test target microversion range.
- min_microversion: 2.2
-
- # Upper version of the test target microversion range.
- max_microversion: latest
-
-
Making Sense of CVP Test Results
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -561,26 +583,27 @@ When a tester is performing trial runs, Dovetail stores results in a local file
* web site url
- * Wait for the complement of LF, test community, etc.
+ * https://cvp.opnfv.org
* Sign in / Sign up
* You need to sign in you account, then you can upload results, and check your private results.
- CVP is now using openstack id as account provider, but will soon support Linux Foundation ID
- as well.
+ CVP is now using Linux Foundation ID as account provider.
- * If you already have a openstack id, you can sign in directly with your id.
+ * If you already have a Linux Foundation ID, you can sign in directly with your id.
- * If you do not have a openstack id, you can sign up a new one on the sign up page.
+ * If you do not have a Linux Foundation ID, you can sign up a new one on the sign up page.
* If you do not sign in, you can only check the community results.
* My results
- * This page lists all results uploaded by you after you signed in,
+ * This page lists all results uploaded by you after you signed in.
* You can also upload your results on this page.
+ * The generated results tar file is located at ``${DOVETAIL_HOME}/``, named as ``logs_29_07_22.tar.gz``.
+
* There is a *choose file* button, once you click it, you can choose your reuslt file in your harddisk
then click the *upload* button, and you will see a results id once your uploading succeed.