diff options
Diffstat (limited to 'docs/testing/user/userguide/testing_guide.rst')
| -rw-r--r-- | docs/testing/user/userguide/testing_guide.rst | 616 |
1 files changed, 368 insertions, 248 deletions
diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst index e55c1595..d1c31683 100644 --- a/docs/testing/user/userguide/testing_guide.rst +++ b/docs/testing/user/userguide/testing_guide.rst @@ -2,16 +2,19 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. -========================================== -Conducting CVP Testing with Dovetail -========================================== +========================================= +Conducting OVP NFVI Testing with Dovetail +========================================= Overview ------------------------------- +-------- -The Dovetail testing framework for CVP consists of two major parts: the testing client that +This guide provides the instructions for the OVP Infrastructure testing. For the +OVP VNF testing, please refer to the next section. + +The Dovetail testing framework for OVP consists of two major parts: the testing client which executes all test cases in a lab (vendor self-testing or a third party lab), -and the server system that is hosted by the CVP administrator to store and +and the server system which is hosted by the OVP administrator to store and view test results based on a web API. The following diagram illustrates this overall framework. @@ -25,10 +28,10 @@ the System Under Test (SUT) itself. The above diagram assumes that the tester's Test Host is situated in a DMZ, which has internal network access to the SUT and external access via the public Internet. The public Internet connection allows for easy installation of the Dovetail containers. -A singular compressed file that includes all the underlying results can be pulled from -the Test Host and uploaded to the OPNFV CVP server. +A single compressed file that includes all the underlying results can be pulled from +the Test Host and uploaded to the OPNFV OVP server. This arrangement may not be supported in some labs. Dovetail also supports an offline mode of -installation that is illustrated in the next diagram. +installation which is illustrated in the next diagram. .. image:: ../../../images/dovetail_offline_mode.png :align: center @@ -41,11 +44,10 @@ the Test Host. While it is possible to run the Test Host as a virtual machine, this user guide assumes it is a physical machine for simplicity. The rest of this guide will describe how to install the Dovetail tool as a -Docker container image, go over the steps of running the CVP test suite, and +Docker container image, go over the steps of running the OVP test suite, and then discuss how to view test results and make sense of them. -Readers interested -in using Dovetail for its functionalities beyond CVP testing, e.g. for in-house +Readers interested in using Dovetail for its functionalities beyond OVP testing, e.g. for in-house or extended testing, should consult the Dovetail developer's guide for additional information. @@ -107,23 +109,7 @@ Or, if the lab environment does not allow ping, try validating it using HTTPS in Installing Prerequisite Packages on the Test Host ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The main prerequisite software for Dovetail are Python and Docker. - -In the CVP test suite for the Danube release, Dovetail requires Python 2.7. Various minor -versions of Python 2.7.x are known to work Dovetail, but there are no assurances. Python 3.x -is not supported at this time. - -Use the following steps to check if the right version of python is already installed, -and if not, install it. - -.. code-block:: bash - - $ python --version - Python 2.7.6 - -If your Test Host does not have Python installed, or the version is not 2.7, you -should consult Python installation guides corresponding to the operating system -in your Test Host on how to install Python 2.7. +The main prerequisite software for Dovetail is Docker. Dovetail does not work with Docker versions prior to 1.12.3. We have validated Dovetail with Docker 17.03 CE. Other versions of Docker later than 1.12.3 may @@ -152,20 +138,13 @@ also work, but community support may be more available on Docker 17.03 CE or gre If your Test Host does not have Docker installed, or Docker is older than 1.12.3, or you have Docker version other than 17.03 CE and wish to change, you will need to install, upgrade, or re-install in order to run Dovetail. -The Docker installation process -can be more complex, you should refer to the official +If you need further assistance with Docker installation process, you should refer to the official Docker installation guide that is relevant to your Test Host's operating system. -The above installation steps assume that the Test Host is in the online mode. For offline -testing, use the following offline installation steps instead. - -In order to install or upgrade Python offline, you may download packaged Python 2.7 -for your Test Host's operating system on a connected host, copy the packge to -the Test Host, then install from that local copy. - -In order to install Docker offline, download Docker static binaries and copy the -tar file to the Test Host, such as for Ubuntu14.04, you may follow the following link -to install, +The above installation steps assume that the Test Host is in the online mode. +For offline testing, use the following offline installation steps instead. +For instance, download Docker static binaries and copy the tar file to the +Test Host, such as for Ubuntu14.04, you may follow the following link: .. code-block:: bash @@ -175,31 +154,33 @@ Configuring the Test Host Environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Test Host needs a few environment variables set correctly in order to access the -Openstack API required to drive the Dovetail tests. For convenience and as a convention, +OpenStack API which is required to drive the Dovetail tests. For convenience and as a convention, we will also create a home directory for storing all Dovetail related config files and results files: .. code-block:: bash - $ mkdir -p /home/dovetail - $ export DOVETAIL_HOME=/home/dovetail + $ mkdir -p ${HOME}/dovetail + $ export DOVETAIL_HOME=${HOME}/dovetail -Here we set dovetail home directory to be ``/home/dovetail`` for an example. -Then create a directory named ``pre_config`` in this directory to store all -Dovetail related config files: +For example, here we set dovetail home directory to be ``${HOME}/dovetail``. +Then create two directories named ``pre_config`` and ``images`` under this directory +to store all Dovetail related config files and all test images respectively: .. code-block:: bash $ mkdir -p ${DOVETAIL_HOME}/pre_config + $ mkdir -p ${DOVETAIL_HOME}/images + Setting up Primary Configuration File ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -At this point, you will need to consult your SUT (Openstack) administrator to correctly set +At this point, you will need to consult your SUT (OpenStack) administrator to correctly set the configurations in a file named ``env_config.sh``. -The Openstack settings need to be configured such that the Dovetail client has all the necessary +The OpenStack settings need to be configured such that the Dovetail client has all the necessary credentials and privileges to execute all test operations. If the SUT uses terms -somewhat differently from the standard Openstack naming, you will need to adjust +somewhat differently from the standard OpenStack naming, you will need to adjust this file accordingly. Create and edit the file ``${DOVETAIL_HOME}/pre_config/env_config.sh`` so that @@ -210,20 +191,17 @@ this file should contain. $ cat ${DOVETAIL_HOME}/pre_config/env_config.sh - # Project-level authentication scope (name or ID), recommend admin project. + # Project-level authentication scope (name or ID), admin project is recommended. export OS_PROJECT_NAME=admin - # For identity v2, it uses OS_TENANT_NAME rather than OS_PROJECT_NAME. - export OS_TENANT_NAME=admin - - # Authentication username, belongs to the project above, recommend admin user. + # Authentication username, belongs to the project above, admin user is recommended. export OS_USERNAME=admin # Authentication password. Use your own password export OS_PASSWORD=xxxxxxxx # Authentication URL, one of the endpoints of keystone service. If this is v3 version, - # there need some extra variables as follows. + # there needs some extra variables as follows. export OS_AUTH_URL='http://xxx.xxx.xxx.xxx:5000/v3' # Default is 2.0. If use keystone v3 API, this should be set as 3. @@ -245,34 +223,45 @@ this file should contain. # If using https + no cacert, should add OS_INSECURE environment parameter. export OS_INSECURE=True + # The name of a network with external connectivity for allocating floating + # IPs. It is required that at least one Neutron network with the attribute + # 'router:external=True' is pre-configured on the system under test. + # This network is used by test cases to SSH into tenant VMs and perform + # operations there. + export EXTERNAL_NETWORK=xxx -The OS_AUTH_URL variable is key to configure correctly, as the other admin services -are gleaned from the identity service. HTTPS should be configured in the SUT so the -final two variables should be uncommented. However, if SSL is disabled in the SUT, comment -out the OS_CACERT and OS_INSECURE variables. Ensure the '/path/to/pre_config' directory in -the above file matches the directory location of the cacert file for the OS_CACERT variable. + # Set an existing role used to create project and user for vping test cases. + # Otherwise, it will create a role 'Member' to do that. + export NEW_USER_ROLE=xxx -Export all these variables into environment by, + # For XCI installer the following environment parameters should be added in + # this file. Otherwise, those parameters could be ignored. + export INSTALLER_TYPE=osa + export DEPLOY_SCENARIO=os-nosdn-nofeature + export XCI_FLAVOR=noha -.. code-block:: bash - $ source ${DOVETAIL_HOME}/pre_config/env_config.sh -The above line may be added to your .bashrc file for convenience when repeatedly using -Dovetail. +The OS_AUTH_URL variable is key to configure correctly, as the other admin services +are collected from the identity service. HTTPS should be configured in the SUT so +either OS_CACERT or OS_INSECURE should be uncommented. +However, if SSL is disabled in the SUT, comment out both OS_CACERT and OS_INSECURE variables. +Ensure the '/path/to/pre_config' directory in +the above file matches the directory location of the cacert file for the OS_CACERT variable. The next three sections outline additional configuration files used by Dovetail. The -tempest (tempest_conf.yaml) configuration file is required for executing the mandatory -osinterop test cases and the optional ipv6/tempest test cases. The HA (pod.yaml) configuration -file is required for the mandatory HA test cases and is also employed to collect SUT hardware +tempest (tempest_conf.yaml) configuration file is required for executing all tempest +test cases (e.g. functest.tempest.compute, functest.tempest.ipv6 ...) and +functest.security.patrole. The HA (pod.yaml) configuration +file is required for HA test cases and is also employed to collect SUT hardware info. The hosts.yaml is optional for hostname/IP resolution. Configuration for Running Tempest Test Cases (Mandatory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The test cases in the test areas `osinterop` (OpenStack Interoperability tests), -`ipv6` and `tempest` are based on Tempest. A SUT-specific configuration of -Tempest is required in order to run those test cases successfully. The +The test cases in the test areas `tempest` and `security` +are based on Tempest. A SUT-specific configuration of +Tempest is required in order to run those test cases successfully. The corresponding SUT-specific configuration options must be supplied in the file ``$DOVETAIL_HOME/pre_config/tempest_conf.yaml``. @@ -289,15 +278,34 @@ Here is an example of what this file should contain. # Expected device name when a volume is attached to an instance. volume_device_name: vdb -Use the listing above at a minimum to execute the mandatory test areas. + # One sub test case of functest.tempest.osinterop will be skipped if not provide this version. + # The default range of microversion for tempest is [None - None]. + # Test case functest.tempest.osinterop required the range to be [2.2 - latest]. + max_microversion: 2.65 + +Use the listing above as a minimum to execute the mandatory test cases. + +If the optional BGPVPN Tempest API tests shall be run, Tempest needs to be told +that the BGPVPN service is available. To do that, add the following to the +``$DOVETAIL_HOME/pre_config/tempest_conf.yaml`` configuration file: + +.. code-block:: bash + + service_available: + bgpvpn: True + Configuration for Running HA Test Cases (Mandatory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The mandatory HA test cases require OpenStack controller node info. It must include the node's +The HA test cases require OpenStack controller node info. It must include the node's name, role, ip, as well as the user and key_filename or password to login to the node. Users -must create the file ``${DOVETAIL_HOME}/pre_config/pod.yaml`` to store the info. This file is -also used as basis to collect SUT hardware information that is stored alongside results and +must create the file ``${DOVETAIL_HOME}/pre_config/pod.yaml`` to store the info. +For some HA test cases, they will log in the controller node 'node1' and kill the specific processes. +The names of the specific processes may be different with the actual ones of the SUTs. +The processes' names can also be changed with file ``${DOVETAIL_HOME}/pre_config/pod.yaml``. + +This file is also used as a basis to collect SUT hardware information which is stored alongside results and uploaded to the OVP web portal. The SUT hardware information can be viewed within the 'My Results' view in the OVP web portal by clicking the SUT column 'info' link. In order to collect SUT hardware information holistically, ensure this file has an entry for each of @@ -309,21 +317,62 @@ Below is a sample with the required syntax when password is employed by the cont nodes: - - # This can not be changed and must be node1. + # This info of node0 is used only for one optional test case 'yardstick.ha.controller_restart'. + # If you don't plan to test it, this Jumpserver node can be ignored. + # This can not be changed and **must** be node0. + name: node0 + + # This **must** be Jumpserver. + role: Jumpserver + + # This is the instance IP of a node which has ipmitool installed. + ip: xx.xx.xx.xx + + # User name of the user of this node. This user **must** have sudo privileges. + user: root + + # Password of the user. + password: root + + - + # Almost all HA test cases are trying to login to a controller node named 'node1' + # and then kill some processes running on it. + # If you don't want to reset the attack node name for each test case, this + # name can not be changed and **must** be node1. name: node1 - # This must be controller. + # This **must** be controller. role: Controller - # This is the install IP of a controller node, which is the haproxy primary node + # This is the instance IP of a controller node, which is the haproxy primary node ip: xx.xx.xx.xx - # User name of this node. This user must have sudo privileges. + # User name of the user of this node. This user **must** have sudo privileges. user: root # Password of the user. password: root + process_info: + - + # For all HA test cases, there are 2 parameters, 'attack_process' and 'attack_host', + # which support to be set by users instead of using the default values. + # The 'attack_process' is the process name of one HA test case which it try to kill. + # The 'attack_host' is the host name which the test case try to login and then kill + # the process running on it. + # Fllowing is 2 samples. + + # The default attack process of yardstick.ha.rabbitmq is 'rabbitmq-server'. + # Here can be reset to 'rabbitmq'. + testcase_name: yardstick.ha.rabbitmq + attack_process: rabbitmq + + - + # The default attack host for all HA test cases is 'node1'. + # Here can be reset to any other node given in the section 'nodes'. + testcase_name: yardstick.ha.glance_api + attack_host: node2 + Besides the 'password', a 'key_filename' entry can be provided to login to the controller node. Users need to create file ``$DOVETAIL_HOME/pre_config/id_rsa`` to store the private key. A sample is provided below to show the required syntax when using a key file. @@ -337,15 +386,43 @@ A sample is provided below to show the required syntax when using a key file. ip: 10.1.0.50 user: root - # Private key of this node. It must be /root/.ssh/id_rsa - # Dovetail will move the key file from $DOVETAIL_HOME/pre_config/id_rsa - # to /root/.ssh/id_rsa of Yardstick container - key_filename: /root/.ssh/id_rsa + # Private ssh key for accessing the controller nodes. If a keyfile is + # being used instead of password, it **must** be put under + # $DOVETAIL_HOME/pre_config/ and named 'id_rsa'. + key_filename: /home/dovetail/pre_config/id_rsa Under nodes, repeat entries for name, role, ip, user and password or key file for each of the controller/compute nodes that comprise the SUT. Use a '-' to separate each of the entries. Specify the value for the role key to be either 'Controller' or 'Compute' for each node. +Under process_info, repeat entries for testcase_name, attack_host and attack_process +for each HA test case. Use a '-' to separate each of the entries. +The default attack host of all HA test cases is **node1**. +The default attack processes of all HA test cases are list here: + + +-------------------------------+-------------------------+ + | Test Case Name | Attack Process Name | + +===============================+=========================+ + | yardstick.ha.cinder_api | cinder-api | + +-------------------------------+-------------------------+ + | yardstick.ha.database | mysql | + +-------------------------------+-------------------------+ + | yardstick.ha.glance_api | glance-api | + +-------------------------------+-------------------------+ + | yardstick.ha.haproxy | haproxy | + +-------------------------------+-------------------------+ + | yardstick.ha.keystone | keystone | + +-------------------------------+-------------------------+ + | yardstick.ha.neutron_l3_agent | neutron-l3-agent | + +-------------------------------+-------------------------+ + | yardstick.ha.neutron_server | neutron-server | + +-------------------------------+-------------------------+ + | yardstick.ha.nova_api | nova-api | + +-------------------------------+-------------------------+ + | yardstick.ha.rabbitmq | rabbitmq-server | + +-------------------------------+-------------------------+ + + Configuration of Hosts File (Optional) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -353,7 +430,7 @@ If your SUT uses a hosts file to translate hostnames into the IP of OS_AUTH_URL, to provide the hosts info in a file ``$DOVETAIL_HOME/pre_config/hosts.yaml``. Create and edit file ``$DOVETAIL_HOME/pre_config/hosts.yaml``. Below is an example of what -this file should contain. Note, that multiple hostnames can be specified for each IP address, +this file should contain. Note that multiple hostnames can be specified for each IP address, as shown in the generic syntax below the example. .. code-block:: bash @@ -363,7 +440,8 @@ as shown in the generic syntax below the example. --- hosts_info: 192.168.141.101: - - ha-vip + - identity.endpoint.url + - compute.endpoint.url <ip>: - <hostname1> @@ -375,69 +453,84 @@ Installing Dovetail on the Test Host The Dovetail project maintains a Docker image that has Dovetail test tools preinstalled. This Docker image is tagged with versions. Before pulling the Dovetail image, check the -OPNFV's CVP web page first to determine the right tag for CVP testing. +OPNFV's OVP web page first to determine the right tag for OVP testing. Online Test Host """""""""""""""" -If the Test Host is online, you can directly pull Dovetail Docker image and download Ubuntu -and Cirros images. All other dependent docker images will automatically be downloaded. The -Ubuntu and Cirros images are used by Dovetail for image creation and VM instantiation within -the SUT. +If the Test Host is online, you can directly pull Dovetail Docker image, then all +other dependent docker images will automatically be downloaded. Also you can download +other related VM images such as Ubuntu and Cirros images which are used by Dovetail +for image creation and VM instantiation within the SUT. + +Following given the download url for each VM images. Cirros-0.4.0 and Ubuntu-16.04 +are used by mandatory test cases, so they are the only 2 images **must** be downloaded +before doing the test. There are also 2 other optional VM images, Ubuntu-14.04 and +Cloudify-manager, which are used by optional test cases functest.vnf.vepc and functest.vnf.vims. +If you don't plan to test these 2 test cases, you can skip downloading these 2 images. .. code-block:: bash - $ wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/pre_config - $ wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P ${DOVETAIL_HOME}/pre_config - - $ sudo docker pull opnfv/dovetail:ovp.1.0.0 - ovp.1.0.0: Pulling from opnfv/dovetail - 30d541b48fc0: Pull complete - 8ecd7f80d390: Pull complete - 46ec9927bb81: Pull complete - 2e67a4d67b44: Pull complete - 7d9dd9155488: Pull complete - cc79be29f08e: Pull complete - e102eed9bf6a: Pull complete - 952b8a9d2150: Pull complete - bfbb639d1f38: Pull complete - bf7c644692de: Pull complete - cdc345e3f363: Pull complete - Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee - Status: Downloaded newer image for opnfv/dovetail:ovp.1.0.0 - -An example of the <tag> is *ovp.1.0.0*. + $ wget -nc http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -P ${DOVETAIL_HOME}/images + $ wget -nc https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/images + $ wget -nc https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/images + $ wget -nc http://repository.cloudifysource.org/cloudify/19.01.24/community-release/cloudify-docker-manager-community-19.01.24.tar -P ${DOVETAIL_HOME}/images + + $ sudo docker pull opnfv/dovetail:ovp-3.0.0 + ovp-3.0.0: Pulling from opnfv/dovetail + 324d088ce065: Pull complete + 2ab951b6c615: Pull complete + 9b01635313e2: Pull complete + 04510b914a6c: Pull complete + 83ab617df7b4: Pull complete + 40ebbe7294ae: Pull complete + d5db7e3e81ae: Pull complete + 0701bf048879: Pull complete + 0ad9f4168266: Pull complete + d949894f87f6: Pull complete + Digest: sha256:7449601108ebc5c40f76a5cd9065ca5e18053be643a0eeac778f537719336c29 + Status: Downloaded newer image for opnfv/dovetail:ovp-3.0.0 Offline Test Host """"""""""""""""" -If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the +If the Test Host is offline, you will need to first pull the Dovetail Docker image and all the dependent images that Dovetail uses, to a host that is online. The reason that you need to pull all dependent images is because Dovetail normally does dependency checking at run-time and automatically pulls images as needed, if the Test Host is online. If the Test Host is offline, then all these dependencies will need to be manually copied. +The Docker images, Ubuntu and Cirros image below are necessary for all mandatory test cases. + +.. code-block:: bash + + $ sudo docker pull opnfv/dovetail:ovp-3.0.0 + $ sudo docker pull opnfv/functest-smoke:hunter + $ sudo docker pull opnfv/functest-healthcheck:hunter + $ sudo docker pull opnfv/yardstick:opnfv-8.0.0 + $ sudo docker pull opnfv/bottlenecks:8.0.1-latest + $ wget -nc http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -P {ANY_DIR} + $ wget -nc https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P ${DOVETAIL_HOME}/images + +The other Docker images and test images below are only used by optional test cases. + .. code-block:: bash - $ sudo docker pull opnfv/dovetail:ovp.1.0.0 - $ sudo docker pull opnfv/functest:ovp.1.0.0 - $ sudo docker pull opnfv/yardstick:danube.3.2 - $ sudo docker pull opnfv/testapi:ovp.1.0.0 - $ sudo docker pull mongo:3.2.1 - $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} - $ sudo wget -nc http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img -P {ANY_DIR} + $ sudo docker pull opnfv/functest-vnf:hunter + $ wget -nc https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} + $ wget -nc http://repository.cloudifysource.org/cloudify/19.01.24/community-release/cloudify-docker-manager-community-19.01.24.tar -P ${DOVETAIL_HOME}/images -Once all these images are pulled, save the images, copy to the Test Host, and then load -the Dovetail image and all dependent images at the Test Host. The final two lines above are -to obtain the test images for transfer to the Test Host. +Once all these images are pulled, save the images, copy them to the Test Host, and then load +the Dovetail image and all dependent images at the Test Host. At the online host, save the images with the command below. .. code-block:: bash - $ sudo docker save -o dovetail.tar opnfv/dovetail:ovp.1.0.0 \ - opnfv/functest:ovp.1.0.0 opnfv/yardstick:danube.3.2 \ - opnfv/testapi:ovp.1.0.0 mongo:3.2.1 + $ sudo docker save -o dovetail.tar opnfv/dovetail:ovp-3.0.0 \ + opnfv/functest-smoke:hunter opnfv/functest-healthcheck:hunter \ + opnfv/functest-vnf:hunter \ + opnfv/yardstick:opnfv-8.0.0 opnfv/bottlenecks:8.0.1-latest The command above creates a dovetail.tar file with all the images, which can then be copied to the Test Host. To load the Dovetail images on the Test Host execute the command below. @@ -451,22 +544,26 @@ Now check to see that all Docker images have been pulled or loaded properly. .. code-block:: bash $ sudo docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest ovp.1.0.0 e2b286547478 6 weeks ago 1.26 GB - opnfv/dovetail ovp.1.0.0 5d25b289451c 8 days ago 516MB - opnfv/yardstick danube.3.2 df830d5c2cb2 6 weeks ago 1.21 GB - opnfv/testapi ovp.1.0.0 05c6d5ebce6c 2 months ago 448 MB - mongo 3.2.1 7e350b877a9a 19 months ago 317 MB + REPOSITORY TAG IMAGE ID CREATED SIZE + opnfv/dovetail ovp-3.0.0 4b68659da24d 22 hours ago 825MB + opnfv/functest-smoke hunter c0253f6de153 3 weeks ago 556MB + opnfv/functest-healthcheck hunter fb6d766e38e0 3 weeks ago 379MB + opnfv/functest-vnf hunter 31466d52d155 21 hours ago 1.1GB + opnfv/yardstick opnfv-8.0.0 189d7d9fbcb2 7 months ago 2.54GB + opnfv/bottlenecks 8.0.1-latest 44c1b9fb25aa 5 hours ago 837MB After copying and loading the Dovetail images at the Test Host, also copy the test images -(Ubuntu, Cirros) to the Test Host. Copy image ubuntu-16.04-server-cloudimg-amd64-disk1.img -to ``${DOVETAIL_HOME}/pre_config/``. Copy image cirros-0.3.5-x86_64-disk.img to -``${DOVETAIL_HOME}/pre_config/``. +(Ubuntu, Cirros and cloudify-manager) to the Test Host. + +- Copy image ``cirros-0.4.0-x86_64-disk.img`` to ``${DOVETAIL_HOME}/images/``. +- Copy image ``ubuntu-14.04-server-cloudimg-amd64-disk1.img`` to ``${DOVETAIL_HOME}/images/``. +- Copy image ``ubuntu-16.04-server-cloudimg-amd64-disk1.img`` to ``${DOVETAIL_HOME}/images/``. +- Copy image ``cloudify-docker-manager-community-19.01.24.tar`` to ``${DOVETAIL_HOME}/images/``. Starting Dovetail Docker ------------------------ -Regardless of whether you pulled down the Dovetail image directly online, or loaded from +Regardless of whether you pulled down the Dovetail image directly online, or loaded it from a static image tar file, you are now ready to run Dovetail. Use the command below to create a Dovetail container and get access to its shell. @@ -478,194 +575,211 @@ create a Dovetail container and get access to its shell. -v /var/run/docker.sock:/var/run/docker.sock \ opnfv/dovetail:<tag> /bin/bash -The ``-e`` option sets the DOVETAIL_HOME environment variable in the container and the -``-v`` options map files in the Test Host to files in the container. The latter option -allows the Dovetail container to read the configuration files and write result files into -DOVETAIL_HOME on the Test Host. The user should be within the Dovetail container shell, -once the command above is executed. +The ``-e`` option sets the DOVETAIL_HOME environment variable in the container +and the ``-v`` options mounts files from the test host to the destination path +inside the container. The latter option allows the Dovetail container to read +the configuration files and write result files into DOVETAIL_HOME on the Test +Host. The user should be within the Dovetail container shell, once the command +above is executed. + -Build Local DB and Testapi Services -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Running the OVP Test Suite +---------------------------- -The steps in this section only need to be executed if the user plans on storing consolidated -results on the Test Host that can be uploaded to the CVP portal. +All or a subset of the available tests can be executed at any location within the +Dovetail container prompt. You can refer to :ref:`cli-reference` +for the details of the CLI. -Dovetail needs to build the local DB and testapi service for storing and reporting results -to the CVP web portal. There is a script in the Dovetail container for building the local DB. -The ports 27017 and 8000 are used by the DB and testapi respectively. If the Test Host is -using these ports for existing services, to avoid conflicts, remap the ports to values that -are unused. Execute the commands below in the Dovetail container to remap ports, as required. -This step can be skipped if there are no port conflicts with the Test Host. .. code-block:: bash - $ export mongodb_port=<new_DB_port> - $ export testapi_port=<new_testapi_port> + $ dovetail run --testsuite <test-suite-name> -Within the Dovetail container, navigate to the directory and execute the shell script using -the commands below to build the local DB and testapi services. +The ``--testsuite`` option is used to control the set of tests intended for execution +at a high level. For the purposes of running the OVP test suite, the test suite name follows +the following format, ``ovp.<release-version>``. The latest and default test suite is +ovp.2019.12. .. code-block:: bash - $ cd /home/opnfv/dovetail/dovetail/utils/local_db/ - $ ./launch_db.sh + $ dovetail run -To validate the DB and testapi services are running successfully, navigate to the URL -``http://<test_host_ip>:<testapi_port>/api/v1/results``, substituting within the URL -the IP address of the Test Host and the testapi port number. If you can access this URL -successfully, the services are up and running. +This command is equal to -Running the CVP Test Suite --------------------------- +.. code-block:: bash -All or a subset of the available tests can be executed at any location within the -Dovetail container prompt. You can refer to :ref:`cli-reference` -for the details of the CLI. + $ dovetail run --testsuite ovp.2019.12 +Without any additional options, the above command will attempt to execute all mandatory and +optional test cases with test suite ovp.2019.12. +To restrict the breadth of the test scope, it can also be specified using options +``--mandatory`` or ``--optional``. .. code-block:: bash - $ dovetail run --testsuite <test-suite-name> + $ dovetail run --mandatory -The '--testsuite' option is used to control the set of tests intended for execution -at a high level. For the purposes of running the CVP test suite, the test suite name follows -the following format, ``ovp.<major>.<minor>.<patch>``. The latest and default test suite is - ovp.1.0.0. +Also there is a ``--testcase`` option provided to run a specified test case. .. code-block:: bash - $ dovetail run + $ dovetail run --testcase functest.tempest.osinterop -This command is equal to +Dovetail allows the user to disable strict API response validation implemented +by Nova Tempest tests by means of the ``--no-api-validation`` option. Usage of +this option is only advisable if the SUT returns Nova API responses that +contain additional attributes. For more information on this command line option +and its intended usage, refer to +:ref:`dovetail-exemption_process_api_response_validation`. .. code-block:: bash - $ dovetail run --testsuite ovp.1.0.0 + $ dovetail run --testcase functest.tempest.osinterop --no-api-validation -Without any additional options, the above command will attempt to execute all mandatory and -optional test cases. To restrict the breadth of the test scope, test areas can also be -specified using the '--testarea' option. The test area can be specified broadly using arguments -'mandatory' and 'optional'. The mandatory tests can be narrowed further using test area arguments -'osinterop', 'vping' and 'ha'. The optional tests can be narrowed further using test area -arguments 'ipv6', 'sdnvpn' and 'tempest'. +By default, during test case execution, the respective feature is responsible to +decide what flavor is going to use for the execution of each test scenario which is under +of its umbrella. +In parallel, there is also implemented a mechanism in order for the extra specs in flavors of +executing test scenarios to be hugepages instead of the default option. +This is happening if the name of the scenario contains the substring "ovs". +In this case, the flavor which is going to be used for the running test case has +'hugepage' characteristics. + +Taking the above into consideration and having in mind that the DEPLOY_SCENARIO +environment parameter is not used by dovetail framework (the initial value is 'unknown'), +we set as input, for the features that they are responsible for the test case execution, +the DEPLOY_SCENARIO environment parameter having as substring the feature name "ovs" +(e.g. os-nosdn-ovs-ha). + +Note for the users: + - if their system uses DPDK, they should run with ``--deploy-scenario <xx-yy-ovs-zz>`` + (e.g. os-nosdn-ovs-ha) + - this is an experimental feature .. code-block:: bash - $ dovetail run --testarea mandatory + $ dovetail run --testcase functest.tempest.osinterop --deploy-scenario os-nosdn-ovs-ha By default, results are stored in local files on the Test Host at ``$DOVETAIL_HOME/results``. Each time the 'dovetail run' command is executed, the results in the aforementioned directory -are overwritten. To create a singular compressed result file for upload to the CVP portal or -for archival purposes, the results need to pushed to the local DB. This can be achieved by -using the '--report' option with an argument syntax as shown below. Note, that the Test Host -IP address and testapi port number must be substituted with appropriate values. +are overwritten. To create a singular compressed result file for upload to the OVP portal or +for archival purposes, the tool provides an option ``--report``. .. code-block:: bash - $ dovetail run --report http://<test_host_ip>:<testapi_port>/api/v1/results + $ dovetail run --report If the Test Host is offline, ``--offline`` should be added to support running with -local resources. +local resources. Otherwise, it will try to download resources online during the run time. .. code-block:: bash - $ dovetail run --offline --report http://<test_host_ip>:<testapi_port>/api/v1/results + $ dovetail run --offline -Below is an example of running the entire mandatory test area and the creation of the compressed +Below is an example of running one test case and the creation of the compressed result file on the Test Host. .. code-block:: bash - $ dovetail run --offline --testarea mandatory --report http://192.168.135.2:8000/api/v1/results - 2017-09-29 07:00:55,718 - run - INFO - ================================================ - 2017-09-29 07:00:55,718 - run - INFO - Dovetail compliance: ovp.1.0.0! - 2017-09-29 07:00:55,718 - run - INFO - ================================================ - 2017-09-29 07:00:55,719 - run - INFO - Build tag: daily-master-f0795af6-a4e3-11e7-acc5-0242ac110004 - 2017-09-29 07:00:55,956 - run - INFO - >>[testcase]: dovetail.osinterop.tc001 - 2017-09-29 07:15:19,514 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. - 2017-09-29 07:15:19,514 - run - INFO - >>[testcase]: dovetail.vping.tc001 - 2017-09-29 07:17:24,095 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. - 2017-09-29 07:17:24,095 - run - INFO - >>[testcase]: dovetail.vping.tc002 - 2017-09-29 07:20:42,434 - run - INFO - Results have been pushed to database and stored with local file /home/dovetail/results/results.json. - 2017-09-29 07:20:42,434 - run - INFO - >>[testcase]: dovetail.ha.tc001 - ... + $ dovetail run --offline --testcase functest.vping.userdata --report + 2019-12-04 07:31:13,156 - run - INFO - ================================================ + 2019-12-04 07:31:13,157 - run - INFO - Dovetail compliance: ovp.2019.12! + 2019-12-04 07:31:13,157 - run - INFO - ================================================ + 2019-12-04 07:31:13,157 - run - INFO - Build tag: daily-master-0c9184e6-1668-11ea-b1cd-0242ac110002 + 2019-12-04 07:31:13,610 - run - INFO - >>[testcase]: functest.vping.userdata + 2019-12-04 07:31:13,612 - dovetail.test_runner.DockerRunner - WARNING - There is no hosts file /home/ovp/pre_config/hosts.yaml. This may cause some issues with domain name resolution. + 2019-12-04 07:31:14,587 - dovetail.test_runner.DockerRunner - INFO - Get hardware info of all nodes list in file /home/ovp/pre_config/pod.yaml ... + 2019-12-04 07:31:14,587 - dovetail.test_runner.DockerRunner - INFO - Hardware info of all nodes are stored in file /home/dovetail/results/all_hosts_info.json. + 2019-12-04 07:31:14,612 - dovetail.container.Container - WARNING - There is no hosts file /home/ovp/pre_config/hosts.yaml. This may cause some issues with domain name resolution. + 2019-12-04 07:32:13,804 - dovetail.report.Report - INFO - Results have been stored with files: ['/home/ovp/results/functest_results.txt']. + 2019-12-04 07:32:13,808 - dovetail.report.Report - INFO - + + Dovetail Report + Version: 2019.12 + Build Tag: daily-master-0c9184e6-1668-11ea-b1cd-0242ac110002 + Test Date: 2019-12-04 07:32:13 UTC + Duration: 60.20 s + + Pass Rate: 100.00% (1/1) + vping: pass rate 100.00% + -functest.vping.userdata PASS + When test execution is complete, a tar file with all result and log files is written in ``$DOVETAIL_HOME`` on the Test Host. An example filename is -``${DOVETAIL_HOME}/logs_20180105_0858.tar.gz``. The file is named using a -timestamp that follows the convention 'YearMonthDay-HourMinute'. In this case, it was generated -at 08:58 on January 5th, 2018. This tar file is used to upload to the CVP portal. +``${DOVETAIL_HOME}/logs_20191204_0732.tar.gz``. The file is named using a timestamp +that follows the convention 'YearMonthDay_HourMinute'. In this case, it was generated +at 07:32 on December 4th, 2019. This tar file is used for uploading the logs and +results to the OVP portal. -Making Sense of CVP Test Results +Making Sense of OVP Test Results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When a tester is performing trial runs, Dovetail stores results in local files on the Test -Host by default within the directory specified below. Note, that if the '--report' option -is used to execute tests, results are written to results.json and the files functest_results.txt -and dovetail_ha_tcXXX.out will not be created. - +Host by default within directory ``$DOVETAIL_HOME/results``. -.. code-block:: bash + * Log file: dovetail.log - cd $DOVETAIL_HOME/results + * Review the dovetail.log to see if all important information has been captured -#. Local file + * In default mode without DEBUG. - * Log file: dovetail.log + * Adding option ``-d/--debug`` to change the mode to be DEBUG. - * Review the dovetail.log to see if all important information has been captured - - in default mode without DEBUG. + * Result file: results.json * Review the results.json to see all results data including criteria for PASS or FAIL. - * Example: OpenStack Interoperability test cases + * Tempest and security test cases - * Can see the log details in ``osinterop_logs/dovetail.osinterop.tc001.log``, + * Can see the log details in ``tempest_logs/functest.tempest.XXX.html`` and + ``security_logs/functest.security.XXX.html`` respectively, which has the passed, skipped and failed test cases results. - * The skipped test cases have the reason for the users to see why these test cases skipped. + * This kind of files need to be opened with a web browser. + + * The skipped test cases are accompanied with the reason tag for the users to see why these test cases skipped. - * The failed test cases have rich debug information for the users to see why these test cases fail. + * The failed test cases have rich debug information for the users to see why these test cases failed. - * Example: vping test case example + * Vping test cases - * Its log is stored in dovetail.log. + * Its log is stored in ``vping_logs/functest.vping.XXX.log``. - * Its result is stored in functest_results.txt. + * HA test cases - * Example: ha test case example + * Its log is stored in ``ha_logs/yardstick.ha.XXX.log``. - * Its log is stored in dovetail.log. + * Stress test cases - * Its result is stored in dovetail_ha_tcXXX.out. + * Its log is stored in ``stress_logs/bottlenecks.stress.XXX.log``. - * Example: ipv6, sdnvpn and tempest test cases examples + * VNF test cases - * Can see the log details in ``ipv6_logs/dovetail.ipv6.tcXXX.log``, - ``sdnvpn_logs/dovetail.sdnvpn.tcXXX.log`` and ``tempest_logs/dovetail.tempest.tcXXX.log``, - respectively. They all have the passed, skipped and failed test cases results. + * Its log is stored in ``vnf_logs/functest.vnf.XXX.log``. -CVP Portal Web Interface + +OVP Portal Web Interface ------------------------ -The CVP portal is a public web interface for the community to collaborate on results +The OVP portal is a public web interface for the community to collaborate on results and to submit results for official OPNFV compliance verification. The portal can be used as a -resource by users and testers to navigate and inspect results more easily than by manually +resource by users to navigate and inspect results more easily than by manually inspecting the log files. The portal also allows users to share results in a private manner until they are ready to submit results for peer community review. * Web Site URL - * https://verified.opnfv.org + * https://nfvi-verified.lfnetworking.org * Sign In / Sign Up Links - * Accounts are exposed through Linux Foundation or OpenStack account credentials. + * Accounts are exposed through Linux Foundation. * If you already have a Linux Foundation ID, you can sign in directly with your ID. - * If you do not have a Linux Foundation ID, you can sign up for a new one using 'Sign Up' + * If you do not have a Linux Foundation ID, you can sign up for a new one using 'Sign Up'. * My Results Tab @@ -673,20 +787,25 @@ until they are ready to submit results for peer community review. * This page lists all results uploaded by you after signing in. - * You can also upload results on this page with the two steps below. + * Following the two steps below, the results are uploaded and in status 'private'. + + * Obtain results tar file located at ``${DOVETAIL_HOME}/``, e.g. ``logs_20180105_0858.tar.gz``. - * Obtain results tar file located at ``${DOVETAIL_HOME}/``, example ``logs_20180105_0858.tar.gz`` + * Use the *Choose File* button where a file selection dialog allows you to choose your result file from the hard-disk. Then click the *Upload result* button and see a results ID once your upload succeeds. - * Use the *Choose File* button where a file selection dialog allows you to choose your result - file from the hard-disk. Then click the *Upload* button and see a results ID once your - upload succeeds. + * Results are remaining in status 'private' until they are submitted for review. - * Results are status 'private' until they are submitted for review. + * Use the *Operation* column drop-down option *submit to review*, to expose results to + OPNFV community peer reviewers. Use the *withdraw submit* option to reverse this action. - * Use the *Operation* column drop-down option 'submit to review', to expose results to - OPNFV community peer reviewers. Use the 'withdraw submit' option to reverse this action. + * Results status are changed to be 'review' after submit to review. - * Use the *Operation* column drop-down option 'share with' to share results with other + * Use the *View Reviews* to find the review status including reviewers' names and the outcome. + + * The administrator will approve the results which have got 2 positive outcome from 2 reviewers. + Then the status will be changed to be 'verified'. + + * Use the *Operation* column drop-down option *share with* to share results with other users by supplying either the login user ID or the email address associated with the share target account. The result is exposed to the share target but remains private otherwise. @@ -695,18 +814,19 @@ until they are ready to submit results for peer community review. * This page shows your account info after you sign in. + * There are 3 different roles: administrator, user and reviewer. + Updating Dovetail or a Test Suite --------------------------------- Follow the instructions in section `Installing Dovetail on the Test Host`_ and -`Running the CVP Test Suite`_ by replacing the docker images with new_tags, +`Running the OVP Test Suite`_ by replacing the docker images with new_tags: .. code-block:: bash sudo docker pull opnfv/dovetail:<dovetail_new_tag> sudo docker pull opnfv/functest:<functest_new_tag> sudo docker pull opnfv/yardstick:<yardstick_new_tag> + sudo docker pull opnfv/bottlenecks:<bottlenecks_new_tag> -This step is necessary if dovetail software or the CVP test suite have updates. - - +This step is necessary if dovetail software or the OVP test suite have updates. |