diff options
Diffstat (limited to 'docs/testing/user')
| -rw-r--r-- | docs/testing/user/testspecification/vping/index.rst | 279 | ||||
| -rw-r--r-- | docs/testing/user/userguide/cli_reference.rst | 9 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 477 | ||||
| -rw-r--r-- | docs/testing/user/userguide/testing_guide.rst | 483 |
4 files changed, 776 insertions, 472 deletions
diff --git a/docs/testing/user/testspecification/vping/index.rst b/docs/testing/user/testspecification/vping/index.rst new file mode 100644 index 00000000..d7a207c0 --- /dev/null +++ b/docs/testing/user/testspecification/vping/index.rst @@ -0,0 +1,279 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +======================== +Vping test specification +======================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The vping test area evaluates basic NFVi capabilities of the system under test. +These capabilities include creating a small number of virtual machines, +establishing basic L3 connectivity between them and verifying connectivity by +means of ICMP packets. + + +References +========== + +- Neutron Client + + - https://docs.openstack.org/developer/python-neutronclient/usage/library.html + +- Nova Client + + - https://docs.openstack.org/developer/python-novaclient/ref/v2/servers.html + +- SSHClient + + - http://docs.paramiko.org/en/2.2/ + +- SCPClient + + - https://pypi.python.org/pypi/scp + + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- ICMP - Internet Control Message Protocol +- L3 - Layer 3 +- NFVi - Network functions virtualization infrastructure +- SCP - Secure Copy +- SSH - Secure Shell +- VM - Virtual machine + + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + + +Test Area Structure +=================== + +The test area is structured in two separate tests which are executed +sequentially. The order of the tests is arbitrary as there are no dependencies +across the tests. + + +Test Descriptions +================= + +-------------------------------------------------------------------- +Test Case 1 - vPing using userdata provided by nova metadata service +-------------------------------------------------------------------- + +Short name +---------- + +opnfv.vping.userdata + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and +requires L3 connectivity between those VMs. The target IP is passed to the VM +that will initiate pings by using a custom userdata script provided by nova metadata service. + + +Test preconditions +------------------ + +At least one compute node is available. No further pre-configuration needed. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IP is passed to the VM sending pings by using a +custom userdata script by means of the config driver mechanism provided by +Nova metadata service. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Test action 1: + * Create a private tenant network by using neutron client + * Create one subnet and one router in the network by neutron client + * Add one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Generate ping script with the IP of VM1 to be passed as userdata provided by + the **nova metadata service**. +* Test action 5: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2, userdata created + in test action 4 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 6: Inside VM2, the ping script is executed automatically when booted and it contains a + loop doing the ping until the return code is 0 or timeout reached. For each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 5:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 7: delete VM1, VM2 +* **Test assertion 6:** VM1 and VM2 are not present in the VM list +* Test action 8: delete security group, gateway, interface, router, subnet and network +* **Test assertion 7:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router returns success code +* Neutron client gateway add command to add to router returns success code +* Neutron client security group create command returns valid "id" parameter which is shown in + the response message +* Nova client VM create command returns valid VM attributes response message +* Nova metadata server can transfer userdata configuration at nova client VM booting time +* Ping command from one VM to the other in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None + + +---------------------------------------------- +Test Case 2 - vPing using SSH to a floating IP +---------------------------------------------- + +Short name +---------- + +opnfv.vping.ssh + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and requires +L3 connectivity between those VMs. An SSH connection is establised from the host to +a floating IP associated with VM2 and ``ping`` is executed on VM2 with the IP of VM1 as target. + + +Test preconditions +------------------ + +At least one compute node is available. There should exist an OpenStack external network +and can assign floating IP. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. To this end, the test establishes an SSH connection from the host +running the test suite to a floating IP associated with VM2 and executes ``ping`` +on VM2 with the IP of VM1 as target. + + +Test execution +'''''''''''''' + + +* Test action 1: + * Create a private tenant network by neutron client + * Create one subnet and one router are created in the network by using neutron client + * Create one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: Boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 5: create one floating IP by using neutron client, storing the floating IP address + returned in the response +* **Test assertion 5:** Floating IP address can be found in the response +* Test action 6: Assign the floating IP address created in test action 5 to VM2 by using nova client +* **Test assertion 6:** The assigned floating IP can be found in the VM2 console log file +* Test action 7: Establish SSH connection between the test host and VM2 through the floating IP +* **Test assertion 7:** SSH connection between the test host and VM2 is established within + 300 seconds +* Test action 8: Copy the Ping script from the test host to VM2 by using SCPClient +* **Test assertion 8:** The Ping script can be found inside VM2 +* Test action 9: Inside VM2, to execute the Ping script to ping VM1, the Ping script contains a + loop doing the ping until the return code is 0 or timeout reached, for each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 9:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 10: delete VM1, VM2 +* **Test assertion 10:** VM1 and VM2 are not present in the VM list +* Test action 11: delete floating IP, security group, gateway, interface, router, subnet and network +* **Test assertion 11:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router return success code +* Neutron client gateway add command to add to router return success code +* Neutron client security group create command returns valid "id" parameter which is shown in the + response message +* Nova client VM create command returns valid VM attributes response message +* Neutron client floating IP create command return valid floating IP address +* Nova client add floating IP command returns valid response message +* SSH connection can be established using a floating IP +* Ping command from one VM to another in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None diff --git a/docs/testing/user/userguide/cli_reference.rst b/docs/testing/user/userguide/cli_reference.rst new file mode 100644 index 00000000..719a991f --- /dev/null +++ b/docs/testing/user/userguide/cli_reference.rst @@ -0,0 +1,9 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV + +========================================= +Dovetail Command Line Interface Reference +========================================= + + diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index d8eb124b..aec3e861 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -1,479 +1,12 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB +.. (c) OPNFV -============================================== -Compliance and Verification program user guide -============================================== +******************************************************** +Compliance Verification Program Testing User Guide +******************************************************** .. toctree:: :maxdepth: 2 -Version history -=============== - -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2017-03-15 | 0.0.1 | Chris Price | Draft version | -| | | | | -+------------+----------+------------------+----------------------------------+ - - -Dovetail CVP Testing Overview -============================= - -The Dovetail testing framework consists of two major parts: the testing client that executes -all test cases in a vendor lab (self-testing) or a third party lab, and the server system that -is under the OPNFV's administration to store and view test results based on OPNFV Test API. The -following diagram illustrates this overall framework. - -/* here is a draft diagram that needs to be revised when exact information is known and fixed */ - -This section mainly focuses on helping the testers in the vendor's domain attempting to run the -CVP tests. - -Dovetail client tool (or just Dovetail tool or Dovetail for short) can be installed in the -jumphost either directly as Python software, or as a Docker(r) container. Comments of pros -and cons of the two options TBD. - -The section 'Installing the test tool'_ describes the steps the tester needs to take to install -Dovetail directly from the source. In 2.3, we describe steps needed for installing Dovetail -Docker(r) container. Once installed, and properly configured, the remaining test process is mostly -identical for the two options. In 2.4, we go over the steps of actually running the test suite. -In 2.5, we discuss how to view test results and make sense of them, for example, what the tester -may do in case of unexpected test failures. Section 2.6 describes additional Dovetail features -that are not absolutely necessary in CVP testing but users may find useful for other purposes. -One example is to run Dovetail for in-house testing as preparation before official CVP testing; -another example is to run Dovetail experimental test suites other than the CVP test suite. -Experimental tests may be made available by the community for experimenting less mature test -cases or functionalities for the purpose of getting feedbacks for improvement. - -Installing the test tool -======================== - -Before taking this step, testers should check the hardware and networking requirements of -the POD, and the jumphost in particular, to make sure they are compliant. - -In this section, we describe the procedure to install Dovetail client tool that runs the CVP -test suite from the jumphost. The jumphost must have network access to both the public Internet -and to the O&M (Operation and Management) network with access rights to all VIM APIs being tested. - -------------------------------- -Checking the Jumphost Readiness -------------------------------- - -While Dovetail does not have hard requirement on a specific operating system type or version, -these have been validated by the community through some level of exercise in OPNFV labs or PlugFests. - -Ubuntu 16.04.2 LTS (Xenial) for x86_64 -Ubuntu 14.04 LTS (Trusty) for x86_64 -CentOS-7-1611 for x86_64 -Red Hat Enterprise Linux 7.3 for x86_64 -Fedora 24 Server for x86_64 -Fedora 25 Server for x86_64 - ------------------------------------- -Configuring the Jumphost Environment ------------------------------------- - -/* First, openstack env variables to be passed to Functest */ - -The jumphost needs to have the right environmental variable setting to enable access to the -Openstack API. This is usually done through the Openstack credential file. - -Sample Openstack credential file environment_config.sh: - -/*Project-level authentication scope (name or ID), recommend admin project.*/ - -export OS_PROJECT_NAME=admin - -/* Authentication username, belongs to the project above, recommend admin user.*/ - -export OS_USERNAME=admin - - -/* Authentication password.*/ - -export OS_PASSWORD=secret - - -/* Authentication URL, one of the endpoints of keystone service. If this is v3 version, there need some extra variables as follows.*/ - -export OS_AUTH_URL='http://xxx.xxx.xxx.xxx:5000/v3' - - -/* Default is 2.0. If use keystone v3 API, this should be set as 3.*/ - -export OS_IDENTITY_API_VERSION=3 - - -/* Domain name or ID containing the user above. Command to check the domain: openstack -user show <OS_USERNAME>*/ - -export OS_USER_DOMAIN_NAME=default - - -/* Domain name or ID containing the project above. Command to check the domain: openstack -project show <OS_PROJECT_NAME>*/ - -export OS_PROJECT_DOMAIN_NAME=default - - -/* home directory for dovetail, if install Dovetail Docker container, DOVETAIL_HOME can -just be /home/opnfv*/ - -export DOVETAIL_HOME=$HOME/cvp - -Export all these variables into environment by, - -% source <OpenStack-credential-file-path> - - -The tester should validate that the Openstack environmental settings are correct by, -% openstack service list - ------------------------------------ -Installing Prerequisite on Jumphost ------------------------------------ - -1. Dovetail requires Python 2.7 and later - -Use the following steps to check if the right version of python is already installed, -and if not, install it. - -% python --version - -2. Dovetail requires Docker 1.8.0 and later - -Use the following steps to check if the right version of Docker is already installed, -and if not, install it. - -% docker --version - -As the docker installation process is much complex, you can refer to the official -document: https://docs.docker.com/engine/installation/linux/ - -------------------------------------- -2.2.4 Installing Dovetail on Jumphost -------------------------------------- - -A tester can choose one of the following two methods for installing and running Dovetail. -In part1, we explain the steps to install Dovetail from the source. In part2, an alternative -using a Docker image with preinstalled Dovetail is introduced. part1. Installing Dovetail directly - -Update and install packages - -a) Ubuntu - -sudo apt-get update - -sudo apt-get -y install gcc git vim python-dev python-pip --no-install-recommends - -b) centos and redhat - -sudo yum -y update - -sudo yum -y install epel-release - -sudo yum -y install gcc git vim-enhanced python-devel python-pip - -c) fedora - -sudo dnf -y update - -sudo dnf -y install gcc git vim-enhanced python-devel python-pip redhat-rpm-config - -p.s When testing SUT's https service, there need some extra packages, such as -apt-transport-https. This still remains to be verified. - - -Installing Dovetail - -Now we are ready to install Dovetail. - -/* Version of dovetail is not specified yet? we are still using the latest in the master -- this needs to be fixed before launch. */ - -First change directory to $DOVETAIL_HOME, - -% cd $DOVETAIL_HOME - -% sudo git clone https://git.opnfv.org/dovetail - -% cd $DOVETAIL_HOME/dovetail - -% sudo pip install -e ./ - -/* test dovetail install is successful */ - -% dovetail -h -part2. Installing Dovetail Docker Container - -The Dovetail project also maintains a Docker image that has Dovetail test tools preinstalled. - -Running CVP Test Suite -====================== - ------------------- -Running Test Suite ------------------- - -The Dovetail client CLI allows the tester to specify which test suite to run. -By default the results are stored in a local file $DOVETAIL_HOME/dovetail/results. - -% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> /*?? */ - -Multiple test suites may be available, testsuites named "debug" and "proposed_tests" are just provided for testing. But for the purpose of running CVP test suite, the test suite name follows the following format, - -CVP.<major>.<minor>.<patch> /* test if this format works */ - -For example, CVP_1_0_0 - -% dovetail run --testsuite CVP_1_0_0 - -When the SUT's VIM (Virtual Infrastructure Manager) is Openstack, its configuration is commonly defined in the openrc file. In that case, you can specify the openrc file in the command line, - -% dovetail run --testsuite CVP_1_0_0 --openrc <path-to-openrc-file> - -In order to report official results to OPNFV, run the CVP test suite and report to OPNFV official URL, - -% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> --report https://www.opnfv.org/cvp - -The official server https://www.opnfv.org/cvp is still under development, there is a temporal server to use http://205.177.226.237:9997/api/v1/results - --------------------------------- -Making Sense of CVP Test Results --------------------------------- - -When a tester is performing trial runs, Dovetail stores results in a local file by default. - -% cd $DOVETAIL_HOME/dovetail/results - - - -1. local file - -a) Log file: dovetail.log - -/* review the dovetail.log to see if all important information has been captured - in default mode without DEBUG */ - -/* the end of the log file has a summary of all test case test results */ - -Additional log files may be of interests: refstack.log, opnfv_yardstick_tcXXX.out ... - -b) Example: Openstack refstack test case example - -can see the log details in refstack.log, which has the passed/skipped/failed test cases result, the failed test cases have rich debug information - -for the users to see why this test case fails. - -c) Example: OPNFV Yardstick test case example - -for yardstick tool, its log is stored in yardstick.log - -for each test case result in Yardstick, the logs are stored in opnfv_yardstick_tcXXX.out, respectively. - - - -2. OPNFV web interface - -wait for the complement of LF, test community, etc. -2.3.3 Updating Dovetail or Test Suite - -% cd $DOVETAIL_HOME/dovetail - -% sudo git pull - -% sudo pip install -e ./ - -This step is necessary if dovetail software or the CVP test suite have updates. - - -Other Dovetail Usage -==================== - ------------------------- -Running Dovetail Locally ------------------------- - -/*DB*/ - ---------------------------------------------- -Running Dovetail with Experimental Test Cases ---------------------------------------------- - - --------------------------------------------------- -Running Individual Test Cases or for Special Cases --------------------------------------------------- - -1. Refstack client to run Defcore testcases - -a) By default, for Defcore test cases run by Refstack-client, which are consumed by -DoveTail, are run followed with automatically generated configuration file, i.e., -refstack_tempest.conf. - -In some circumstances, the automatic configuration file may not quite satisfied with -the SUT, DoveTail provide a way for users to set its configuration file according -to its own SUT manually, - -besides, the users should define Defcore testcase file, i.e., defcore.txt, at the -same time. The steps are shown as, - -when "Installing Dovetail Docker Container" method is used, - - -% sudo mkdir /home/opnfv/dovetail/userconfig - -% cd /home/opnfv/dovetail/userconfig - -% touch refstack_tempest.conf defcore.txt - -% vim refstack_tempest.conf - -% vim defcore.txt - - -the recommend way to set refstack_tempest.conf is shown in -https://aptira.com/testing-openstack-tempest-part-1/ - -the recommended way to edit defcore.txt is to open -https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false -and copy all the test cases into defcore.txt. - -Then use “docker run” to create a container, - - -% sudo docker run --privileged=true -it -v <openrc_path>:<openrc_path> \ - --v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \ - --v /home/opnfv/dovetail/userconfig:/home/opnfv/dovetail/userconfig \ - --v /var/run/docker.sock:/var/run/docker.sock \ - ---name <DoveTail_Container_Name> (optional) \ - -opnfv/dovetail:<Tag> /bin/bash - - - -there is a need to adjust the CVP_1_0_0 testsuite, for dovetail, -defcore.tc001.yml and defcore.tc002.yml are used for automatic and -manual running method, respectively. - -Inside the dovetail container, - - -% cd /home/opnfv/dovetail/compliance - -% vim CVP_1_0_0.yml - - -to add defcore.tc002 and annotate defcore.tc001. - - -b) when "Installing Dovetail Directly" method is used, before to run -the dovetail commands, there is a need to set configuration file and -defcore test cases file - - -% cd $DOVETAIL_HOME/dovetail - -% mkdir userconfig - -% cd userconfig - -% touch refstack_tempest.conf defcore.txt - -% vim refstack_tempest.conf - -% vim defcore.txt - -recommended way to set refstack_tempest.conf and defcore.txt is -same as above in "Installing Dovetail Docker Container" method section. - - - -For Defcore test cases manually running method, there is a need to adjust -the compliance_set test suite, - -for dovetail, defcore.tc001.yml and defcore.tc002.yml are used for automatic -and manual running method, respectively. - - - -% cd $DOVETAIL_HOME/dovetail/compliance - -% vim CVP_1_0_0.yml - - -to add defcore.tc002 and annotate defcore.tc001 - -3 Dovetail Client CLI Manual - -This section contains a brief manual for all the features available through the Dovetail client command line interface (CLI). -3.1 Check dovetail commands - -% dovetail -h - -dovetail.PNG - -Dovetail has three commands: list, run and show. -6.2 List -6.2.1 List help - -% dovetail list -h - -list-help.PNG -6.2.2 List a test suite - -List command will list all test cases belong to the given test suite. - -% dovetail list compliance_set - -list-compliance.PNG - -% dovetail list debug - -list-debug.PNG - -The ipv6, example and nfvi are test areas. If no <TESTSUITE> is given, it will list all testsuites. -6.3 Show - -Show command will give the detailed info of one certain test case. -6.3.1 Show help - -% dovetail show -h - -show-help.PNG -6.3.2 Show test case - -show-ipv6.PNG -6.4 Run - -Dovetail supports running a named test suite, or one named test area of a test suite. -6.4.1 Run help - -% dovetail run -h - -run-help.PNGThere are some options: - -func_tag: set FuncTest’s Docker tag, for example stable,latest and danube.1.0 - -openrc: give the path of OpenStack credential file - -yard_tag: set Yardstick’s Docker tag - -testarea: set a certain testarea within a certain testsuite - -offline: run without pull the docker images, and it requires the jumphost to have these images locally. This will ensure DoveTail run in an offline environment. - -report: push results to DB or store with files - -testsuite: set the testsuite to be tested - -debug: flag to show the debug log messages - + testing_guide.rst diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst new file mode 100644 index 00000000..9e46b561 --- /dev/null +++ b/docs/testing/user/userguide/testing_guide.rst @@ -0,0 +1,483 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +========================================== +Conducting CVP Testing with Dovetail +========================================== + +Overview +------------------------------ + +The Dovetail testing framework for CVP consists of two major parts: the testing client that +executes all test cases in a lab (vendor self-testing or a third party lab), +and the server system that is hosted by the CVP administrator to store and +view test results based on a web API. The following diagram illustrates +this overall framework. + +.. image:: ../../../images/dovetail_online_mode.png + :align: center + :scale: 50% + +Within the tester's lab, the Test Host is the machine where Dovetail executes all +automated test cases. As it hosts the test harness, the Test Host must not be part of +the System Under Test (SUT) itself. +The above diagram assumes that the tester's Test Host is situated in a DMZ which +has internal network access to the SUT and external access to the OPNFV server +via the public Internet. +This arrangement may not be supported in some labs. +Dovetail also supports an offline mode of testing that is +illustrated in the next diagram. + +.. image:: ../../../images/dovetail_offline_mode.png + :align: center + :scale: 50% + +In the offline mode, the Test Host only needs to have access to the SUT +via the internal network, but does not need to connect to the public Internet. This +user guide will highlight differences between the online and offline modes of +the Test Host. While it is possible to run the Test Host as a virtual machine, +this user guide assumes it is a physical machine for simplicity. + +The rest of this guide will describe how to install the Dovetail tool as a +Docker container image, go over the steps of running the CVP test suite, and +then discuss how to view test results and make sense of them. + +Readers interested +in using Dovetail for its functionalities beyond CVP testing, e.g. for in-house +or extended testing, should consult the Dovetail developer's guide for additional +information. + +Installing Dovetail +-------------------- + +In this section, we describe the procedure to install Dovetail client tool on the Test Host. +The Test Host must have network access to the management network with access rights to +the Virtual Infrastructure Manager's API. + +Checking the Test Host Readiness +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Test Host must have network access to the Virtual Infrastructure Manager's API +hosted in the SUT so that the Dovetail tool can exercise the API from the Test Host. +It must also have ``ssh`` access to the Linux operating system +of the compute nodes in the SUT. The ``ssh`` mechanism is used by some test cases +to generate test events in the compute nodes. You can find out which test cases +use this mechanism in the test specification document. + +We have tested the Dovetail tool on the following host operating systems. Other versions +or distribution of Linux may also work, but community support may be more available on +these versions. + +- Ubuntu 16.04.2 LTS (Xenial) or 14.04 LTS (Trusty) +- CentOS-7-1611 +- Red Hat Enterprise Linux 7.3 +- Fedora 24 or 25 Server + +Non-Linux operating systems, such as Windows, Mac OS, have not been tested +and are not supported. + +If online mode is used, the tester should also validate that the Test Host can reach +the public Internet. For example, + +.. code-block:: bash + + $ ping www.opnfv.org + PING www.opnfv.org (50.56.49.117): 56 data bytes + 64 bytes from 50.56.49.117: icmp_seq=0 ttl=48 time=52.952 ms + 64 bytes from 50.56.49.117: icmp_seq=1 ttl=48 time=53.805 ms + 64 bytes from 50.56.49.117: icmp_seq=2 ttl=48 time=53.349 ms + ... + + +Or, if the lab environment does not allow ping, try validating it using HTTPS instead. + +.. code-block:: bash + + $ curl https://www.opnfv.org + <!doctype html> + + + <html lang="en-US" class="no-js"> + <head> + ... + + +Configuring the Test Host Environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Test Host needs a few environment variables set correctly in order to access the +Openstack API required to drive the Dovetail tests. For convenience and as a convention, +we will also create a home directory for storing all Dovetail related config files and +results files: + +.. code-block:: bash + + $ mkdir -p /home/dovetail + $ export DOVETAIL_HOME=/home/dovetail + +Here we set dovetail home directory to be ``/home/dovetail`` for an example. +Then create a directory named ``pre_config`` in this directory to store all +Dovetail related config files: + +.. code-block:: bash + + $ mkdir -p ${DOVETAIL_HOME}/pre_config + +At this point, you will need to consult your SUT (Openstack) administrator to correctly set +the configurations in a file named ``env_config.sh``. +The Openstack settings need to be configured such that the Dovetail client has all the necessary +credentials and privileges to execute all test operations. If the SUT uses terms +somewhat differently from the standard Openstack naming, you will need to adjust +this file accordingly. + +In our example, we will use the file '${DOVETAIL_HOME}/pre_config/env_config.sh'. Create and edit +the file so that all parameters are set correctly to match your SUT. Here is an example of what +this file should contain. + +.. code-block:: bash + + $ cat ${DOVETAIL_HOME}/pre_config/env_config.sh + + # Project-level authentication scope (name or ID), recommend admin project. + export OS_PROJECT_NAME=admin + + # For identity v2, it uses OS_TENANT_NAME rather than OS_PROJECT_NAME. + export OS_TENANT_NAME=admin + + # Authentication username, belongs to the project above, recommend admin user. + export OS_USERNAME=admin + + # Authentication password. Use your own password + export OS_PASSWORD=xxxxxxxx + + # Authentication URL, one of the endpoints of keystone service. If this is v3 version, + # there need some extra variables as follows. + export OS_AUTH_URL='http://xxx.xxx.xxx.xxx:5000/v3' + + # Default is 2.0. If use keystone v3 API, this should be set as 3. + export OS_IDENTITY_API_VERSION=3 + + # Domain name or ID containing the user above. + # Command to check the domain: openstack user show <OS_USERNAME> + export OS_USER_DOMAIN_NAME=default + + # Domain name or ID containing the project above. + # Command to check the domain: openstack project show <OS_PROJECT_NAME> + export OS_PROJECT_DOMAIN_NAME=default + + # Home directory for dovetail that you have created before. + export DOVETAIL_HOME=/home/dovetail + +Export all these variables into environment by, + +.. code-block:: bash + + $ source ${DOVETAIL_HOME}/pre_config/env_config.sh + +If OpenStack client is installed, you can validate that the OpenStack environmental +settings are correct by, + +.. code-block:: bash + + $ openstack service list + + +Installing Prerequisite on the Test Host +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The main prerequisite software for Dovetail are Python and Docker. + +In the CVP test suite for the Danube release, Dovetail requires Python 2.7. Python 3.x +is not supported at this time. + +Use the following steps to check if the right version of python is already installed, +and if not, install it. + +.. code-block:: bash + + $ python --version + Python 2.7.6 + +If your Test Host does not have Python installed, or the version is not 2.7, you +should consult Python installation guides corresponding to the operating system +in your Test Host on how to install Python 2.7. + +Dovetail does not work with Docker versions prior to 1.12.3. We have validated +Dovetail with Docker 17.03 CE. Other versions of Docker later than 1.12.3 may +also work, but community support may be more available on Docker 17.03 CE. + +.. code-block:: bash + + $ sudo docker version + Client: + Version: 17.03.1-ce + API version: 1.27 + Go version: go1.7.5 + Git commit: c6d412e + Built: Mon Mar 27 17:10:36 2017 + OS/Arch: linux/amd64 + + Server: + Version: 17.03.1-ce + API version: 1.27 (minimum version 1.12) + Go version: go1.7.5 + Git commit: c6d412e + Built: Mon Mar 27 17:10:36 2017 + OS/Arch: linux/amd64 + Experimental: false + +If your Test Host does not have Docker installed, or Docker is older than 1.12.3, +or you have Docker version other than 17.03 CE and wish to change, +you will need to install, upgrade, or re-install in order to run Dovetail. +The Docker installation process +can be more complex, you should refer to the official +Docker installation guide that is relevant to your Test Host's operating system. + +The above installation steps assume that the Test Host is in the online mode. For offline +testing, use the following offline installation steps instead. + +In order to install or upgrade Python offline, you may download packaged Python 2.7 +for your Test Host's operating system on a connected host, copy the packge to +the Test Host, then install from that local copy. + +In order to install Docker offline, download Docker static binaries and copy the +tar file to the Test Host, such as for Ubuntu14.04, you may follow the following link +to install, + +.. code-block:: bash + + https://github.com/meetyg/docker-offline-install + + +Installing Dovetail on the Test Host +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Dovetail project maintains a Docker image that has Dovetail test tools preinstalled. +This Docker image is tagged with versions. Before pulling the Dovetail image, check the +OPNFV's CVP web page first to determine the right tag for CVP testing. + +If the Test Host is online, you can directly pull. + +.. code-block:: bash + + $ sudo docker pull opnfv/dovetail:cvp.0.3.0 + cvp.0.3.0: Pulling from opnfv/dovetail + 30d541b48fc0: Pull complete + 8ecd7f80d390: Pull complete + 46ec9927bb81: Pull complete + 2e67a4d67b44: Pull complete + 7d9dd9155488: Pull complete + cc79be29f08e: Pull complete + e102eed9bf6a: Pull complete + 952b8a9d2150: Pull complete + bfbb639d1f38: Pull complete + bf7c644692de: Pull complete + cdc345e3f363: Pull complete + Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee + Status: Downloaded newer image for opnfv/dovetail:cvp.0.3.0 + +An example of the <tag> is *cvp.0.3.0*. + +If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the +dependent images that Dovetail uses, to a host that is online. The reason that you need +to pull all dependent images is because Dovetail normally does dependency checking at run-time +and automatically pull images as needed, if the Test Host is online. If the Test Host is +offline, then all these dependencies will also need to be manually copied. + +.. code-block:: bash + + $ sudo docker pull opnfv/dovetail:cvp.0.3.0 + $ sudo docker pull opnfv/functest:cvp.0.2.0 + $ sudo docker pull opnfv/yardstick:danube.3.1 + $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} + +Once all these images are pulled, save the images, copy to the Test Host, and then load +the Dovetail and all dependent images at the Test Host. + +At the online host, save images. + +.. code-block:: bash + + $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.3.0 opnfv/functest:cvp.0.2.0 opnfv/yardstick:danube.3.1 + +Copy dovetail.tar file to the Test Host, and then load the images on the Test Host. + +.. code-block:: bash + + $ sudo docker load --input dovetail.tar + +Copy sdnvpn test area image ubuntu-16.04-server-cloudimg-amd64-disk1.img to ${DOVETAIL_HOME}/pre_config/. + +Now check to see that the Dovetail image has been pulled or loaded properly. + +.. code-block:: bash + + $ sudo docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + opnfv/functest cvp.0.2.0 9eaeaea5f203 8 days ago 1.53GB + opnfv/dovetail cvp.0.3.0 5d25b289451c 8 days ago 516MB + opnfv/yardstick danube.3.1 574596b6ea12 8 days ago 1.2GB + +Regardless of whether you pulled down the Dovetail image directly online, or loaded from +a static image tar file, you are ready to run Dovetail. + +.. code-block:: bash + + $ sudo docker run --privileged=true -it \ + -e DOVETAIL_HOME=$DOVETAIL_HOME \ + -v $DOVETAIL_HOME:$DOVETAIL_HOME \ + -v /var/run/docker.sock:/var/run/docker.sock \ + opnfv/dovetail:<tag> /bin/bash + +The ``-e`` options set the env variables in the container and the ``-v`` options map files +in the host to files in the container. + +Running the CVP Test Suite +---------------------------- + +Now you should be in the Dovetail container's prompt and ready to execute +test suites. + +The Dovetail client CLI allows the tester to specify which test suite to run. +By default the results are stored in a local file +``$DOVETAIL_HOME/results``. + +.. code-block:: bash + + $ dovetail run --testsuite <test-suite-name> + +Multiple test suites may be available. For the purpose of running +CVP test suite, the test suite name follows the following format, +``CVP_<major>_<minor>_<patch>`` +For example, CVP_1_0_0. + +.. code-block:: bash + + $ dovetail run --testsuite CVP_1_0_0 + +If you are not running the entire test suite, you can choose to run an +individual test area instead. + +.. code-block:: bash + + $ dovetail run --testsuite CVP_1_0_0 --testarea ipv6 + +Until the official test suite is approved and released, you can use +the *proposed_tests* for your trial runs, like this. + +.. code-block:: bash + + $ dovetail run --testsuite proposed_tests --testarea ipv6 + 2017-05-23 05:01:49,488 - run - INFO - ================================================ + 2017-05-23 05:01:49,488 - run - INFO - Dovetail compliance: proposed_tests! + 2017-05-23 05:01:49,488 - run - INFO - ================================================ + 2017-05-23 05:01:49,488 - run - INFO - Build tag: daily-master-4bdde6b8-afa6-40bb-8fc9-5d568d74c8d7 + 2017-05-23 05:01:49,536 - run - INFO - + 2017-05-23 05:01:49,710 - run - INFO - >>[testcase]: dovetail.ipv6.tc001 + 2017-05-23 05:08:22,532 - run - INFO - Results have been stored with file /home/dovetail/results/functest_results.txt. + 2017-05-23 05:08:22,538 - run - INFO - >>[testcase]: dovetail.ipv6.tc002 + ... + +Special Configuration for Running HA Test Cases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +HA test cases need to know the info of a controller node of the OpenStack. +It should include the node's name, role, ip, as well as the user and key_filename +or password to login the node. Users should create file ${DOVETAIL_HOME}/pre_config/pod.yaml +to store the info. + +There is a sample file for users. + +.. code-block:: bash + + nodes: + - + # This can not be changed and must be node1. + name: node1 + + # This must be controller. + role: Controller + + # This is the install IP of a controller node. + ip: xx.xx.xx.xx + + # User name of this node. This user must have sudo privileges. + user: root + + # Password of the user. + password: root + +Besides the 'password', user could also provide 'key_filename' to login the node. +Users need to create file $DOVETAIL_HOME/pre_config/id_rsa to store the private key. + +.. code-block:: bash + + name: node1 + role: Controller + ip: 10.1.0.50 + user: root + + # Private key of this node. It must be /root/.ssh/id_rsa + # Dovetail will move the key file from $DOVETAIL_HOME/pre_config/id_rsa + # to /root/.ssh/id_rsa of Yardstick container + key_filename: /root/.ssh/id_rsa + + +Making Sense of CVP Test Results +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a tester is performing trial runs, Dovetail stores results in a local file by default. + +.. code-block:: bash + + cd $DOVETAIL_HOME/results + +#. Local file + + * Log file: dovetail.log + + * Review the dovetail.log to see if all important information has been captured + - in default mode without DEBUG. + + * The end of the log file has a summary of all test case test results. + + * Additional log files may be of interests: refstack.log, dovetail_ha_tcXXX.out ... + + * Example: Openstack refstack test case example + + * Can see the log details in refstack.log, which has the passed/skipped/failed + test cases result, the failed test cases have rich debug information for the + users to see why this test case fails. + + * Example: OPNFV Functest test case example + + * For Functest tool, its log is stored in functest.log + + * For each test case result in Functest, the logs are stored in functest_results.txt. + + * Example: OPNFV Yardstick test case example + + * For Yardstick tool, its log is stored in yardstick.log + + * For each test case result in Yardstick, the logs are stored in dovetail_ha_tcXXX.out, respectively. + +#. OPNFV web interface + + Wait for the complement of LF, test community, etc. + +Updating Dovetail or a Test Suite +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Follow the instructions in section `Installing Dovetail on the Test Host`_ and +`Running the CVP Test Suite`_ by replacing the docker images with new_tags, + +.. code-block:: bash + + sudo docker pull opnfv/dovetail:<dovetail_new_tag> + sudo docker pull opnfv/functest:<functest_new_tag> + sudo docker pull opnfv/yardstick:<yardstick_new_tag> + +This step is necessary if dovetail software or the CVP test suite have updates. + + |