aboutsummaryrefslogtreecommitdiffstats
path: root/docs/userguide/opnfv_yardstick_tc052.rst
Path not found
a id='n14' href='#n14'>14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140
Executing the functest suites
=============================

Manual testing
--------------

Once the Functest docker container is running and Functest environment ready
(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is
ready to run the tests.

The script *run_tests.sh* is located in $repos_dir/functest/docker and it has
several options::

    ./run_tests.sh -h
    Script to trigger the tests automatically.

    usage:
        bash run_tests.sh [--offline] [-h|--help] [-t <test_name>]

    where:
        -h|--help         show this help text
        -r|--report       push results to database (false by default)
        -n|--no-clean     do not clean up OpenStack resources after test run
        -t|--test         run specific set of tests
          <test_name>     one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma.

    examples:
        run_tests.sh
        run_tests.sh --test vping,odl
        run_tests.sh -t tempest,rally --no-clean

The *-r* option is used by the Continuous Integration in order to push the test
results into a test collection database, see in next section for details.
In manual mode, you must not use it, your try will be anyway probably rejected
as your POD must be declared in the database to collect the data.

The *-n* option is used for preserving all the existing OpenStack resources after
execution test cases.

The *-t* option can be used to specify the list of test you want to launch, by
default Functest will try to launch all its test suites in the following order
vPing, odl, Tempest, vIMS, Rally.
You may launch only one single test by using *-t <the test you want to launch>*.

Within Tempest test suite you can define which test cases you want to execute in
your environment by editing test_list.txt file before executing *run_tests.sh*
script.

Please note that Functest includes cleaning mechanism in order to remove
everything except what was present after a fresh install.
If you create your own VMs, tenants, networks etc. and then launch Functest,
they all will be deleted after executing the tests. Use the *--no-clean* option with
run_test.sh in order to preserve all the existing resources.
However, be aware that Tempest and Rally create of lot of resources (users,
tenants, networks, volumes etc.) that are not always properly cleaned, so this
cleaning function has been set to keep the system as clean as possible after a
full Functest run.

You may also add you own test by adding a section into the function run_test().


Automated testing
-----------------

As mentioned in `[1]`, the *prepare-env.sh* and *run_test.sh* can be executed within
the container from jenkins.
2 jobs have been created, one to run all the test and one that allows testing
test suite by test suite.
You thus just have to launch the acurate jenkins job on the target lab, all the
tests shall be automatically run.

When the tests are automatically started from CI, a basic algorithm has been
created in order to detect whether the test is runnable or not on the given
scenario.
In fact, one of the most challenging task in Brahmaputra consists in dealing
with lots of scenario and installers.
Functest test suites cannot be systematically run (e.g. run the ODL suite on an
ONOS scenario).

CI provides several information:

 * The installer (apex|compass|fuel|joid)
 * The scenario [controller]-[feature]-[mode] with

   * controller = (odl|onos|ocl|nosdn)
   * feature = (ovs(dpdk)|kvm)
   * mode = (ha|noha)

Constraints per test case are defined in the Functest configuration file
/home/opnfv/functest/config/config_functest.yaml::

 test-dependencies:
    functest:
        vims:
            scenario: '(ocl)|(odl)|(nosdn)'
        vping:
        vping_userdata:
            scenario: '(ocl)|(odl)|(nosdn)'
        tempest:
        rally:
        odl:
            scenario: 'odl'
        onos:
            scenario: 'onos'
        ....

At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a
file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of
all the runnable tests.
We consider the static constraints as regex and compare them with the scenario.
For instance, odl can be run only on scenario including odl in its name.

The order of execution is also described in the Functest configuration file::

 test_exec_priority:

    1: vping
    2: vping_userdata
    3: tempest
    4: odl
    5: onos
    6: ovno
    7: doctor
    8: promise
    9: odl-vpnservice
    10: bgpvpn
    11: openstack-neutron-bgpvpn-api-extension-tests
    12: vims
    13: rally

The tests are executed as follow:

 * Basic scenario (vPing, vPing_userdata, Tempest)
 * Controller suites: ODL or ONOS or OpenContrail
 * Feature projects
 * vIMS
 * Rally (benchmark scenario)

At the end of an automated execution, everything is cleaned.
We keep only the users/networks that have been statically declared in 'https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml'_