diff options
Diffstat (limited to 'docs/userguide/runfunctest.rst')
| -rw-r--r-- | docs/userguide/runfunctest.rst | 135 |
1 files changed, 82 insertions, 53 deletions
diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst index b186e539..36433374 100644 --- a/docs/userguide/runfunctest.rst +++ b/docs/userguide/runfunctest.rst @@ -5,89 +5,120 @@ Manual testing -------------- Once the Functest docker container is running and Functest environment ready -(through /home/opnfv/repos/functest/docker/prepare_env.sh script), the system is +(through */home/opnfv/repos/functest/docker/prepare_env.sh* script), the system is ready to run the tests. -The script *run_tests.sh* is located in $repos_dir/functest/docker and it has +The script *run_tests.sh* launches the test in an automated way. +Although it is possible to execute the different tests manually, it is +recommended to use the previous shell script which makes the call +to the actual scripts with the appropriate parameters. + +It is located in *$repos_dir/functest/docker* and it has several options:: ./run_tests.sh -h Script to trigger the tests automatically. usage: - bash run_tests.sh [--offline] [-h|--help] [-t <test_name>] + bash run_tests.sh [-h|--help] [-r|--report] [-n|--no-clean] [-t|--test <test_name>] where: -h|--help show this help text -r|--report push results to database (false by default) -n|--no-clean do not clean up OpenStack resources after test run + -s|--serial run tests in one thread -t|--test run specific set of tests - <test_name> one or more of the following: vping,vping_userdata,odl,rally,tempest,vims,onos,promise. Separated by comma. + <test_name> one or more of the following separated by comma: + vping_ssh,vping_userdata,odl,rally,tempest,vims,onos,promise,ovno examples: run_tests.sh run_tests.sh --test vping,odl run_tests.sh -t tempest,rally --no-clean -The *-r* option is used by the Continuous Integration in order to push the test -results into a test collection database, see in next section for details. -In manual mode, you must not use it, your try will be anyway probably rejected -as your POD must be declared in the database to collect the data. +The *-r* option is used by the OPNFV Continuous Integration automation mechanisms +in order to push the test results into the NoSQL results collection database. +This database is read only for a regular user given that it needs special rights +and special conditions to push data. -The *-n* option is used for preserving all the existing OpenStack resources after -execution test cases. +The *-t* option can be used to specify the list of a desired test to be launched, +by default Functest will launch all the test suites in the following order: +vPing, Tempest, vIMS, Rally. -The *-t* option can be used to specify the list of test you want to launch, by -default Functest will try to launch all its test suites in the following order -vPing, odl, Tempest, vIMS, Rally. -You may launch only one single test by using *-t <the test you want to launch>*. +A single or set of test may be launched at once using *-t <test_name>* specifying +the test name or names separated by commas in the following list: +*[vping,vping_userdata,odl,rally,tempest,vims,onos,promise]*. -Within Tempest test suite you can define which test cases you want to execute in -your environment by editing test_list.txt file before executing *run_tests.sh* -script. +The *-n* option is used for preserving all the possible OpenStack resources created +by the tests after their execution. Please note that Functest includes cleaning mechanism in order to remove -everything except what was present after a fresh install. -If you create your own VMs, tenants, networks etc. and then launch Functest, -they all will be deleted after executing the tests. Use the *--no-clean* option with -run_test.sh in order to preserve all the existing resources. -However, be aware that Tempest and Rally create of lot of resources (users, +all the VIM resources except what was present before running any test. The script +*$repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/generate_defaults.py* +is called once by *prepare_env.sh* when setting up the Functest environment +to snapshot all the OpenStack resources (images, networks, volumes, security groups, +tenants, users) so that an eventual cleanup does not remove any of this defaults. + +The *-s* option forces execution of test cases in a single thread. Currently this +option affects Tempest test cases only and can be used e.g. for troubleshooting +concurrency problems. + +The script +*$repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/clean_openstack.py* +is normally called after a test execution if the *-n* is not specified. It +is in charge of cleaning the OpenStack resources that are not specified +in the defaults file generated previously which is stored in +*/home/opnfv/functest/conf/os_defaults.yaml* in the docker +container. + +It is important to mention that if there are new OpenStack resources created +manually after preparing the Functest environment, they will be removed if this +flag is not specified in the *run_tests.sh* command. +The reason to include this cleanup meachanism in Functest is because some +test suites such as Tempest or Rally create a lot of resources (users, tenants, networks, volumes etc.) that are not always properly cleaned, so this -cleaning function has been set to keep the system as clean as possible after a -full Functest run. +cleaning function has been set to keep the system as clean as it was before a +full Functest execution. -You may also add you own test by adding a section into the function run_test(). +Within the Tempest test suite it is possible to define which test cases to execute +by editing *test_list.txt* file before executing *run_tests.sh* script. This file +is located in *$repos_dir/functest/testcases/VIM/OpenStack/CI/custom_tests/test_list.txt* +Although *run_tests.sh* provides an easy way to run any test, it is possible to +do a direct call to the desired test script. For example:: + python $repos_dir/functest/testcases/vPing/vPing.py -d Automated testing ----------------- -As mentioned in `[1]`, the *prepare-env.sh* and *run_test.sh* can be executed within -the container from jenkins. -2 jobs have been created, one to run all the test and one that allows testing -test suite by test suite. -You thus just have to launch the acurate jenkins job on the target lab, all the -tests shall be automatically run. +As mentioned in `[1]`, the *prepare-env.sh* and *run_test.sh* can be called within +the container from Jenkins. There are 2 jobs that automate all the manual steps +explained in the previous section. One job runs all the tests and the other one allows testing +test suite by test suite specifying the test name. The user might use one or +the other job to execute the desired test suites. + +One of the most challenging task in the Brahmaputra release consists +in dealing with lots of scenarios and installers. Thus, when the tests are +automatically started from CI, a basic algorithm has been created in order to +detect whether a given test is runnable or not on the given scenario. +Some Functest test suites cannot be systematically run (e.g. ODL suite can not +be run on an ONOS scenario). + -When the tests are automatically started from CI, a basic algorithm has been -created in order to detect whether the test is runnable or not on the given -scenario. -In fact, one of the most challenging task in Brahmaputra consists in dealing -with lots of scenario and installers. -Functest test suites cannot be systematically run (e.g. run the ODL suite on an -ONOS scenario). -CI provides several information: +CI provides some useful information passed to the container as environment +variables: - * The installer (apex|compass|fuel|joid) - * The scenario [controller]-[feature]-[mode] with + * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE + * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP + * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with * controller = (odl|onos|ocl|nosdn) * feature = (ovs(dpdk)|kvm) * mode = (ha|noha) -Constraints per test case are defined in the Functest configuration file -/home/opnfv/functest/config/config_functest.yaml:: +The constraints per test case are defined in the Functest configuration file +*/home/opnfv/functest/config/config_functest.yaml*:: test-dependencies: functest: @@ -105,16 +136,17 @@ Constraints per test case are defined in the Functest configuration file .... At the end of the Functest environment creation (prepare_env.sh see `[1]`_), a -file (/home/opnfv/functest/conf/testcase-list.txt) is created with the list of +file */home/opnfv/functest/conf/testcase-list.txt* is created with the list of all the runnable tests. -We consider the static constraints as regex and compare them with the scenario. -For instance, odl can be run only on scenario including odl in its name. +Functest considers the static constraints as regular expressions and compare them +with the given scenario name. +For instance, ODL suite can be run only on an scenario including 'odl' in its name. The order of execution is also described in the Functest configuration file:: test_exec_priority: - 1: vping + 1: vping_ssh 2: vping_userdata 3: tempest 4: odl @@ -130,13 +162,10 @@ The order of execution is also described in the Functest configuration file:: The tests are executed in the following order: - * Basic scenario (vPing, vPing_userdata, Tempest) + * Basic scenario (vPing_ssh, vPing_userdata, Tempest) * Controller suites: ODL or ONOS or OpenContrail * Feature projects (promise, vIMS) * Rally (benchmark scenario) -At the end of an automated execution, everything is cleaned. -Before running Functest, a snapshot of the OpenStack configuration (users, -tenants, networks, ....) is performed. After Functest, a clean mechanism is -launched to delete everything that would not have been properly deleted in order -to restitute the system as it was prior to the tests. +As explained before, at the end of an automated execution, the OpenStack resources +might be eventually removed. |