.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 Executing the functest suites ============================= Manual testing -------------- This section assumes the following: * The Functest Docker container is running * The docker prompt is shown * The Functest environment is ready (Functest CLI command 'functest env prepare' has been executed) If any of the above steps are missing please refer to the Functest Config Guide as they are a prerequisite and all the commands explained in this section **must** be performed **inside the container**. Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a new Functest CLI. One difference, is that tests run through the Functest CLI will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this document, where this difference is discussed. The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) for the execution of Test Tiers or Test Cases:: root@22e436918db0:~/repos/functest/ci# functest tier --help Usage: functest tier [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. Commands: get-tests Prints the tests in a tier. list Lists the available tiers. run Executes all the tests within a tier. show Shows information about a tier. root@22e436918db0:~/repos/functest/ci# functest testcase --help and Usage: functest testcase [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. Commands: list Lists the available testcases. run Executes a test case. show Shows information about a test case. More details on the existing Tiers and Test Cases can be seen with the 'list' command:: root@22e436918db0:~/repos/functest/ci# functest tier list - 0. healthcheck: ['healthcheck'] - 1. smoke: ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - 2. sdn_suites: ['odl'] - 3. features: ['doctor', 'security_scan'] - 4. openstack: ['tempest_full_parallel', 'rally_full'] - 5. vnf: ['vims'] and root@22e436918db0:~/repos/functest/ci# functest testcase list healthcheck vping_ssh vping_userdata tempest_smoke_serial rally_sanity odl doctor security_scan tempest_full_parallel rally_full vims More specific details on specific Tiers or Test Cases can be seen wih the 'show' command:: root@22e436918db0:~/repos/functest/ci# functest tier show smoke +======================================================================+ | Tier: smoke | +======================================================================+ | Order: 1 | | CI Loop: (daily)|(weekly) | | Description: | | Set of basic Functional tests to validate the OpenStack | | deployment. | | Test cases: | | - vping_ssh | | - vping_userdata | | - tempest_smoke_serial | | - rally_sanity | | | +----------------------------------------------------------------------+ and root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial +======================================================================+ | Testcase: tempest_smoke_serial | +======================================================================+ | Description: | | This test case runs the smoke subset of the OpenStack Tempest | | suite. The list of test cases is generated by Tempest | | automatically and depends on the parameters of the OpenStack | | deplopyment. | | Dependencies: | | - Installer: | | - Scenario : | | | +----------------------------------------------------------------------+ To execute a Test Tier or Test Case, the 'run' command is used:: root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck' 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file... 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################ 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck' 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################ 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'... 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API... 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK! 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API... 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK! 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API... 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK! 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API... 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks... 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets... 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers... 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK! 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API... 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK! 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP... : : 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK! 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed! 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources... 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ Version 1 is deprecated, use alternative version 2 instead. WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead. 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances... 2016-06-30 11:48:24,439 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images... 2016-06-30 11:48:35,853 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes... 2016-06-30 11:48:37,344 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs... 2016-06-30 11:48:37,467 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects 2016-06-30 11:48:53,633 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups... 2016-06-30 11:48:53,689 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users... 2016-06-30 11:48:54,444 - clean_openstack - INFO - ------------------------------------------- 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants... 2016-06-30 11:48:54,711 - clean_openstack - INFO - ------------------------------------------- and root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh' 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file... 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'... 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'... 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net... 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'... 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'... 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48' 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'... 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE. 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'... 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'... 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE. 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'... 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'... 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140' 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2' 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140... 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping... 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected! 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s. 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources... 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ Version 1 is deprecated, use alternative version 2 instead. : : etc. To list the test cases which are part of a specific Test Tier, the 'get-tests' command is used with 'functest tier':: root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites Test cases in tier 'sdn_suites': ['odl'] Please note that for some scenarios some test cases might not be launched. For example, the last example displayed only the 'odl' testcase for the given environment. In this system the deployment does not support the 'onos' SDN Controller Test Case; for example. **Important** If you use the command 'functest tier run ', then the Functest CLI utility will call **all valid Test Cases**, which belong to the specified Test Tier, as relevant to scenarios deployed to the SUT environment. Thus, the Functest CLI utility calculates automatically which tests can be executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, which is passed in to the Functest docker container. Currently, the Functest CLI command 'functest testcase run ', supports two possibilities:: * Run a single Test Case, specified by a valid choice of * Run ALL test Test Cases (for all Tiers) by specifying = 'all' Example:: root@22e436918db0:~/repos/functest/ci# functest testcase run all Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all' 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file... 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed: - 0. healthcheck: ['healthcheck'] - 1. smoke: ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - 2. sdn_suites: ['odl'] - 3. features: ['doctor', 'security_scan'] - 4. openstack: ['tempest_full_parallel', 'rally_full'] - 5. vnf: ['vims'] 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################ 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck' 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################ 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'... 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API... 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK! 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API... : : etc. Functest includes cleaning mechanism in order to remove all the OpenStack resources except what was present before running any test. The script *$repos_dir/functest/utils/generate_defaults.py* is called once when setting up the Functest environment (i.e. CLI command 'functest env prepare') to snapshot all the OpenStack resources (images, networks, volumes, security groups, tenants, users) so that an eventual cleanup does not remove any of this defaults. The script **clean_openstack.py** which is located in *$repos_dir/functest/utils/* is normally called after a test execution. It is in charge of cleaning the OpenStack resources that are not specified in the defaults file generated previously which is stored in */home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container. It is important to mention that if there are new OpenStack resources created manually after preparing the Functest environment, they will be removed, unless you use the special method of invoking the test case with specific suppression of clean up. (See the `Troubleshooting`_ section). The reason to include this cleanup meachanism in Functest is because some test suites such as Tempest or Rally create a lot of resources (users, tenants, networks, volumes etc.) that are not always properly cleaned, so this function has been set to keep the system as clean as it was before a full Functest execution. Although the Functest CLI provides an easy way to run any test, it is possible to do a direct call to the desired test script. For example: python $repos_dir/functest/testcases/OpenStack/vPing/vPing_ssh.py -d Automated testing ----------------- **TODO** Jose, the next section has not yet been revised! As mentioned previously, the Functest Docker container preparation as well as invocation of Test Cases can be called within the container from the Jenkins CI system. There are 2 jobs that automate the whole process. The first job runs all the tests and the second job allows testing test suite by test suite specifying the test suite name. The user may also use either of these Jenkins jobs to execute the desired test suites. One of the most challenging task in the Brahmaputra release consists in dealing with lots of scenarios and installers. Thus, when the tests are automatically started from CI, a basic algorithm has been created in order to detect whether a given test is runnable or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite can not be run on an ONOS scenario). CI provides some useful information passed to the container as environment variables: * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with * controller = (odl|onos|ocl|nosdn) * feature = (ovs(dpdk)|kvm|sfc|bgpvpn) * mode = (ha|noha) The constraints per test case are defined in the Functest configuration file */home/opnfv/functest/config/config_functest.yaml*:: test-dependencies: functest: vims: scenario: '(ocl)|(odl)|(nosdn)' vping: vping_userdata: scenario: '(ocl)|(odl)|(nosdn)' tempest: rally: odl: scenario: 'odl' onos: scenario: 'onos' .... At the end of the Functest environment creation, a file */home/opnfv/functest/conf/testcase-list.txt* is created with the list of all the runnable tests. Functest considers the static constraints as regular expressions and compare them with the given scenario name. For instance, ODL suite can be run only on an scenario including 'odl' in its name. The order of execution is also described in the Functest configuration file:: test_exec_priority: 1: vping_ssh 2: vping_userdata 3: tempest 4: odl 5: onos 6: ovno 7: doctor 8: promise 9: odl-vpnservice 10: bgpvpn #11: openstack-neutron-bgpvpn-api-extension-tests 12: vims 13: rally The tests are executed in the following order: 1) vPing test cases 2) Tempest suite 3) SDN controller suites 4) Feature project tests cases (Promise, Doctor, BGPVPN...) 5) vIMS suite 6) Rally suite As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed. **END of TODO**