.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 Executing the functest suites ============================= Manual testing -------------- This section assumes the following: * The Functest Docker container is running * The docker prompt is shown * The Functest environment is ready (Functest CLI command 'functest env prepare' has been executed) If any of the above steps are missing please refer to the Functest Config Guide as they are a prerequisite and all the commands explained in this section **must** be performed **inside the container**. Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a new Functest CLI. One difference, is that tests run through the Functest CLI will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this document, where this difference is discussed. The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) for the execution of Test Tiers or Test Cases:: root@22e436918db0:~/repos/functest/ci# functest tier --help Usage: functest tier [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. Commands: get-tests Prints the tests in a tier. list Lists the available tiers. run Executes all the tests within a tier. show Shows information about a tier. root@22e436918db0:~/repos/functest/ci# functest testcase --help Usage: functest testcase [OPTIONS] COMMAND [ARGS]... Options: -h, --help Show this message and exit. Commands: list Lists the available testcases. run Executes a test case. show Shows information about a test case. More details on the existing Tiers and Test Cases can be seen with the 'list' command:: root@22e436918db0:~/repos/functest/ci# functest tier list - 0. healthcheck: ['healthcheck'] - 1. smoke: ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - 2. sdn_suites: ['odl'] - 3. features: ['doctor', 'security_scan'] - 4. openstack: ['tempest_full_parallel', 'rally_full'] - 5. vnf: ['vims'] and root@22e436918db0:~/repos/functest/ci# functest testcase list healthcheck vping_ssh vping_userdata tempest_smoke_serial rally_sanity odl doctor security_scan tempest_full_parallel rally_full vims More specific details on specific Tiers or Test Cases can be seen wih the 'show' command:: root@22e436918db0:~/repos/functest/ci# functest tier show smoke +======================================================================+ | Tier: smoke | +======================================================================+ | Order: 1 | | CI Loop: (daily)|(weekly) | | Description: | | Set of basic Functional tests to validate the OpenStack | | deployment. | | Test cases: | | - vping_ssh | | - vping_userdata | | - tempest_smoke_serial | | - rally_sanity | | | +----------------------------------------------------------------------+ and root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial +======================================================================+ | Testcase: tempest_smoke_serial | +======================================================================+ | Description: | | This test case runs the smoke subset of the OpenStack Tempest | | suite. The list of test cases is generated by Tempest | | automatically and depends on the parameters of the OpenStack | | deplopyment. | | Dependencies: | | - Installer: | | - Scenario : | | | +----------------------------------------------------------------------+ To execute a Test Tier or Test Case, the 'r
##############################################################################
# Copyright (c) 2017 Huawei Technologies Co.,Ltd and others.
#
# All rights reserved. This program and the accompanying materials
# are made available under the terms of the Apache License, Version 2.0
# which accompanies this distribution, and is available at
# http://www.apache.org/licenses/LICENSE-2.0
##############################################################################
---
# Fuel integration test task suite

schema: "yardstick:suite:0.1"

name: "fuel_test_suite"
test_cases_dir: "samples/"
test_cases:
-
  file_name: ping.yaml
-
  file_name: iperf3.yaml
le or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite can not be run on an ONOS scenario). Moreover since Colorado, we also introduce the notion of daily/weekly in order to save CI time and avoid running systematically long duration tests. CI provides some useful information passed to the container as environment variables: * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with * controller = (odl|onos|ocl|nosdn) * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites) * mode = (ha|noha) The constraints per test case are defined in the Functest configuration file */home/opnfv/repos/functest/functest/ci/testcases.yaml*:: tiers: - name: healthcheck order: 0 ci_loop: '(daily)|(weekly)' description : >- First tier to be executed to verify the basic operations in the VIM. testcases: - name: healthcheck criteria: 'status == "PASS"' blocking: true description: >- This test case verifies the basic OpenStack services like Keystone, Glance, Cinder, Neutron and Nova. dependencies: installer: '' scenario: '' - name: smoke order: 1 ci_loop: '(daily)|(weekly)' description : >- Set of basic Functional tests to validate the OpenStack deployment. testcases: - name: vping_ssh criteria: 'status == "PASS"' blocking: true description: >- This test case verifies: 1) SSH to an instance using floating IPs over the public network. 2) Connectivity between 2 instances over a private network. dependencies: installer: '' scenario: '^((?!bgpvpn|odl_l3).)*$' .... We may distinguish 2 levels in the test case description: * Tier level * Test case level At the tier level, we define the following parameters: * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs * description: a high level view of the test case For a given test case we defined: * the name of the test case * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL * blocking: if set to true, if the test is failed, the execution of the following tests is canceled * the description of the test case * the dependencies: a combination of 2 regex on the scenario and the installer name The order of execution is the one defined in the file if all test cases are selected. In CI daily job the tests are executed in the following order: 1) healthcheck (blocking) 2) smoke: both vPings are blocking 3) SDN controller suites (blocking) 4) Feature project tests cases In CI weekly job we add 2 tiers: 5) vIMS suite 6) Rally suite As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed. Please note that a system snapshot is taken before any test case execution. This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.