diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configguide/configguide.rst | 158 | ||||
| -rw-r--r-- | docs/configguide/index.rst | 78 | ||||
| -rw-r--r-- | docs/userguide/index.rst | 92 | ||||
| -rw-r--r-- | docs/userguide/introduction.rst | 58 | ||||
| -rw-r--r-- | docs/userguide/runfunctest.rst | 94 | ||||
| -rw-r--r-- | docs/userguide/troubleshooting.rst | 7 |
6 files changed, 296 insertions, 191 deletions
diff --git a/docs/configguide/configguide.rst b/docs/configguide/configguide.rst index ea5b01ea6..c03760c5c 100644 --- a/docs/configguide/configguide.rst +++ b/docs/configguide/configguide.rst @@ -30,9 +30,9 @@ following docker command:: [functester@jumphost ~]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.611 GB - opnfv/functest colorado.1.0 94b78faa94f7 4 weeks ago 874.9 MB - hello-world latest 94df4f0ce8a4 7 weeks ago 967 B + opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB + opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB + opnfv/functest colorado.1.0 94b78faa94f7 9 weeks ago 968 MB The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g. 'devstack'), with the @@ -104,10 +104,14 @@ recommended parameters for invoking docker container -e "INSTALLER_TYPE=<type>" # Use one of following apex, compass, fuel or joid - #. Functest needs to know the IP of the installer:: + #. Functest needs to know the IP of some installers:: -e "INSTALLER_IP=<Specific IP Address>" + This IP is needed to fetch RC file from deployment, fetch logs, ... + If not provided, there is no way to fetch the RC file. It must be + provided manually as a volume + #. Credentials for accessing the Openstack. Most convenient way of passing them to container is by having a local copy of the credentials file in Jumphost and then using the @@ -138,13 +142,24 @@ recommended parameters for invoking docker container If several features are pertinent then use the underscore character '_' to separate each feature (e.g. ovs_kvm) 'nofeature' indicates no NFV feature is deployed - ha_mode is one of ( ha | noha ) + ha_mode (high availability) is one of ( ha | noha ) **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are supported. The name passed in to the Functest Docker container must match the scenario used when the actual OPNFV platform was deployed. See release note to see the list of supported scenarios. + **NOTE:** The scenario name is mainly used to automatically detect + if a test suite is runnable or not (e.g. it will prevent ONOS test suite + to be run on ODL scenarios). If not set, Functest will try to run the + default test cases that might not include SDN controller or a specific + feature + + **NOTE:** A HA scenario means that 3 OpenStack controller nodes are + deployed. It does not necessarily mean that the whole system is HA. See + installer release notes for details. + + Putting all above together, when using installer 'fuel' and an invented INSTALLER_IP of '10.20.0.2', the recommended command to create the Functest Docker container is as follows:: @@ -311,14 +326,25 @@ should now be in place:: | `-- results `-- repos |-- bgpvpn + |-- copper |-- doctor + |-- domino |-- functest - |-- odl_integration + |-- kingbird + |-- moon + |-- odl_test |-- onos + |-- ovno + |-- parser |-- promise |-- rally |-- releng - `-- vims-test + |-- sdnvpn + |-- securityscanning + |-- sfc + |-- tempest + |-- vims_test + `-- vnfs Underneath the '/home/opnfv/' directory, the Functest docker container includes two main directories: @@ -339,24 +365,10 @@ follows:: . |-- INFO |-- LICENSE - |-- __init__.py - |-- ci - | |-- __init__.py - | |-- check_os.sh - | |-- config_functest.yaml - | |-- generate_report.py - | |-- exec_test.sh - | |-- prepare_env.py - | |-- run_tests.py - | |-- testcases.yaml - | |-- tier_builder.py - | `-- tier_handler.py - |-- cli - | |-- __init__.py - | |-- cli_base.py - | |-- commands - | |-- functest-complete.sh - | `-- setup.py + |-- requirements.txt + |-- run_unit_tests.sh + |-- setup.py + |-- test-requirements.txt |-- commons | |-- ims | |-- mobile @@ -364,50 +376,96 @@ follows:: |-- docker | |-- Dockerfile | |-- config_install_env.sh - | `-- requirements.pip + | `-- docker_remote_api |-- docs | |-- com | |-- configguide | |-- devguide | |-- images + | |-- internship | |-- release-notes | |-- results | `--userguide - |-- testcases - | |-- __init__.py - | |-- Controllers - | |-- OpenStack - | |-- features - | |-- security_scan - | `-- vnf - `-- utils + |-- functest |-- __init__.py - |-- functest_logger.py - |-- functest_utils.py - |-- openstack_clean.py - |-- openstack_snapshot.py - `-- openstack_utils.py + |-- ci + | |-- __init__.py + | |-- check_os.sh + | |-- config_functest.yaml + | |-- config_patch.yaml + | |-- exec_test.sh + | |-- generate_report.py + | |-- prepare_env.py + | |-- run_tests.py + | |-- testcases.yaml + | |-- tier_builder.py + | `-- tier_handler.py + |-- cli + | |-- __init__.py + | |-- cli_base.py + | |-- commands + | |-- functest-complete.sh + | `-- setup.py + |-- core + | |-- __init__.py + | |-- feature_base.py + | |-- pytest_suite_runner.py + | |-- testcase_base.py + | |-- vnf_base.py + |-- opnfv_tests + | |-- __init__.py + | |-- features + | |-- mano + | |-- openstack + | |-- sdn + | |-- security_scan + | `-- vnf + |-- tests + | |-- __init__.py + | `-- unit + `-- utils + |-- __init__.py + |-- config.py + |-- constants.py + |-- env.py + |-- functest_constants.py + |-- functest_logger.py + |-- functest_utils.py + |-- openstack + |-- openstack_clean.py + |-- openstack_snapshot.py + |-- openstack_tacker.py + `-- openstack_utils.py + (Note: All *.pyc files removed from above list for brevity...) -We may distinguish 7 different directories: +We may distinguish several directories, the first level has 4 directories: + +* **commons**: This directory is dedicated for storage of traffic + profile or any other test inputs that could be reused by any test + project. +* **docker**: This directory includes the needed files and tools to + build the Funtest Docker image. +* **docs**: This directory includes documentation: Release Notes, + User Guide, Configuration Guide and Developer Guide. Test results + are also located in a sub--directory called 'results'. +* **functest**: This directory contains all the code needed to run + functest internal cases and OPNFV onboarded feature or VNF test cases. +Functest directory has 6 directories: * **ci**: This directory contains test structure definition files (e.g <filename>.yaml) and bash shell/python scripts used to configure and execute Functional tests. The test execution script can be executed under the control of Jenkins CI jobs. * **cli**: This directory holds the python based Functest CLI utility source code, which is based on the Python 'click' framework. - * **commons**: This directory is dedicated for storage of traffic - profile or any other test inputs that could be reused by any test - project. - * **docker**: This directory includes the needed files and tools to - build the Funtest Docker container image. - * **docs**: This directory includes documentation: Release Notes, - User Guide, Configuration Guide and Developer Guide. Test results - are also located in a sub--directory called 'results'. - * **testcases**: This directory includes the scripts required by + * **core**: This directory holds the python based Functest core + source code. Three abstraction classes have been created to ease + the integration of internal, feature or vnf cases. + * **opnfv_tests**: This directory includes the scripts required by Functest internal test cases and other feature projects test cases. + * **tests**: This directory includes the functest unit tests * **utils**: this directory holds Python source code for some general purpose helper utilities, which testers can also re-use in their own test code. See for an example the Openstack helper utility: @@ -452,7 +510,7 @@ It is useful sometimes to remove a container if there are some problems:: Use the *-f* option if the container is still running, it will force to destroy it:: - docker -f rm <CONTAINER_ID> + docker rm -f <CONTAINER_ID> Check the Docker documentation dockerdocs_ for more information. diff --git a/docs/configguide/index.rst b/docs/configguide/index.rst index b61bf8ff5..58e8a1343 100644 --- a/docs/configguide/index.rst +++ b/docs/configguide/index.rst @@ -9,6 +9,21 @@ OPNFV FUNCTEST Configuration Guide :numbered: :maxdepth: 2 +Version history +=============== + ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-19 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | * update testcase list | +| | | | * update docker command | ++------------+----------+------------------+----------------------------------+ + Introduction ============ This document describes how to install and configure Functest in OPNFV. @@ -43,6 +58,7 @@ follows:: | | | -- healthcheck | | | | | | | | | -- vPing_ssh | | | | | | | | | -- vPing_userdata | | | | | | + | | | -- SNAPS_cases | | | | | | | | | -- Tempest_smoke | | | | | | | | | -- Rally_sanity | | | | | | | | | -- Tempest_full | | | | | | @@ -52,12 +68,13 @@ follows:: | | | -- odl | | | | | | | | | -- onos | | | | | | | | | | | | | | | + | | | - Features | | | | | | + | | | | | | | | | | | | - VNF | | | | | | - | | | -- vIMS | | | | | | | | | | | | | | | | | +--------------------+ | | | | | - | | +-------------------------+ | - | | Functest Docker | | | | | + | | Functest Docker +-------------------------+ | + | | | | | | | | | +---------------------------------+ | | | | | | | | +----------------------------+ | | | | @@ -107,7 +124,7 @@ The functional test cases are described in the Functest User Guide `[2]`_ Prerequisites ============= The OPNFV deployment is out of the scope of this document but it can be -found in http://artifacts.opnfv.org/opnfvdocs/colorado/docs/configguide/index.html. +found in http://docs.opnfv.org. The OPNFV platform is considered as the System Under Test (SUT) in this document. @@ -217,15 +234,14 @@ container from Jenkins. Docker creation in set-functest-env builder `[3]`_:: - envs="INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" + envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" [...] - docker pull opnfv/functest:latest_stable - cmd="docker run -id -e $envs ${labconfig} ${sshkey} ${res_volume} opnfv/functest:latest_stable /bin/bash" + docker pull opnfv/functest:$DOCKER_TAG >/dev/null + cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash" echo "Functest: Running docker run command: ${cmd}" - ${cmd} - docker ps -a + ${cmd} >${redirect} sleep 5 - container_id=$(docker ps | grep 'opnfv/functest:latest_stable' | awk '{print $1}' | head -1) + container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1) echo "Container ID=${container_id}" if [ -z ${container_id} ]; then echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing." @@ -235,43 +251,29 @@ Docker creation in set-functest-env builder `[3]`_:: echo "Starting the container: docker start ${container_id}" docker start ${container_id} sleep 5 - docker ps - if [ $(docker ps | grep 'opnfv/functest:latest_stable' | wc -l) == 0 ]; then + docker ps >${redirect} + if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..." exit 1 fi - cmd="${FUNCTEST_REPO_DIR}/docker/prepare_env.sh" + + cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start" echo "Executing command inside the docker: ${cmd}" docker exec ${container_id} ${cmd} Test execution in functest-all builder `[3]`_:: - echo "Functest: run $FUNCTEST_SUITE_NAME" - cmd="${FUNCTEST_REPO_DIR}/docker/run_tests.sh --test $FUNCTEST_SUITE_NAME ${flag}" - container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) - docker exec $container_id $cmd - -Docker clean in functest-cleanup builder `[3]`_:: - - echo "Cleaning up docker containers/images..." - # Remove previous running containers if exist - if [[ ! -z $(docker ps -a | grep opnfv/functest) ]]; then - echo "Removing existing opnfv/functest containers..." - docker ps | grep opnfv/functest | awk '{print $1}' | xargs docker stop - docker ps -a | grep opnfv/functest | awk '{print $1}' | xargs docker rm + branch=${GIT_BRANCH##*/} + echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}" + cmd="functest testcase run $FUNCTEST_SUITE_NAME" fi + container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) + docker exec $container_id $cmd + ret_value=$? + exit $ret_value - # Remove existing images if exist - if [[ ! -z $(docker images | grep opnfv/functest) ]]; then - echo "Docker images to remove:" - docker images | head -1 && docker images | grep opnfv/functest - image_tags=($(docker images | grep opnfv/functest | awk '{print $2}')) - for tag in "${image_tags[@]}"; do - echo "Removing docker image opnfv/functest:$tag..." - docker rmi opnfv/functest:$tag - done - fi +Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi References @@ -279,7 +281,7 @@ References .. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html .. _`[3]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-ci-jobs.yml -.. _`[4]`: http://artifacts.opnfv.org/opnfvdocs/colorado/docs/configguide/index.html +.. _`[4]`: http://artifacts.opnfv.org/functest/danube/docs/configguide/index.html OPNFV main site: opnfvmain_. @@ -289,4 +291,4 @@ OPNFV functional test page: opnfvfunctest_. IRC support channel: #opnfv-functest .. _opnfvmain: http://www.opnfv.org -.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing +.. _opnfvfunctest: https://wiki.opnfv.org/functest diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index d2467df71..bc02776aa 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -8,18 +8,28 @@ OPNFV FUNCTEST user guide .. toctree:: :maxdepth: 2 +Version history +=============== ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | | +| | | | | ++------------+----------+------------------+----------------------------------+ + Introduction ============ The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. In the OPNFV Colorado system release, +provide a procedure to execute them. In the OPNFV Danube system release, a Functest CLI utility is introduced for easier execution of test procedures. -An overview presentation has been created for the first OPNFV Summit `[4]`_. - -This document is a continuation of the OPNFV Functest Configuration Guide `[1]`_. - **IMPORTANT**: It is assumed here that the Functest Docker container is already properly deployed and that all instructions described in this guide are to be performed from *inside* the deployed Functest Docker container. @@ -34,7 +44,7 @@ VIM (Virtualized Infrastructure Manager) Healthcheck ^^^^^^^^^^^ In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' -is introduced. The healthcheck testcase verifies that some basic IP connectivity +was introduced. The healthcheck testcase verifies that some basic IP connectivity and essential operations of OpenStack functionality over the command line are working correctly. @@ -174,7 +184,7 @@ The Tempest testcases are distributed accross two Tiers: * Smoke Tier - Test Case 'tempest_smoke_serial' - * Openstack Tier - Test case 'tempest_full_parallel' + * Components Tier - Test case 'tempest_full_parallel' NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' @@ -214,11 +224,44 @@ A basic SLA (stop test on errors) has been implemented. The Rally testcases are distributed accross two Tiers: * Smoke Tier - Test Case 'rally_sanity' - * Openstack Tier - Test case 'rally_full' + * Components Tier - Test case 'rally_full' NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. +SNAPS +----- + +SNAPS stands for "SNA/NFV Application development Platform and Stack". +This project seeks to develop baseline OpenStack NFV installations. It has been +developed by Steven Pisarski and provided an object oriented library to perform +functional and performance tests. It has been declined in several test suites in +Functest. + +connection check +^^^^^^^^^^^^^^^^ +Connection_check consists in 9 test cases (test duration < 5s) checking the +connectivity with Glance, Keystone, Neutron, Nova and the external network. + +api_check +^^^^^^^^^ +This test case verifies the retrieval of OpenStack clients: Keystone, Glance, +Neutron and Nova and may perform some simple queries. When the config value of +snaps.use_keystone is True, functest must have access to the cloud's private +network. +This suite consists in 49 tests (test duration< 2 minutes) + +Snaps_smoke +^^^^^^^^^^^ +This test case contains tests that setup and destroy environments with VMs with +and without Floating IPs with a newly created user and project. Set the config +value snaps.use_floating_ips (True|False) to toggle this functionality. When +the config value of snaps.use_keystone is True, functest must have access +the cloud's private network. +This suite consists in 38 tests (test duration < 10 minutes) + +More information on SNAPS can be found in `[13]`_ + SDN Controllers --------------- @@ -323,21 +366,19 @@ The test cases are described as follows: Features -------- -Most of the features have been developped by feature projects. -Security_scan has been initiated in Functest repository but should soon -be declared in its own repository as well. - Please refer to the dedicated feature user guides for details: - * bgpvpn: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html - * copper: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html - * doctor: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html + * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html + * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html + * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html * moon: http://artifacts.opnfv.org/moon/docs/userguide/index.html * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html - * onos-sfc: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html - * odl-sfc: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html - * promise: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html + * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html + * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html + * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html + * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html + * TODO security_scan ^^^^^^^^^^^^^ @@ -345,11 +386,12 @@ security_scan Security Scanning, is a project to insure security compliance and vulnerability checks, as part of an automated CI / CD platform delivery process. -The project makes use of the existing SCAP format[6] to perform deep scanning of -NFVi nodes, to insure they are hardened and free of known CVE reported vulnerabilities. +The project makes use of the existing SCAP format `[6]`_ to perform deep +scanning of NFVI nodes, to insure they are hardened and free of known CVE +reported vulnerabilities. The SCAP content itself, is then consumed and run using an upstream opensource tool -known as OpenSCAP[7]. +known as OpenSCAP `[7]`_. The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins build platform, to perform a complete scan. Resulting reports are then copied to the @@ -369,8 +411,7 @@ The current work flow is as follows: * If the config file value 'clean' is set to 'True' then the application installed in step 5 is removed, and all reports created at step 6 are deleted. -At present, only the Apex installer is supported, with support for other installers due -within D-release. +Security scan is supported by Apex, TODO.... @@ -385,8 +426,8 @@ architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the -essential functions needed for a NFV platform. +functional VNF allows the test of most of the essential functions needed for a +NFV platform. The goal of this test suite consists of: @@ -491,6 +532,7 @@ References .. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml .. _`[11]`: http://robotframework.org/ .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[13]`: TODO URL doc SNAPS OPNFV main site: opnfvmain_. diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst index 76aadc842..e5a090ed5 100644 --- a/docs/userguide/introduction.rst +++ b/docs/userguide/introduction.rst @@ -9,14 +9,26 @@ In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 4 main domains: VIM +The current list of test suites can be distributed over 5 main domains: VIM (Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features and VNF (Virtual Network Functions). +Features, VNF (Virtual Network Functions) and MANO stacks. + +Functest test suites are also distributed in the OPNFV testing categories: +healthcheck, smoke, features, components, performance, VNF, Stress tests. + +All the Healthcheck and smoke tests of a given scenario must be succesful to +validate the scenario for the release. +-------------+---------------+----------------+----------------------------------+ | Domain | Tier | Test case | Comments | +=============+===============+================+==================================+ | VIM | healthcheck | healthcheck | Verify basic operation in VIM | +| | +----------------+----------------------------------+ +| | | connection | Check OpenStack connectivity | +| | | _check | through SNAPS framework | +| | +----------------+----------------------------------+ +| | | api_check | Check OpenStack API through | +| | | | SNAPS framework | | +---------------+----------------+----------------------------------+ | | smoke | vPing_SSH | NFV "Hello World" using an SSH | | | | | connection to a destination VM | @@ -45,8 +57,11 @@ Features and VNF (Virtual Network Functions). | | +----------------+----------------------------------+ | | | rally_sanity | Run a subset of the OpenStack | | | | | Rally Test Suite in smoke mode | +| | +----------------+----------------------------------+ +| | | snaps_smoke | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | | +---------------+----------------+----------------------------------+ -| | openstack | tempest_full | Generate and run a full set of | +| | components | tempest_full | Generate and run a full set of | | | | \_parallel | the OpenStack Tempest Test Suite.| | | | | See the OpenStack reference test | | | | | suite `[2]`_. The generated | @@ -57,7 +72,7 @@ Features and VNF (Virtual Network Functions). | | | | benchmarking OpenStack modules | | | | | See the Rally documents `[3]`_. | +-------------+---------------+----------------+----------------------------------+ -| Controllers | sdn_suites | odl | Opendaylight Test suite | +| Controllers | smoke | odl | Opendaylight Test suite | | | | | Limited test suite to check the | | | | | basic neutron (Layer 2) | | | | | operations mainly based on | @@ -69,7 +84,7 @@ Features and VNF (Virtual Network Functions). | | | | See `ONOSFW User Guide`_ for | | | | | details. | +-------------+---------------+----------------+----------------------------------+ -| Features | features | Promise | Resource reservation and | +| Features | features | promise | Resource reservation and | | | | | management project to identify | | | | | NFV related requirements and | | | | | realize resource reservation for | @@ -80,7 +95,7 @@ Features and VNF (Virtual Network Functions). | | | | See `Promise User Guide`_ for | | | | | details. | | | +----------------+----------------------------------+ -| | | Doctor | Doctor platform, as of Colorado | +| | | doctor | Doctor platform, as of Colorado | | | | | release, provides the three | | | | | features: | | | | | * Immediate Notification | @@ -119,7 +134,7 @@ Features and VNF (Virtual Network Functions). | | | | See `Domino User Guide`_ for | | | | | details | | | +----------------+----------------------------------+ -| | | Copper | Copper develops OPNFV platform | +| | | copper | Copper develops OPNFV platform | | | | | support for policy management, | | | | | using open source projects such | | | | | as OpenStack Congress, focused | @@ -139,13 +154,17 @@ Features and VNF (Virtual Network Functions). | | | | See `Moon User Guide`_ for | | | | | details | +-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | vims | Example of a real VNF deployment | +| VNF | vnf | cloudify_ims | Example of a real VNF deployment | | | | | to show the NFV capabilities of | | | | | the platform. The IP Multimedia | | | | | Subsytem is a typical Telco test | | | | | case, referenced by ETSI. | | | | | It provides a fully functional | | | | | VoIP System | +| | +----------------+----------------------------------+ +| | | opera_ims | vIMS deployment using openBaton | +| | +----------------+----------------------------------+ +| | | orchestra_ims | vIMS deployment using open-O | + +---------------+----------------+----------------------------------+ | | | parser | Parser is an integration project | | | | | which aims to provide | @@ -166,6 +185,8 @@ Test cases also have an implicit execution order. For example, if the early 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' Tier testcases, there is little point to launch a full testcase execution round. +In Danube, we merged smoke and sdn controller tiers in smoke tier. + An overview of the Functest Structural Concept is depicted graphically below: .. figure:: ../images/concepts_mapping_final.png @@ -186,19 +207,24 @@ NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot of a live Functest Dashboard is shown below: -.. figure:: ../images/FunctestDashboardColorado.png +** TODO ** +.. figure:: ../images/FunctestDashboardDanube.png :align: center :alt: Functest Dashboard -There is no real notion of Test domain or Test coverage. Basic components -(VIM, SDN controllers) are tested through their own suites. Feature projects -also provide their own test suites with different ways of running their tests. +Basic components (VIM, SDN controllers) are tested through their own suites. +Feature projects also provide their own test suites with different ways of +running their tests. + +The notion of domain has been introduced in the description of the test cases +stored in the Database. +This parameters as well as possible tags can be used for the Test case catalog. vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure. -Functest considers OPNFV as a black box. As of Colorado release the OPNFV +Functest considers OPNFV as a black box. As of Danube release the OPNFV offers a lot of potential combinations: * 3 controllers (OpenDaylight, ONOS, OpenContrail) @@ -210,9 +236,9 @@ deployed features. The system uses the environment variables (INSTALLER_IP and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment. -In the Colorado OPNFV System release a convenience Functest CLI utility is also -introduced to simplify setting up the Functest evironment, management of the -OpenStack environment (e.g. resource clean-up) and for executing tests. +A convenience Functest CLI utility is also available to simplify setting up the +Functest evironment, management of the OpenStack environment (e.g. resource +clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst index a6a2c1756..ef1017104 100644 --- a/docs/userguide/runfunctest.rst +++ b/docs/userguide/runfunctest.rst @@ -17,11 +17,6 @@ If any of the above steps are missing please refer to the Functest Config Guide as they are a prerequisite and all the commands explained in this section **must** be performed **inside the container**. -Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a -new Functest CLI. One difference, is that tests run through the Functest CLI -will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this -document, where this difference is discussed. - The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) for the execution of Test Tiers or Test Cases:: @@ -53,17 +48,15 @@ command:: root@22e436918db0:~/repos/functest/ci# functest tier list - 0. healthcheck: - ['healthcheck'] + ['healthcheck', 'connection_check', 'api_check',] - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: + ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] + - 2. features: ['doctor', 'security_scan'] - - 4. openstack: + - 3. components: ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] + - 4. vnf: + ['cloudify_ims'] and @@ -71,6 +64,9 @@ command:: healthcheck vping_ssh vping_userdata + connection_check + api_check + snaps_smoke tempest_smoke_serial rally_sanity odl @@ -78,7 +74,7 @@ command:: security_scan tempest_full_parallel rally_full - vims + cloudify_ims More specific details on specific Tiers or Test Cases can be seen wih the 'show' command:: @@ -208,9 +204,9 @@ To execute a Test Tier or Test Case, the 'run' command is used:: To list the test cases which are part of a specific Test Tier, the 'get-tests' command is used with 'functest tier':: - root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites - Test cases in tier 'sdn_suites': - ['odl'] + root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck + Test cases in tier 'healthcheck': + ['healthcheck'] Please note that for some scenarios some test cases might not be launched. @@ -231,37 +227,6 @@ two possibilities:: * Run a single Test Case, specified by a valid choice of <testcase_name> * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all' -Example:: - - root@22e436918db0:~/repos/functest/ci# functest testcase run all - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all' - 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed: - - 0. healthcheck: - ['healthcheck'] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: - ['doctor', 'security_scan'] - - 4. openstack: - ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] - 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################ - 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck' - 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################ - 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ - 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'... - 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ - 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API... - 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK! - 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API... - : - : - etc. - Functest includes a cleaning mechanism in order to remove all the OpenStack resources except those present before running any test. The script *$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up @@ -304,19 +269,20 @@ once a week maximum) and the third job allows testing test suite by test suite s the test suite name. The user may also use either of these Jenkins jobs to execute the desired test suites. -One of the most challenging task in the Colorado release consists +One of the most challenging task in the Danube release consists in dealing with lots of scenarios and installers. Thus, when the tests are automatically started from CI, a basic algorithm has been created in order to detect whether a given test is runnable or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). Moreover since Colorado, we also introduce the -notion of daily/weekly in order to save CI time and avoid running systematically -long duration tests. +be run on an ONOS scenario). The daily/weekly notion has been introduces in +Colorado in order to save CI time and avoid running systematically +long duration tests. It was not used in Colorado due to CI resource shortage. +The mechanism remains however as part of the CI evolution. CI provides some useful information passed to the container as environment variables: - * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE + * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with @@ -366,6 +332,9 @@ The constraints per test case are defined in the Functest configuration file dependencies: installer: '' scenario: '^((?!bgpvpn|odl_l3).)*$' + run: + module: 'functest.opnfv_tests.openstack.vping.vping_ssh' + class: 'VPingSSH' .... We may distinguish 2 levels in the test case description: @@ -383,6 +352,18 @@ For a given test case we defined: * blocking: if set to true, if the test is failed, the execution of the following tests is canceled * the description of the test case * the dependencies: a combination of 2 regex on the scenario and the installer name + * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests + +For further details on abstraction classes, see developper guide. + +Additional parameters have been added in the desription in the Database. +The target is to use the configuration stored in the Database and consider the +local file as backup if the Database is not reachable. +The additional fields related to a test case are: + * trust: we introduced this notion to put in place a mechanism of scenario promotion. + * Version: it indicates since which version you can run this test + * domains: the main domain covered by the test suite + * tags: a list of tags related to the test suite The order of execution is the one defined in the file if all test cases are selected. @@ -390,13 +371,12 @@ In CI daily job the tests are executed in the following order: 1) healthcheck (blocking) 2) smoke: both vPings are blocking - 3) SDN controller suites (blocking) - 4) Feature project tests cases + 3) Feature project tests cases In CI weekly job we add 2 tiers: - 5) vIMS suite - 6) Rally suite + 4) VNFs (vIMS) + 5) Components (Rally and Tempest long duration suites) As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed. diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst index bf94935e6..1b7bf9b38 100644 --- a/docs/userguide/troubleshooting.rst +++ b/docs/userguide/troubleshooting.rst @@ -48,9 +48,6 @@ python scripts, located in paths: *$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* Notes: - #. In this Colorado Funtest Userguide, the use of the Functest CLI is - emphasized. The Functest CLI replaces the earlier Bash shell script - *run_tests.sh*. #. There is one difference, between the Functest CLI based test case execution compared to the earlier used Bash shell script, which is @@ -348,8 +345,8 @@ See OpenSCAP web site: https://www.open-scap.org/ NFV --- -vIMS -^^^^ +cloudify_ims +^^^^^^^^^^^^ vIMS deployment may fail for several reasons, the most frequent ones are described in the following table: |