From 131ac18e57e0b0413e00be4a763e5f0dbdca047c Mon Sep 17 00:00:00 2001 From: Morgan Richomme Date: Mon, 23 Jan 2017 15:49:14 +0100 Subject: First update of Functest user guide for Danube JIRA: FUNCTEST-560 Change-Id: I90d7c3447dfb15b880704cb25da080221cb7b063 Signed-off-by: Morgan Richomme --- docs/userguide/index.rst | 92 +++++++++++++++++++++++++++---------- docs/userguide/introduction.rst | 58 ++++++++++++++++------- docs/userguide/runfunctest.rst | 94 +++++++++++++++----------------------- docs/userguide/troubleshooting.rst | 7 +-- 4 files changed, 148 insertions(+), 103 deletions(-) diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index d2467df71..bc02776aa 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -8,18 +8,28 @@ OPNFV FUNCTEST user guide .. toctree:: :maxdepth: 2 +Version history +=============== ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | | +| | | | | ++------------+----------+------------------+----------------------------------+ + Introduction ============ The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. In the OPNFV Colorado system release, +provide a procedure to execute them. In the OPNFV Danube system release, a Functest CLI utility is introduced for easier execution of test procedures. -An overview presentation has been created for the first OPNFV Summit `[4]`_. - -This document is a continuation of the OPNFV Functest Configuration Guide `[1]`_. - **IMPORTANT**: It is assumed here that the Functest Docker container is already properly deployed and that all instructions described in this guide are to be performed from *inside* the deployed Functest Docker container. @@ -34,7 +44,7 @@ VIM (Virtualized Infrastructure Manager) Healthcheck ^^^^^^^^^^^ In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' -is introduced. The healthcheck testcase verifies that some basic IP connectivity +was introduced. The healthcheck testcase verifies that some basic IP connectivity and essential operations of OpenStack functionality over the command line are working correctly. @@ -174,7 +184,7 @@ The Tempest testcases are distributed accross two Tiers: * Smoke Tier - Test Case 'tempest_smoke_serial' - * Openstack Tier - Test case 'tempest_full_parallel' + * Components Tier - Test case 'tempest_full_parallel' NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' @@ -214,11 +224,44 @@ A basic SLA (stop test on errors) has been implemented. The Rally testcases are distributed accross two Tiers: * Smoke Tier - Test Case 'rally_sanity' - * Openstack Tier - Test case 'rally_full' + * Components Tier - Test case 'rally_full' NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. +SNAPS +----- + +SNAPS stands for "SNA/NFV Application development Platform and Stack". +This project seeks to develop baseline OpenStack NFV installations. It has been +developed by Steven Pisarski and provided an object oriented library to perform +functional and performance tests. It has been declined in several test suites in +Functest. + +connection check +^^^^^^^^^^^^^^^^ +Connection_check consists in 9 test cases (test duration < 5s) checking the +connectivity with Glance, Keystone, Neutron, Nova and the external network. + +api_check +^^^^^^^^^ +This test case verifies the retrieval of OpenStack clients: Keystone, Glance, +Neutron and Nova and may perform some simple queries. When the config value of +snaps.use_keystone is True, functest must have access to the cloud's private +network. +This suite consists in 49 tests (test duration< 2 minutes) + +Snaps_smoke +^^^^^^^^^^^ +This test case contains tests that setup and destroy environments with VMs with +and without Floating IPs with a newly created user and project. Set the config +value snaps.use_floating_ips (True|False) to toggle this functionality. When +the config value of snaps.use_keystone is True, functest must have access +the cloud's private network. +This suite consists in 38 tests (test duration < 10 minutes) + +More information on SNAPS can be found in  `[13]`_ + SDN Controllers --------------- @@ -323,21 +366,19 @@ The test cases are described as follows: Features -------- -Most of the features have been developped by feature projects. -Security_scan has been initiated in Functest repository but should soon -be declared in its own repository as well. - Please refer to the dedicated feature user guides for details: - * bgpvpn: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html - * copper: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html - * doctor: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html + * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html + * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html + * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html * moon: http://artifacts.opnfv.org/moon/docs/userguide/index.html * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html - * onos-sfc: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html - * odl-sfc: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html - * promise: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html + * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html + * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html + * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html + * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html + * TODO security_scan ^^^^^^^^^^^^^ @@ -345,11 +386,12 @@ security_scan Security Scanning, is a project to insure security compliance and vulnerability checks, as part of an automated CI / CD platform delivery process. -The project makes use of the existing SCAP format[6] to perform deep scanning of -NFVi nodes, to insure they are hardened and free of known CVE reported vulnerabilities. +The project makes use of the existing SCAP format `[6]`_ to perform deep +scanning of NFVI nodes, to insure they are hardened and free of known CVE +reported vulnerabilities. The SCAP content itself, is then consumed and run using an upstream opensource tool -known as OpenSCAP[7]. +known as OpenSCAP `[7]`_. The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins build platform, to perform a complete scan. Resulting reports are then copied to the @@ -369,8 +411,7 @@ The current work flow is as follows: * If the config file value 'clean' is set to 'True' then the application installed in step 5 is removed, and all reports created at step 6 are deleted. -At present, only the Apex installer is supported, with support for other installers due -within D-release. +Security scan is supported by Apex, TODO.... @@ -385,8 +426,8 @@ architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the -essential functions needed for a NFV platform. +functional VNF allows the test of most of the essential functions needed for a +NFV platform. The goal of this test suite consists of: @@ -491,6 +532,7 @@ References .. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml .. _`[11]`: http://robotframework.org/ .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[13]`: TODO URL doc SNAPS OPNFV main site: opnfvmain_. diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst index 76aadc842..e5a090ed5 100644 --- a/docs/userguide/introduction.rst +++ b/docs/userguide/introduction.rst @@ -9,14 +9,26 @@ In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 4 main domains: VIM +The current list of test suites can be distributed over 5 main domains: VIM (Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features and VNF (Virtual Network Functions). +Features, VNF (Virtual Network Functions) and MANO stacks. + +Functest test suites are also distributed in the OPNFV testing categories: +healthcheck, smoke, features, components, performance, VNF, Stress tests. + +All the Healthcheck and smoke tests of a given scenario must be succesful to +validate the scenario for the release. +-------------+---------------+----------------+----------------------------------+ | Domain | Tier | Test case | Comments | +=============+===============+================+==================================+ | VIM | healthcheck | healthcheck | Verify basic operation in VIM | +| | +----------------+----------------------------------+ +| | | connection | Check OpenStack connectivity | +| | | _check | through SNAPS framework | +| | +----------------+----------------------------------+ +| | | api_check | Check OpenStack API through | +| | | | SNAPS framework | | +---------------+----------------+----------------------------------+ | | smoke | vPing_SSH | NFV "Hello World" using an SSH | | | | | connection to a destination VM | @@ -45,8 +57,11 @@ Features and VNF (Virtual Network Functions). | | +----------------+----------------------------------+ | | | rally_sanity | Run a subset of the OpenStack | | | | | Rally Test Suite in smoke mode | +| | +----------------+----------------------------------+ +| | | snaps_smoke | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | | +---------------+----------------+----------------------------------+ -| | openstack | tempest_full | Generate and run a full set of | +| | components | tempest_full | Generate and run a full set of | | | | \_parallel | the OpenStack Tempest Test Suite.| | | | | See the OpenStack reference test | | | | | suite `[2]`_. The generated | @@ -57,7 +72,7 @@ Features and VNF (Virtual Network Functions). | | | | benchmarking OpenStack modules | | | | | See the Rally documents `[3]`_. | +-------------+---------------+----------------+----------------------------------+ -| Controllers | sdn_suites | odl | Opendaylight Test suite | +| Controllers | smoke | odl | Opendaylight Test suite | | | | | Limited test suite to check the | | | | | basic neutron (Layer 2) | | | | | operations mainly based on | @@ -69,7 +84,7 @@ Features and VNF (Virtual Network Functions). | | | | See `ONOSFW User Guide`_ for | | | | | details. | +-------------+---------------+----------------+----------------------------------+ -| Features | features | Promise | Resource reservation and | +| Features | features | promise | Resource reservation and | | | | | management project to identify | | | | | NFV related requirements and | | | | | realize resource reservation for | @@ -80,7 +95,7 @@ Features and VNF (Virtual Network Functions). | | | | See `Promise User Guide`_ for | | | | | details. | | | +----------------+----------------------------------+ -| | | Doctor | Doctor platform, as of Colorado | +| | | doctor | Doctor platform, as of Colorado | | | | | release, provides the three | | | | | features: | | | | | * Immediate Notification | @@ -119,7 +134,7 @@ Features and VNF (Virtual Network Functions). | | | | See `Domino User Guide`_ for | | | | | details | | | +----------------+----------------------------------+ -| | | Copper | Copper develops OPNFV platform | +| | | copper | Copper develops OPNFV platform | | | | | support for policy management, | | | | | using open source projects such | | | | | as OpenStack Congress, focused | @@ -139,13 +154,17 @@ Features and VNF (Virtual Network Functions). | | | | See `Moon User Guide`_ for | | | | | details | +-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | vims | Example of a real VNF deployment | +| VNF | vnf | cloudify_ims | Example of a real VNF deployment | | | | | to show the NFV capabilities of | | | | | the platform. The IP Multimedia | | | | | Subsytem is a typical Telco test | | | | | case, referenced by ETSI. | | | | | It provides a fully functional | | | | | VoIP System | +| | +----------------+----------------------------------+ +| | | opera_ims | vIMS deployment using openBaton | +| | +----------------+----------------------------------+ +| | | orchestra_ims | vIMS deployment using open-O | + +---------------+----------------+----------------------------------+ | | | parser | Parser is an integration project | | | | | which aims to provide | @@ -166,6 +185,8 @@ Test cases also have an implicit execution order. For example, if the early 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' Tier testcases, there is little point to launch a full testcase execution round. +In Danube, we merged smoke and sdn controller tiers in smoke tier. + An overview of the Functest Structural Concept is depicted graphically below: .. figure:: ../images/concepts_mapping_final.png @@ -186,19 +207,24 @@ NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot of a live Functest Dashboard is shown below: -.. figure:: ../images/FunctestDashboardColorado.png +** TODO ** +.. figure:: ../images/FunctestDashboardDanube.png :align: center :alt: Functest Dashboard -There is no real notion of Test domain or Test coverage. Basic components -(VIM, SDN controllers) are tested through their own suites. Feature projects -also provide their own test suites with different ways of running their tests. +Basic components (VIM, SDN controllers) are tested through their own suites. +Feature projects also provide their own test suites with different ways of +running their tests. + +The notion of domain has been introduced in the description of the test cases +stored in the Database. +This parameters as well as possible tags can be used for the Test case catalog. vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure. -Functest considers OPNFV as a black box. As of Colorado release the OPNFV +Functest considers OPNFV as a black box. As of Danube release the OPNFV offers a lot of potential combinations: * 3 controllers (OpenDaylight, ONOS, OpenContrail) @@ -210,9 +236,9 @@ deployed features. The system uses the environment variables (INSTALLER_IP and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment. -In the Colorado OPNFV System release a convenience Functest CLI utility is also -introduced to simplify setting up the Functest evironment, management of the -OpenStack environment (e.g. resource clean-up) and for executing tests. +A convenience Functest CLI utility is also available to simplify setting up the +Functest evironment, management of the OpenStack environment (e.g. resource +clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst index a6a2c1756..ef1017104 100644 --- a/docs/userguide/runfunctest.rst +++ b/docs/userguide/runfunctest.rst @@ -17,11 +17,6 @@ If any of the above steps are missing please refer to the Functest Config Guide as they are a prerequisite and all the commands explained in this section **must** be performed **inside the container**. -Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a -new Functest CLI. One difference, is that tests run through the Functest CLI -will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this -document, where this difference is discussed. - The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) for the execution of Test Tiers or Test Cases:: @@ -53,17 +48,15 @@ command:: root@22e436918db0:~/repos/functest/ci# functest tier list - 0. healthcheck: - ['healthcheck'] + ['healthcheck', 'connection_check', 'api_check',] - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: + ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] + - 2. features: ['doctor', 'security_scan'] - - 4. openstack: + - 3. components: ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] + - 4. vnf: + ['cloudify_ims'] and @@ -71,6 +64,9 @@ command:: healthcheck vping_ssh vping_userdata + connection_check + api_check + snaps_smoke tempest_smoke_serial rally_sanity odl @@ -78,7 +74,7 @@ command:: security_scan tempest_full_parallel rally_full - vims + cloudify_ims More specific details on specific Tiers or Test Cases can be seen wih the 'show' command:: @@ -208,9 +204,9 @@ To execute a Test Tier or Test Case, the 'run' command is used:: To list the test cases which are part of a specific Test Tier, the 'get-tests' command is used with 'functest tier':: - root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites - Test cases in tier 'sdn_suites': - ['odl'] + root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck + Test cases in tier 'healthcheck': + ['healthcheck'] Please note that for some scenarios some test cases might not be launched. @@ -231,37 +227,6 @@ two possibilities:: * Run a single Test Case, specified by a valid choice of * Run ALL test Test Cases (for all Tiers) by specifying = 'all' -Example:: - - root@22e436918db0:~/repos/functest/ci# functest testcase run all - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all' - 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed: - - 0. healthcheck: - ['healthcheck'] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: - ['doctor', 'security_scan'] - - 4. openstack: - ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] - 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################ - 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck' - 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################ - 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ - 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'... - 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================ - 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API... - 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK! - 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API... - : - : - etc. - Functest includes a cleaning mechanism in order to remove all the OpenStack resources except those present before running any test. The script *$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up @@ -304,19 +269,20 @@ once a week maximum) and the third job allows testing test suite by test suite s the test suite name. The user may also use either of these Jenkins jobs to execute the desired test suites. -One of the most challenging task in the Colorado release consists +One of the most challenging task in the Danube release consists in dealing with lots of scenarios and installers. Thus, when the tests are automatically started from CI, a basic algorithm has been created in order to detect whether a given test is runnable or not on the given scenario. Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). Moreover since Colorado, we also introduce the -notion of daily/weekly in order to save CI time and avoid running systematically -long duration tests. +be run on an ONOS scenario). The daily/weekly notion has been introduces in +Colorado in order to save CI time and avoid running systematically +long duration tests. It was not used in Colorado due to CI resource shortage. +The mechanism remains however as part of the CI evolution. CI provides some useful information passed to the container as environment variables: - * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE + * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with @@ -366,6 +332,9 @@ The constraints per test case are defined in the Functest configuration file dependencies: installer: '' scenario: '^((?!bgpvpn|odl_l3).)*$' + run: + module: 'functest.opnfv_tests.openstack.vping.vping_ssh' + class: 'VPingSSH' .... We may distinguish 2 levels in the test case description: @@ -383,6 +352,18 @@ For a given test case we defined: * blocking: if set to true, if the test is failed, the execution of the following tests is canceled * the description of the test case * the dependencies: a combination of 2 regex on the scenario and the installer name + * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests + +For further details on abstraction classes, see developper guide. + +Additional parameters have been added in the desription in the Database. +The target is to use the configuration stored in the Database and consider the +local file as backup if the Database is not reachable. +The additional fields related to a test case are: + * trust: we introduced this notion to put in place a mechanism of scenario promotion. + * Version: it indicates since which version you can run this test + * domains: the main domain covered by the test suite + * tags: a list of tags related to the test suite The order of execution is the one defined in the file if all test cases are selected. @@ -390,13 +371,12 @@ In CI daily job the tests are executed in the following order: 1) healthcheck (blocking) 2) smoke: both vPings are blocking - 3) SDN controller suites (blocking) - 4) Feature project tests cases + 3) Feature project tests cases In CI weekly job we add 2 tiers: - 5) vIMS suite - 6) Rally suite + 4) VNFs (vIMS) + 5) Components (Rally and Tempest long duration suites) As explained before, at the end of an automated execution, the OpenStack resources might be eventually removed. diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst index bf94935e6..1b7bf9b38 100644 --- a/docs/userguide/troubleshooting.rst +++ b/docs/userguide/troubleshooting.rst @@ -48,9 +48,6 @@ python scripts, located in paths: *$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* Notes: - #. In this Colorado Funtest Userguide, the use of the Functest CLI is - emphasized. The Functest CLI replaces the earlier Bash shell script - *run_tests.sh*. #. There is one difference, between the Functest CLI based test case execution compared to the earlier used Bash shell script, which is @@ -348,8 +345,8 @@ See OpenSCAP web site: https://www.open-scap.org/ NFV --- -vIMS -^^^^ +cloudify_ims +^^^^^^^^^^^^ vIMS deployment may fail for several reasons, the most frequent ones are described in the following table: -- cgit 1.2.3-korg