diff options
Diffstat (limited to 'docs/testing/user/userguide')
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 548 | ||||
| -rw-r--r-- | docs/testing/user/userguide/introduction.rst | 255 | ||||
| -rw-r--r-- | docs/testing/user/userguide/runfunctest.rst | 385 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 387 |
4 files changed, 1575 insertions, 0 deletions
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst new file mode 100644 index 00000000..9436de2b --- /dev/null +++ b/docs/testing/user/userguide/index.rst @@ -0,0 +1,548 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +========================= +OPNFV FUNCTEST user guide +========================= + +.. toctree:: + :maxdepth: 2 + +Version history +=============== ++------------+----------+------------------+----------------------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++------------+----------+------------------+----------------------------------+ +| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | +| | | Column Gaynor | | ++------------+----------+------------------+----------------------------------+ +| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | +| | | | | +| | | | | ++------------+----------+------------------+----------------------------------+ + + +Introduction +============ + +The goal of this document is to describe the OPNFV Functest test cases and to +provide a procedure to execute them. In the OPNFV Danube system release, +a Functest CLI utility is introduced for easier execution of test procedures. + +**IMPORTANT**: It is assumed here that the Functest Docker container is already +properly deployed and that all instructions described in this guide are to be +performed from *inside* the deployed Functest Docker container. + +.. include:: ./introduction.rst + +The different test cases are described in the remaining sections of this document. + +VIM (Virtualized Infrastructure Manager) +---------------------------------------- + +Healthcheck +^^^^^^^^^^^ +In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' +was introduced. The healthcheck testcase verifies that some basic IP connectivity +and essential operations of OpenStack functionality over the command line are +working correctly. + +In particular, the following verifications are performed: + + * DHCP agent functionality for IP address allocation + * Openstack Authentication management functionality via the Keystone API + * OpenStack Image management functionality via the Glance API + * OpenStack Block Storage management functionality via the Cinder API + * OpenStack Networking management functionality via the Neutron API + * Openstack Compute management functionality via the NOVA API + +Self-obviously, successful completion of the 'healthcheck' testcase is a +necessary pre-requisite for the execution of all other test Tiers. + + +vPing_ssh +^^^^^^^^^ + +Given the script **ping.sh**:: + + #!/bin/sh + while true; do + ping -c 1 $1 2>&1 >/dev/null + RES=$? + if [ "Z$RES" = "Z0" ] ; then + echo 'vPing OK' + break + else + echo 'vPing KO' + fi + sleep 1 + done + + +The goal of this test is to establish an SSH connection using a floating IP +on the Public/External network and verify that 2 instances can talk over a Private +Tenant network:: + + vPing_ssh test case + +-------------+ +-------------+ + | | | | + | | Boot VM1 with IP1 | | + | +------------------->| | + | Tester | | System | + | | Boot VM2 | Under | + | +------------------->| Test | + | | | | + | | Create floating IP | | + | +------------------->| | + | | | | + | | Assign floating IP | | + | | to VM2 | | + | +------------------->| | + | | | | + | | Establish SSH | | + | | connection to VM2 | | + | | through floating IP| | + | +------------------->| | + | | | | + | | SCP ping.sh to VM2 | | + | +------------------->| | + | | | | + | | VM2 executes | | + | | ping.sh to VM1 | | + | +------------------->| | + | | | | + | | If ping: | | + | | exit OK | | + | | else (timeout): | | + | | exit Failed | | + | | | | + +-------------+ +-------------+ + +This test can be considered as an "Hello World" example. +It is the first basic use case which **must** work on any deployment. + +vPing_userdata +^^^^^^^^^^^^^^ + +This test case is similar to vPing_ssh but without the use of Floating IPs +and the Public/External network to transfer the ping script. +Instead, it uses Nova metadata service to pass it to the instance at booting time. +As vPing_ssh, it checks that 2 instances can talk to +each other on a Private Tenant network:: + + vPing_userdata test case + +-------------+ +-------------+ + | | | | + | | Boot VM1 with IP1 | | + | +------------------->| | + | | | | + | | Boot VM2 with | | + | | ping.sh as userdata| | + | | with IP1 as $1. | | + | +------------------->| | + | Tester | | System | + | | VM2 exeutes ping.sh| Under | + | | (ping IP1) | Test | + | +------------------->| | + | | | | + | | Monitor nova | | + | | console-log VM 2 | | + | | If ping: | | + | | exit OK | | + | | else (timeout) | | + | | exit Failed | | + | | | | + +-------------+ +-------------+ + +When the second VM boots it will execute the script passed as userdata +automatically. The ping will be detected by periodically capturing the output +in the console-log of the second VM. + + +Tempest +^^^^^^^ + +Tempest `[2]`_ is the reference OpenStack Integration test suite. +It is a set of integration tests to be run against a live OpenStack cluster. +Tempest has suites of tests for: + + * OpenStack API validation + * Scenarios + * Other specific tests useful in validating an OpenStack deployment + +Functest uses Rally `[3]`_ to run the Tempest suite. +Rally generates automatically the Tempest configuration file **tempest.conf**. +Before running the actual test cases, +Functest creates the needed resources (user, tenant) and +updates the appropriate parameters into the configuration file. + +When the Tempest suite is executed, each test duration is measured and the full +console output is stored to a *log* file for further analysis. + +The Tempest testcases are distributed accross two +Tiers: + + * Smoke Tier - Test Case 'tempest_smoke_serial' + * Components Tier - Test case 'tempest_full_parallel' + +NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke +tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' +executes all defined Tempest tests using several concurrent threads +(i.e. parallel mode). The number of threads activated corresponds to the number +of available logical CPUs. + +The goal of the Tempest test suite is to check the basic functionalities of the +different OpenStack components on an OPNFV fresh installation, using the +corresponding REST API interfaces. + + +Rally bench test suites +^^^^^^^^^^^^^^^^^^^^^^^ + +Rally `[3]`_ is a benchmarking tool that answers the question: + +*How does OpenStack work at scale?* + +The goal of this test suite is to benchmark all the different OpenStack modules and +get significant figures that could help to define Telco Cloud KPIs. + +The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: + + * authenticate + * cinder + * glance + * heat + * keystone + * neutron + * nova + * quotas + * requests + +A basic SLA (stop test on errors) has been implemented. + +The Rally testcases are distributed accross two Tiers: + + * Smoke Tier - Test Case 'rally_sanity' + * Components Tier - Test case 'rally_full' + +NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test +cases. Test case 'rally_full' executes the full defined set of Rally tests. + +SNAPS +----- + +SNAPS stands for "SNA/NFV Application development Platform and Stack". +This project seeks to develop baseline OpenStack NFV installations. It has been +developed by Steven Pisarski and provided an object oriented library to perform +functional and performance tests. It has been declined in several test suites in +Functest, 2 are part of healthcheck tier, one belongs to smoke tier. + +connection check +^^^^^^^^^^^^^^^^ +Connection_check consists in 9 test cases (test duration < 5s) checking the +connectivity with Glance, Keystone, Neutron, Nova and the external network. + +api_check +^^^^^^^^^ +This test case verifies the retrieval of OpenStack clients: Keystone, Glance, +Neutron and Nova and may perform some simple queries. When the config value of +snaps.use_keystone is True, functest must have access to the cloud's private +network. +This suite consists in 49 tests (test duration< 2 minutes) + +snaps_smoke +^^^^^^^^^^^ +This test case contains tests that setup and destroy environments with VMs with +and without Floating IPs with a newly created user and project. Set the config +value snaps.use_floating_ips (True|False) to toggle this functionality. When +the config value of snaps.use_keystone is True, functest must have access +the cloud's private network. +This suite consists in 38 tests (test duration < 10 minutes) + +More information on SNAPS can be found in `[13]`_ + + +SDN Controllers +--------------- + +There are currently 2 available controllers: + + * OpenDaylight (ODL) + * ONOS + +OpenDaylight +^^^^^^^^^^^^ + +The OpenDaylight (ODL) test suite consists of a set of basic tests inherited +from the ODL project using the Robot `[11]`_ framework. +The suite verifies creation and deletion of networks, subnets and ports with +OpenDaylight and Neutron. + +The list of tests can be described as follows: + + * Basic Restconf test cases + * Connect to Restconf URL + * Check the HTTP code status + + * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) + + * Neutron Network test cases + * Check OpenStack networks + * Check OpenDaylight networks + * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Check that the network has also been successfully created in OpenDaylight + + * Neutron Subnet test cases + * Check OpenStack subnets + * Check OpenDaylight subnets + * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Check that the subnet has also been successfully created in OpenDaylight + + * Neutron Port test cases + * Check OpenStack Neutron for known ports + * Check OpenDaylight ports + * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Check that the new port has also been successfully created in OpenDaylight + + * Delete operations + * Delete the port previously created via OpenStack + * Check that the port has been also succesfully deleted in OpenDaylight + * Delete previously subnet created via OpenStack + * Check that the subnet has also been successfully deleted in OpenDaylight + * Delete the network created via OpenStack + * Check that the network has also been succesfully deleted in OpenDaylight + +Note: the checks in OpenDaylight are based on the returned HTTP status +code returned by OpenDaylight. + + +ONOS +^^^^ + +TestON Framework is used to test the ONOS SDN controller functions. +The test cases deal with L2 and L3 functions. +The ONOS test suite can be run on any ONOS compliant scenario. + +The test cases are described as follows: + + * onosfunctest: The main executable file contains the initialization of + the docker environment and functions called by FUNCvirNetNB and + FUNCvirNetNBL3 + + * FUNCvirNetNB + + * Create Network: Post Network data and check it in ONOS + * Update Network: Update the Network and compare it in ONOS + * Delete Network: Delete the Network and check if it's NULL in ONOS or + not + * Create Subnet: Post Subnet data and check it in ONOS + * Update Subnet: Update the Subnet and compare it in ONOS + * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not + * Create Port: Post Port data and check it in ONOS + * Update Port: Update the Port and compare it in ONOS + * Delete Port: Delete the Port and check if it's NULL in ONOS or not + + * FUNCvirNetNBL3 + + * Create Router: Post data for create Router and check it in ONOS + * Update Router: Update the Router and compare it in ONOS + * Delete Router: Delete the Router data and check it in ONOS + * Create RouterInterface: Post Router Interface data to an existing Router + and check it in ONOS + * Delete RouterInterface: Delete the RouterInterface and check the Router + * Create FloatingIp: Post data for create FloatingIp and check it in ONOS + * Update FloatingIp: Update the FloatingIp and compare it in ONOS + * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in + ONOS + * Create External Gateway: Post data to create an External Gateway for an + existing Router and check it in ONOS + * Update External Gateway: Update the External Gateway and compare the change + * Delete External Gateway: Delete the External Gateway and check that it is + 'NULL' in ONOS + + +Features +-------- + +Please refer to the dedicated feature user guides for details: + + * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html + * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html + * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html + * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html + * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html + * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html + * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html + * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html + * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html + * TODO + +security_scan +^^^^^^^^^^^^^ + +Security Scanning, is a project to insure security compliance and vulnerability +checks, as part of an automated CI / CD platform delivery process. + +The project makes use of the existing SCAP format `[6]`_ to perform deep +scanning of NFVI nodes, to insure they are hardened and free of known CVE +reported vulnerabilities. + +The SCAP content itself, is then consumed and run using an upstream opensource tool +known as OpenSCAP `[7]`_. + +The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins +build platform, to perform a complete scan. Resulting reports are then copied to the +OPNFV functest dashboard. + +The current work flow is as follows: + + * Jenkins Build Initiated + * security_scan.py script is called, and a config file is passed to the script as + an argument. + * The IP addresses of each NFVi node (compute / control) are gathered + * A scan profile is matched to the node type. + * The OpenSCAP application is remotely installed onto each target node gathered + on step 3, using upstream packaging (rpm and .deb). + * A scan is made against each node gathered within step 3. + * HTML Reports are downloaded for rendering on a dashboard. + * If the config file value 'clean' is set to 'True' then the application installed in + step 5 is removed, and all reports created at step 6 are deleted. + +Security scan is supported by Apex, TODO.... + + + +VNF +--- + + +vIMS +^^^^ +The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an +architectural framework for delivering IP multimedia services. + +vIMS has been integrated in Functest to demonstrate the capability to deploy a +relatively complex NFV scenario on the OPNFV platform. The deployment of a complete +functional VNF allows the test of most of the essential functions needed for a +NFV platform. + +The goal of this test suite consists of: + + * deploy a VNF orchestrator (Cloudify) + * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this + orchestrator based on a TOSCA blueprint defined in `[5]`_ + * run suite of signaling tests on top of this VNF + +The Clearwater architecture is described as follows: + +.. figure:: ../images/clearwater-architecture.png + :align: center + :alt: vIMS architecture + + +parser +^^^^^^ + +See parser user guide for details: `[12]`_ + + +.. include:: ./runfunctest.rst + + +Test results +============ + +Manual testing +-------------- + +In manual mode test results are displayed in the console and result files +are put in /home/opnfv/functest/results. + +Automated testing +-------------- + +In automated mode, test results are displayed in jenkins logs, a summary is provided +at the end of the job and can be described as follow:: + + +==================================================================================================================================================+ + | FUNCTEST REPORT | + +==================================================================================================================================================+ + | | + | Deployment description: | + | INSTALLER: fuel | + | SCENARIO: os-odl_l2-nofeature-ha | + | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 | + | CI LOOP: daily | + | | + +=========================+===============+============+===============+===========================================================================+ + | TEST CASE | TIER | DURATION | RESULT | URL | + +=========================+===============+============+===============+===========================================================================+ + | healthcheck | healthcheck | 03:07 | PASS | | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf | + +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ + +Results are automatically pushed to the test results database, some additional +result files are pushed to OPNFV artifact web sites. + +Based on the results stored in the result database, a `Functest reporting`_ +portal is also automatically updated. This portal provides information on: + + * The overall status per scenario and per installer + * Tempest: Tempest test case including reported errors per scenario and installer + * vIMS: vIMS details per scenario and installer + +.. figure:: ../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page + + +Test Dashboard +============== + +Based on results collected in CI, a test dashboard is dynamically generated. + + +.. include:: ./troubleshooting.rst + + +References +========== + +.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# +.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html +.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf +.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml +.. _`[6]`: https://scap.nist.gov/ +.. _`[7]`: https://github.com/OpenSCAP/openscap +.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml +.. _`[11]`: http://robotframework.org/ +.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[13]`: TODO URL doc SNAPS + +OPNFV main site: opnfvmain_. + +OPNFV functional test page: opnfvfunctest_. + +IRC support chan: #opnfv-testperf + +.. _opnfvmain: http://www.opnfv.org +.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing +.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html +.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html +.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py +.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html diff --git a/docs/testing/user/userguide/introduction.rst b/docs/testing/user/userguide/introduction.rst new file mode 100644 index 00000000..4dfe7937 --- /dev/null +++ b/docs/testing/user/userguide/introduction.rst @@ -0,0 +1,255 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Overview of the Functest suites +=============================== + +Functest is the OPNFV project primarily targeting function testing. +In the Continuous Integration pipeline, it is launched after an OPNFV fresh +installation to validate and verify the basic functions of the +infrastructure. + +The current list of test suites can be distributed over 5 main domains: VIM +(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), +Features, VNF (Virtual Network Functions) and MANO stacks. + +Functest test suites are also distributed in the OPNFV testing categories: +healthcheck, smoke, features, components, performance, VNF, Stress tests. + +All the Healthcheck and smoke tests of a given scenario must be succesful to +validate the scenario for the release. + ++-------------+---------------+----------------+----------------------------------+ +| Domain | Tier | Test case | Comments | ++=============+===============+================+==================================+ +| VIM | healthcheck | healthcheck | Verify basic operation in VIM | +| | +----------------+----------------------------------+ +| | | connection | Check OpenStack connectivity | +| | | _check | through SNAPS framework | +| | +----------------+----------------------------------+ +| | | api_check | Check OpenStack API through | +| | | | SNAPS framework | +| +---------------+----------------+----------------------------------+ +| | smoke | vPing_SSH | NFV "Hello World" using an SSH | +| | | | connection to a destination VM | +| | | | over a created floating IP | +| | | | address on the SUT Public / | +| | | | External network. Using the SSH | +| | | | connection a test script is then | +| | | | copied to the destination | +| | | | VM and then executed via SSH. | +| | | | The script will ping another | +| | | | VM on a specified IP address over| +| | | | the SUT Private Tenant network. | +| | +----------------+----------------------------------+ +| | | vPing_userdata | Uses Ping with given userdata | +| | | | to test intra-VM connectivity | +| | | | over the SUT Private Tenant | +| | | | network. The correct operation | +| | | | of the NOVA Metadata service is | +| | | | also verified in this test. | +| | +----------------+----------------------------------+ +| | | tempest_smoke | Generate and run a relevant | +| | | \_serial | Tempest Test Suite in smoke mode.| +| | | | The generated test set is | +| | | | dependent on the OpenStack | +| | | | deployment environment. | +| | +----------------+----------------------------------+ +| | | rally_sanity | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | +| | +----------------+----------------------------------+ +| | | snaps_smoke | Run a subset of the OpenStack | +| | | | Rally Test Suite in smoke mode | +| +---------------+----------------+----------------------------------+ +| | components | tempest_full | Generate and run a full set of | +| | | \_parallel | the OpenStack Tempest Test Suite.| +| | | | See the OpenStack reference test | +| | | | suite `[2]`_. The generated | +| | | | test set is dependent on the | +| | | | OpenStack deployment environment.| +| | +----------------+----------------------------------+ +| | | rally_full | Run the OpenStack testing tool | +| | | | benchmarking OpenStack modules | +| | | | See the Rally documents `[3]`_. | ++-------------+---------------+----------------+----------------------------------+ +| Controllers | smoke | odl | Opendaylight Test suite | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | +| | +----------------+----------------------------------+ +| | | onos | Test suite of ONOS L2 and L3 | +| | | | functions. | +| | | | See `ONOSFW User Guide`_ for | +| | | | details. | ++-------------+---------------+----------------+----------------------------------+ +| Features | features | promise | Resource reservation and | +| | | | management project to identify | +| | | | NFV related requirements and | +| | | | realize resource reservation for | +| | | | future usage by capacity | +| | | | management of resource pools | +| | | | regarding compute, network and | +| | | | storage. | +| | | | See `Promise User Guide`_ for | +| | | | details. | +| | +----------------+----------------------------------+ +| | | doctor | Doctor platform, as of Colorado | +| | | | release, provides the three | +| | | | features: | +| | | | * Immediate Notification | +| | | | * Consistent resource state | +| | | | awareness for compute host down | +| | | | * Valid compute host status | +| | | | given to VM owner | +| | | | See `Doctor User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | bgpvpn | Implementation of the OpenStack | +| | | | bgpvpn API from the SDNVPN | +| | | | feature project. It allows for | +| | | | the creation of BGP VPNs. | +| | | | See `SDNVPN User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | security_scan | Implementation of a simple | +| | | | security scan. (Currently | +| | | | available only for the Apex | +| | | | installer environment) | +| | +----------------+----------------------------------+ +| | | onos-sfc | SFC testing for onos scenarios | +| | | | See `ONOSFW User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | odl-sfc | SFC testing for odl scenarios | +| | | | See `SFC User Guide`_ for details| +| | +----------------+----------------------------------+ +| | | domino | Domino provides TOSCA template | +| | | | distribution service for network | +| | | | service and VNF descriptors | +| | | | among MANO components e.g., | +| | | | NFVO, VNFM, VIM, SDN-C, etc., | +| | | | as well as OSS/BSS functions. | +| | | | See `Domino User Guide`_ for | +| | | | details | +| | +----------------+----------------------------------+ +| | | copper | Copper develops OPNFV platform | +| | | | support for policy management, | +| | | | using open source projects such | +| | | | as OpenStack Congress, focused | +| | | | on helping ensure that virtual | +| | | | infrastructure and the apps that | +| | | | execute on it comply with the | +| | | | configuration policy intent of | +| | | | service providers, developers, | +| | | | and end users. See more detail | +| | | | in the `Copper User Guide`_. | +| | +----------------+----------------------------------+ +| | | multisites | Multisites | +| | | | See `Multisite User Guide`_ for | +| | | | details | ++-------------+---------------+----------------+----------------------------------+ +| VNF | vnf | cloudify_ims | Example of a real VNF deployment | +| | | | to show the NFV capabilities of | +| | | | the platform. The IP Multimedia | +| | | | Subsytem is a typical Telco test | +| | | | case, referenced by ETSI. | +| | | | It provides a fully functional | +| | | | VoIP System | +| | +----------------+----------------------------------+ +| | | opera_ims | vIMS deployment using openBaton | +| | +----------------+----------------------------------+ +| | | orchestra_ims | vIMS deployment using open-O | ++ +---------------+----------------+----------------------------------+ +| | | parser | Parser is an integration project | +| | | | which aims to provide | +| | | | placement/deployment templates | +| | | | translation for OPNFV platform, | +| | | | including TOSCA -> HOT, POLICY ->| +| | | | TOSCA and YANG -> TOSCA. | +| | | | See `Parser User Guide`_ for | +| | | | details | ++-------------+---------------+----------------+----------------------------------+ + + +As shown in the above table, Functest is structured into different 'domains', +'tiers' and 'test cases'. Each 'test case' usually represents an actual +'Test Suite' comprised -in turn- of several test cases internally. + +Test cases also have an implicit execution order. For example, if the early +'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' +Tier testcases, there is little point to launch a full testcase execution round. + +In Danube, we merged smoke and sdn controller tiers in smoke tier. + +An overview of the Functest Structural Concept is depicted graphically below: + +.. figure:: ../images/concepts_mapping_final.png + :align: center + :alt: Functest Concepts Structure + +Some of the test cases are developed by Functest team members, whereas others +are integrated from upstream communities or other OPNFV projects. For example, +`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the +OpenStack integration test suite and Functest is in charge of the selection, +integration and automation of those tests that fit suitably to OPNFV. + +The Tempest test suite is the default OpenStack smoke test suite but no new test +cases have been created in OPNFV Functest. + +The results produced by the tests run from CI are pushed and collected into a +NoSQL database. The goal is to populate the database with results from different +sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot +of a live Functest Dashboard is shown below: + +** TODO ** +.. figure:: ../images/FunctestDashboardDanube.png + :align: center + :alt: Functest Dashboard + + +Basic components (VIM, SDN controllers) are tested through their own suites. +Feature projects also provide their own test suites with different ways of +running their tests. + +The notion of domain has been introduced in the description of the test cases +stored in the Database. +This parameters as well as possible tags can be used for the Test case catalog. + +vIMS test case was integrated to demonstrate the capability to deploy a +relatively complex NFV scenario on top of the OPNFV infrastructure. + +Functest considers OPNFV as a black box. As of Danube release the OPNFV +offers a lot of potential combinations: + + * 3 controllers (OpenDaylight, ONOS, OpenContrail) + * 4 installers (Apex, Compass, Fuel, Joid) + +Most of the tests are runnable by any combination, but some tests might have +restrictions imposed by the utilized installers or due to the available +deployed features. The system uses the environment variables (INSTALLER_IP and +DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given +environment. + +A convenience Functest CLI utility is also available to simplify setting up the +Functest evironment, management of the OpenStack environment (e.g. resource +clean-up) and for executing tests. +The Functest CLI organised the testcase into logical Tiers, which contain in +turn one or more testcases. The CLI allows execution of a single specified +testcase, all test cases in a specified Tier, or the special case of execution +of **ALL** testcases. The Functest CLI is introduced in more detail in the +section `Executing the functest suites`_ of this document. + +.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html +.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html +.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html +.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html +.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html +.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html +.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html +.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ +.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html +.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst new file mode 100644 index 00000000..e7ab84b2 --- /dev/null +++ b/docs/testing/user/userguide/runfunctest.rst @@ -0,0 +1,385 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Executing the functest suites +============================= + +Manual testing +-------------- + +This section assumes the following: + * The Functest Docker container is running + * The docker prompt is shown + * The Functest environment is ready (Functest CLI command 'functest env prepare' + has been executed) + +If any of the above steps are missing please refer to the Functest Config Guide +as they are a prerequisite and all the commands explained in this section **must** be +performed **inside the container**. + +The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) +for the execution of Test Tiers or Test Cases:: + + root@22e436918db0:~/repos/functest/ci# functest tier --help + Usage: functest tier [OPTIONS] COMMAND [ARGS]... + + Options: + -h, --help Show this message and exit. + + Commands: + get-tests Prints the tests in a tier. + list Lists the available tiers. + run Executes all the tests within a tier. + show Shows information about a tier. + root@22e436918db0:~/repos/functest/ci# functest testcase --help + + Usage: functest testcase [OPTIONS] COMMAND [ARGS]... + + Options: + -h, --help Show this message and exit. + + Commands: + list Lists the available testcases. + run Executes a test case. + show Shows information about a test case. + +More details on the existing Tiers and Test Cases can be seen with the 'list' +command:: + + root@22e436918db0:~/repos/functest/ci# functest tier list + - 0. healthcheck: + ['healthcheck', 'connection_check', 'api_check',] + - 1. smoke: + ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] + - 2. features: + ['doctor', 'security_scan'] + - 3. components: + ['tempest_full_parallel', 'rally_full'] + - 4. vnf: + ['cloudify_ims'] + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase list + healthcheck + api_check + connection_check + vping_ssh + vping_userdata + snaps_smoke + tempest_smoke_serial + rally_sanity + odl + doctor + security_scan + tempest_full_parallel + rally_full + cloudify_ims + +More specific details on specific Tiers or Test Cases can be seen wih the +'show' command:: + + root@22e436918db0:~/repos/functest/ci# functest tier show smoke + +======================================================================+ + | Tier: smoke | + +======================================================================+ + | Order: 1 | + | CI Loop: (daily)|(weekly) | + | Description: | + | Set of basic Functional tests to validate the OpenStack | + | deployment. | + | Test cases: | + | - vping_ssh | + | - vping_userdata | + | - tempest_smoke_serial | + | - rally_sanity | + | | + +----------------------------------------------------------------------+ + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial + +======================================================================+ + | Testcase: tempest_smoke_serial | + +======================================================================+ + | Description: | + | This test case runs the smoke subset of the OpenStack Tempest | + | suite. The list of test cases is generated by Tempest | + | automatically and depends on the parameters of the OpenStack | + | deplopyment. | + | Dependencies: | + | - Installer: | + | - Scenario : | + | | + +----------------------------------------------------------------------+ + + +To execute a Test Tier or Test Case, the 'run' command is used:: + + root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck + Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck' + 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file... + 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################ + 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck' + 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################ + 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ + 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'... + 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ + 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API... + 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK! + 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API... + 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK! + 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API... + 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK! + 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API... + 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca + 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks... + 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets... + 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers... + 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK! + 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API... + 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK! + 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP... + : + : + 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK! + 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed! + 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources... + 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + Version 1 is deprecated, use alternative version 2 instead. + WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead. + 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances... + 2016-06-30 11:48:24,439 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images... + 2016-06-30 11:48:35,853 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes... + 2016-06-30 11:48:37,344 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs... + 2016-06-30 11:48:37,467 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects + 2016-06-30 11:48:53,633 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups... + 2016-06-30 11:48:53,689 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users... + 2016-06-30 11:48:54,444 - clean_openstack - INFO - ------------------------------------------- + 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants... + 2016-06-30 11:48:54,711 - clean_openstack - INFO - ------------------------------------------- + + and + + root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh + Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh' + 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file... + 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ + 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'... + 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ + 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'... + 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net... + 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'... + 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'... + 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48' + 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'... + 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE. + 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'... + 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'... + 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE. + 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'... + 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'... + 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140' + 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2' + 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140... + 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping... + 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected! + 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s. + 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK + 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources... + 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ + Version 1 is deprecated, use alternative version 2 instead. + : + : + etc. + +To list the test cases which are part of a specific Test Tier, the 'get-tests' +command is used with 'functest tier':: + + root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck + Test cases in tier 'healthcheck': + ['healthcheck'] + + +Please note that for some scenarios some test cases might not be launched. +For example, the last example displayed only the 'odl' testcase for the given +environment. In this particular system the deployment does not support the 'onos' SDN +Controller Test Case; for example. + +**Important** If you use the command 'functest tier run <tier_name>', then the +Functest CLI utility will call **all valid Test Cases**, which belong to the +specified Test Tier, as relevant to scenarios deployed to the SUT environment. +Thus, the Functest CLI utility calculates automatically which tests can be +executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, +which is passed in to the Functest docker container. + +Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports +two possibilities:: + + * Run a single Test Case, specified by a valid choice of <testcase_name> + * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all' + +Functest includes a cleaning mechanism in order to remove all the OpenStack +resources except those present before running any test. The script +*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up +the Functest environment (i.e. CLI command 'functest env prepare') to snapshot +all the OpenStack resources (images, networks, volumes, security groups, tenants, +users) so that an eventual cleanup does not remove any of these defaults. + +The script **clean_openstack.py** which is located in +*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is +in charge of cleaning the OpenStack resources that are not specified in the +defaults file generated previously which is stored in +*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container. + +It is important to mention that if there are new OpenStack resources created +manually after preparing the Functest environment, they will be removed, unless +you use the special method of invoking the test case with specific suppression +of clean up. (See the `Troubleshooting`_ section). + +The reason to include this cleanup meachanism in Functest is because some +test suites such as Tempest or Rally create a lot of resources (users, +tenants, networks, volumes etc.) that are not always properly cleaned, so this +function has been set to keep the system as clean as it was before a +full Functest execution. + +Although the Functest CLI provides an easy way to run any test, it is possible to +do a direct call to the desired test script. For example: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d + + +Automated testing +----------------- + +As mentioned previously, the Functest Docker container preparation as well as +invocation of Test Cases can be called within the container from the Jenkins CI +system. There are 3 jobs that automate the whole process. The first job runs all +the tests referenced in the daily loop (i.e. that must been run daily), the second +job runs the tests referenced in the weekly loop (usually long duration tests run +once a week maximum) and the third job allows testing test suite by test suite specifying +the test suite name. The user may also use either of these Jenkins jobs to execute +the desired test suites. + +One of the most challenging task in the Danube release consists +in dealing with lots of scenarios and installers. Thus, when the tests are +automatically started from CI, a basic algorithm has been created in order to +detect whether a given test is runnable or not on the given scenario. +Some Functest test suites cannot be systematically run (e.g. ODL suite can not +be run on an ONOS scenario). The daily/weekly notion has been introduces in +Colorado in order to save CI time and avoid running systematically +long duration tests. It was not used in Colorado due to CI resource shortage. +The mechanism remains however as part of the CI evolution. + +CI provides some useful information passed to the container as environment +variables: + + * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE + * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP + * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with + + * controller = (odl|onos|ocl|nosdn) + * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites) + * mode = (ha|noha) + +The constraints per test case are defined in the Functest configuration file +*/home/opnfv/repos/functest/functest/ci/testcases.yaml*:: + + tiers: + - + name: healthcheck + order: 0 + ci_loop: '(daily)|(weekly)' + description : >- + First tier to be executed to verify the basic + operations in the VIM. + testcases: + - + name: healthcheck + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies the basic OpenStack services like + Keystone, Glance, Cinder, Neutron and Nova. + + dependencies: + installer: '' + scenario: '' + + - + name: smoke + order: 1 + ci_loop: '(daily)|(weekly)' + description : >- + Set of basic Functional tests to validate the OpenStack deployment. + testcases: + - + name: vping_ssh + criteria: 'status == "PASS"' + blocking: true + description: >- + This test case verifies: 1) SSH to an instance using floating + IPs over the public network. 2) Connectivity between 2 instances + over a private network. + dependencies: + installer: '' + scenario: '^((?!bgpvpn|odl_l3).)*$' + run: + module: 'functest.opnfv_tests.openstack.vping.vping_ssh' + class: 'VPingSSH' + .... + +We may distinguish 2 levels in the test case description: + * Tier level + * Test case level + +At the tier level, we define the following parameters: + + * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs + * description: a high level view of the test case + +For a given test case we defined: + * the name of the test case + * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL + * blocking: if set to true, if the test is failed, the execution of the following tests is canceled + * the description of the test case + * the dependencies: a combination of 2 regex on the scenario and the installer name + * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests + +For further details on abstraction classes, see developper guide. + +Additional parameters have been added in the desription in the Database. +The target is to use the configuration stored in the Database and consider the +local file as backup if the Database is not reachable. +The additional fields related to a test case are: + * trust: we introduced this notion to put in place a mechanism of scenario promotion. + * Version: it indicates since which version you can run this test + * domains: the main domain covered by the test suite + * tags: a list of tags related to the test suite + +The order of execution is the one defined in the file if all test cases are selected. + +In CI daily job the tests are executed in the following order: + + 1) healthcheck (blocking) + 2) smoke: both vPings are blocking + 3) Feature project tests cases + +In CI weekly job we add 2 tiers: + + 4) VNFs (vIMS) + 5) Components (Rally and Tempest long duration suites) + +As explained before, at the end of an automated execution, the OpenStack resources +might be eventually removed. +Please note that a system snapshot is taken before any test case execution. + +This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst new file mode 100644 index 00000000..84550191 --- /dev/null +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -0,0 +1,387 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +Troubleshooting +=============== + +This section gives some guidelines about how to troubleshoot the test cases +owned by Functest. + +**IMPORTANT**: As in the previous section, the steps defined below must be +executed inside the Functest Docker container and after sourcing the OpenStack +credentials:: + + . $creds + +or:: + + source /home/opnfv/functest/conf/openstack.creds + +VIM +--- + +This section covers the test cases related to the VIM (healthcheck, vping_ssh, +vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, +rally_full). + +vPing common +^^^^^^^^^^^^ +For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are +similar: + + * Create Glance image + * Create Network + * Create Security Group + * Create Instances + +After these actions, the test cases differ and will be explained in their +respective section. + +These test cases can be run inside the container, using new Functest CLI as follows:: + + $ functest testcase run vping_ssh + $ functest testcase run vping_userdata + +The Functest CLI is designed to route a call to the corresponding internal +python scripts, located in paths: +*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_ssh.py* and +*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* + +Notes: + + #. There is one difference, between the Functest CLI based test case + execution compared to the earlier used Bash shell script, which is + relevant to point out in troubleshooting scenarios: + + The Functest CLI does **not yet** support the option to suppress + clean-up of the generated OpenStack resources, following the execution + of a test case. + + Explanation: After finishing the test execution, the corresponding + script will remove, by default, all created resources in OpenStack + (image, instances, network and security group). When troubleshooting, + it is advisable sometimes to keep those resources in case the test + fails and a manual testing is needed. + + It is actually still possible to invoke test execution, with suppression + of OpenStack resource cleanup, however this requires invocation of a + **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'. + The `OPNFV Functest Developer Guide`_ provides guidance on the use of that + Python script in such troubleshooting cases. + +Some of the common errors that can appear in this test case are:: + + vPing_ssh- ERROR - There has been a problem when creating the neutron network.... + +This means that there has been some problems with Neutron, even before creating the +instances. Try to create manually a Neutron network and a Subnet to see if that works. +The debug messages will also help to see when it failed (subnet and router creation). +Example of Neutron commands (using 10.6.0.0/24 range for example):: + + neutron net-create net-test + neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ + --gateway 10.6.0.254 net-test 10.6.0.0/24 + neutron router-create test_router + neutron router-interface-add <ROUTER_ID> test_subnet + neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> + +Another related error can occur while creating the Security Groups for the instances:: + + vPing_ssh- ERROR - Failed to create the security group... + +In this case, proceed to create it manually. These are some hints:: + + neutron security-group-create sg-test + neutron security-group-rule-create sg-test --direction ingress --protocol icmp \ + --remote-ip-prefix 0.0.0.0/0 + neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \ + --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 + neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \ + --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 + +The next step is to create the instances. The image used is located in +*/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img* and a Glance image is created +with the name **functest-vping**. If booting the instances fails (i.e. the status +is not **ACTIVE**), you can check why it failed by doing:: + + nova list + nova show <INSTANCE_ID> + +It might show some messages about the booting failure. To try that manually:: + + nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test + +This will spawn a VM using the network created previously manually. +In all the OPNFV tested scenarios from CI, it never has been a problem with the +previous actions. Further possible problems are explained in the following sections. + + +vPing_SSH +^^^^^^^^^ +This test case creates a floating IP on the external network and assigns it to +the second instance **opnfv-vping-2**. The purpose of this is to establish +a SSH connection to that instance and SCP a script that will ping the first +instance. This script is located in the repository under +*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/ping.sh* and takes an IP as +a parameter. When the SCP is completed, the test will do an SSH call to that script +inside the second instance. Some problems can happen here:: + + vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting + +If this is displayed, stop the test or wait for it to finish, if you have used +the special method of test invocation with specific supression of OpenStack +resource clean-up, as explained earler. It means that the Container can not +reach the Public/External IP assigned to the instance **opnfv-vping-2**. There +are many possible reasons, and they really depend on the chosen scenario. For +most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known +limitation. + +First, make sure that the instance **opnfv-vping-2** succeeded to get an IP +from the DHCP agent. It can be checked by doing:: + + nova console-log opnfv-vping-2 + +If the message *Sending discover* and *No lease, failing* is shown, it probably +means that the Neutron dhcp-agent failed to assign an IP or even that it was not +responding. At this point it does not make sense to try to ping the floating IP. + +If the instance got an IP properly, try to ping manually the VM from the container:: + + nova list + <grab the public IP> + ping <public IP> + +If the ping does not return anything, try to ping from the Host where the Docker +container is running. If that solves the problem, check the iptable rules because +there might be some rules rejecting ICMP or TCP traffic coming/going from/to the +container. + +At this point, if the ping does not work either, try to reproduce the test +manually with the steps described above in the vPing common section with the +addition:: + + neutron floatingip-create <EXT_NET_NAME> + nova floating-ip-associate nova-test <FLOATING_IP> + + +Further troubleshooting is out of scope of this document, as it might be due to +problems with the SDN controller. Contact the installer team members or send an +email to the corresponding OPNFV mailing list for more information. + + + +vPing_userdata +^^^^^^^^^^^^^^ +This test case does not create any floating IP neither establishes an SSH +connection. Instead, it uses nova-metadata service when creating an instance +to pass the same script as before (ping.sh) but as 1-line text. This script +will be executed automatically when the second instance **opnfv-vping-2** is booted. + +The only known problem here for this test to fail is mainly the lack of support +of cloud-init (nova-metadata service). Check the console of the instance:: + + nova console-log opnfv-vping-2 + +If this text or similar is shown:: + + checking http://169.254.169.254/2009-04-04/instance-id + failed 1/20: up 1.13. request failed + failed 2/20: up 13.18. request failed + failed 3/20: up 25.20. request failed + failed 4/20: up 37.23. request failed + failed 5/20: up 49.25. request failed + failed 6/20: up 61.27. request failed + failed 7/20: up 73.29. request failed + failed 8/20: up 85.32. request failed + failed 9/20: up 97.34. request failed + failed 10/20: up 109.36. request failed + failed 11/20: up 121.38. request failed + failed 12/20: up 133.40. request failed + failed 13/20: up 145.43. request failed + failed 14/20: up 157.45. request failed + failed 15/20: up 169.48. request failed + failed 16/20: up 181.50. request failed + failed 17/20: up 193.52. request failed + failed 18/20: up 205.54. request failed + failed 19/20: up 217.56. request failed + failed 20/20: up 229.58. request failed + failed to read iid from metadata. tried 20 + +it means that the instance failed to read from the metadata service. Contact +the Functest or installer teams for more information. + +NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests +have been excluded from CI in those scenarios. + + +Tempest +^^^^^^^ + +In the upstream OpenStack CI all the Tempest test cases are supposed to pass. +If some test cases fail in an OPNFV deployment, the reason is very probably one +of the following + ++-----------------------------+-----------------------------------------------------+ +| Error | Details | ++=============================+=====================================================+ +| Resources required for test | Such resources could be e.g. an external network | +| case execution are missing | and access to the management subnet (adminURL) from | +| | the Functest docker container. | ++-----------------------------+-----------------------------------------------------+ +| OpenStack components or | Check running services in the controller and compute| +| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| +| configured properly | Configuration parameters can be verified from the | +| | related .conf files located under '/etc/<component>'| +| | directories. | ++-----------------------------+-----------------------------------------------------+ +| Some resources required for | The tempest.conf file, automatically generated by | +| execution test cases are | Rally in Functest, does not contain all the needed | +| missing | parameters or some parameters are not set properly. | +| | The tempest.conf file is located in directory | +| | '/home/opnfv/.rally/tempest/for-deployment-<UUID>' | +| | in the Functest Docker container. Use the "rally | +| | deployment list" command in order to check the UUID | +| | the UUID of the current deployment. | ++-----------------------------+-----------------------------------------------------+ + + +When some Tempest test case fails, captured traceback and possibly also the +related REST API requests/responses are output to the console. More detailed debug +information can be found from tempest.log file stored into related Rally deployment +folder. + + +Rally +^^^^^ + +The same error causes which were mentioned above for Tempest test cases, may also +lead to errors in Rally as well. + +It is possible to run only one Rally scenario, instead of the whole suite. +To do that, call the alternative python script as follows:: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -h + usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name + + positional arguments: + test_name Module name to be tested. Possible values are : [ + authenticate | glance | cinder | heat | keystone | neutron | + nova | quotas | requests | vm | all ] The 'all' value + performs all possible test scenarios + + optional arguments: + -h, --help show this help message and exit + -d, --debug Debug mode + -r, --report Create json result file + -s, --smoke Smoke test mode + -v, --verbose Print verbose info about the progress + -n, --noclean Don't clean the created resources for this test. + +For example, to run the Glance scenario with debug information:: + + python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -d glance + +Possible scenarios are: + * authenticate + * glance + * cinder + * heat + * keystone + * neutron + * nova + * quotas + * requests + * vm + +To know more about what those scenarios are doing, they are defined in directory: +*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/scenario* +For more info about Rally scenario definition please refer to the Rally official +documentation. `[3]`_ + +If the flag *all* is specified, it will run all the scenarios one by one. Please +note that this might take some time (~1,5hr), taking around 1 hour alone to +complete the Nova scenario. + +To check any possible problems with Rally, the logs are stored under +*/home/opnfv/functest/results/rally/* in the Functest Docker container. + + +Controllers +----------- + +Opendaylight +^^^^^^^^^^^^ + +If the Basic Restconf test suite fails, check that the ODL controller is +reachable and its Restconf module has been installed. + +If the Neutron Reachability test fails, verify that the modules +implementing Neutron requirements have been properly installed. + +If any of the other test cases fails, check that Neutron and ODL have +been correctly configured to work together. Check Neutron configuration +files, accounts, IP addresses etc.). + + +ONOS +^^^^ + +Please refer to the ONOS documentation. `ONOSFW User Guide`_ . + + +Features +-------- + +Please refer to the dedicated feature user guides for details. + + +security_scan +^^^^^^^^^^^^^ + +See OpenSCAP web site: https://www.open-scap.org/ + + + +NFV +--- + +cloudify_ims +^^^^^^^^^^^^ +vIMS deployment may fail for several reasons, the most frequent ones are +described in the following table: + ++-----------------------------------+------------------------------------+ +| Error | Comments | ++===================================+====================================+ +| Keystone admin API not reachable | Impossible to create vIMS user and | +| | tenant | ++-----------------------------------+------------------------------------+ +| Impossible to retrieve admin role | Impossible to create vIMS user and | +| id | tenant | ++-----------------------------------+------------------------------------+ +| Error when uploading image from | impossible to deploy VNF | +| OpenStack to glance | | ++-----------------------------------+------------------------------------+ +| Cinder quota cannot be updated | Default quotas not sufficient, they| +| | are adapted in the script | ++-----------------------------------+------------------------------------+ +| Impossible to create a volume | VNF cannot be deployed | ++-----------------------------------+------------------------------------+ +| SSH connection issue between the | if vPing test fails, vIMS test will| +| Test Docker container and the VM | fail... | ++-----------------------------------+------------------------------------+ +| No Internet access from the VM | the VMs of the VNF must have an | +| | external access to Internet | ++-----------------------------------+------------------------------------+ +| No access to OpenStack API from | Orchestrator can be installed but | +| the VM | the vIMS VNF installation fails | ++-----------------------------------+------------------------------------+ + + +parser +^^^^^^ + +For now log info is the only way to do trouble shooting + + +.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# |