diff options
Diffstat (limited to 'docs/userguide')
| -rw-r--r-- | docs/userguide/index.rst | 548 | ||||
| -rw-r--r-- | docs/userguide/introduction.rst | 255 | ||||
| -rw-r--r-- | docs/userguide/runfunctest.rst | 385 | ||||
| -rw-r--r-- | docs/userguide/troubleshooting.rst | 387 |
4 files changed, 0 insertions, 1575 deletions
diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst deleted file mode 100644 index 9436de2b..00000000 --- a/docs/userguide/index.rst +++ /dev/null @@ -1,548 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -========================= -OPNFV FUNCTEST user guide -========================= - -.. toctree:: - :maxdepth: 2 - -Version history -=============== -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release | -| | | Column Gaynor | | -+------------+----------+------------------+----------------------------------+ -| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube | -| | | | | -| | | | | -+------------+----------+------------------+----------------------------------+ - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. In the OPNFV Danube system release, -a Functest CLI utility is introduced for easier execution of test procedures. - -**IMPORTANT**: It is assumed here that the Functest Docker container is already -properly deployed and that all instructions described in this guide are to be -performed from *inside* the deployed Functest Docker container. - -.. include:: ./introduction.rst - -The different test cases are described in the remaining sections of this document. - -VIM (Virtualized Infrastructure Manager) ----------------------------------------- - -Healthcheck -^^^^^^^^^^^ -In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck' -was introduced. The healthcheck testcase verifies that some basic IP connectivity -and essential operations of OpenStack functionality over the command line are -working correctly. - -In particular, the following verifications are performed: - - * DHCP agent functionality for IP address allocation - * Openstack Authentication management functionality via the Keystone API - * OpenStack Image management functionality via the Glance API - * OpenStack Block Storage management functionality via the Cinder API - * OpenStack Networking management functionality via the Neutron API - * Openstack Compute management functionality via the NOVA API - -Self-obviously, successful completion of the 'healthcheck' testcase is a -necessary pre-requisite for the execution of all other test Tiers. - - -vPing_ssh -^^^^^^^^^ - -Given the script **ping.sh**:: - - #!/bin/sh - while true; do - ping -c 1 $1 2>&1 >/dev/null - RES=$? - if [ "Z$RES" = "Z0" ] ; then - echo 'vPing OK' - break - else - echo 'vPing KO' - fi - sleep 1 - done - - -The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: - - vPing_ssh test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +------------------->| | - | Tester | | System | - | | Boot VM2 | Under | - | +------------------->| Test | - | | | | - | | Create floating IP | | - | +------------------->| | - | | | | - | | Assign floating IP | | - | | to VM2 | | - | +------------------->| | - | | | | - | | Establish SSH | | - | | connection to VM2 | | - | | through floating IP| | - | +------------------->| | - | | | | - | | SCP ping.sh to VM2 | | - | +------------------->| | - | | | | - | | VM2 executes | | - | | ping.sh to VM1 | | - | +------------------->| | - | | | | - | | If ping: | | - | | exit OK | | - | | else (timeout): | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -This test can be considered as an "Hello World" example. -It is the first basic use case which **must** work on any deployment. - -vPing_userdata -^^^^^^^^^^^^^^ - -This test case is similar to vPing_ssh but without the use of Floating IPs -and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. -As vPing_ssh, it checks that 2 instances can talk to -each other on a Private Tenant network:: - - vPing_userdata test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +------------------->| | - | | | | - | | Boot VM2 with | | - | | ping.sh as userdata| | - | | with IP1 as $1. | | - | +------------------->| | - | Tester | | System | - | | VM2 exeutes ping.sh| Under | - | | (ping IP1) | Test | - | +------------------->| | - | | | | - | | Monitor nova | | - | | console-log VM 2 | | - | | If ping: | | - | | exit OK | | - | | else (timeout) | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -When the second VM boots it will execute the script passed as userdata -automatically. The ping will be detected by periodically capturing the output -in the console-log of the second VM. - - -Tempest -^^^^^^^ - -Tempest `[2]`_ is the reference OpenStack Integration test suite. -It is a set of integration tests to be run against a live OpenStack cluster. -Tempest has suites of tests for: - - * OpenStack API validation - * Scenarios - * Other specific tests useful in validating an OpenStack deployment - -Functest uses Rally `[3]`_ to run the Tempest suite. -Rally generates automatically the Tempest configuration file **tempest.conf**. -Before running the actual test cases, -Functest creates the needed resources (user, tenant) and -updates the appropriate parameters into the configuration file. - -When the Tempest suite is executed, each test duration is measured and the full -console output is stored to a *log* file for further analysis. - -The Tempest testcases are distributed accross two -Tiers: - - * Smoke Tier - Test Case 'tempest_smoke_serial' - * Components Tier - Test case 'tempest_full_parallel' - -NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. - -The goal of the Tempest test suite is to check the basic functionalities of the -different OpenStack components on an OPNFV fresh installation, using the -corresponding REST API interfaces. - - -Rally bench test suites -^^^^^^^^^^^^^^^^^^^^^^^ - -Rally `[3]`_ is a benchmarking tool that answers the question: - -*How does OpenStack work at scale?* - -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. - -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: - - * authenticate - * cinder - * glance - * heat - * keystone - * neutron - * nova - * quotas - * requests - -A basic SLA (stop test on errors) has been implemented. - -The Rally testcases are distributed accross two Tiers: - - * Smoke Tier - Test Case 'rally_sanity' - * Components Tier - Test case 'rally_full' - -NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test -cases. Test case 'rally_full' executes the full defined set of Rally tests. - -SNAPS ------ - -SNAPS stands for "SNA/NFV Application development Platform and Stack". -This project seeks to develop baseline OpenStack NFV installations. It has been -developed by Steven Pisarski and provided an object oriented library to perform -functional and performance tests. It has been declined in several test suites in -Functest, 2 are part of healthcheck tier, one belongs to smoke tier. - -connection check -^^^^^^^^^^^^^^^^ -Connection_check consists in 9 test cases (test duration < 5s) checking the -connectivity with Glance, Keystone, Neutron, Nova and the external network. - -api_check -^^^^^^^^^ -This test case verifies the retrieval of OpenStack clients: Keystone, Glance, -Neutron and Nova and may perform some simple queries. When the config value of -snaps.use_keystone is True, functest must have access to the cloud's private -network. -This suite consists in 49 tests (test duration< 2 minutes) - -snaps_smoke -^^^^^^^^^^^ -This test case contains tests that setup and destroy environments with VMs with -and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. When -the config value of snaps.use_keystone is True, functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) - -More information on SNAPS can be found in `[13]`_ - - -SDN Controllers ---------------- - -There are currently 2 available controllers: - - * OpenDaylight (ODL) - * ONOS - -OpenDaylight -^^^^^^^^^^^^ - -The OpenDaylight (ODL) test suite consists of a set of basic tests inherited -from the ODL project using the Robot `[11]`_ framework. -The suite verifies creation and deletion of networks, subnets and ports with -OpenDaylight and Neutron. - -The list of tests can be described as follows: - - * Basic Restconf test cases - * Connect to Restconf URL - * Check the HTTP code status - - * Neutron Reachability test cases - * Get the complete list of neutron resources (networks, subnets, ports) - - * Neutron Network test cases - * Check OpenStack networks - * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron - * Check that the network has also been successfully created in OpenDaylight - - * Neutron Subnet test cases - * Check OpenStack subnets - * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron - * Check that the subnet has also been successfully created in OpenDaylight - - * Neutron Port test cases - * Check OpenStack Neutron for known ports - * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron - * Check that the new port has also been successfully created in OpenDaylight - - * Delete operations - * Delete the port previously created via OpenStack - * Check that the port has been also succesfully deleted in OpenDaylight - * Delete previously subnet created via OpenStack - * Check that the subnet has also been successfully deleted in OpenDaylight - * Delete the network created via OpenStack - * Check that the network has also been succesfully deleted in OpenDaylight - -Note: the checks in OpenDaylight are based on the returned HTTP status -code returned by OpenDaylight. - - -ONOS -^^^^ - -TestON Framework is used to test the ONOS SDN controller functions. -The test cases deal with L2 and L3 functions. -The ONOS test suite can be run on any ONOS compliant scenario. - -The test cases are described as follows: - - * onosfunctest: The main executable file contains the initialization of - the docker environment and functions called by FUNCvirNetNB and - FUNCvirNetNBL3 - - * FUNCvirNetNB - - * Create Network: Post Network data and check it in ONOS - * Update Network: Update the Network and compare it in ONOS - * Delete Network: Delete the Network and check if it's NULL in ONOS or - not - * Create Subnet: Post Subnet data and check it in ONOS - * Update Subnet: Update the Subnet and compare it in ONOS - * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not - * Create Port: Post Port data and check it in ONOS - * Update Port: Update the Port and compare it in ONOS - * Delete Port: Delete the Port and check if it's NULL in ONOS or not - - * FUNCvirNetNBL3 - - * Create Router: Post data for create Router and check it in ONOS - * Update Router: Update the Router and compare it in ONOS - * Delete Router: Delete the Router data and check it in ONOS - * Create RouterInterface: Post Router Interface data to an existing Router - and check it in ONOS - * Delete RouterInterface: Delete the RouterInterface and check the Router - * Create FloatingIp: Post data for create FloatingIp and check it in ONOS - * Update FloatingIp: Update the FloatingIp and compare it in ONOS - * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in - ONOS - * Create External Gateway: Post data to create an External Gateway for an - existing Router and check it in ONOS - * Update External Gateway: Update the External Gateway and compare the change - * Delete External Gateway: Delete the External Gateway and check that it is - 'NULL' in ONOS - - -Features --------- - -Please refer to the dedicated feature user guides for details: - - * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html - * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html - * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html - * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html - * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html - * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html - * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html - * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html - * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html - * TODO - -security_scan -^^^^^^^^^^^^^ - -Security Scanning, is a project to insure security compliance and vulnerability -checks, as part of an automated CI / CD platform delivery process. - -The project makes use of the existing SCAP format `[6]`_ to perform deep -scanning of NFVI nodes, to insure they are hardened and free of known CVE -reported vulnerabilities. - -The SCAP content itself, is then consumed and run using an upstream opensource tool -known as OpenSCAP `[7]`_. - -The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins -build platform, to perform a complete scan. Resulting reports are then copied to the -OPNFV functest dashboard. - -The current work flow is as follows: - - * Jenkins Build Initiated - * security_scan.py script is called, and a config file is passed to the script as - an argument. - * The IP addresses of each NFVi node (compute / control) are gathered - * A scan profile is matched to the node type. - * The OpenSCAP application is remotely installed onto each target node gathered - on step 3, using upstream packaging (rpm and .deb). - * A scan is made against each node gathered within step 3. - * HTML Reports are downloaded for rendering on a dashboard. - * If the config file value 'clean' is set to 'True' then the application installed in - step 5 is removed, and all reports created at step 6 are deleted. - -Security scan is supported by Apex, TODO.... - - - -VNF ---- - - -vIMS -^^^^ -The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an -architectural framework for delivering IP multimedia services. - -vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. - -The goal of this test suite consists of: - - * deploy a VNF orchestrator (Cloudify) - * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this - orchestrator based on a TOSCA blueprint defined in `[5]`_ - * run suite of signaling tests on top of this VNF - -The Clearwater architecture is described as follows: - -.. figure:: ../images/clearwater-architecture.png - :align: center - :alt: vIMS architecture - - -parser -^^^^^^ - -See parser user guide for details: `[12]`_ - - -.. include:: ./runfunctest.rst - - -Test results -============ - -Manual testing --------------- - -In manual mode test results are displayed in the console and result files -are put in /home/opnfv/functest/results. - -Automated testing --------------- - -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: - - +==================================================================================================================================================+ - | FUNCTEST REPORT | - +==================================================================================================================================================+ - | | - | Deployment description: | - | INSTALLER: fuel | - | SCENARIO: os-odl_l2-nofeature-ha | - | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 | - | CI LOOP: daily | - | | - +=========================+===============+============+===============+===========================================================================+ - | TEST CASE | TIER | DURATION | RESULT | URL | - +=========================+===============+============+===============+===========================================================================+ - | healthcheck | healthcheck | 03:07 | PASS | | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf | - +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+ - -Results are automatically pushed to the test results database, some additional -result files are pushed to OPNFV artifact web sites. - -Based on the results stored in the result database, a `Functest reporting`_ -portal is also automatically updated. This portal provides information on: - - * The overall status per scenario and per installer - * Tempest: Tempest test case including reported errors per scenario and installer - * vIMS: vIMS details per scenario and installer - -.. figure:: ../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - - -Test Dashboard -============== - -Based on results collected in CI, a test dashboard is dynamically generated. - - -.. include:: ./troubleshooting.rst - - -References -========== - -.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf -.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[6]`: https://scap.nist.gov/ -.. _`[7]`: https://github.com/OpenSCAP/openscap -.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`[13]`: TODO URL doc SNAPS - -OPNFV main site: opnfvmain_. - -OPNFV functional test page: opnfvfunctest_. - -IRC support chan: #opnfv-testperf - -.. _opnfvmain: http://www.opnfv.org -.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing -.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html -.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py -.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst deleted file mode 100644 index 4dfe7937..00000000 --- a/docs/userguide/introduction.rst +++ /dev/null @@ -1,255 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Overview of the Functest suites -=============================== - -Functest is the OPNFV project primarily targeting function testing. -In the Continuous Integration pipeline, it is launched after an OPNFV fresh -installation to validate and verify the basic functions of the -infrastructure. - -The current list of test suites can be distributed over 5 main domains: VIM -(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features, VNF (Virtual Network Functions) and MANO stacks. - -Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. - -All the Healthcheck and smoke tests of a given scenario must be succesful to -validate the scenario for the release. - -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | healthcheck | Verify basic operation in VIM | -| | +----------------+----------------------------------+ -| | | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| +---------------+----------------+----------------------------------+ -| | smoke | vPing_SSH | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network. | -| | +----------------+----------------------------------+ -| | | vPing_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test. | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment. | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment.| -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_. | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | onos | Test suite of ONOS L2 and L3 | -| | | | functions. | -| | | | See `ONOSFW User Guide`_ for | -| | | | details. | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | promise | Resource reservation and | -| | | | management project to identify | -| | | | NFV related requirements and | -| | | | realize resource reservation for | -| | | | future usage by capacity | -| | | | management of resource pools | -| | | | regarding compute, network and | -| | | | storage. | -| | | | See `Promise User Guide`_ for | -| | | | details. | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | security_scan | Implementation of a simple | -| | | | security scan. (Currently | -| | | | available only for the Apex | -| | | | installer environment) | -| | +----------------+----------------------------------+ -| | | onos-sfc | SFC testing for onos scenarios | -| | | | See `ONOSFW User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | domino | Domino provides TOSCA template | -| | | | distribution service for network | -| | | | service and VNF descriptors | -| | | | among MANO components e.g., | -| | | | NFVO, VNFM, VIM, SDN-C, etc., | -| | | | as well as OSS/BSS functions. | -| | | | See `Domino User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | copper | Copper develops OPNFV platform | -| | | | support for policy management, | -| | | | using open source projects such | -| | | | as OpenStack Congress, focused | -| | | | on helping ensure that virtual | -| | | | infrastructure and the apps that | -| | | | execute on it comply with the | -| | | | configuration policy intent of | -| | | | service providers, developers, | -| | | | and end users. See more detail | -| | | | in the `Copper User Guide`_. | -| | +----------------+----------------------------------+ -| | | multisites | Multisites | -| | | | See `Multisite User Guide`_ for | -| | | | details | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | opera_ims | vIMS deployment using openBaton | -| | +----------------+----------------------------------+ -| | | orchestra_ims | vIMS deployment using open-O | -+ +---------------+----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. | -| | | | See `Parser User Guide`_ for | -| | | | details | -+-------------+---------------+----------------+----------------------------------+ - - -As shown in the above table, Functest is structured into different 'domains', -'tiers' and 'test cases'. Each 'test case' usually represents an actual -'Test Suite' comprised -in turn- of several test cases internally. - -Test cases also have an implicit execution order. For example, if the early -'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. - -In Danube, we merged smoke and sdn controller tiers in smoke tier. - -An overview of the Functest Structural Concept is depicted graphically below: - -.. figure:: ../images/concepts_mapping_final.png - :align: center - :alt: Functest Concepts Structure - -Some of the test cases are developed by Functest team members, whereas others -are integrated from upstream communities or other OPNFV projects. For example, -`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the -OpenStack integration test suite and Functest is in charge of the selection, -integration and automation of those tests that fit suitably to OPNFV. - -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. - -The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: - -** TODO ** -.. figure:: ../images/FunctestDashboardDanube.png - :align: center - :alt: Functest Dashboard - - -Basic components (VIM, SDN controllers) are tested through their own suites. -Feature projects also provide their own test suites with different ways of -running their tests. - -The notion of domain has been introduced in the description of the test cases -stored in the Database. -This parameters as well as possible tags can be used for the Test case catalog. - -vIMS test case was integrated to demonstrate the capability to deploy a -relatively complex NFV scenario on top of the OPNFV infrastructure. - -Functest considers OPNFV as a black box. As of Danube release the OPNFV -offers a lot of potential combinations: - - * 3 controllers (OpenDaylight, ONOS, OpenContrail) - * 4 installers (Apex, Compass, Fuel, Joid) - -Most of the tests are runnable by any combination, but some tests might have -restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_IP and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. - -A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource -clean-up) and for executing tests. -The Functest CLI organised the testcase into logical Tiers, which contain in -turn one or more testcases. The CLI allows execution of a single specified -testcase, all test cases in a specified Tier, or the special case of execution -of **ALL** testcases. The Functest CLI is introduced in more detail in the -section `Executing the functest suites`_ of this document. - -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html -.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html -.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst deleted file mode 100644 index e7ab84b2..00000000 --- a/docs/userguide/runfunctest.rst +++ /dev/null @@ -1,385 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Executing the functest suites -============================= - -Manual testing --------------- - -This section assumes the following: - * The Functest Docker container is running - * The docker prompt is shown - * The Functest environment is ready (Functest CLI command 'functest env prepare' - has been executed) - -If any of the above steps are missing please refer to the Functest Config Guide -as they are a prerequisite and all the commands explained in this section **must** be -performed **inside the container**. - -The Functest CLI offers two commands (functest tier ...) and (functest testcase ... ) -for the execution of Test Tiers or Test Cases:: - - root@22e436918db0:~/repos/functest/ci# functest tier --help - Usage: functest tier [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - get-tests Prints the tests in a tier. - list Lists the available tiers. - run Executes all the tests within a tier. - show Shows information about a tier. - root@22e436918db0:~/repos/functest/ci# functest testcase --help - - Usage: functest testcase [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - list Lists the available testcases. - run Executes a test case. - show Shows information about a test case. - -More details on the existing Tiers and Test Cases can be seen with the 'list' -command:: - - root@22e436918db0:~/repos/functest/ci# functest tier list - - 0. healthcheck: - ['healthcheck', 'connection_check', 'api_check',] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl'] - - 2. features: - ['doctor', 'security_scan'] - - 3. components: - ['tempest_full_parallel', 'rally_full'] - - 4. vnf: - ['cloudify_ims'] - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase list - healthcheck - api_check - connection_check - vping_ssh - vping_userdata - snaps_smoke - tempest_smoke_serial - rally_sanity - odl - doctor - security_scan - tempest_full_parallel - rally_full - cloudify_ims - -More specific details on specific Tiers or Test Cases can be seen wih the -'show' command:: - - root@22e436918db0:~/repos/functest/ci# functest tier show smoke - +======================================================================+ - | Tier: smoke | - +======================================================================+ - | Order: 1 | - | CI Loop: (daily)|(weekly) | - | Description: | - | Set of basic Functional tests to validate the OpenStack | - | deployment. | - | Test cases: | - | - vping_ssh | - | - vping_userdata | - | - tempest_smoke_serial | - | - rally_sanity | - | | - +----------------------------------------------------------------------+ - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial - +======================================================================+ - | Testcase: tempest_smoke_serial | - +======================================================================+ - | Description: | - | This test case runs the smoke subset of the OpenStack Tempest | - | suite. The list of test cases is generated by Tempest | - | automatically and depends on the parameters of the OpenStack | - | deplopyment. | - | Dependencies: | - | - Installer: | - | - Scenario : | - | | - +----------------------------------------------------------------------+ - - -To execute a Test Tier or Test Case, the 'run' command is used:: - - root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck' - 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################ - 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck' - 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################ - 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ - 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'... - 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================ - 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API... - 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK! - 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API... - 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK! - 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API... - 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK! - 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API... - 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca - 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks... - 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets... - 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers... - 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK! - 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API... - 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK! - 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP... - : - : - 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK! - 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed! - 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources... - 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - Version 1 is deprecated, use alternative version 2 instead. - WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead. - 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances... - 2016-06-30 11:48:24,439 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images... - 2016-06-30 11:48:35,853 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes... - 2016-06-30 11:48:37,344 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs... - 2016-06-30 11:48:37,467 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects - 2016-06-30 11:48:53,633 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups... - 2016-06-30 11:48:53,689 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users... - 2016-06-30 11:48:54,444 - clean_openstack - INFO - ------------------------------------------- - 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants... - 2016-06-30 11:48:54,711 - clean_openstack - INFO - ------------------------------------------- - - and - - root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh - Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh' - 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file... - 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ - 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'... - 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================ - 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'... - 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net... - 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'... - 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'... - 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48' - 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'... - 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE. - 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'... - 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'... - 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE. - 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'... - 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'... - 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140' - 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2' - 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140... - 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping... - 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected! - 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s. - 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK - 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources... - 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++ - Version 1 is deprecated, use alternative version 2 instead. - : - : - etc. - -To list the test cases which are part of a specific Test Tier, the 'get-tests' -command is used with 'functest tier':: - - root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck - Test cases in tier 'healthcheck': - ['healthcheck'] - - -Please note that for some scenarios some test cases might not be launched. -For example, the last example displayed only the 'odl' testcase for the given -environment. In this particular system the deployment does not support the 'onos' SDN -Controller Test Case; for example. - -**Important** If you use the command 'functest tier run <tier_name>', then the -Functest CLI utility will call **all valid Test Cases**, which belong to the -specified Test Tier, as relevant to scenarios deployed to the SUT environment. -Thus, the Functest CLI utility calculates automatically which tests can be -executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, -which is passed in to the Functest docker container. - -Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports -two possibilities:: - - * Run a single Test Case, specified by a valid choice of <testcase_name> - * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all' - -Functest includes a cleaning mechanism in order to remove all the OpenStack -resources except those present before running any test. The script -*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up -the Functest environment (i.e. CLI command 'functest env prepare') to snapshot -all the OpenStack resources (images, networks, volumes, security groups, tenants, -users) so that an eventual cleanup does not remove any of these defaults. - -The script **clean_openstack.py** which is located in -*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is -in charge of cleaning the OpenStack resources that are not specified in the -defaults file generated previously which is stored in -*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container. - -It is important to mention that if there are new OpenStack resources created -manually after preparing the Functest environment, they will be removed, unless -you use the special method of invoking the test case with specific suppression -of clean up. (See the `Troubleshooting`_ section). - -The reason to include this cleanup meachanism in Functest is because some -test suites such as Tempest or Rally create a lot of resources (users, -tenants, networks, volumes etc.) that are not always properly cleaned, so this -function has been set to keep the system as clean as it was before a -full Functest execution. - -Although the Functest CLI provides an easy way to run any test, it is possible to -do a direct call to the desired test script. For example: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d - - -Automated testing ------------------ - -As mentioned previously, the Functest Docker container preparation as well as -invocation of Test Cases can be called within the container from the Jenkins CI -system. There are 3 jobs that automate the whole process. The first job runs all -the tests referenced in the daily loop (i.e. that must been run daily), the second -job runs the tests referenced in the weekly loop (usually long duration tests run -once a week maximum) and the third job allows testing test suite by test suite specifying -the test suite name. The user may also use either of these Jenkins jobs to execute -the desired test suites. - -One of the most challenging task in the Danube release consists -in dealing with lots of scenarios and installers. Thus, when the tests are -automatically started from CI, a basic algorithm has been created in order to -detect whether a given test is runnable or not on the given scenario. -Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). The daily/weekly notion has been introduces in -Colorado in order to save CI time and avoid running systematically -long duration tests. It was not used in Colorado due to CI resource shortage. -The mechanism remains however as part of the CI evolution. - -CI provides some useful information passed to the container as environment -variables: - - * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE - * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP - * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with - - * controller = (odl|onos|ocl|nosdn) - * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites) - * mode = (ha|noha) - -The constraints per test case are defined in the Functest configuration file -*/home/opnfv/repos/functest/functest/ci/testcases.yaml*:: - - tiers: - - - name: healthcheck - order: 0 - ci_loop: '(daily)|(weekly)' - description : >- - First tier to be executed to verify the basic - operations in the VIM. - testcases: - - - name: healthcheck - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies the basic OpenStack services like - Keystone, Glance, Cinder, Neutron and Nova. - - dependencies: - installer: '' - scenario: '' - - - - name: smoke - order: 1 - ci_loop: '(daily)|(weekly)' - description : >- - Set of basic Functional tests to validate the OpenStack deployment. - testcases: - - - name: vping_ssh - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies: 1) SSH to an instance using floating - IPs over the public network. 2) Connectivity between 2 instances - over a private network. - dependencies: - installer: '' - scenario: '^((?!bgpvpn|odl_l3).)*$' - run: - module: 'functest.opnfv_tests.openstack.vping.vping_ssh' - class: 'VPingSSH' - .... - -We may distinguish 2 levels in the test case description: - * Tier level - * Test case level - -At the tier level, we define the following parameters: - - * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs - * description: a high level view of the test case - -For a given test case we defined: - * the name of the test case - * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL - * blocking: if set to true, if the test is failed, the execution of the following tests is canceled - * the description of the test case - * the dependencies: a combination of 2 regex on the scenario and the installer name - * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests - -For further details on abstraction classes, see developper guide. - -Additional parameters have been added in the desription in the Database. -The target is to use the configuration stored in the Database and consider the -local file as backup if the Database is not reachable. -The additional fields related to a test case are: - * trust: we introduced this notion to put in place a mechanism of scenario promotion. - * Version: it indicates since which version you can run this test - * domains: the main domain covered by the test suite - * tags: a list of tags related to the test suite - -The order of execution is the one defined in the file if all test cases are selected. - -In CI daily job the tests are executed in the following order: - - 1) healthcheck (blocking) - 2) smoke: both vPings are blocking - 3) Feature project tests cases - -In CI weekly job we add 2 tiers: - - 4) VNFs (vIMS) - 5) Components (Rally and Tempest long duration suites) - -As explained before, at the end of an automated execution, the OpenStack resources -might be eventually removed. -Please note that a system snapshot is taken before any test case execution. - -This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst deleted file mode 100644 index 84550191..00000000 --- a/docs/userguide/troubleshooting.rst +++ /dev/null @@ -1,387 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Troubleshooting -=============== - -This section gives some guidelines about how to troubleshoot the test cases -owned by Functest. - -**IMPORTANT**: As in the previous section, the steps defined below must be -executed inside the Functest Docker container and after sourcing the OpenStack -credentials:: - - . $creds - -or:: - - source /home/opnfv/functest/conf/openstack.creds - -VIM ---- - -This section covers the test cases related to the VIM (healthcheck, vping_ssh, -vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, -rally_full). - -vPing common -^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: - - * Create Glance image - * Create Network - * Create Security Group - * Create Instances - -After these actions, the test cases differ and will be explained in their -respective section. - -These test cases can be run inside the container, using new Functest CLI as follows:: - - $ functest testcase run vping_ssh - $ functest testcase run vping_userdata - -The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths: -*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_ssh.py* and -*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py* - -Notes: - - #. There is one difference, between the Functest CLI based test case - execution compared to the earlier used Bash shell script, which is - relevant to point out in troubleshooting scenarios: - - The Functest CLI does **not yet** support the option to suppress - clean-up of the generated OpenStack resources, following the execution - of a test case. - - Explanation: After finishing the test execution, the corresponding - script will remove, by default, all created resources in OpenStack - (image, instances, network and security group). When troubleshooting, - it is advisable sometimes to keep those resources in case the test - fails and a manual testing is needed. - - It is actually still possible to invoke test execution, with suppression - of OpenStack resource cleanup, however this requires invocation of a - **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'. - The `OPNFV Functest Developer Guide`_ provides guidance on the use of that - Python script in such troubleshooting cases. - -Some of the common errors that can appear in this test case are:: - - vPing_ssh- ERROR - There has been a problem when creating the neutron network.... - -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: - - neutron net-create net-test - neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ - --gateway 10.6.0.254 net-test 10.6.0.0/24 - neutron router-create test_router - neutron router-interface-add <ROUTER_ID> test_subnet - neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> - -Another related error can occur while creating the Security Groups for the instances:: - - vPing_ssh- ERROR - Failed to create the security group... - -In this case, proceed to create it manually. These are some hints:: - - neutron security-group-create sg-test - neutron security-group-rule-create sg-test --direction ingress --protocol icmp \ - --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - -The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: - - nova list - nova show <INSTANCE_ID> - -It might show some messages about the booting failure. To try that manually:: - - nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test - -This will spawn a VM using the network created previously manually. -In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. - - -vPing_SSH -^^^^^^^^^ -This test case creates a floating IP on the external network and assigns it to -the second instance **opnfv-vping-2**. The purpose of this is to establish -a SSH connection to that instance and SCP a script that will ping the first -instance. This script is located in the repository under -*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/ping.sh* and takes an IP as -a parameter. When the SCP is completed, the test will do an SSH call to that script -inside the second instance. Some problems can happen here:: - - vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting - -If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not -reach the Public/External IP assigned to the instance **opnfv-vping-2**. There -are many possible reasons, and they really depend on the chosen scenario. For -most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known -limitation. - -First, make sure that the instance **opnfv-vping-2** succeeded to get an IP -from the DHCP agent. It can be checked by doing:: - - nova console-log opnfv-vping-2 - -If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. - -If the instance got an IP properly, try to ping manually the VM from the container:: - - nova list - <grab the public IP> - ping <public IP> - -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. - -At this point, if the ping does not work either, try to reproduce the test -manually with the steps described above in the vPing common section with the -addition:: - - neutron floatingip-create <EXT_NET_NAME> - nova floating-ip-associate nova-test <FLOATING_IP> - - -Further troubleshooting is out of scope of this document, as it might be due to -problems with the SDN controller. Contact the installer team members or send an -email to the corresponding OPNFV mailing list for more information. - - - -vPing_userdata -^^^^^^^^^^^^^^ -This test case does not create any floating IP neither establishes an SSH -connection. Instead, it uses nova-metadata service when creating an instance -to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. - -The only known problem here for this test to fail is mainly the lack of support -of cloud-init (nova-metadata service). Check the console of the instance:: - - nova console-log opnfv-vping-2 - -If this text or similar is shown:: - - checking http://169.254.169.254/2009-04-04/instance-id - failed 1/20: up 1.13. request failed - failed 2/20: up 13.18. request failed - failed 3/20: up 25.20. request failed - failed 4/20: up 37.23. request failed - failed 5/20: up 49.25. request failed - failed 6/20: up 61.27. request failed - failed 7/20: up 73.29. request failed - failed 8/20: up 85.32. request failed - failed 9/20: up 97.34. request failed - failed 10/20: up 109.36. request failed - failed 11/20: up 121.38. request failed - failed 12/20: up 133.40. request failed - failed 13/20: up 145.43. request failed - failed 14/20: up 157.45. request failed - failed 15/20: up 169.48. request failed - failed 16/20: up 181.50. request failed - failed 17/20: up 193.52. request failed - failed 18/20: up 205.54. request failed - failed 19/20: up 217.56. request failed - failed 20/20: up 229.58. request failed - failed to read iid from metadata. tried 20 - -it means that the instance failed to read from the metadata service. Contact -the Functest or installer teams for more information. - -NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests -have been excluded from CI in those scenarios. - - -Tempest -^^^^^^^ - -In the upstream OpenStack CI all the Tempest test cases are supposed to pass. -If some test cases fail in an OPNFV deployment, the reason is very probably one -of the following - -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/<component>'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | '/home/opnfv/.rally/tempest/for-deployment-<UUID>' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ - - -When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. - - -Rally -^^^^^ - -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. - -It is possible to run only one Rally scenario, instead of the whole suite. -To do that, call the alternative python script as follows:: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -h - usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name - - positional arguments: - test_name Module name to be tested. Possible values are : [ - authenticate | glance | cinder | heat | keystone | neutron | - nova | quotas | requests | vm | all ] The 'all' value - performs all possible test scenarios - - optional arguments: - -h, --help show this help message and exit - -d, --debug Debug mode - -r, --report Create json result file - -s, --smoke Smoke test mode - -v, --verbose Print verbose info about the progress - -n, --noclean Don't clean the created resources for this test. - -For example, to run the Glance scenario with debug information:: - - python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -d glance - -Possible scenarios are: - * authenticate - * glance - * cinder - * heat - * keystone - * neutron - * nova - * quotas - * requests - * vm - -To know more about what those scenarios are doing, they are defined in directory: -*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/scenario* -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ - -If the flag *all* is specified, it will run all the scenarios one by one. Please -note that this might take some time (~1,5hr), taking around 1 hour alone to -complete the Nova scenario. - -To check any possible problems with Rally, the logs are stored under -*/home/opnfv/functest/results/rally/* in the Functest Docker container. - - -Controllers ------------ - -Opendaylight -^^^^^^^^^^^^ - -If the Basic Restconf test suite fails, check that the ODL controller is -reachable and its Restconf module has been installed. - -If the Neutron Reachability test fails, verify that the modules -implementing Neutron requirements have been properly installed. - -If any of the other test cases fails, check that Neutron and ODL have -been correctly configured to work together. Check Neutron configuration -files, accounts, IP addresses etc.). - - -ONOS -^^^^ - -Please refer to the ONOS documentation. `ONOSFW User Guide`_ . - - -Features --------- - -Please refer to the dedicated feature user guides for details. - - -security_scan -^^^^^^^^^^^^^ - -See OpenSCAP web site: https://www.open-scap.org/ - - - -NFV ---- - -cloudify_ims -^^^^^^^^^^^^ -vIMS deployment may fail for several reasons, the most frequent ones are -described in the following table: - -+-----------------------------------+------------------------------------+ -| Error | Comments | -+===================================+====================================+ -| Keystone admin API not reachable | Impossible to create vIMS user and | -| | tenant | -+-----------------------------------+------------------------------------+ -| Impossible to retrieve admin role | Impossible to create vIMS user and | -| id | tenant | -+-----------------------------------+------------------------------------+ -| Error when uploading image from | impossible to deploy VNF | -| OpenStack to glance | | -+-----------------------------------+------------------------------------+ -| Cinder quota cannot be updated | Default quotas not sufficient, they| -| | are adapted in the script | -+-----------------------------------+------------------------------------+ -| Impossible to create a volume | VNF cannot be deployed | -+-----------------------------------+------------------------------------+ -| SSH connection issue between the | if vPing test fails, vIMS test will| -| Test Docker container and the VM | fail... | -+-----------------------------------+------------------------------------+ -| No Internet access from the VM | the VMs of the VNF must have an | -| | external access to Internet | -+-----------------------------------+------------------------------------+ -| No access to OpenStack API from | Orchestrator can be installed but | -| the VM | the vIMS VNF installation fails | -+-----------------------------------+------------------------------------+ - - -parser -^^^^^^ - -For now log info is the only way to do trouble shooting - - -.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# |