aboutsummaryrefslogtreecommitdiffstats
path: root/docs/userguide/introduction.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/userguide/introduction.rst')
-rw-r--r--docs/userguide/introduction.rst223
1 files changed, 155 insertions, 68 deletions
diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst
index 65f8dc48b..8624bee83 100644
--- a/docs/userguide/introduction.rst
+++ b/docs/userguide/introduction.rst
@@ -1,94 +1,181 @@
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-Overview of the functest suites
+Overview of the Functest suites
===============================
Functest is the OPNFV project primarily targeting function testing.
In the Continuous Integration pipeline, it is launched after an OPNFV fresh
-installation to validate and verify the basic functions of the infrastructure.
-
-The current list of test suites can be distributed in 3 main domains: VIM
-(Virtualised Infrastructure Manager), Controllers and Features.
-
-+----------------+----------------+-------------------------------------------+
-| Domain | Test suite | Comments |
-+================+================+===========================================+
-| VIM | vPing | NFV "Hello World" using SSH connection |
-| | | and floatting IP |
-| +----------------+-------------------------------------------+
-| | vPing_userdata | Ping using userdata and cloud-init |
-| | | mechanism |
-| +----------------+-------------------------------------------+
-| | Tempest | OpenStack reference test suite `[2]`_ |
-| +----------------+-------------------------------------------+
-| | Rally bench | OpenStack testing tool benchmarking |
-| | | OpenStack modules `[3]`_ |
-+----------------+----------------+-------------------------------------------+
-| | OpenDaylight | Opendaylight Test suite |
-| +----------------+-------------------------------------------+
-| Controllers | ONOS | Test suite of ONOS L2 and L3 functions |
-| | | See `ONOSFW User Guide`_ for details |
-+----------------+----------------+-------------------------------------------+
-| Features | vIMS | Example of a real VNF deployment to show |
-| | | the NFV capabilities of the platform. |
-| | | The IP Multimedia Subsytem is a typical |
-| | | Telco test case, referenced by ETSI. |
-| | | It provides a fully functional VoIP System|
-| +----------------+-------------------------------------------+
-| | Promise | Resource reservation and management |
-| | | project to identify NFV related |
-| | | requirements and realize resource |
-| | | reservation for future usage by capacity |
-| | | management of resource pools regarding |
-| | | compute, network and storage. |
-| | | See `Promise User Guide`_ for details |
-| +----------------+-------------------------------------------+
-| | Doctor | Doctor platform, as of Brahmaputra release|
-| | | , provides the two features: |
-| | | * Immediate Notification |
-| | | * Consistent resource state awareness |
-| | | (compute), see `Doctor User Guide`_ for |
-| | | details |
-| +----------------+-------------------------------------------+
-| | SDNVPN | Implementation of the OpenStack bgpvpn API|
-| | | from the SDNVPN feature project. |
-| | | It allowing the cration of BGP VPNs |
-| | | see SDNVPN User Guide for |
-+----------------+----------------+-------------------------------------------+
-
-
-Functest includes different test suites with several test cases within. Some
-of the tests are developed by Functest team members whereas others are
-integrated from upstream communities or other OPNFV projects. For example,
+installation to validate and verify the basic functions of the
+infrastructure.
+
+The current list of test suites can be distributed over 4 main domains: VIM
+(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
+Features and VNF (Virtual Network Functions).
+
++-------------+---------------+----------------+----------------------------------+
+| Domain | Tier | Test case | Comments |
++=============+===============+================+==================================+
+| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
+| +---------------+----------------+----------------------------------+
+| | smoke | vPing_SSH | NFV "Hello World" using an SSH |
+| | | | connection to a destination VM |
+| | | | over a created floating IP |
+| | | | address on the SUT Public / |
+| | | | External network. Using the SSH |
+| | | | connection a test script is then |
+| | | | copied to the destination |
+| | | | VM and then executed via SSH. |
+| | | | The script will ping another |
+| | | | VM on a specified IP address over|
+| | | | the SUT Private Tenant network. |
+| | +----------------+----------------------------------+
+| | | vPing_userdata | Uses Ping with given userdata |
+| | | | to test intra-VM connectivity |
+| | | | over the SUT Private Tenant |
+| | | | network. The correct operation |
+| | | | of the NOVA Metadata service is |
+| | | | also verified in this test. |
+| | +----------------+----------------------------------+
+| | | tempest_smoke | Generate and run a relevant |
+| | | \_serial | Tempest Test Suite in smoke mode.|
+| | | | The generated test set is |
+| | | | dependent on the OpenStack |
+| | | | deployment environment. |
+| | +----------------+----------------------------------+
+| | | rally_sanity | Run a subset of the OpenStack |
+| | | | Rally Test Suite in smoke mode |
+| +---------------+----------------+----------------------------------+
+| | openstack | tempest_full | Generate and run a full set of |
+| | | \_parallel | the OpenStack Tempest Test Suite.|
+| | | | See the OpenStack reference test |
+| | | | suite `[2]`_. The generated |
+| | | | test set is dependent on the |
+| | | | OpenStack deployment environment.|
+| | +----------------+----------------------------------+
+| | | rally_full | Run the OpenStack testing tool |
+| | | | benchmarking OpenStack modules |
+| | | | See the Rally documents `[3]`_. |
++-------------+---------------+----------------+----------------------------------+
+| Controllers | sdn_suites | odl | Opendaylight Test suite |
+| | | | TODO: Find a document reference! |
+| | +----------------+----------------------------------+
+| | | onos | Test suite of ONOS L2 and L3 |
+| | | | functions. |
+| | | | See `ONOSFW User Guide`_ for |
+| | | | details. |
++-------------+---------------+----------------+----------------------------------+
+| Features | features | Promise | Resource reservation and |
+| | | | management project to identify |
+| | | | NFV related requirements and |
+| | | | realize resource reservation for |
+| | | | future usage by capacity |
+| | | | management of resource pools |
+| | | | regarding compute, network and |
+| | | | storage. |
+| | | | See `Promise User Guide`_ for |
+| | | | details. |
+| | +----------------+----------------------------------+
+| | | Doctor | Doctor platform, as of Colorado |
+| | | | release, provides the two |
+| | | | features: |
+| | | | * Immediate Notification |
+| | | | * Consistent resource state |
+| | | | awareness (compute). See the |
+| | | | See `Doctor User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | bgpvpn | Implementation of the OpenStack |
+| | | | bgpvpn API from the SDNVPN |
+| | | | feature project. It allows for |
+| | | | the creation of BGP VPNs. |
+| | | | See `SDNVPN User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | security_scan | Implementation of a simple |
+| | | | security scan. (Currently |
+| | | | available only for the Apex |
+| | | | installer environment) |
+| | | | TODO: Add document link from |
+| | | | Luke Hinds; when received. |
++-------------+---------------+----------------+----------------------------------+
+| VNF | vnf | vims | Example of a real VNF deployment |
+| | | | to show the NFV capabilities of |
+| | | | the platform. The IP Multimedia |
+| | | | Subsytem is a typical Telco test |
+| | | | case, referenced by ETSI. |
+| | | | It provides a fully functional |
+| | | | VoIP System, |
++-------------+---------------+----------------+----------------------------------+
+
+
+As shown in the above table, Functest is structured into different 'domains',
+'tiers' and 'test cases'. Each 'test case' usually represents an actual
+'Test Suite' comprised -in turn- of several test cases internally.
+
+Test cases also have an implicit execution order. For example, if the early
+'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
+Tier testcases, there is little point to launch a full testcase execution round.
+
+An overview of the Functest Structural Concept is depicted graphically below:
+
+.. figure:: ../images/concepts_mapping_final.png
+ :align: center
+ :alt: Functest Concepts Structure
+
+Some of the test cases are developed by Functest team members, whereas others
+are integrated from upstream communities or other OPNFV projects. For example,
`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
OpenStack integration test suite and Functest is in charge of the selection,
-integration and automation of the tests that fit in OPNFV.
+integration and automation of those tests that fit suitably to OPNFV.
+
+The Tempest test suite has been customized but no new test cases have been
+created in OPNFV Functest.
+
+The results produced by the tests run from CI are pushed and collected into a
+NoSQL database. The goal is to populate the database with results from different
+sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
+of a live Functest Dashboard is shown below:
-The Tempest suite has been customized but no new test cases have been created.
+.. figure:: ../images/FunctestDashboard.png
+ :align: center
+ :alt: Functest Dashboard
-The results produced by the tests run from CI are pushed and collected in a NoSQL
-database. The goal is to populate the database with results from different sources
-and scenarios and to show them on a Dashboard.
There is no real notion of Test domain or Test coverage. Basic components
-(VIM, controllers) are tested through their own suites. Feature projects also
-provide their own test suites with different ways of running their tests.
+(VIM, SDN controllers) are tested through their own suites. Feature projects
+also provide their own test suites with different ways of running their tests.
vIMS test case was integrated to demonstrate the capability to deploy a
relatively complex NFV scenario on top of the OPNFV infrastructure.
-Functest considers OPNFV as a black box.
-OPNFV, since the Brahmaputra release, offers lots of potential combinations:
+Functest considers OPNFV as a black box. As of Colorado release the OPNFV
+offers a lot of potential combinations:
- * 2 controllers (OpenDayligh, ONOS)
+ * 3 controllers (OpenDaylight, ONOS, OpenContrail)
* 4 installers (Apex, Compass, Fuel, Joid)
-Most of the tests are runnable on any combination, but some others might have
-restrictions imposed by the installers or the available deployed features.
+Most of the tests are runnable by any combination, but some tests might have
+restrictions imposed by the utilized installers or due to the available
+deployed features. The system uses the environment variables (INSTALLER_IP and
+DEPLOY_SCENARIO) to automatically determine the valid test cases; for each given
+environment.
+
+In the Colorado OPNFV System release a convenience Functest CLI utility is also
+introduced to simplify setting up the Functest evironment, management of the
+OpenStack environment (e.g. resource clean-up) and for executing tests.
+The Functest CLI organised the testcase into logical Tiers, which contain in
+turn one or more testcases. The CLI allow execution of a single specified
+testcase, all test cases in a specified Tier, or the special case of execution
+of **ALL** testcases. The Functest CLI is introduced in more detail in the
+section `Executing the functest suites`_ of this document.
.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
.. _`Doctor User Guide`: http://artifacts.opnfv.org/opnfvdocs/brahmaputra/docs/userguide/featureusage-doctor.html
.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/brahmaputra/docs/userguide/index.html
.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html
+.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html
+.. _`Functest Dashboard`: http://testresults.opnfv.org/dashboard/
+
+