Functest Userguide update for Colorado

Text updates to reflect the introduction of the Functest CLI for
Functest Docker container environment.

JIRA: FUNCTEST-238

Change-Id: Ifa742fb5c4a24ae94cc1e7062bcb8c0a2f49bf02
Signed-off-by: columgaynor <colum.gaynor@nokia.com>

1 files changed, 155 insertions, 68 deletions

diff --git a/docs/userguide/introduction.rst b/docs/userguide/introduction.rst
index 65f8dc48b..8624bee83 100644
--- a/docs/userguide/introduction.rst
+++ b/docs/userguide/introduction.rst
@@ -1,94 +1,181 @@
 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
 .. http://creativecommons.org/licenses/by/4.0
 
-Overview of the functest suites
+Overview of the Functest suites
 ===============================
 
 Functest is the OPNFV project primarily targeting function testing.
 In the Continuous Integration pipeline, it is launched after an OPNFV fresh
-installation to validate and verify the basic functions of the infrastructure.
-
-The current list of test suites can be distributed in 3 main domains: VIM
-(Virtualised Infrastructure Manager), Controllers and Features.
-
-+----------------+----------------+-------------------------------------------+
-| Domain         | Test suite     | Comments                                  |
-+================+================+===========================================+
-| VIM            | vPing          | NFV "Hello World" using SSH connection    |
-|                |                | and floatting IP                          |
-|                +----------------+-------------------------------------------+
-|                | vPing_userdata | Ping using userdata and cloud-init        |
-|                |                | mechanism                                 |
-|                +----------------+-------------------------------------------+
-|                | Tempest        | OpenStack reference test suite `[2]`_     |
-|                +----------------+-------------------------------------------+
-|                | Rally bench    | OpenStack testing tool benchmarking       |
-|                |                | OpenStack modules `[3]`_                  |
-+----------------+----------------+-------------------------------------------+
-|                | OpenDaylight   | Opendaylight Test suite                   |
-|                +----------------+-------------------------------------------+
-| Controllers    | ONOS           | Test suite of ONOS L2 and L3 functions    |
-|                |                | See `ONOSFW User Guide`_ for details      |
-+----------------+----------------+-------------------------------------------+
-| Features       | vIMS           | Example of a real VNF deployment to show  |
-|                |                | the NFV capabilities of the platform.     |
-|                |                | The IP Multimedia Subsytem is a typical   |
-|                |                | Telco test case, referenced by ETSI.      |
-|                |                | It provides a fully functional VoIP System|
-|                +----------------+-------------------------------------------+
-|                | Promise        | Resource reservation and management       |
-|                |                | project to identify NFV related           |
-|                |                | requirements and realize resource         |
-|                |                | reservation for future usage by capacity  |
-|                |                | management of resource pools regarding    |
-|                |                | compute, network and storage.             |
-|                |                | See `Promise User Guide`_ for details     |
-|                +----------------+-------------------------------------------+
-|                | Doctor         | Doctor platform, as of Brahmaputra release|
-|                |                | , provides the two features:              |
-|                |                | * Immediate Notification                  |
-|                |                | * Consistent resource state awareness     |
-|                |                | (compute), see `Doctor User Guide`_ for   |
-|                |                | details                                   |
-|                +----------------+-------------------------------------------+
-|                | SDNVPN         | Implementation of the OpenStack bgpvpn API|
-|                |                | from the SDNVPN feature project.          |
-|                |                | It allowing the cration of BGP VPNs       |
-|                |                | see SDNVPN User Guide for                 |
-+----------------+----------------+-------------------------------------------+
-
-
-Functest includes different test suites with several test cases within. Some
-of the tests are developed by Functest team members whereas others are
-integrated from upstream communities or other OPNFV projects. For example,
+installation to validate and verify the basic functions of the
+infrastructure.
+
+The current list of test suites can be distributed over 4 main domains: VIM
+(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
+Features and VNF (Virtual Network Functions).
+
++-------------+---------------+----------------+----------------------------------+
+| Domain      | Tier          | Test case      | Comments                         |
++=============+===============+================+==================================+
+| VIM         | healthcheck   | healthcheck    | Verify basic operation in VIM    |
+|             +---------------+----------------+----------------------------------+
+|             | smoke         | vPing_SSH      | NFV "Hello World" using an SSH   |
+|             |               |                | connection to a destination VM   |
+|             |               |                | over a created floating IP       |
+|             |               |                | address on the SUT Public /      |
+|             |               |                | External network. Using the SSH  |
+|             |               |                | connection a test script is then |
+|             |               |                | copied to the destination        |
+|             |               |                | VM and then executed via SSH.    |
+|             |               |                | The script will ping another     |
+|             |               |                | VM on a specified IP address over|
+|             |               |                | the SUT Private Tenant network.  |
+|             |               +----------------+----------------------------------+
+|             |               | vPing_userdata | Uses Ping with given userdata    |
+|             |               |                | to test intra-VM connectivity    |
+|             |               |                | over the SUT Private Tenant      |
+|             |               |                | network. The correct operation   |
+|             |               |                | of the NOVA Metadata service is  |
+|             |               |                | also verified in this test.      |
+|             |               +----------------+----------------------------------+
+|             |               | tempest_smoke  | Generate and run a relevant      |
+|             |               | \_serial       | Tempest Test Suite in smoke mode.|
+|             |               |                | The generated test set is        |
+|             |               |                | dependent on the OpenStack       |
+|             |               |                | deployment environment.          |
+|             |               +----------------+----------------------------------+
+|             |               | rally_sanity   | Run a subset of the OpenStack    |
+|             |               |                | Rally Test Suite in smoke mode   |
+|             +---------------+----------------+----------------------------------+
+|             | openstack     | tempest_full   | Generate and run a full set of   |
+|             |               | \_parallel     | the OpenStack Tempest Test Suite.|
+|             |               |                | See the OpenStack reference test |
+|             |               |                | suite `[2]`_. The generated      |
+|             |               |                | test set is dependent on the     |
+|             |               |                | OpenStack deployment environment.|
+|             |               +----------------+----------------------------------+
+|             |               | rally_full     | Run the OpenStack testing tool   |
+|             |               |                | benchmarking OpenStack modules   |
+|             |               |                | See the Rally documents `[3]`_.  |
++-------------+---------------+----------------+----------------------------------+
+| Controllers | sdn_suites    | odl            | Opendaylight Test suite          |
+|             |               |                | TODO: Find a document reference! |
+|             |               +----------------+----------------------------------+
+|             |               | onos           | Test suite of ONOS L2 and L3     |
+|             |               |                | functions.                       |
+|             |               |                | See `ONOSFW User Guide`_  for    |
+|             |               |                | details.                         |
++-------------+---------------+----------------+----------------------------------+
+| Features    | features      | Promise        | Resource reservation and         |
+|             |               |                | management project to identify   |
+|             |               |                | NFV related requirements and     |
+|             |               |                | realize resource reservation for |
+|             |               |                | future usage by capacity         |
+|             |               |                | management of resource pools     |
+|             |               |                | regarding compute, network and   |
+|             |               |                | storage.                         |
+|             |               |                | See `Promise User Guide`_ for    |
+|             |               |                | details.                         |
+|             |               +----------------+----------------------------------+
+|             |               | Doctor         | Doctor platform, as of Colorado  |
+|             |               |                | release, provides the two        |
+|             |               |                | features:                        |
+|             |               |                | * Immediate Notification         |
+|             |               |                | * Consistent resource state      |
+|             |               |                | awareness (compute). See the     |
+|             |               |                | See `Doctor User Guide`_ for     |
+|             |               |                | details                          |
+|             |               +----------------+----------------------------------+
+|             |               | bgpvpn         | Implementation of the OpenStack  |
+|             |               |                | bgpvpn API from the SDNVPN       |
+|             |               |                | feature project. It allows for   |
+|             |               |                | the creation of BGP VPNs.        |
+|             |               |                | See `SDNVPN User Guide`_ for     |
+|             |               |                | details                          |
+|             |               +----------------+----------------------------------+
+|             |               | security_scan  | Implementation of a simple       |
+|             |               |                | security scan. (Currently        |
+|             |               |                | available only for the Apex      |
+|             |               |                | installer environment)           |
+|             |               |                | TODO: Add document link from     |
+|             |               |                | Luke Hinds; when received.       |
++-------------+---------------+----------------+----------------------------------+
+| VNF         | vnf           | vims           | Example of a real VNF deployment |
+|             |               |                | to show the NFV capabilities of  |
+|             |               |                | the platform. The IP Multimedia  |
+|             |               |                | Subsytem is a typical Telco test |
+|             |               |                | case, referenced by ETSI.        |
+|             |               |                | It provides a fully functional   |
+|             |               |                | VoIP System,                     |
++-------------+---------------+----------------+----------------------------------+
+
+
+As shown in the above table, Functest is structured into different 'domains',
+'tiers' and 'test cases'. Each 'test case' usually represents an actual
+'Test Suite' comprised -in turn- of several test cases internally.
+
+Test cases also have an implicit execution order. For example, if the early
+'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
+Tier testcases, there is little point to launch a full testcase execution round.
+
+An overview of the Functest Structural Concept is depicted graphically below:
+
+.. figure:: ../images/concepts_mapping_final.png
+   :align: center
+   :alt: Functest Concepts Structure
+
+Some of the test cases are developed by Functest team members, whereas others
+are integrated from upstream communities or other OPNFV projects. For example,
 `Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
 OpenStack integration test suite and Functest is in charge of the selection,
-integration and automation of the tests that fit in OPNFV.
+integration and automation of those tests that fit suitably to OPNFV.
+
+The Tempest test suite has been customized but no new test cases have been
+created in OPNFV Functest.
+
+The results produced by the tests run from CI are pushed and collected into a
+NoSQL database. The goal is to populate the database with results from different
+sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
+of a live Functest Dashboard is shown below:
 
-The Tempest suite has been customized but no new test cases have been created.
+.. figure:: ../images/FunctestDashboard.png
+   :align: center
+   :alt: Functest Dashboard
 
-The results produced by the tests run from CI are pushed and collected in a NoSQL
-database. The goal is to populate the database with results from different sources
-and scenarios and to show them on a Dashboard.
 
 There is no real notion of Test domain or Test coverage. Basic components
-(VIM, controllers) are tested through their own suites. Feature projects also
-provide their own test suites with different ways of running their tests.
+(VIM, SDN controllers) are tested through their own suites. Feature projects
+also provide their own test suites with different ways of running their tests.
 
 vIMS test case was integrated to demonstrate the capability to deploy a
 relatively complex NFV scenario on top of the OPNFV infrastructure.
 
-Functest considers OPNFV as a black box.
-OPNFV, since the Brahmaputra release, offers lots of potential combinations:
+Functest considers OPNFV as a black box. As of Colorado release the OPNFV
+offers a lot of potential combinations:
 
-  * 2 controllers (OpenDayligh, ONOS)
+  * 3 controllers (OpenDaylight, ONOS, OpenContrail)
   * 4 installers (Apex, Compass, Fuel, Joid)
 
-Most of the tests are runnable on any combination, but some others might have
-restrictions imposed by the installers or the available deployed features.
+Most of the tests are runnable by any combination, but some tests might have
+restrictions imposed by the utilized installers or due to the available
+deployed features. The system uses the environment variables (INSTALLER_IP and
+DEPLOY_SCENARIO) to automatically determine the valid test cases; for each given
+environment.
+
+In the Colorado OPNFV System release a convenience Functest CLI utility is also
+introduced to simplify setting up the Functest evironment, management of the
+OpenStack environment (e.g. resource clean-up) and for executing tests.
+The Functest CLI organised the testcase into logical Tiers, which contain in
+turn one or more testcases. The CLI allow execution of a single specified
+testcase, all test cases in a specified Tier, or the special case of execution
+of **ALL** testcases. The Functest CLI is introduced in more detail in the
+section `Executing the functest suites`_ of this document.
 
 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
 .. _`Doctor User Guide`: http://artifacts.opnfv.org/opnfvdocs/brahmaputra/docs/userguide/featureusage-doctor.html
 .. _`Promise User Guide`: http://artifacts.opnfv.org/promise/brahmaputra/docs/userguide/index.html
 .. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html
+.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html
+.. _`Functest Dashboard`: http://testresults.opnfv.org/dashboard/
+
+