Update user guide for Colorado

- Automated section
- troubleshooting
- add new feature projects

JIRA: FUNCTEST-406

Change-Id: I6186b287b4defb9dff4547dc228fe5a9326fb6f3
Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>

1 files changed, 85 insertions, 64 deletions

diff --git a/docs/userguide/runfunctest.rst b/docs/userguide/runfunctest.rst
index 4c2e3f54b..f3da540a4 100644
--- a/docs/userguide/runfunctest.rst
+++ b/docs/userguide/runfunctest.rst
@@ -38,8 +38,6 @@ for the execution of Test Tiers or Test Cases::
     show       Shows information about a tier.
   root@22e436918db0:~/repos/functest/ci# functest testcase --help
 
-  and
-
   Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
 
   Options:
@@ -217,7 +215,7 @@ command is used with 'functest tier'::
 
 Please note that for some scenarios some test cases might not be launched.
 For example, the last example displayed only the 'odl' testcase for the given
-environment. In this system the deployment does not support the 'onos' SDN
+environment. In this particular system the deployment does not support the 'onos' SDN
 Controller Test Case; for example.
 
 **Important** If you use the command 'functest tier run <tier_name>', then the
@@ -264,12 +262,12 @@ Example::
   :
   etc.
 
-Functest includes cleaning mechanism in order to remove all the OpenStack
-resources except what was present before running any test. The script
+Functest includes a cleaning mechanism in order to remove all the OpenStack
+resources except those present before running any test. The script
 *$repos_dir/functest/utils/generate_defaults.py* is called once when setting up
 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
 all the OpenStack resources (images, networks, volumes, security groups, tenants,
-users) so that an eventual cleanup does not remove any of this defaults.
+users) so that an eventual cleanup does not remove any of these defaults.
 
 The script **clean_openstack.py** which is located in
 *$repos_dir/functest/utils/* is normally called after a test execution. It is
@@ -297,21 +295,23 @@ do a direct call to the desired test script. For example:
 Automated testing
 -----------------
 
-**TODO** Jose, the next section has not yet been revised!
-
 As mentioned previously, the Functest Docker container preparation as well as
 invocation of Test Cases can be called within the container from the Jenkins CI
-system. There are 2 jobs that automate the whole process. The first job runs all
-the tests and the second job allows testing test suite by test suite specifying
+system. There are 3 jobs that automate the whole process. The first job runs all
+the tests referenced in the daily loop (i.e. that must been run daily), the second
+job runs the tests referenced in the weekly loop (usually long duration tests run
+once a week maximum) and the third job allows testing test suite by test suite specifying
 the test suite name. The user may also use either of these Jenkins jobs to execute
 the desired test suites.
 
-One of the most challenging task in the Brahmaputra release consists
+One of the most challenging task in the Colorado release consists
 in dealing with lots of scenarios and installers. Thus, when the tests are
 automatically started from CI, a basic algorithm has been created in order to
 detect whether a given test is runnable or not on the given scenario.
 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
-be run on an ONOS scenario).
+be run on an ONOS scenario). Moreover since Colorado, we also introduce the
+notion of daily/weekly in order to save CI time and avoid running systematically
+long duration tests.
 
 CI provides some useful information passed to the container as environment
 variables:
@@ -320,65 +320,86 @@ variables:
  * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
  * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
 
- * controller = (odl|onos|ocl|nosdn)
- * feature = (ovs(dpdk)|kvm|sfc|bgpvpn)
- * mode = (ha|noha)
+   * controller = (odl|onos|ocl|nosdn)
+   * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites)
+   * mode = (ha|noha)
 
 The constraints per test case are defined in the Functest configuration file
-*/home/opnfv/functest/config/config_functest.yaml*::
-
- test-dependencies:
-    functest:
-        vims:
-            scenario: '(ocl)|(odl)|(nosdn)'
-        vping:
-        vping_userdata:
-            scenario: '(ocl)|(odl)|(nosdn)'
-        tempest:
-        rally:
-        odl:
-            scenario: 'odl'
-        onos:
-            scenario: 'onos'
+*/home/opnfv/repos/functest/ci/testcases.yaml*::
+
+ tiers:
+    -
+        name: healthcheck
+        order: 0
+        ci_loop: '(daily)|(weekly)'
+        description : >-
+            First tier to be executed to verify the basic
+            operations in the VIM.
+        testcases:
+            -
+                name: healthcheck
+                criteria: 'status == "PASS"'
+                blocking: true
+                description: >-
+                    This test case verifies the basic OpenStack services like
+                    Keystone, Glance, Cinder, Neutron and Nova.
+
+                dependencies:
+                    installer: ''
+                    scenario: ''
+
+   -
+        name: smoke
+        order: 1
+        ci_loop: '(daily)|(weekly)'
+        description : >-
+            Set of basic Functional tests to validate the OpenStack deployment.
+        testcases:
+            -
+                name: vping_ssh
+                criteria: 'status == "PASS"'
+                blocking: true
+                description: >-
+                    This test case verifies: 1) SSH to an instance using floating
+                    IPs over the public network. 2) Connectivity between 2 instances
+                    over a private network.
+                dependencies:
+                    installer: ''
+                    scenario: '^((?!bgpvpn|odl_l3).)*$'
         ....
 
-At the end of the Functest environment creation, a file
-*/home/opnfv/functest/conf/testcase-list.txt* is created with the list of
-all the runnable tests.
-Functest considers the static constraints as regular expressions and compare them
-with the given scenario name.
-For instance, ODL suite can be run only on an scenario including 'odl' in its name.
-
-The order of execution is also described in the Functest configuration file::
-
- test_exec_priority:
-
-    1: vping_ssh
-    2: vping_userdata
-    3: tempest
-    4: odl
-    5: onos
-    6: ovno
-    7: doctor
-    8: promise
-    9: odl-vpnservice
-    10: bgpvpn
-    #11: openstack-neutron-bgpvpn-api-extension-tests
-    12: vims
-    13: rally
-
-The tests are executed in the following order:
-
-  1) vPing test cases
-  2) Tempest suite
-  3) SDN controller suites
-  4) Feature project tests cases (Promise, Doctor, BGPVPN...)
+We may distinguish 2 levels in the test case description:
+  * Tier level
+  * Test case level
+
+At the tier level, we define the following parameters:
+
+ * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
+ * description: a high level view of the test case
+
+For a given test case we defined:
+  * the name of the test case
+  * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
+  * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
+  * the description of the test case
+  * the dependencies: a combination of 2 regex on the scenario and the installer name
+
+The order of execution is the one defined in the file if all test cases are selected.
+
+In CI daily job the tests are executed in the following order:
+
+  1) healthcheck (blocking)
+  2) smoke: both vPings are blocking
+  3) SDN controller suites (blocking)
+  4) Feature project tests cases
+
+In CI weekly job we add 2 tiers:
+
   5) vIMS suite
   6) Rally suite
 
 As explained before, at the end of an automated execution, the OpenStack resources
 might be eventually removed.
+Please note that a system snapshot is taken before any test case execution.
 
-**END of TODO**
-
-
+This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.