Restructure docs

DOCS directory restructured according to:
https://wiki.opnfv.org/display/DOC/Documentation+Guide

Now:
 - release
    - release notes

 - testing
    - developer
       - dev guide (this is to be released on docs.opnfv.org)
       - internship (docs about intern projects)

    - user (this is to be released on docs.opnfv.org)
       - config guide
       - user guide

Change-Id: I1851189601aac3c5989f19f99d779efe23dbf3d1
Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>

1 files changed, 385 insertions, 0 deletions

diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst
new file mode 100644
index 00000000..e7ab84b2
--- /dev/null
+++ b/docs/testing/user/userguide/runfunctest.rst
@@ -0,0 +1,385 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Executing the functest suites
+=============================
+
+Manual testing
+--------------
+
+This section assumes the following:
+ * The Functest Docker container is running
+ * The docker prompt is shown
+ * The Functest environment is ready (Functest CLI command 'functest env prepare'
+   has been executed)
+
+If any of the above steps are missing please refer to the Functest Config Guide
+as they are a prerequisite and all the commands explained in this section **must** be
+performed **inside the container**.
+
+The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
+for the execution of Test Tiers or Test Cases::
+
+  root@22e436918db0:~/repos/functest/ci# functest tier --help
+  Usage: functest tier [OPTIONS] COMMAND [ARGS]...
+
+  Options:
+    -h, --help  Show this message and exit.
+
+  Commands:
+    get-tests  Prints the tests in a tier.
+    list       Lists the available tiers.
+    run        Executes all the tests within a tier.
+    show       Shows information about a tier.
+  root@22e436918db0:~/repos/functest/ci# functest testcase --help
+
+  Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
+
+  Options:
+    -h, --help  Show this message and exit.
+
+  Commands:
+    list  Lists the available testcases.
+    run   Executes a test case.
+    show  Shows information about a test case.
+
+More details on the existing Tiers and Test Cases can be seen with the 'list'
+command::
+
+  root@22e436918db0:~/repos/functest/ci# functest tier list
+      - 0. healthcheck:
+             ['healthcheck', 'connection_check', 'api_check',]
+      - 1. smoke:
+             ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl']
+      - 2. features:
+             ['doctor', 'security_scan']
+      - 3. components:
+             ['tempest_full_parallel', 'rally_full']
+      - 4. vnf:
+             ['cloudify_ims']
+
+  and
+
+  root@22e436918db0:~/repos/functest/ci# functest testcase list
+  healthcheck
+  api_check
+  connection_check
+  vping_ssh
+  vping_userdata
+  snaps_smoke
+  tempest_smoke_serial
+  rally_sanity
+  odl
+  doctor
+  security_scan
+  tempest_full_parallel
+  rally_full
+  cloudify_ims
+
+More specific details on specific Tiers or Test Cases can be seen wih the
+'show' command::
+
+  root@22e436918db0:~/repos/functest/ci# functest tier show smoke
+  +======================================================================+
+  | Tier:  smoke                                                         |
+  +======================================================================+
+  | Order: 1                                                             |
+  | CI Loop: (daily)|(weekly)                                            |
+  | Description:                                                         |
+  |    Set of basic Functional tests to validate the OpenStack           |
+  |    deployment.                                                       |
+  | Test cases:                                                          |
+  |    - vping_ssh                                                       |
+  |    - vping_userdata                                                  |
+  |    - tempest_smoke_serial                                            |
+  |    - rally_sanity                                                    |
+  |                                                                      |
+  +----------------------------------------------------------------------+
+
+  and
+
+  root@22e436918db0:~/repos/functest/ci# functest testcase  show tempest_smoke_serial
+  +======================================================================+
+  | Testcase:  tempest_smoke_serial                                      |
+  +======================================================================+
+  | Description:                                                         |
+  |    This test case runs the smoke subset of the OpenStack Tempest     |
+  |    suite. The list of test cases is generated by Tempest             |
+  |    automatically and depends on the parameters of the OpenStack      |
+  |    deplopyment.                                                      |
+  | Dependencies:                                                        |
+  |   - Installer:                                                       |
+  |   - Scenario :                                                       |
+  |                                                                      |
+  +----------------------------------------------------------------------+
+
+
+To execute a Test Tier or Test Case, the 'run' command is used::
+
+  root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
+  Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
+  2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
+  2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
+  2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
+  2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
+  2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
+  2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
+  2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
+  2016-06-30 11:44:56,953 - healtcheck - INFO -  Testing Keystone API...
+  2016-06-30 11:45:05,351 - healtcheck - INFO -  ...Keystone OK!
+  2016-06-30 11:45:05,354 - healtcheck - INFO -  Testing Glance API...
+  2016-06-30 11:45:29,746 - healtcheck - INFO -  ... Glance OK!
+  2016-06-30 11:45:29,749 - healtcheck - INFO -  Testing Cinder API...
+  2016-06-30 11:45:37,502 - healtcheck - INFO -  ...Cinder OK!
+  2016-06-30 11:45:37,505 - healtcheck - INFO -  Testing Neutron API...
+  2016-06-30 11:45:39,664 - healtcheck - INFO -  External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
+  2016-06-30 11:45:39,667 - healtcheck - INFO -  1. Create Networks...
+  2016-06-30 11:45:44,227 - healtcheck - INFO -  2. Create subnets...
+  2016-06-30 11:45:46,805 - healtcheck - INFO -  4. Create Routers...
+  2016-06-30 11:45:54,261 - healtcheck - INFO -  ...Neutron OK!
+  2016-06-30 11:45:54,264 - healtcheck - INFO -  Testing Nova API...
+  2016-06-30 11:47:12,272 - healtcheck - INFO -  ...Nova OK!
+  2016-06-30 11:47:12,274 - healtcheck - INFO -  Checking if instances get an IP from DHCP...
+  :
+  :
+  2016-06-30 11:48:17,832 - healtcheck - INFO -  ...DHCP OK!
+  2016-06-30 11:48:17,835 - healtcheck - INFO -  Health check passed!
+  2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+  2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
+  2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+  Version 1 is deprecated, use alternative version 2 instead.
+  WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
+  2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
+  2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
+  2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
+  2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
+  2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
+  2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
+  2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
+  2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
+  2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
+  2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
+
+  and
+
+  root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
+  Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
+  2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
+  2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
+  2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
+  2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
+  2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'...
+  2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
+  2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group  'vPing-sg'...
+  2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
+  2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
+  2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
+  2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
+  2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
+  2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
+  2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
+  2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
+  2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
+  2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
+  2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
+  2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
+  2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
+  2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
+  2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
+  2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
+  2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+  2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
+  2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+  Version 1 is deprecated, use alternative version 2 instead.
+  :
+  :
+  etc.
+
+To list the test cases which are part of a specific Test Tier, the 'get-tests'
+command is used with 'functest tier'::
+
+  root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
+  Test cases in tier 'healthcheck':
+   ['healthcheck']
+
+
+Please note that for some scenarios some test cases might not be launched.
+For example, the last example displayed only the 'odl' testcase for the given
+environment. In this particular system the deployment does not support the 'onos' SDN
+Controller Test Case; for example.
+
+**Important** If you use the command 'functest tier run <tier_name>', then the
+Functest CLI utility will call **all valid Test Cases**, which belong to the
+specified Test Tier, as relevant to scenarios deployed to the SUT environment.
+Thus, the Functest CLI utility calculates automatically which tests can be
+executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
+which is passed in to the Functest docker container.
+
+Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
+two possibilities::
+
+ *  Run a single Test Case, specified by a valid choice of <testcase_name>
+ *  Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
+
+Functest includes a cleaning mechanism in order to remove all the OpenStack
+resources except those present before running any test. The script
+*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
+the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
+all the OpenStack resources (images, networks, volumes, security groups, tenants,
+users) so that an eventual cleanup does not remove any of these defaults.
+
+The script **clean_openstack.py** which is located in
+*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
+in charge of cleaning the OpenStack resources that are not specified in the
+defaults file generated previously which is stored in
+*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
+
+It is important to mention that if there are new OpenStack resources created
+manually after preparing the Functest environment, they will be removed, unless
+you use the special method of invoking the test case with specific suppression
+of clean up. (See the `Troubleshooting`_ section).
+
+The reason to include this cleanup meachanism in Functest is because some
+test suites such as Tempest or Rally create a lot of resources (users,
+tenants, networks, volumes etc.) that are not always properly cleaned, so this
+function has been set to keep the system as clean as it was before a
+full Functest execution.
+
+Although the Functest CLI provides an easy way to run any test, it is possible to
+do a direct call to the desired test script. For example:
+
+    python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
+
+
+Automated testing
+-----------------
+
+As mentioned previously, the Functest Docker container preparation as well as
+invocation of Test Cases can be called within the container from the Jenkins CI
+system. There are 3 jobs that automate the whole process. The first job runs all
+the tests referenced in the daily loop (i.e. that must been run daily), the second
+job runs the tests referenced in the weekly loop (usually long duration tests run
+once a week maximum) and the third job allows testing test suite by test suite specifying
+the test suite name. The user may also use either of these Jenkins jobs to execute
+the desired test suites.
+
+One of the most challenging task in the Danube release consists
+in dealing with lots of scenarios and installers. Thus, when the tests are
+automatically started from CI, a basic algorithm has been created in order to
+detect whether a given test is runnable or not on the given scenario.
+Some Functest test suites cannot be systematically run (e.g. ODL suite can not
+be run on an ONOS scenario). The daily/weekly notion has been introduces in
+Colorado in order to save CI time and avoid running systematically
+long duration tests. It was not used in Colorado due to CI resource shortage.
+The mechanism remains however as part of the CI evolution.
+
+CI provides some useful information passed to the container as environment
+variables:
+
+ * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE
+ * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
+ * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
+
+   * controller = (odl|onos|ocl|nosdn)
+   * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites)
+   * mode = (ha|noha)
+
+The constraints per test case are defined in the Functest configuration file
+*/home/opnfv/repos/functest/functest/ci/testcases.yaml*::
+
+ tiers:
+    -
+        name: healthcheck
+        order: 0
+        ci_loop: '(daily)|(weekly)'
+        description : >-
+            First tier to be executed to verify the basic
+            operations in the VIM.
+        testcases:
+            -
+                name: healthcheck
+                criteria: 'status == "PASS"'
+                blocking: true
+                description: >-
+                    This test case verifies the basic OpenStack services like
+                    Keystone, Glance, Cinder, Neutron and Nova.
+
+                dependencies:
+                    installer: ''
+                    scenario: ''
+
+   -
+        name: smoke
+        order: 1
+        ci_loop: '(daily)|(weekly)'
+        description : >-
+            Set of basic Functional tests to validate the OpenStack deployment.
+        testcases:
+            -
+                name: vping_ssh
+                criteria: 'status == "PASS"'
+                blocking: true
+                description: >-
+                    This test case verifies: 1) SSH to an instance using floating
+                    IPs over the public network. 2) Connectivity between 2 instances
+                    over a private network.
+                dependencies:
+                    installer: ''
+                    scenario: '^((?!bgpvpn|odl_l3).)*$'
+                run:
+                    module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
+                    class: 'VPingSSH'
+        ....
+
+We may distinguish 2 levels in the test case description:
+  * Tier level
+  * Test case level
+
+At the tier level, we define the following parameters:
+
+ * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
+ * description: a high level view of the test case
+
+For a given test case we defined:
+  * the name of the test case
+  * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
+  * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
+  * the description of the test case
+  * the dependencies: a combination of 2 regex on the scenario and the installer name
+  * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
+
+For further details on abstraction classes, see developper guide.
+
+Additional parameters have been added in the desription in the Database.
+The target is to use the configuration stored in the Database and consider the
+local file as backup if the Database is not reachable.
+The additional fields related to a test case are:
+  * trust: we introduced this notion to put in place a mechanism of scenario promotion.
+  * Version: it indicates since which version you can run this test
+  * domains: the main domain covered by the test suite
+  * tags: a list of tags related to the test suite
+
+The order of execution is the one defined in the file if all test cases are selected.
+
+In CI daily job the tests are executed in the following order:
+
+  1) healthcheck (blocking)
+  2) smoke: both vPings are blocking
+  3) Feature project tests cases
+
+In CI weekly job we add 2 tiers:
+
+  4) VNFs (vIMS)
+  5) Components (Rally and Tempest long duration suites)
+
+As explained before, at the end of an automated execution, the OpenStack resources
+might be eventually removed.
+Please note that a system snapshot is taken before any test case execution.
+
+This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.