Functest Userguide update for Colorado

Text updates to reflect the introduction of the Functest CLI for
Functest Docker container environment.

JIRA: FUNCTEST-238

Change-Id: Ifa742fb5c4a24ae94cc1e7062bcb8c0a2f49bf02
Signed-off-by: columgaynor <colum.gaynor@nokia.com>

1 files changed, 142 insertions, 104 deletions

diff --git a/docs/userguide/troubleshooting.rst b/docs/userguide/troubleshooting.rst
index 99b92977b..cd79bafcf 100644
--- a/docs/userguide/troubleshooting.rst
+++ b/docs/userguide/troubleshooting.rst
@@ -8,7 +8,8 @@ This section gives some guidelines about how to troubleshoot the test cases
 owned by Functest.
 
 **IMPORTANT**: As in the previous section, the steps defined below must be
-executed inside the Functest Docker container and after sourcing the OpenStack credentials::
+executed inside the Functest Docker container and after sourcing the OpenStack
+credentials::
 
     . $creds
 
@@ -19,7 +20,9 @@ or::
 VIM
 ---
 
-This section covers the test cases related to the VIM (vPing, Tempest, Rally).
+This section covers the test cases related to the VIM (healthcheck, vping_ssh,
+vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity,
+rally_full).
 
 vPing common
 ^^^^^^^^^^^^
@@ -29,29 +32,45 @@ similar:
     * Create Glance image
     * Create Network
     * Create Security Group
-    * Create instances
+    * Create Instances
 
-After these actions, the test cases differ and will be explained in their respective section.
+After these actions, the test cases differ and will be explained in their
+respective section.
 
-These test cases can be run inside the container as follows::
+These test cases can be run inside the container, using new Functest CLI as follows::
 
-    $repos_dir/functest/docker/run_tests.sh -t vping_ssh
-    $repos_dir/functest/docker/run_tests.sh -t vping_userdata
+    $ functest testcase run vping_ssh
+    $ functest testcase run vping_userdata
 
-The **run_tests.sh** script is basically calling internally the corresponding
-vPing scripts, located in
+The Functest CLI is designed to route a call to the corresponding internal
+python scripts, located in paths:
 *$repos_dir/functest/testcases/vPing/CI/libraries/vPing_ssh.py* and
-*$repos_dir/functest/testcases/vPing/CI/libraries/vPing_userdata.py* with the
-appropriate flags.
+*$repos_dir/functest/testcases/vPing/CI/libraries/vPing_userdata.py*
 
-After finishing the test execution, the corresponding script will remove all
-created resources in OpenStack (image, instances, network and security group).
-When troubleshooting, it is advisable sometimes to keep those resources in case the
-test fails and a manual testing is needed. This can be achieved by adding the flag *-n*::
+Notes:
+  #. In this Colorado Funtest Userguide, the use of the Functest CLI is
+     emphasized. The Functest CLI replaces the earlier Bash shell script
+     *run_tests.sh*.
 
-    $repos_dir/functest/docker/run_tests.sh -n -t vping_ssh
-    $repos_dir/functest/docker/run_tests.sh -n -t vping_userdata
+  #. There is one difference, between the Functest CLI based test case
+     execution compared to the earlier used Bash shell script, which is
+     relevant to point out in troubleshooting scenarios:
 
+         The Functest CLI does **not yet** support the option to suppress
+         clean-up of the generated OpenStack resources, following the execution
+         of a test case.
+
+     Explanation: After finishing the test execution, the corresponding
+     script will remove, by default, all created resources in OpenStack
+     (image, instances, network and security group). When troubleshooting,
+     it is advisable sometimes to keep those resources in case the test
+     fails and a manual testing is needed.
+
+     It is actually still possible to invoke test execution, with suppression
+     of OpenStack resource cleanup, however this requires invocation of a
+     **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'.
+     The `OPNFV Functest Developer Guide`_ provides guidance on the use of that
+     Python script in such troubleshooting cases.
 
 Some of the common errors that can appear in this test case are::
 
@@ -63,7 +82,8 @@ The debug messages will also help to see when it failed (subnet and router creat
 Example of Neutron commands (using 10.6.0.0/24 range for example)::
 
     neutron net-create net-test
-    neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 --gateway 10.6.0.254 net-test 10.6.0.0/24
+    neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
+    --gateway 10.6.0.254 net-test 10.6.0.0/24
     neutron router-create test_router
     neutron router-interface-add <ROUTER_ID> test_subnet
     neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
@@ -75,9 +95,12 @@ Another related error can occur while creating the Security Groups for the insta
 In this case, proceed to create it manually. These are some hints::
 
     neutron security-group-create sg-test
-    neutron security-group-rule-create sg-test --direction ingress --protocol icmp --remote-ip-prefix 0.0.0.0/0
-    neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
-    neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
+    neutron security-group-rule-create sg-test --direction ingress --protocol icmp \
+    --remote-ip-prefix 0.0.0.0/0
+    neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \
+    --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
+    neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \
+    --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
 
 The next step is to create the instances. The image used is located in
 */home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img* and a Glance image is created
@@ -100,23 +123,24 @@ vPing_SSH
 ^^^^^^^^^
 This test case creates a floating IP on the external network and assigns it to
 the second instance **opnfv-vping-2**. The purpose of this is to establish
-a SSH connection to that instance and SCP a script that will ping the first instance.
-This script is located in the repository under
-*$repos_dir/functest/testcases/vPing/CI/libraries/ping.sh* and takes an IP as
+a SSH connection to that instance and SCP a script that will ping the first
+instance. This script is located in the repository under
+*$repos_dir/functest/testcases/OpenStack/vPing/ping.sh* and takes an IP as
 a parameter. When the SCP is completed, the test will do an SSH call to that script
 inside the second instance. Some problems can happen here::
 
     vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
 
-If this is displayed, stop the test or wait for it to finish (if you have used the flag
-*-n* in **run_tests.sh** explained previously) so that the test does not clean
-the OpenStack resources. It means that the Container can not reach the public
-IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and
-they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios
-this has been noticed and it is a known limitation.
+If this is displayed, stop the test or wait for it to finish, if you have used
+the special method of test invocation with specific supression of OpenStack
+resource clean-up, as explained earler. It means that the Container can not
+reach the Public/External IP assigned to the instance **opnfv-vping-2**. There
+are many possible reasons, and they really depend on the chosen scenario. For
+most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known
+limitation.
 
-First, make sure that the instance **opnfv-vping-2** succeeded to get an IP from
-the DHCP agent. It can be checked by doing::
+First, make sure that the instance **opnfv-vping-2** succeeded to get an IP
+from the DHCP agent. It can be checked by doing::
 
     nova console-log opnfv-vping-2
 
@@ -132,10 +156,12 @@ If the instance got an IP properly, try to ping manually the VM from the contain
 
 If the ping does not return anything, try to ping from the Host where the Docker
 container is running. If that solves the problem, check the iptable rules because
-there might be some rules rejecting ICMP or TCP traffic coming/going from/to the container.
+there might be some rules rejecting ICMP or TCP traffic coming/going from/to the
+container.
 
 At this point, if the ping does not work either, try to reproduce the test
-manually with the steps described above in the vPing common section with the addition::
+manually with the steps described above in the vPing common section with the
+addition::
 
     neutron floatingip-create <EXT_NET_NAME>
     nova floating-ip-associate nova-test <FLOATING_IP>
@@ -187,8 +213,8 @@ If this text or similar is shown::
 it means that the instance failed to read from the metadata service. Contact
 the Functest or installer teams for more information.
 
-NOTE: Cloud-init in not supported on scenario dealing with ONOS and the tests have been
-excluded from CI in those scenarios.
+NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests
+have been excluded from CI in those scenarios.
 
 
 Tempest
@@ -198,65 +224,65 @@ In the upstream OpenStack CI all the Tempest test cases are supposed to pass.
 If some test cases fail in an OPNFV deployment, the reason is very probably one
 of the following
 
-+-----------------------------+------------------------------------------------+
-| Error                       | Details                                        |
-+=============================+================================================+
-| Resources required for test | Such resources could be e.g. an external       |
-| case execution are missing  | network and access to the management subnet    |
-|                             | (adminURL) from the Functest docker container. |
-+-----------------------------+------------------------------------------------+
-| OpenStack components or     | Check running services in the controller and   |
-| services are missing or not | compute nodes (e.g. with "systemctl" or        |
-| configured properly         | "service" commands). Configuration parameters  |
-|                             | can be verified from related .conf files       |
-|                             | located under /etc/<component> directories.    |
-+-----------------------------+------------------------------------------------+
-| Some resources required for | The tempest.conf file, automatically generated |
-| execution test cases are    | by Rally in Functest, does not contain all the |
-| missing                     | needed parameters or some parameters are not   |
-|                             | set properly.                                  |
-|                             | The tempest.conf file is located in /home/opnfv|
-|                             | /.rally/tempest/for-deployment-<UUID> in       |
-|                             | Functest container                             |
-|                             | Use "rally deployment list" command in order to|
-|                             | check UUID of current deployment.              |
-+-----------------------------+------------------------------------------------+
-
-
-When some Tempest test case fails, captured traceback and possibly also related
-REST API requests/responses are output to the console.
-More detailed debug information can be found from tempest.log file stored into
-related Rally deployment folder.
++-----------------------------+-----------------------------------------------------+
+| Error                       | Details                                             |
++=============================+=====================================================+
+| Resources required for test | Such resources could be e.g. an external network    |
+| case execution are missing  | and access to the management subnet (adminURL) from |
+|                             | the Functest docker container.                      |
++-----------------------------+-----------------------------------------------------+
+| OpenStack components or     | Check running services in the controller and compute|
+| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).|
+| configured properly         | Configuration parameters can be verified from the   |
+|                             | related .conf files located under '/etc/<component>'|
+|                             | directories.                                        |
++-----------------------------+-----------------------------------------------------+
+| Some resources required for | The tempest.conf file, automatically generated by   |
+| execution test cases are    | Rally in Functest, does not contain all the needed  |
+| missing                     | parameters or some parameters are not set properly. |
+|                             | The tempest.conf file is located in directory       |
+|                             | '/home/opnfv/.rally/tempest/for-deployment-<UUID>'  |
+|                             | in the Functest Docker container. Use the "rally    |
+|                             | deployment list" command in order to check the UUID |
+|                             | the UUID of the current deployment.                 |
++-----------------------------+-----------------------------------------------------+
+
+
+When some Tempest test case fails, captured traceback and possibly also the
+related REST API requests/responses are output to the console. More detailed debug
+information can be found from tempest.log file stored into related Rally deployment
+folder.
 
 
 Rally
 ^^^^^
 
-Same error causes than for Tempest mentioned above may lead to errors in Rally.
+The same error causes which were mentioned above for Tempest test cases, may also
+lead to errors in Rally as well.
 
 It is possible to run only one Rally scenario, instead of the whole suite.
-To do that, call the python script (instead of *run_tests.sh*) as follows::
+To do that, call the alternative python script as follows::
 
-    python $repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/run_rally-cert.py -h
-    usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name
+  python $repos_dir/functest/testcases/OpenStack/rally/run_rally-cert.py -h
+  usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name
 
-    positional arguments:
-      test_name      Module name to be tested. Possible values are : [
-                     authenticate | glance | cinder | heat | keystone | neutron |
-                     nova | quotas | requests | vm | all ] The 'all' value
-                     performs all possible test scenarios
+  positional arguments:
+    test_name      Module name to be tested. Possible values are : [
+                   authenticate | glance | cinder | heat | keystone | neutron |
+                   nova | quotas | requests | vm | all ] The 'all' value
+                   performs all possible test scenarios
 
-    optional arguments:
-      -h, --help     show this help message and exit
-      -d, --debug    Debug mode
-      -r, --report   Create json result file
-      -s, --smoke    Smoke test mode
-      -v, --verbose  Print verbose info about the progress
-      -n, --noclean  Don't clean the created resources for this test.
+  optional arguments:
+    -h, --help     show this help message and exit
+    -d, --debug    Debug mode
+    -r, --report   Create json result file
+    -s, --smoke    Smoke test mode
+    -v, --verbose  Print verbose info about the progress
+    -n, --noclean  Don't clean the created resources for this test.
 
 For example, to run the Glance scenario with debug information::
 
-    python $repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/run_rally-cert.py -d glance
+  python $repos_dir/functest/testcases/OpenStack/rally/run_rally-cert.py -d glance
 
 Possible scenarios are:
  * authenticate
@@ -270,16 +296,17 @@ Possible scenarios are:
  * requests
  * vm
 
-To know more about what those scenarios are doing, they are defined in:
-*$repos_dir/functest/testcases/VIM/OpenStack/CI/rally_cert/scenario*. For more info about
-Rally scenario definition please refer to the Rally official documentation.
+To know more about what those scenarios are doing, they are defined in directory:
+*$repos_dir/functest/testcases/OpenStack/rally/scenario*
+For more info about Rally scenario definition please refer to the Rally official
+documentation. `[3]`_
 
 If the flag *all* is specified, it will run all the scenarios one by one. Please
-note that this might take some time (~1,5hr), taking around 1 hour to complete the
-Nova scenario.
+note that this might take some time (~1,5hr), taking around 1 hour alone to
+complete the Nova scenario.
 
-To check any possible problems with rally, the logs are stored under
-*/home/opnfv/functest/results/rally/* in the Functest container.
+To check any possible problems with Rally, the logs are stored under
+*/home/opnfv/functest/results/rally/* in the Functest Docker container.
 
 
 Controllers
@@ -287,7 +314,7 @@ Controllers
 
 ODL
 ^^^
-2 versions are supported in Brahmaputra depending on the scenario:
+2 versions are supported in Colorado, depending on the scenario:
  * Lithium
  * Berylium
 
@@ -298,10 +325,30 @@ due to wrong return code.
 ONOS
 ^^^^
 
-Please refer to the ONOS documentation.
+Please refer to the ONOS documentation. `ONOSFW User Guide`_ .
+
+
+Features
+--------
+
+
+Doctor
+^^^^^^
+Please refer to the Doctor documentation. `Doctor User Guide`_
+
+
+Promise
+^^^^^^^
+Please refer to the Promise documentation. `Promise User Guide`_
+
 
-Feature
--------
+bgpvpn
+^^^^^^
+Please refer to the SNVPN documentation. `SDNVPN User Guide`_
+
+
+NFV
+---
 
 vIMS
 ^^^^
@@ -326,7 +373,7 @@ described in the following table:
 | Impossible to create a volume     | VNF cannot be deployed             |
 +-----------------------------------+------------------------------------+
 | SSH connection issue between the  | if vPing test fails, vIMS test will|
-| Test container and the VM         | fail...                            |
+| Test Docker container and the VM  | fail...                            |
 +-----------------------------------+------------------------------------+
 | No Internet access from the VM    | the VMs of the VNF must have an    |
 |                                   | external access to Internet        |
@@ -336,13 +383,4 @@ described in the following table:
 +-----------------------------------+------------------------------------+
 
 
-Promise
-^^^^^^^
-
-Please refer to the Promise documentation.
-
-
-SDNVPN
-^^^^^^^
-
-Please refer to the SNVPN documentation.
+.. _`OPNFV Functest Developer Guide`:  http://artifacts.opnfv.org/functest/docs/devguide/#