Updates all testcases and their results

Change-Id: I127146b4d6face1b6221e049636a29e601415fd8
Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
(cherry picked from commit 37251af62ca89b55f09dfd6ddce15ee179c580f3)

3 files changed, 118 insertions, 98 deletions

diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index 2f8b3e28d..5c668dcc5 100644
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -32,21 +32,26 @@ The internal test cases are:
  * vping_ssh
  * vping_userdata
  * cinder_test
- * odl
  * tempest_smoke
+ * odl
  * neutron-tempest-plugin-api
  * rally_sanity
  * refstack_compute
  * refstack_object
  * refstack_platform
+ * tempest_full
+ * tempest_scenario
+ * tempest_slow
  * patrole
+ * barbican
  * neutron_trunk
  * networking-bgpvpn
  * networking-sfc
- * barbican
- * tempest_full
- * tempest_scenario
+ * octavia
  * rally_full
+ * rally_jobs
+ * vmtp
+ * shaker
  * cloudify
  * cloudify_ims
  * heat_ims
@@ -59,6 +64,7 @@ Kubernetes
 The internal test cases are:
 
  * k8s_smoke
+ * xrally_kubernetes
  * k8s_conformance
 
 Release Data
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index c50fd1f1f..cb722bc2e 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -29,6 +29,9 @@ cat env::
   NAMESERVER=XXX  # if not 8.8.8.8
   EXTERNAL_NETWORK=XXX # if not first network with router:external=True
   NEW_USER_ROLE=XXX # if not member
+  SDN_CONTROLLER_IP=XXX  # if odl scenario
+  VOLUME_DEVICE_NAME=XXX  # if not vdb
+  FLAVOR_EXTRA_SPECS=hw:mem_page_size:large    # if fdio scenarios
 
 See section on environment variables for details.
 
@@ -41,6 +44,7 @@ cat env_file::
   export OS_PROJECT_NAME=XXX
   export OS_PASSWORD=XXX
   export OS_IDENTITY_API_VERSION=3
+  export OS_REGION_NAME=XXX
 
 See section on OpenStack credentials for details.
 
@@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume)::
   images/cirros-0.4.0-aarch64-disk.img
   images/cirros-0.4.0-x86_64-disk.img
   images/cloudify-docker-manager-community-19.01.24.tar
-  images/shaker-image-arm64.qcow2
-  images/shaker-image.qcow2
+  images/shaker-image-1.3.0+stretch.qcow2
   images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
   images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
   images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
@@ -73,17 +76,17 @@ Results shall be displayed as follows::
   +--------------------------+------------------+---------------------+------------------+----------------+
   |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
   +--------------------------+------------------+---------------------+------------------+----------------+
-  |     connection_check     |     functest     |     healthcheck     |      00:01       |      PASS      |
+  |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
   |      tenantnetwork1      |     functest     |     healthcheck     |      00:06       |      PASS      |
-  |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |      tenantnetwork2      |     functest     |     healthcheck     |      00:08       |      PASS      |
   |         vmready1         |     functest     |     healthcheck     |      00:08       |      PASS      |
-  |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
-  |        singlevm1         |     functest     |     healthcheck     |      00:35       |      PASS      |
-  |        singlevm2         |     functest     |     healthcheck     |      00:38       |      PASS      |
-  |        vping_ssh         |     functest     |     healthcheck     |      00:47       |      PASS      |
-  |      vping_userdata      |     functest     |     healthcheck     |      00:33       |      PASS      |
-  |       cinder_test        |     functest     |     healthcheck     |      01:16       |      PASS      |
-  |      tempest_smoke       |     functest     |     healthcheck     |      07:05       |      PASS      |
+  |         vmready2         |     functest     |     healthcheck     |      00:10       |      PASS      |
+  |        singlevm1         |     functest     |     healthcheck     |      00:31       |      PASS      |
+  |        singlevm2         |     functest     |     healthcheck     |      00:29       |      PASS      |
+  |        vping_ssh         |     functest     |     healthcheck     |      00:37       |      PASS      |
+  |      vping_userdata      |     functest     |     healthcheck     |      00:36       |      PASS      |
+  |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
+  |      tempest_smoke       |     functest     |     healthcheck     |      04:58       |      PASS      |
   |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
   +--------------------------+------------------+---------------------+------------------+----------------+
 
@@ -104,18 +107,20 @@ Results shall be displayed as follows::
   +------------------------------------+------------------+---------------+------------------+----------------+
   |             TEST CASE              |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
   +------------------------------------+------------------+---------------+------------------+----------------+
-  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      13:50       |      PASS      |
-  |            rally_sanity            |     functest     |     smoke     |      14:13       |      PASS      |
-  |          refstack_compute          |     functest     |     smoke     |      07:24       |      PASS      |
-  |          refstack_object           |     functest     |     smoke     |      02:08       |      PASS      |
-  |         refstack_platform          |     functest     |     smoke     |      08:32       |      PASS      |
-  |            tempest_full            |     functest     |     smoke     |      58:40       |      PASS      |
-  |          tempest_scenario          |     functest     |     smoke     |      09:47       |      PASS      |
-  |              patrole               |     functest     |     smoke     |      05:34       |      PASS      |
+  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      10:35       |      PASS      |
+  |            rally_sanity            |     functest     |     smoke     |      17:11       |      PASS      |
+  |          refstack_compute          |     functest     |     smoke     |      06:09       |      PASS      |
+  |          refstack_object           |     functest     |     smoke     |      01:48       |      PASS      |
+  |         refstack_platform          |     functest     |     smoke     |      05:53       |      PASS      |
+  |            tempest_full            |     functest     |     smoke     |      30:02       |      PASS      |
+  |          tempest_scenario          |     functest     |     smoke     |      09:26       |      PASS      |
+  |            tempest_slow            |     functest     |     smoke     |      18:53       |      PASS      |
+  |              patrole               |     functest     |     smoke     |      02:28       |      PASS      |
+  |              barbican              |     functest     |     smoke     |      02:13       |      PASS      |
   |           neutron_trunk            |     functest     |     smoke     |      00:00       |      SKIP      |
-  |         networking-bgpvpn          |     functest     |     smoke     |      01:43       |      PASS      |
-  |           networking-sfc           |     functest     |     smoke     |      04:49       |      PASS      |
-  |              barbican              |     functest     |     smoke     |      05:09       |      PASS      |
+  |         networking-bgpvpn          |     functest     |     smoke     |      00:00       |      SKIP      |
+  |           networking-sfc           |     functest     |     smoke     |      00:00       |      SKIP      |
+  |              octavia               |     functest     |     smoke     |      00:00       |      SKIP      |
   +------------------------------------+------------------+---------------+------------------+----------------+
 
 Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -136,10 +141,10 @@ Results shall be displayed as follows::
   +--------------------+------------------+----------------------+------------------+----------------+
   |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
   +--------------------+------------------+----------------------+------------------+----------------+
-  |     rally_full     |     functest     |     benchmarking     |      121:55      |      PASS      |
-  |     rally_jobs     |     functest     |     benchmarking     |      24:21       |      PASS      |
-  |        vmtp        |     functest     |     benchmarking     |      00:00       |      SKIP      |
-  |       shaker       |     functest     |     benchmarking     |      00:00       |      SKIP      |
+  |     rally_full     |     functest     |     benchmarking     |      92:42       |      PASS      |
+  |     rally_jobs     |     functest     |     benchmarking     |      18:44       |      PASS      |
+  |        vmtp        |     functest     |     benchmarking     |      15:20       |      PASS      |
+  |       shaker       |     functest     |     benchmarking     |      23:54       |      PASS      |
   +--------------------+------------------+----------------------+------------------+----------------+
 
 Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -160,11 +165,11 @@ Results shall be displayed as follows::
   +----------------------+------------------+--------------+------------------+----------------+
   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
   +----------------------+------------------+--------------+------------------+----------------+
-  |       cloudify       |     functest     |     vnf      |      06:42       |      PASS      |
-  |     cloudify_ims     |     functest     |     vnf      |      39:21       |      PASS      |
-  |       heat_ims       |     functest     |     vnf      |      34:33       |      PASS      |
-  |     vyos_vrouter     |     functest     |     vnf      |      20:20       |      PASS      |
-  |       juju_epc       |     functest     |     vnf      |      56:55       |      PASS      |
+  |       cloudify       |     functest     |     vnf      |      03:50       |      PASS      |
+  |     cloudify_ims     |     functest     |     vnf      |      22:34       |      PASS      |
+  |       heat_ims       |     functest     |     vnf      |      32:39       |      PASS      |
+  |     vyos_vrouter     |     functest     |     vnf      |      15:10       |      PASS      |
+  |       juju_epc       |     functest     |     vnf      |      37:03       |      PASS      |
   +----------------------+------------------+--------------+------------------+----------------+
 
 Functest Dockers for Kubernetes deployment
@@ -199,7 +204,7 @@ Results shall be displayed as follows::
   +-------------------+------------------+---------------------+------------------+----------------+
   |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
   +-------------------+------------------+---------------------+------------------+----------------+
-  |     k8s_smoke     |     functest     |     healthcheck     |      02:27       |      PASS      |
+  |     k8s_smoke     |     functest     |     healthcheck     |      01:44       |      PASS      |
   +-------------------+------------------+---------------------+------------------+----------------+
 
 Testing smoke suite
@@ -213,11 +218,12 @@ Run smoke suite::
 
 Results shall be displayed as follows::
 
-  +-------------------------+------------------+---------------+------------------+----------------+
-  |        TEST CASE        |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
-  +-------------------------+------------------+---------------+------------------+----------------+
-  |     k8s_conformance     |     functest     |     smoke     |      57:14       |      PASS      |
-  +-------------------------+------------------+---------------+------------------+----------------+
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |     xrally_kubernetes     |     functest     |     smoke     |      16:12       |      PASS      |
+  |      k8s_conformance      |     functest     |     smoke     |      149:59      |      PASS      |
+  +---------------------------+------------------+---------------+------------------+----------------+
 
 Environment variables
 =====================
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index 8023c4661..457d7a1d3 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -21,75 +21,83 @@ end of each suite and can be described as follow.
 
 Healthcheck suite::
 
- +----------------------------+------------------+---------------------+------------------+----------------+
- |         TEST CASE          |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
- +----------------------------+------------------+---------------------+------------------+----------------+
- |      connection_check      |     functest     |     healthcheck     |      00:07       |      PASS      |
- |         api_check          |     functest     |     healthcheck     |      07:46       |      PASS      |
- |     snaps_health_check     |     functest     |     healthcheck     |      00:36       |      PASS      |
- +----------------------------+------------------+---------------------+------------------+----------------+
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+  +--------------------------+------------------+---------------------+------------------+----------------+
+  |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
+  |      tenantnetwork1      |     functest     |     healthcheck     |      00:06       |      PASS      |
+  |      tenantnetwork2      |     functest     |     healthcheck     |      00:08       |      PASS      |
+  |         vmready1         |     functest     |     healthcheck     |      00:08       |      PASS      |
+  |         vmready2         |     functest     |     healthcheck     |      00:10       |      PASS      |
+  |        singlevm1         |     functest     |     healthcheck     |      00:31       |      PASS      |
+  |        singlevm2         |     functest     |     healthcheck     |      00:29       |      PASS      |
+  |        vping_ssh         |     functest     |     healthcheck     |      00:37       |      PASS      |
+  |      vping_userdata      |     functest     |     healthcheck     |      00:36       |      PASS      |
+  |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
+  |      tempest_smoke       |     functest     |     healthcheck     |      04:58       |      PASS      |
+  |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
+  +--------------------------+------------------+---------------------+------------------+----------------+
 
 Smoke suite::
 
- +------------------------------+------------------+---------------+------------------+----------------+
- |          TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
- +------------------------------+------------------+---------------+------------------+----------------+
- |          vping_ssh           |     functest     |     smoke     |      00:57       |      PASS      |
- |        vping_userdata        |     functest     |     smoke     |      00:33       |      PASS      |
- |     tempest_smoke_serial     |     functest     |     smoke     |      13:22       |      PASS      |
- |         rally_sanity         |     functest     |     smoke     |      24:07       |      PASS      |
- |       refstack_defcore       |     functest     |     smoke     |      05:21       |      PASS      |
- |           patrole            |     functest     |     smoke     |      04:29       |      PASS      |
- |         snaps_smoke          |     functest     |     smoke     |      46:54       |      PASS      |
- |             odl              |     functest     |     smoke     |      00:00       |      SKIP      |
- |        neutron_trunk         |     functest     |     smoke     |      00:00       |      SKIP      |
- +------------------------------+------------------+---------------+------------------+----------------+
-
-Components suite::
-
- +-------------------------------+------------------+--------------------+------------------+----------------+
- |           TEST CASE           |     PROJECT      |        TIER        |     DURATION     |     RESULT     |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- |     tempest_full_parallel     |     functest     |     components     |      48:28       |      PASS      |
- |           rally_full          |     functest     |     components     |      126:02      |      PASS      |
- +-------------------------------+------------------+--------------------+------------------+----------------+
+  +------------------------------------+------------------+---------------+------------------+----------------+
+  |             TEST CASE              |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+  +------------------------------------+------------------+---------------+------------------+----------------+
+  |     neutron-tempest-plugin-api     |     functest     |     smoke     |      10:35       |      PASS      |
+  |            rally_sanity            |     functest     |     smoke     |      17:11       |      PASS      |
+  |          refstack_compute          |     functest     |     smoke     |      06:09       |      PASS      |
+  |          refstack_object           |     functest     |     smoke     |      01:48       |      PASS      |
+  |         refstack_platform          |     functest     |     smoke     |      05:53       |      PASS      |
+  |            tempest_full            |     functest     |     smoke     |      30:02       |      PASS      |
+  |          tempest_scenario          |     functest     |     smoke     |      09:26       |      PASS      |
+  |            tempest_slow            |     functest     |     smoke     |      18:53       |      PASS      |
+  |              patrole               |     functest     |     smoke     |      02:28       |      PASS      |
+  |              barbican              |     functest     |     smoke     |      02:13       |      PASS      |
+  |           neutron_trunk            |     functest     |     smoke     |      00:00       |      SKIP      |
+  |         networking-bgpvpn          |     functest     |     smoke     |      00:00       |      SKIP      |
+  |           networking-sfc           |     functest     |     smoke     |      00:00       |      SKIP      |
+  |              octavia               |     functest     |     smoke     |      00:00       |      SKIP      |
+  +------------------------------------+------------------+---------------+------------------+----------------+
+
+Benchmarking suite::
+
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
+  +--------------------+------------------+----------------------+------------------+----------------+
+  |     rally_full     |     functest     |     benchmarking     |      92:42       |      PASS      |
+  |     rally_jobs     |     functest     |     benchmarking     |      18:44       |      PASS      |
+  |        vmtp        |     functest     |     benchmarking     |      15:20       |      PASS      |
+  |       shaker       |     functest     |     benchmarking     |      23:54       |      PASS      |
+  +--------------------+------------------+----------------------+------------------+----------------+
 
 Vnf suite::
 
- +----------------------+------------------+--------------+------------------+----------------+
- |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
- +----------------------+------------------+--------------+------------------+----------------+
- |     cloudify_ims     |     functest     |     vnf      |      28:15       |      PASS      |
- |     vyos_vrouter     |     functest     |     vnf      |      17:59       |      PASS      |
- |       juju_epc       |     functest     |     vnf      |      46:44       |      PASS      |
- +----------------------+------------------+--------------+------------------+----------------+
-
-Functest Kubernetes test result::
-
- +--------------------------------------+------------------------------------------------------------+
- |               ENV VAR                |                           VALUE                            |
- +--------------------------------------+------------------------------------------------------------+
- |            INSTALLER_TYPE            |                          compass                           |
- |           DEPLOY_SCENARIO            |                   k8-nosdn-nofeature-ha                    |
- |              BUILD_TAG               |     jenkins-functest-compass-baremetal-daily-master-75     |
- |               CI_LOOP                |                           daily                            |
- +--------------------------------------+------------------------------------------------------------+
+  +----------------------+------------------+--------------+------------------+----------------+
+  |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
+  +----------------------+------------------+--------------+------------------+----------------+
+  |       cloudify       |     functest     |     vnf      |      03:50       |      PASS      |
+  |     cloudify_ims     |     functest     |     vnf      |      22:34       |      PASS      |
+  |       heat_ims       |     functest     |     vnf      |      32:39       |      PASS      |
+  |     vyos_vrouter     |     functest     |     vnf      |      15:10       |      PASS      |
+  |       juju_epc       |     functest     |     vnf      |      37:03       |      PASS      |
+  +----------------------+------------------+--------------+------------------+----------------+
 
 Kubernetes healthcheck suite::
 
- +-------------------+------------------+---------------------+------------------+----------------+
- |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
- +-------------------+------------------+---------------------+------------------+----------------+
- |     k8s_smoke     |     functest     |     healthcheck     |      01:54       |      PASS      |
- +-------------------+------------------+---------------------+------------------+----------------+
+  +-------------------+------------------+---------------------+------------------+----------------+
+  |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
+  +-------------------+------------------+---------------------+------------------+----------------+
+  |     k8s_smoke     |     functest     |     healthcheck     |      01:44       |      PASS      |
+  +-------------------+------------------+---------------------+------------------+----------------+
 
 Kubernetes smoke suite::
 
- +-------------------------+------------------+---------------+------------------+----------------+
- |        TEST CASE        |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
- +-------------------------+------------------+---------------+------------------+----------------+
- |     k8s_conformance     |     functest     |     smoke     |      57:47       |      PASS      |
- +-------------------------+------------------+---------------+------------------+----------------+
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
+  +---------------------------+------------------+---------------+------------------+----------------+
+  |     xrally_kubernetes     |     functest     |     smoke     |      16:12       |      PASS      |
+  |      k8s_conformance      |     functest     |     smoke     |      149:59      |      PASS      |
+  +---------------------------+------------------+---------------+------------------+----------------+
 
 Results are automatically pushed to the test results database, some additional
 result files are pushed to OPNFV artifact web sites.