aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorCédric Ollivier <cedric.ollivier@orange.com>2019-10-13 14:43:16 +0200
committerCédric Ollivier <cedric.ollivier@orange.com>2019-10-13 15:17:24 +0200
commit2c4348d726e1c6bd4582182a10104d89dac16056 (patch)
tree596413f3be491fade97c7cf4e28b6ba8f83e3797
parent3e503a868cc7211324b472767648023b3706497c (diff)
Updates all testcases and their results
Change-Id: I127146b4d6face1b6221e049636a29e601415fd8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com> (cherry picked from commit 37251af62ca89b55f09dfd6ddce15ee179c580f3)
-rw-r--r--docs/release/release-notes/functest-release.rst14
-rw-r--r--docs/testing/user/configguide/configguide.rst80
-rw-r--r--docs/testing/user/userguide/test_results.rst122
3 files changed, 118 insertions, 98 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index 2f8b3e28d..5c668dcc5 100644
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -32,21 +32,26 @@ The internal test cases are:
* vping_ssh
* vping_userdata
* cinder_test
- * odl
* tempest_smoke
+ * odl
* neutron-tempest-plugin-api
* rally_sanity
* refstack_compute
* refstack_object
* refstack_platform
+ * tempest_full
+ * tempest_scenario
+ * tempest_slow
* patrole
+ * barbican
* neutron_trunk
* networking-bgpvpn
* networking-sfc
- * barbican
- * tempest_full
- * tempest_scenario
+ * octavia
* rally_full
+ * rally_jobs
+ * vmtp
+ * shaker
* cloudify
* cloudify_ims
* heat_ims
@@ -59,6 +64,7 @@ Kubernetes
The internal test cases are:
* k8s_smoke
+ * xrally_kubernetes
* k8s_conformance
Release Data
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index c50fd1f1f..cb722bc2e 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -29,6 +29,9 @@ cat env::
NAMESERVER=XXX # if not 8.8.8.8
EXTERNAL_NETWORK=XXX # if not first network with router:external=True
NEW_USER_ROLE=XXX # if not member
+ SDN_CONTROLLER_IP=XXX # if odl scenario
+ VOLUME_DEVICE_NAME=XXX # if not vdb
+ FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios
See section on environment variables for details.
@@ -41,6 +44,7 @@ cat env_file::
export OS_PROJECT_NAME=XXX
export OS_PASSWORD=XXX
export OS_IDENTITY_API_VERSION=3
+ export OS_REGION_NAME=XXX
See section on OpenStack credentials for details.
@@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume)::
images/cirros-0.4.0-aarch64-disk.img
images/cirros-0.4.0-x86_64-disk.img
images/cloudify-docker-manager-community-19.01.24.tar
- images/shaker-image-arm64.qcow2
- images/shaker-image.qcow2
+ images/shaker-image-1.3.0+stretch.qcow2
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
@@ -73,17 +76,17 @@ Results shall be displayed as follows::
+--------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:01 | PASS |
+ | connection_check | functest | healthcheck | 00:03 | PASS |
| tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
- | tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:08 | PASS |
| vmready1 | functest | healthcheck | 00:08 | PASS |
- | vmready2 | functest | healthcheck | 00:08 | PASS |
- | singlevm1 | functest | healthcheck | 00:35 | PASS |
- | singlevm2 | functest | healthcheck | 00:38 | PASS |
- | vping_ssh | functest | healthcheck | 00:47 | PASS |
- | vping_userdata | functest | healthcheck | 00:33 | PASS |
- | cinder_test | functest | healthcheck | 01:16 | PASS |
- | tempest_smoke | functest | healthcheck | 07:05 | PASS |
+ | vmready2 | functest | healthcheck | 00:10 | PASS |
+ | singlevm1 | functest | healthcheck | 00:31 | PASS |
+ | singlevm2 | functest | healthcheck | 00:29 | PASS |
+ | vping_ssh | functest | healthcheck | 00:37 | PASS |
+ | vping_userdata | functest | healthcheck | 00:36 | PASS |
+ | cinder_test | functest | healthcheck | 01:05 | PASS |
+ | tempest_smoke | functest | healthcheck | 04:58 | PASS |
| odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
@@ -104,18 +107,20 @@ Results shall be displayed as follows::
+------------------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+------------------------------------+------------------+---------------+------------------+----------------+
- | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS |
- | rally_sanity | functest | smoke | 14:13 | PASS |
- | refstack_compute | functest | smoke | 07:24 | PASS |
- | refstack_object | functest | smoke | 02:08 | PASS |
- | refstack_platform | functest | smoke | 08:32 | PASS |
- | tempest_full | functest | smoke | 58:40 | PASS |
- | tempest_scenario | functest | smoke | 09:47 | PASS |
- | patrole | functest | smoke | 05:34 | PASS |
+ | neutron-tempest-plugin-api | functest | smoke | 10:35 | PASS |
+ | rally_sanity | functest | smoke | 17:11 | PASS |
+ | refstack_compute | functest | smoke | 06:09 | PASS |
+ | refstack_object | functest | smoke | 01:48 | PASS |
+ | refstack_platform | functest | smoke | 05:53 | PASS |
+ | tempest_full | functest | smoke | 30:02 | PASS |
+ | tempest_scenario | functest | smoke | 09:26 | PASS |
+ | tempest_slow | functest | smoke | 18:53 | PASS |
+ | patrole | functest | smoke | 02:28 | PASS |
+ | barbican | functest | smoke | 02:13 | PASS |
| neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 01:43 | PASS |
- | networking-sfc | functest | smoke | 04:49 | PASS |
- | barbican | functest | smoke | 05:09 | PASS |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ | octavia | functest | smoke | 00:00 | SKIP |
+------------------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -136,10 +141,10 @@ Results shall be displayed as follows::
+--------------------+------------------+----------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------+------------------+----------------------+------------------+----------------+
- | rally_full | functest | benchmarking | 121:55 | PASS |
- | rally_jobs | functest | benchmarking | 24:21 | PASS |
- | vmtp | functest | benchmarking | 00:00 | SKIP |
- | shaker | functest | benchmarking | 00:00 | SKIP |
+ | rally_full | functest | benchmarking | 92:42 | PASS |
+ | rally_jobs | functest | benchmarking | 18:44 | PASS |
+ | vmtp | functest | benchmarking | 15:20 | PASS |
+ | shaker | functest | benchmarking | 23:54 | PASS |
+--------------------+------------------+----------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -160,11 +165,11 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 06:42 | PASS |
- | cloudify_ims | functest | vnf | 39:21 | PASS |
- | heat_ims | functest | vnf | 34:33 | PASS |
- | vyos_vrouter | functest | vnf | 20:20 | PASS |
- | juju_epc | functest | vnf | 56:55 | PASS |
+ | cloudify | functest | vnf | 03:50 | PASS |
+ | cloudify_ims | functest | vnf | 22:34 | PASS |
+ | heat_ims | functest | vnf | 32:39 | PASS |
+ | vyos_vrouter | functest | vnf | 15:10 | PASS |
+ | juju_epc | functest | vnf | 37:03 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Functest Dockers for Kubernetes deployment
@@ -199,7 +204,7 @@ Results shall be displayed as follows::
+-------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 02:27 | PASS |
+ | k8s_smoke | functest | healthcheck | 01:44 | PASS |
+-------------------+------------------+---------------------+------------------+----------------+
Testing smoke suite
@@ -213,11 +218,12 @@ Run smoke suite::
Results shall be displayed as follows::
- +-------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:14 | PASS |
- +-------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | xrally_kubernetes | functest | smoke | 16:12 | PASS |
+ | k8s_conformance | functest | smoke | 149:59 | PASS |
+ +---------------------------+------------------+---------------+------------------+----------------+
Environment variables
=====================
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index 8023c4661..457d7a1d3 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -21,75 +21,83 @@ end of each suite and can be described as follow.
Healthcheck suite::
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:07 | PASS |
- | api_check | functest | healthcheck | 07:46 | PASS |
- | snaps_health_check | functest | healthcheck | 00:36 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:03 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:08 | PASS |
+ | vmready1 | functest | healthcheck | 00:08 | PASS |
+ | vmready2 | functest | healthcheck | 00:10 | PASS |
+ | singlevm1 | functest | healthcheck | 00:31 | PASS |
+ | singlevm2 | functest | healthcheck | 00:29 | PASS |
+ | vping_ssh | functest | healthcheck | 00:37 | PASS |
+ | vping_userdata | functest | healthcheck | 00:36 | PASS |
+ | cinder_test | functest | healthcheck | 01:05 | PASS |
+ | tempest_smoke | functest | healthcheck | 04:58 | PASS |
+ | odl | functest | healthcheck | 00:00 | SKIP |
+ +--------------------------+------------------+---------------------+------------------+----------------+
Smoke suite::
- +------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------+------------------+----------------+
- | vping_ssh | functest | smoke | 00:57 | PASS |
- | vping_userdata | functest | smoke | 00:33 | PASS |
- | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
- | rally_sanity | functest | smoke | 24:07 | PASS |
- | refstack_defcore | functest | smoke | 05:21 | PASS |
- | patrole | functest | smoke | 04:29 | PASS |
- | snaps_smoke | functest | smoke | 46:54 | PASS |
- | odl | functest | smoke | 00:00 | SKIP |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- +------------------------------+------------------+---------------+------------------+----------------+
-
-Components suite::
-
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full_parallel | functest | components | 48:28 | PASS |
- | rally_full | functest | components | 126:02 | PASS |
- +-------------------------------+------------------+--------------------+------------------+----------------+
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | neutron-tempest-plugin-api | functest | smoke | 10:35 | PASS |
+ | rally_sanity | functest | smoke | 17:11 | PASS |
+ | refstack_compute | functest | smoke | 06:09 | PASS |
+ | refstack_object | functest | smoke | 01:48 | PASS |
+ | refstack_platform | functest | smoke | 05:53 | PASS |
+ | tempest_full | functest | smoke | 30:02 | PASS |
+ | tempest_scenario | functest | smoke | 09:26 | PASS |
+ | tempest_slow | functest | smoke | 18:53 | PASS |
+ | patrole | functest | smoke | 02:28 | PASS |
+ | barbican | functest | smoke | 02:13 | PASS |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ | octavia | functest | smoke | 00:00 | SKIP |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+
+Benchmarking suite::
+
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | rally_full | functest | benchmarking | 92:42 | PASS |
+ | rally_jobs | functest | benchmarking | 18:44 | PASS |
+ | vmtp | functest | benchmarking | 15:20 | PASS |
+ | shaker | functest | benchmarking | 23:54 | PASS |
+ +--------------------+------------------+----------------------+------------------+----------------+
Vnf suite::
- +----------------------+------------------+--------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------+------------------+--------------+------------------+----------------+
- | cloudify_ims | functest | vnf | 28:15 | PASS |
- | vyos_vrouter | functest | vnf | 17:59 | PASS |
- | juju_epc | functest | vnf | 46:44 | PASS |
- +----------------------+------------------+--------------+------------------+----------------+
-
-Functest Kubernetes test result::
-
- +--------------------------------------+------------------------------------------------------------+
- | ENV VAR | VALUE |
- +--------------------------------------+------------------------------------------------------------+
- | INSTALLER_TYPE | compass |
- | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 |
- | CI_LOOP | daily |
- +--------------------------------------+------------------------------------------------------------+
+ +----------------------+------------------+--------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+--------------+------------------+----------------+
+ | cloudify | functest | vnf | 03:50 | PASS |
+ | cloudify_ims | functest | vnf | 22:34 | PASS |
+ | heat_ims | functest | vnf | 32:39 | PASS |
+ | vyos_vrouter | functest | vnf | 15:10 | PASS |
+ | juju_epc | functest | vnf | 37:03 | PASS |
+ +----------------------+------------------+--------------+------------------+----------------+
Kubernetes healthcheck suite::
- +-------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 01:54 | PASS |
- +-------------------+------------------+---------------------+------------------+----------------+
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | k8s_smoke | functest | healthcheck | 01:44 | PASS |
+ +-------------------+------------------+---------------------+------------------+----------------+
Kubernetes smoke suite::
- +-------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:47 | PASS |
- +-------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | xrally_kubernetes | functest | smoke | 16:12 | PASS |
+ | k8s_conformance | functest | smoke | 149:59 | PASS |
+ +---------------------------+------------------+---------------+------------------+----------------+
Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.