summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorCédric Ollivier <cedric.ollivier@orange.com>2019-10-13 14:43:16 +0200
committerCédric Ollivier <cedric.ollivier@orange.com>2019-10-13 14:43:16 +0200
commit37251af62ca89b55f09dfd6ddce15ee179c580f3 (patch)
treecc60bd0b663b7512ed60c88c22527aaa77c0bc64
parent74df72ec294998787a8f0af5e2084db91ba0778b (diff)
Updates all testcases and their results
Change-Id: I127146b4d6face1b6221e049636a29e601415fd8 Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
-rw-r--r--docs/release/release-notes/functest-release.rst14
-rw-r--r--docs/testing/user/configguide/configguide.rst78
-rw-r--r--docs/testing/user/userguide/test_results.rst122
3 files changed, 117 insertions, 97 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index 2f8b3e2..5c668dc 100644
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -32,21 +32,26 @@ The internal test cases are:
* vping_ssh
* vping_userdata
* cinder_test
- * odl
* tempest_smoke
+ * odl
* neutron-tempest-plugin-api
* rally_sanity
* refstack_compute
* refstack_object
* refstack_platform
+ * tempest_full
+ * tempest_scenario
+ * tempest_slow
* patrole
+ * barbican
* neutron_trunk
* networking-bgpvpn
* networking-sfc
- * barbican
- * tempest_full
- * tempest_scenario
+ * octavia
* rally_full
+ * rally_jobs
+ * vmtp
+ * shaker
* cloudify
* cloudify_ims
* heat_ims
@@ -59,6 +64,7 @@ Kubernetes
The internal test cases are:
* k8s_smoke
+ * xrally_kubernetes
* k8s_conformance
Release Data
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index c50fd1f..25aa7cd 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -29,6 +29,9 @@ cat env::
NAMESERVER=XXX # if not 8.8.8.8
EXTERNAL_NETWORK=XXX # if not first network with router:external=True
NEW_USER_ROLE=XXX # if not member
+ SDN_CONTROLLER_IP=XXX # if odl scenario
+ VOLUME_DEVICE_NAME=XXX # if not vdb
+ FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios
See section on environment variables for details.
@@ -41,6 +44,7 @@ cat env_file::
export OS_PROJECT_NAME=XXX
export OS_PASSWORD=XXX
export OS_IDENTITY_API_VERSION=3
+ export OS_REGION_NAME=XXX
See section on OpenStack credentials for details.
@@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume)::
images/cirros-0.4.0-aarch64-disk.img
images/cirros-0.4.0-x86_64-disk.img
images/cloudify-docker-manager-community-19.01.24.tar
- images/shaker-image-arm64.qcow2
- images/shaker-image.qcow2
+ images/shaker-image-1.3.0+stretch.qcow2
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
@@ -73,17 +76,17 @@ Results shall be displayed as follows::
+--------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:01 | PASS |
+ | connection_check | functest | healthcheck | 00:02 | PASS |
| tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
| tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
- | vmready1 | functest | healthcheck | 00:08 | PASS |
+ | vmready1 | functest | healthcheck | 00:06 | PASS |
| vmready2 | functest | healthcheck | 00:08 | PASS |
- | singlevm1 | functest | healthcheck | 00:35 | PASS |
- | singlevm2 | functest | healthcheck | 00:38 | PASS |
- | vping_ssh | functest | healthcheck | 00:47 | PASS |
- | vping_userdata | functest | healthcheck | 00:33 | PASS |
- | cinder_test | functest | healthcheck | 01:16 | PASS |
- | tempest_smoke | functest | healthcheck | 07:05 | PASS |
+ | singlevm1 | functest | healthcheck | 00:28 | PASS |
+ | singlevm2 | functest | healthcheck | 00:25 | PASS |
+ | vping_ssh | functest | healthcheck | 00:36 | PASS |
+ | vping_userdata | functest | healthcheck | 00:34 | PASS |
+ | cinder_test | functest | healthcheck | 01:03 | PASS |
+ | tempest_smoke | functest | healthcheck | 05:13 | PASS |
| odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
@@ -104,18 +107,20 @@ Results shall be displayed as follows::
+------------------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+------------------------------------+------------------+---------------+------------------+----------------+
- | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS |
- | rally_sanity | functest | smoke | 14:13 | PASS |
- | refstack_compute | functest | smoke | 07:24 | PASS |
- | refstack_object | functest | smoke | 02:08 | PASS |
- | refstack_platform | functest | smoke | 08:32 | PASS |
- | tempest_full | functest | smoke | 58:40 | PASS |
- | tempest_scenario | functest | smoke | 09:47 | PASS |
- | patrole | functest | smoke | 05:34 | PASS |
+ | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS |
+ | rally_sanity | functest | smoke | 16:29 | PASS |
+ | refstack_compute | functest | smoke | 06:25 | PASS |
+ | refstack_object | functest | smoke | 01:54 | PASS |
+ | refstack_platform | functest | smoke | 06:52 | PASS |
+ | tempest_full | functest | smoke | 30:26 | PASS |
+ | tempest_scenario | functest | smoke | 09:23 | PASS |
+ | tempest_slow | functest | smoke | 24:42 | PASS |
+ | patrole | functest | smoke | 02:36 | PASS |
+ | barbican | functest | smoke | 02:13 | PASS |
| neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 01:43 | PASS |
- | networking-sfc | functest | smoke | 04:49 | PASS |
- | barbican | functest | smoke | 05:09 | PASS |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ | octavia | functest | smoke | 00:00 | SKIP |
+------------------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -136,10 +141,10 @@ Results shall be displayed as follows::
+--------------------+------------------+----------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------+------------------+----------------------+------------------+----------------+
- | rally_full | functest | benchmarking | 121:55 | PASS |
- | rally_jobs | functest | benchmarking | 24:21 | PASS |
- | vmtp | functest | benchmarking | 00:00 | SKIP |
- | shaker | functest | benchmarking | 00:00 | SKIP |
+ | rally_full | functest | benchmarking | 92:16 | PASS |
+ | rally_jobs | functest | benchmarking | 18:49 | PASS |
+ | vmtp | functest | benchmarking | 15:28 | PASS |
+ | shaker | functest | benchmarking | 24:04 | PASS |
+--------------------+------------------+----------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -160,11 +165,11 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 06:42 | PASS |
- | cloudify_ims | functest | vnf | 39:21 | PASS |
- | heat_ims | functest | vnf | 34:33 | PASS |
- | vyos_vrouter | functest | vnf | 20:20 | PASS |
- | juju_epc | functest | vnf | 56:55 | PASS |
+ | cloudify | functest | vnf | 03:49 | PASS |
+ | cloudify_ims | functest | vnf | 24:20 | PASS |
+ | heat_ims | functest | vnf | 32:13 | PASS |
+ | vyos_vrouter | functest | vnf | 14:55 | PASS |
+ | juju_epc | functest | vnf | 41:24 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Functest Dockers for Kubernetes deployment
@@ -199,7 +204,7 @@ Results shall be displayed as follows::
+-------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 02:27 | PASS |
+ | k8s_smoke | functest | healthcheck | 01:09 | PASS |
+-------------------+------------------+---------------------+------------------+----------------+
Testing smoke suite
@@ -213,11 +218,12 @@ Run smoke suite::
Results shall be displayed as follows::
- +-------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:14 | PASS |
- +-------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | xrally_kubernetes | functest | smoke | 22:04 | PASS |
+ | k8s_conformance | functest | smoke | 173:48 | PASS |
+ +---------------------------+------------------+---------------+------------------+----------------+
Environment variables
=====================
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index 8023c46..511b0b4 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -21,75 +21,83 @@ end of each suite and can be described as follow.
Healthcheck suite::
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:07 | PASS |
- | api_check | functest | healthcheck | 07:46 | PASS |
- | snaps_health_check | functest | healthcheck | 00:36 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:02 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
+ | vmready1 | functest | healthcheck | 00:06 | PASS |
+ | vmready2 | functest | healthcheck | 00:08 | PASS |
+ | singlevm1 | functest | healthcheck | 00:28 | PASS |
+ | singlevm2 | functest | healthcheck | 00:25 | PASS |
+ | vping_ssh | functest | healthcheck | 00:36 | PASS |
+ | vping_userdata | functest | healthcheck | 00:34 | PASS |
+ | cinder_test | functest | healthcheck | 01:03 | PASS |
+ | tempest_smoke | functest | healthcheck | 05:13 | PASS |
+ | odl | functest | healthcheck | 00:00 | SKIP |
+ +--------------------------+------------------+---------------------+------------------+----------------+
Smoke suite::
- +------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------+------------------+----------------+
- | vping_ssh | functest | smoke | 00:57 | PASS |
- | vping_userdata | functest | smoke | 00:33 | PASS |
- | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
- | rally_sanity | functest | smoke | 24:07 | PASS |
- | refstack_defcore | functest | smoke | 05:21 | PASS |
- | patrole | functest | smoke | 04:29 | PASS |
- | snaps_smoke | functest | smoke | 46:54 | PASS |
- | odl | functest | smoke | 00:00 | SKIP |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- +------------------------------+------------------+---------------+------------------+----------------+
-
-Components suite::
-
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full_parallel | functest | components | 48:28 | PASS |
- | rally_full | functest | components | 126:02 | PASS |
- +-------------------------------+------------------+--------------------+------------------+----------------+
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS |
+ | rally_sanity | functest | smoke | 16:29 | PASS |
+ | refstack_compute | functest | smoke | 06:25 | PASS |
+ | refstack_object | functest | smoke | 01:54 | PASS |
+ | refstack_platform | functest | smoke | 06:52 | PASS |
+ | tempest_full | functest | smoke | 30:26 | PASS |
+ | tempest_scenario | functest | smoke | 09:23 | PASS |
+ | tempest_slow | functest | smoke | 24:42 | PASS |
+ | patrole | functest | smoke | 02:36 | PASS |
+ | barbican | functest | smoke | 02:13 | PASS |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ | octavia | functest | smoke | 00:00 | SKIP |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+
+Benchmarking suite::
+
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | rally_full | functest | benchmarking | 92:16 | PASS |
+ | rally_jobs | functest | benchmarking | 18:49 | PASS |
+ | vmtp | functest | benchmarking | 15:28 | PASS |
+ | shaker | functest | benchmarking | 24:04 | PASS |
+ +--------------------+------------------+----------------------+------------------+----------------+
Vnf suite::
- +----------------------+------------------+--------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------+------------------+--------------+------------------+----------------+
- | cloudify_ims | functest | vnf | 28:15 | PASS |
- | vyos_vrouter | functest | vnf | 17:59 | PASS |
- | juju_epc | functest | vnf | 46:44 | PASS |
- +----------------------+------------------+--------------+------------------+----------------+
-
-Functest Kubernetes test result::
-
- +--------------------------------------+------------------------------------------------------------+
- | ENV VAR | VALUE |
- +--------------------------------------+------------------------------------------------------------+
- | INSTALLER_TYPE | compass |
- | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 |
- | CI_LOOP | daily |
- +--------------------------------------+------------------------------------------------------------+
+ +----------------------+------------------+--------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+--------------+------------------+----------------+
+ | cloudify | functest | vnf | 03:49 | PASS |
+ | cloudify_ims | functest | vnf | 24:20 | PASS |
+ | heat_ims | functest | vnf | 32:13 | PASS |
+ | vyos_vrouter | functest | vnf | 14:55 | PASS |
+ | juju_epc | functest | vnf | 41:24 | PASS |
+ +----------------------+------------------+--------------+------------------+----------------+
Kubernetes healthcheck suite::
- +-------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 01:54 | PASS |
- +-------------------+------------------+---------------------+------------------+----------------+
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | k8s_smoke | functest | healthcheck | 01:09 | PASS |
+ +-------------------+------------------+---------------------+------------------+----------------+
Kubernetes smoke suite::
- +-------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:47 | PASS |
- +-------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | xrally_kubernetes | functest | smoke | 22:04 | PASS |
+ | k8s_conformance | functest | smoke | 173:48 | PASS |
+ +---------------------------+------------------+---------------+------------------+----------------+
Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.