aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
-rw-r--r--docs/release/release-notes/functest-release.rst13
-rw-r--r--docs/testing/user/configguide/configguide.rst72
-rw-r--r--docs/testing/user/userguide/test_results.rst122
3 files changed, 111 insertions, 96 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index f3551d2c8..0a443d11a 100644
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -32,21 +32,26 @@ The internal test cases are:
* vping_ssh
* vping_userdata
* cinder_test
- * odl
* tempest_smoke
+ * odl
* neutron-tempest-plugin-api
* rally_sanity
* refstack_compute
* refstack_object
* refstack_platform
+ * tempest_full
+ * tempest_scenario
+ * tempest_slow
* patrole
+ * barbican
* neutron_trunk
* networking-bgpvpn
* networking-sfc
- * barbican
- * tempest_full
- * tempest_scenario
+ * octavia
* rally_full
+ * rally_jobs
+ * vmtp
+ * shaker
* cloudify
* cloudify_ims
* heat_ims
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index a0ef6bf4a..7731ea4ba 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -29,6 +29,9 @@ cat env::
NAMESERVER=XXX # if not 8.8.8.8
EXTERNAL_NETWORK=XXX # if not first network with router:external=True
NEW_USER_ROLE=XXX # if not member
+ SDN_CONTROLLER_IP=XXX # if odl scenario
+ VOLUME_DEVICE_NAME=XXX # if not vdb
+ FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios
See section on environment variables for details.
@@ -41,6 +44,7 @@ cat env_file::
export OS_PROJECT_NAME=XXX
export OS_PASSWORD=XXX
export OS_IDENTITY_API_VERSION=3
+ export OS_REGION_NAME=XXX
See section on OpenStack credentials for details.
@@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume)::
images/cirros-0.4.0-aarch64-disk.img
images/cirros-0.4.0-x86_64-disk.img
images/cloudify-docker-manager-community-19.01.24.tar
- images/shaker-image-arm64.qcow2
- images/shaker-image.qcow2
+ images/shaker-image-1.3.0+stretch.qcow2
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
@@ -73,17 +76,17 @@ Results shall be displayed as follows::
+--------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:01 | PASS |
+ | connection_check | functest | healthcheck | 00:03 | PASS |
| tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
- | tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
- | vmready1 | functest | healthcheck | 00:08 | PASS |
- | vmready2 | functest | healthcheck | 00:08 | PASS |
- | singlevm1 | functest | healthcheck | 00:35 | PASS |
- | singlevm2 | functest | healthcheck | 00:38 | PASS |
- | vping_ssh | functest | healthcheck | 00:47 | PASS |
- | vping_userdata | functest | healthcheck | 00:33 | PASS |
- | cinder_test | functest | healthcheck | 01:16 | PASS |
- | tempest_smoke | functest | healthcheck | 07:05 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:07 | PASS |
+ | vmready1 | functest | healthcheck | 00:09 | PASS |
+ | vmready2 | functest | healthcheck | 00:09 | PASS |
+ | singlevm1 | functest | healthcheck | 00:30 | PASS |
+ | singlevm2 | functest | healthcheck | 00:33 | PASS |
+ | vping_ssh | functest | healthcheck | 00:42 | PASS |
+ | vping_userdata | functest | healthcheck | 00:38 | PASS |
+ | cinder_test | functest | healthcheck | 01:06 | PASS |
+ | tempest_smoke | functest | healthcheck | 04:03 | PASS |
| odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
@@ -104,18 +107,19 @@ Results shall be displayed as follows::
+------------------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+------------------------------------+------------------+---------------+------------------+----------------+
- | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS |
- | rally_sanity | functest | smoke | 14:13 | PASS |
- | refstack_compute | functest | smoke | 07:24 | PASS |
- | refstack_object | functest | smoke | 02:08 | PASS |
- | refstack_platform | functest | smoke | 08:32 | PASS |
- | tempest_full | functest | smoke | 58:40 | PASS |
- | tempest_scenario | functest | smoke | 09:47 | PASS |
- | patrole | functest | smoke | 05:34 | PASS |
+ | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS |
+ | rally_sanity | functest | smoke | 16:43 | PASS |
+ | refstack_compute | functest | smoke | 06:48 | PASS |
+ | refstack_object | functest | smoke | 02:04 | PASS |
+ | refstack_platform | functest | smoke | 07:53 | PASS |
+ | tempest_full | functest | smoke | 31:51 | PASS |
+ | tempest_scenario | functest | smoke | 09:38 | PASS |
+ | tempest_slow | functest | smoke | 22:21 | PASS |
+ | patrole | functest | smoke | 02:32 | PASS |
+ | barbican | functest | smoke | 02:16 | PASS |
| neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 01:43 | PASS |
- | networking-sfc | functest | smoke | 04:49 | PASS |
- | barbican | functest | smoke | 05:09 | PASS |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+------------------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -136,10 +140,10 @@ Results shall be displayed as follows::
+--------------------+------------------+----------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------+------------------+----------------------+------------------+----------------+
- | rally_full | functest | benchmarking | 121:55 | PASS |
- | rally_jobs | functest | benchmarking | 24:21 | PASS |
- | vmtp | functest | benchmarking | 00:00 | SKIP |
- | shaker | functest | benchmarking | 00:00 | SKIP |
+ | rally_full | functest | benchmarking | 92:16 | PASS |
+ | rally_jobs | functest | benchmarking | 17:31 | PASS |
+ | vmtp | functest | benchmarking | 15:20 | PASS |
+ | shaker | functest | benchmarking | 22:44 | PASS |
+--------------------+------------------+----------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -160,11 +164,11 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 06:42 | PASS |
- | cloudify_ims | functest | vnf | 39:21 | PASS |
- | heat_ims | functest | vnf | 34:33 | PASS |
- | vyos_vrouter | functest | vnf | 20:20 | PASS |
- | juju_epc | functest | vnf | 56:55 | PASS |
+ | cloudify | functest | vnf | 03:35 | PASS |
+ | cloudify_ims | functest | vnf | 23:26 | PASS |
+ | heat_ims | functest | vnf | 34:22 | PASS |
+ | vyos_vrouter | functest | vnf | 13:49 | PASS |
+ | juju_epc | functest | vnf | 41:49 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Functest Dockers for Kubernetes deployment
@@ -199,7 +203,7 @@ Results shall be displayed as follows::
+-------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 02:27 | PASS |
+ | k8s_smoke | functest | healthcheck | 01:15 | PASS |
+-------------------+------------------+---------------------+------------------+----------------+
Testing smoke suite
@@ -216,7 +220,7 @@ Results shall be displayed as follows::
+-------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:14 | PASS |
+ | k8s_conformance | functest | smoke | 135:54 | PASS |
+-------------------------+------------------+---------------+------------------+----------------+
Environment variables
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index c4d85f631..59d3d0fba 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -10,7 +10,7 @@ In manual mode test results are displayed in the console and result files
are put in /home/opnfv/functest/results.
If you want additional logs, you may configure the logging.ini under
-/usr/lib/python2.7/site-packages/xtesting/ci.
+/usr/lib/python3.6/site-packages/xtesting/ci.
Automated testing
-----------------
@@ -21,75 +21,81 @@ end of each suite and can be described as follow.
Healthcheck suite::
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:07 | PASS |
- | api_check | functest | healthcheck | 07:46 | PASS |
- | snaps_health_check | functest | healthcheck | 00:36 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:03 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:07 | PASS |
+ | vmready1 | functest | healthcheck | 00:09 | PASS |
+ | vmready2 | functest | healthcheck | 00:09 | PASS |
+ | singlevm1 | functest | healthcheck | 00:30 | PASS |
+ | singlevm2 | functest | healthcheck | 00:33 | PASS |
+ | vping_ssh | functest | healthcheck | 00:42 | PASS |
+ | vping_userdata | functest | healthcheck | 00:38 | PASS |
+ | cinder_test | functest | healthcheck | 01:06 | PASS |
+ | tempest_smoke | functest | healthcheck | 04:03 | PASS |
+ | odl | functest | healthcheck | 00:00 | SKIP |
+ +--------------------------+------------------+---------------------+------------------+----------------+
Smoke suite::
- +------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------+------------------+----------------+
- | vping_ssh | functest | smoke | 00:57 | PASS |
- | vping_userdata | functest | smoke | 00:33 | PASS |
- | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
- | rally_sanity | functest | smoke | 24:07 | PASS |
- | refstack_defcore | functest | smoke | 05:21 | PASS |
- | patrole | functest | smoke | 04:29 | PASS |
- | snaps_smoke | functest | smoke | 46:54 | PASS |
- | odl | functest | smoke | 00:00 | SKIP |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- +------------------------------+------------------+---------------+------------------+----------------+
-
-Components suite::
-
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full_parallel | functest | components | 48:28 | PASS |
- | rally_full | functest | components | 126:02 | PASS |
- +-------------------------------+------------------+--------------------+------------------+----------------+
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS |
+ | rally_sanity | functest | smoke | 16:43 | PASS |
+ | refstack_compute | functest | smoke | 06:48 | PASS |
+ | refstack_object | functest | smoke | 02:04 | PASS |
+ | refstack_platform | functest | smoke | 07:53 | PASS |
+ | tempest_full | functest | smoke | 31:51 | PASS |
+ | tempest_scenario | functest | smoke | 09:38 | PASS |
+ | tempest_slow | functest | smoke | 22:21 | PASS |
+ | patrole | functest | smoke | 02:32 | PASS |
+ | barbican | functest | smoke | 02:16 | PASS |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+
+Benchmarking suite::
+
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | rally_full | functest | benchmarking | 92:16 | PASS |
+ | rally_jobs | functest | benchmarking | 17:31 | PASS |
+ | vmtp | functest | benchmarking | 15:20 | PASS |
+ | shaker | functest | benchmarking | 22:44 | PASS |
+ +--------------------+------------------+----------------------+------------------+----------------+
Vnf suite::
- +----------------------+------------------+--------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------+------------------+--------------+------------------+----------------+
- | cloudify_ims | functest | vnf | 28:15 | PASS |
- | vyos_vrouter | functest | vnf | 17:59 | PASS |
- | juju_epc | functest | vnf | 46:44 | PASS |
- +----------------------+------------------+--------------+------------------+----------------+
-
-Functest Kubernetes test result::
-
- +--------------------------------------+------------------------------------------------------------+
- | ENV VAR | VALUE |
- +--------------------------------------+------------------------------------------------------------+
- | INSTALLER_TYPE | compass |
- | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 |
- | CI_LOOP | daily |
- +--------------------------------------+------------------------------------------------------------+
+ +----------------------+------------------+--------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +----------------------+------------------+--------------+------------------+----------------+
+ | cloudify | functest | vnf | 03:35 | PASS |
+ | cloudify_ims | functest | vnf | 23:26 | PASS |
+ | heat_ims | functest | vnf | 34:22 | PASS |
+ | vyos_vrouter | functest | vnf | 13:49 | PASS |
+ | juju_epc | functest | vnf | 41:49 | PASS |
+ +----------------------+------------------+--------------+------------------+----------------+
Kubernetes healthcheck suite::
- +-------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------+------------------+---------------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 01:54 | PASS |
- +-------------------+------------------+---------------------+------------------+----------------+
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | k8s_smoke | functest | healthcheck | 01:15 | PASS |
+ +-------------------+------------------+---------------------+------------------+----------------+
Kubernetes smoke suite::
- +-------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:47 | PASS |
- +-------------------------+------------------+---------------+------------------+----------------+
+ +-------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------+------------------+---------------+------------------+----------------+
+ | k8s_conformance | functest | smoke | 135:54 | PASS |
+ +-------------------------+------------------+---------------+------------------+----------------+
Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.