aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/user/configguide/configguide.rst122
-rw-r--r--docs/testing/user/userguide/test_results.rst98
2 files changed, 150 insertions, 70 deletions
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index 131bf1191..75af1bb52 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -16,7 +16,9 @@ Docker images are available on the dockerhub:
* opnfv/functest-core
* opnfv/functest-healthcheck
* opnfv/functest-smoke
+ * opnfv/functest-smoke-cntt
* opnfv/functest-benchmarking
+ * opnfv/functest-benchmarking-cntt
* opnfv/functest-vnf
@@ -28,6 +30,7 @@ cat env::
DEPLOY_SCENARIO=XXX # if not os-nosdn-nofeature-noha scenario
NAMESERVER=XXX # if not 8.8.8.8
EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+ DASHBOARD_URL=XXX # else tempest_horizon will be skipped
NEW_USER_ROLE=XXX # if not member
SDN_CONTROLLER_IP=XXX # if odl scenario
VOLUME_DEVICE_NAME=XXX # if not vdb
@@ -55,6 +58,7 @@ Create a directory for the different images (attached as a Docker volume)::
images/cirros-0.4.0-aarch64-disk.img
images/cirros-0.4.0-x86_64-disk.img
images/cloudify-docker-manager-community-19.01.24.tar
+ images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
images/shaker-image-1.3.0+stretch.qcow2
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
@@ -76,17 +80,18 @@ Results shall be displayed as follows::
+--------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:02 | PASS |
- | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | connection_check | functest | healthcheck | 00:03 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:05 | PASS |
| tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
| vmready1 | functest | healthcheck | 00:06 | PASS |
| vmready2 | functest | healthcheck | 00:08 | PASS |
- | singlevm1 | functest | healthcheck | 00:28 | PASS |
- | singlevm2 | functest | healthcheck | 00:25 | PASS |
- | vping_ssh | functest | healthcheck | 00:36 | PASS |
- | vping_userdata | functest | healthcheck | 00:34 | PASS |
- | cinder_test | functest | healthcheck | 01:03 | PASS |
- | tempest_smoke | functest | healthcheck | 05:13 | PASS |
+ | singlevm1 | functest | healthcheck | 00:32 | PASS |
+ | singlevm2 | functest | healthcheck | 00:37 | PASS |
+ | vping_ssh | functest | healthcheck | 00:46 | PASS |
+ | vping_userdata | functest | healthcheck | 00:39 | PASS |
+ | cinder_test | functest | healthcheck | 01:05 | PASS |
+ | tempest_smoke | functest | healthcheck | 05:39 | PASS |
+ | tempest_horizon | functest | healthcheck | 01:05 | PASS |
| odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
@@ -104,28 +109,54 @@ Run smoke suite::
Results shall be displayed as follows::
- +------------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------------+------------------+---------------+------------------+----------------+
- | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS |
- | rally_sanity | functest | smoke | 16:29 | PASS |
- | refstack_compute | functest | smoke | 06:25 | PASS |
- | refstack_object | functest | smoke | 01:54 | PASS |
- | refstack_platform | functest | smoke | 06:52 | PASS |
- | tempest_full | functest | smoke | 30:26 | PASS |
- | tempest_scenario | functest | smoke | 09:23 | PASS |
- | tempest_slow | functest | smoke | 24:42 | PASS |
- | patrole | functest | smoke | 02:36 | PASS |
- | barbican | functest | smoke | 02:13 | PASS |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
- | networking-sfc | functest | smoke | 00:00 | SKIP |
- | octavia | functest | smoke | 00:00 | SKIP |
- +------------------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | tempest_neutron | functest | smoke | 15:30 | PASS |
+ | tempest_cinder | functest | smoke | 02:01 | PASS |
+ | tempest_keystone | functest | smoke | 01:17 | PASS |
+ | tempest_heat | functest | smoke | 22:14 | PASS |
+ | tempest_telemetry | functest | smoke | 00:00 | SKIP |
+ | rally_sanity | functest | smoke | 17:24 | PASS |
+ | refstack_compute | functest | smoke | 07:03 | PASS |
+ | refstack_object | functest | smoke | 02:09 | PASS |
+ | refstack_platform | functest | smoke | 07:31 | PASS |
+ | tempest_full | functest | smoke | 41:52 | PASS |
+ | tempest_scenario | functest | smoke | 08:42 | PASS |
+ | tempest_slow | functest | smoke | 43:42 | PASS |
+ | patrole | functest | smoke | 02:42 | PASS |
+ | tempest_barbican | functest | smoke | 02:30 | PASS |
+ | tempest_octavia | functest | smoke | 00:00 | SKIP |
+ +---------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
+Testing smoke CNTT suite
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run smoke-cntt suite::
+
+ sudo docker run --env-file env \
+ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
+ -v $(pwd)/images:/home/opnfv/functest/images \
+ opnfv/functest-smoke-cntt
+
+Results shall be displayed as follows::
+
+ +-------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------------+------------------+---------------+------------------+----------------+
+ | tempest_neutron_cntt | functest | smoke | 11:35 | PASS |
+ | tempest_cinder_cntt | functest | smoke | 01:58 | PASS |
+ | tempest_keystone_cntt | functest | smoke | 01:13 | PASS |
+ | tempest_heat_cntt | functest | smoke | 22:32 | PASS |
+ | rally_sanity_cntt | functest | smoke | 17:16 | PASS |
+ | tempest_full_cntt | functest | smoke | 41:13 | PASS |
+ | tempest_scenario_cntt | functest | smoke | 08:57 | PASS |
+ | tempest_slow_cntt | functest | smoke | 35:58 | PASS |
+ +-------------------------------+------------------+---------------+------------------+----------------+
+
Testing benchmarking suite
^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -141,15 +172,36 @@ Results shall be displayed as follows::
+--------------------+------------------+----------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------+------------------+----------------------+------------------+----------------+
- | rally_full | functest | benchmarking | 92:16 | PASS |
- | rally_jobs | functest | benchmarking | 18:49 | PASS |
- | vmtp | functest | benchmarking | 15:28 | PASS |
- | shaker | functest | benchmarking | 24:04 | PASS |
+ | rally_full | functest | benchmarking | 93:03 | PASS |
+ | rally_jobs | functest | benchmarking | 27:05 | PASS |
+ | vmtp | functest | benchmarking | 17:56 | PASS |
+ | shaker | functest | benchmarking | 24:02 | PASS |
+--------------------+------------------+----------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
+Testing benchmarking CNTT suite
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run benchmarking-cntt suite::
+
+ sudo docker run --env-file env \
+ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
+ -v $(pwd)/images:/home/opnfv/functest/images \
+ opnfv/functest-benchmarking-cntt
+
+Results shall be displayed as follows::
+
+ +-------------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------+------------------+----------------------+------------------+----------------+
+ | rally_full_cntt | functest | benchmarking | 89:52 | PASS |
+ | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS |
+ | vmtp | functest | benchmarking | 16:59 | PASS |
+ | shaker | functest | benchmarking | 23:43 | PASS |
+ +-------------------------+------------------+----------------------+------------------+----------------+
+
Testing vnf suite
^^^^^^^^^^^^^^^^^
@@ -165,11 +217,11 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 03:49 | PASS |
- | cloudify_ims | functest | vnf | 24:20 | PASS |
- | heat_ims | functest | vnf | 32:13 | PASS |
- | vyos_vrouter | functest | vnf | 14:55 | PASS |
- | juju_epc | functest | vnf | 41:24 | PASS |
+ | cloudify | functest | vnf | 05:08 | PASS |
+ | cloudify_ims | functest | vnf | 24:46 | PASS |
+ | heat_ims | functest | vnf | 33:12 | PASS |
+ | vyos_vrouter | functest | vnf | 15:53 | PASS |
+ | juju_epc | functest | vnf | 27:52 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Functest Dockers for Kubernetes deployment
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
index 5f78c1dc2..8d8f79fe6 100644
--- a/docs/testing/user/userguide/test_results.rst
+++ b/docs/testing/user/userguide/test_results.rst
@@ -24,62 +24,90 @@ Healthcheck suite::
+--------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:02 | PASS |
- | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | connection_check | functest | healthcheck | 00:03 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:05 | PASS |
| tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
| vmready1 | functest | healthcheck | 00:06 | PASS |
| vmready2 | functest | healthcheck | 00:08 | PASS |
- | singlevm1 | functest | healthcheck | 00:28 | PASS |
- | singlevm2 | functest | healthcheck | 00:25 | PASS |
- | vping_ssh | functest | healthcheck | 00:36 | PASS |
- | vping_userdata | functest | healthcheck | 00:34 | PASS |
- | cinder_test | functest | healthcheck | 01:03 | PASS |
- | tempest_smoke | functest | healthcheck | 05:13 | PASS |
+ | singlevm1 | functest | healthcheck | 00:32 | PASS |
+ | singlevm2 | functest | healthcheck | 00:37 | PASS |
+ | vping_ssh | functest | healthcheck | 00:46 | PASS |
+ | vping_userdata | functest | healthcheck | 00:39 | PASS |
+ | cinder_test | functest | healthcheck | 01:05 | PASS |
+ | tempest_smoke | functest | healthcheck | 05:39 | PASS |
+ | tempest_horizon | functest | healthcheck | 01:05 | PASS |
| odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
Smoke suite::
- +------------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------------+------------------+---------------+------------------+----------------+
- | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS |
- | rally_sanity | functest | smoke | 16:29 | PASS |
- | refstack_compute | functest | smoke | 06:25 | PASS |
- | refstack_object | functest | smoke | 01:54 | PASS |
- | refstack_platform | functest | smoke | 06:52 | PASS |
- | tempest_full | functest | smoke | 30:26 | PASS |
- | tempest_scenario | functest | smoke | 09:23 | PASS |
- | tempest_slow | functest | smoke | 24:42 | PASS |
- | patrole | functest | smoke | 02:36 | PASS |
- | barbican | functest | smoke | 02:13 | PASS |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
- | networking-sfc | functest | smoke | 00:00 | SKIP |
- | octavia | functest | smoke | 00:00 | SKIP |
- +------------------------------------+------------------+---------------+------------------+----------------+
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +---------------------------+------------------+---------------+------------------+----------------+
+ | tempest_neutron | functest | smoke | 15:30 | PASS |
+ | tempest_cinder | functest | smoke | 02:01 | PASS |
+ | tempest_keystone | functest | smoke | 01:17 | PASS |
+ | tempest_heat | functest | smoke | 22:14 | PASS |
+ | tempest_telemetry | functest | smoke | 00:00 | SKIP |
+ | rally_sanity | functest | smoke | 17:24 | PASS |
+ | refstack_compute | functest | smoke | 07:03 | PASS |
+ | refstack_object | functest | smoke | 02:09 | PASS |
+ | refstack_platform | functest | smoke | 07:31 | PASS |
+ | tempest_full | functest | smoke | 41:52 | PASS |
+ | tempest_scenario | functest | smoke | 08:42 | PASS |
+ | tempest_slow | functest | smoke | 43:42 | PASS |
+ | patrole | functest | smoke | 02:42 | PASS |
+ | tempest_barbican | functest | smoke | 02:30 | PASS |
+ | tempest_octavia | functest | smoke | 00:00 | SKIP |
+ +---------------------------+------------------+---------------+------------------+----------------+
+
+Smoke CNTT suite::
+
+ +-------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------------+------------------+---------------+------------------+----------------+
+ | tempest_neutron_cntt | functest | smoke | 11:35 | PASS |
+ | tempest_cinder_cntt | functest | smoke | 01:58 | PASS |
+ | tempest_keystone_cntt | functest | smoke | 01:13 | PASS |
+ | tempest_heat_cntt | functest | smoke | 22:32 | PASS |
+ | rally_sanity_cntt | functest | smoke | 17:16 | PASS |
+ | tempest_full_cntt | functest | smoke | 41:13 | PASS |
+ | tempest_scenario_cntt | functest | smoke | 08:57 | PASS |
+ | tempest_slow_cntt | functest | smoke | 35:58 | PASS |
+ +-------------------------------+------------------+---------------+------------------+----------------+
Benchmarking suite::
+--------------------+------------------+----------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+--------------------+------------------+----------------------+------------------+----------------+
- | rally_full | functest | benchmarking | 92:16 | PASS |
- | rally_jobs | functest | benchmarking | 18:49 | PASS |
- | vmtp | functest | benchmarking | 15:28 | PASS |
- | shaker | functest | benchmarking | 24:04 | PASS |
+ | rally_full | functest | benchmarking | 93:03 | PASS |
+ | rally_jobs | functest | benchmarking | 27:05 | PASS |
+ | vmtp | functest | benchmarking | 17:56 | PASS |
+ | shaker | functest | benchmarking | 24:02 | PASS |
+--------------------+------------------+----------------------+------------------+----------------+
+Benchmarking CNTT suite::
+
+ +-------------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------------+------------------+----------------------+------------------+----------------+
+ | rally_full_cntt | functest | benchmarking | 89:52 | PASS |
+ | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS |
+ | vmtp | functest | benchmarking | 16:59 | PASS |
+ | shaker | functest | benchmarking | 23:43 | PASS |
+ +-------------------------+------------------+----------------------+------------------+----------------+
+
Vnf suite::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 03:49 | PASS |
- | cloudify_ims | functest | vnf | 24:20 | PASS |
- | heat_ims | functest | vnf | 32:13 | PASS |
- | vyos_vrouter | functest | vnf | 14:55 | PASS |
- | juju_epc | functest | vnf | 41:24 | PASS |
+ | cloudify | functest | vnf | 05:08 | PASS |
+ | cloudify_ims | functest | vnf | 24:46 | PASS |
+ | heat_ims | functest | vnf | 33:12 | PASS |
+ | vyos_vrouter | functest | vnf | 15:53 | PASS |
+ | juju_epc | functest | vnf | 27:52 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Kubernetes healthcheck suite::