diff options
| -rw-r--r-- | docs/release/release-notes/functest-release.rst | 14 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 78 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 122 |
3 files changed, 117 insertions, 97 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index 2f8b3e28d..5c668dcc5 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -32,21 +32,26 @@ The internal test cases are: * vping_ssh * vping_userdata * cinder_test - * odl * tempest_smoke + * odl * neutron-tempest-plugin-api * rally_sanity * refstack_compute * refstack_object * refstack_platform + * tempest_full + * tempest_scenario + * tempest_slow * patrole + * barbican * neutron_trunk * networking-bgpvpn * networking-sfc - * barbican - * tempest_full - * tempest_scenario + * octavia * rally_full + * rally_jobs + * vmtp + * shaker * cloudify * cloudify_ims * heat_ims @@ -59,6 +64,7 @@ Kubernetes The internal test cases are: * k8s_smoke + * xrally_kubernetes * k8s_conformance Release Data diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index c50fd1f1f..25aa7cd26 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -29,6 +29,9 @@ cat env:: NAMESERVER=XXX # if not 8.8.8.8 EXTERNAL_NETWORK=XXX # if not first network with router:external=True NEW_USER_ROLE=XXX # if not member + SDN_CONTROLLER_IP=XXX # if odl scenario + VOLUME_DEVICE_NAME=XXX # if not vdb + FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios See section on environment variables for details. @@ -41,6 +44,7 @@ cat env_file:: export OS_PROJECT_NAME=XXX export OS_PASSWORD=XXX export OS_IDENTITY_API_VERSION=3 + export OS_REGION_NAME=XXX See section on OpenStack credentials for details. @@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume):: images/cirros-0.4.0-aarch64-disk.img images/cirros-0.4.0-x86_64-disk.img images/cloudify-docker-manager-community-19.01.24.tar - images/shaker-image-arm64.qcow2 - images/shaker-image.qcow2 + images/shaker-image-1.3.0+stretch.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img @@ -73,17 +76,17 @@ Results shall be displayed as follows:: +--------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:01 | PASS | + | connection_check | functest | healthcheck | 00:02 | PASS | | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | - | vmready1 | functest | healthcheck | 00:08 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | | vmready2 | functest | healthcheck | 00:08 | PASS | - | singlevm1 | functest | healthcheck | 00:35 | PASS | - | singlevm2 | functest | healthcheck | 00:38 | PASS | - | vping_ssh | functest | healthcheck | 00:47 | PASS | - | vping_userdata | functest | healthcheck | 00:33 | PASS | - | cinder_test | functest | healthcheck | 01:16 | PASS | - | tempest_smoke | functest | healthcheck | 07:05 | PASS | + | singlevm1 | functest | healthcheck | 00:28 | PASS | + | singlevm2 | functest | healthcheck | 00:25 | PASS | + | vping_ssh | functest | healthcheck | 00:36 | PASS | + | vping_userdata | functest | healthcheck | 00:34 | PASS | + | cinder_test | functest | healthcheck | 01:03 | PASS | + | tempest_smoke | functest | healthcheck | 05:13 | PASS | | odl | functest | healthcheck | 00:00 | SKIP | +--------------------------+------------------+---------------------+------------------+----------------+ @@ -104,18 +107,20 @@ Results shall be displayed as follows:: +------------------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +------------------------------------+------------------+---------------+------------------+----------------+ - | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS | - | rally_sanity | functest | smoke | 14:13 | PASS | - | refstack_compute | functest | smoke | 07:24 | PASS | - | refstack_object | functest | smoke | 02:08 | PASS | - | refstack_platform | functest | smoke | 08:32 | PASS | - | tempest_full | functest | smoke | 58:40 | PASS | - | tempest_scenario | functest | smoke | 09:47 | PASS | - | patrole | functest | smoke | 05:34 | PASS | + | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS | + | rally_sanity | functest | smoke | 16:29 | PASS | + | refstack_compute | functest | smoke | 06:25 | PASS | + | refstack_object | functest | smoke | 01:54 | PASS | + | refstack_platform | functest | smoke | 06:52 | PASS | + | tempest_full | functest | smoke | 30:26 | PASS | + | tempest_scenario | functest | smoke | 09:23 | PASS | + | tempest_slow | functest | smoke | 24:42 | PASS | + | patrole | functest | smoke | 02:36 | PASS | + | barbican | functest | smoke | 02:13 | PASS | | neutron_trunk | functest | smoke | 00:00 | SKIP | - | networking-bgpvpn | functest | smoke | 01:43 | PASS | - | networking-sfc | functest | smoke | 04:49 | PASS | - | barbican | functest | smoke | 05:09 | PASS | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | + | octavia | functest | smoke | 00:00 | SKIP | +------------------------------------+------------------+---------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -136,10 +141,10 @@ Results shall be displayed as follows:: +--------------------+------------------+----------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------+------------------+----------------------+------------------+----------------+ - | rally_full | functest | benchmarking | 121:55 | PASS | - | rally_jobs | functest | benchmarking | 24:21 | PASS | - | vmtp | functest | benchmarking | 00:00 | SKIP | - | shaker | functest | benchmarking | 00:00 | SKIP | + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 18:49 | PASS | + | vmtp | functest | benchmarking | 15:28 | PASS | + | shaker | functest | benchmarking | 24:04 | PASS | +--------------------+------------------+----------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -160,11 +165,11 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify | functest | vnf | 06:42 | PASS | - | cloudify_ims | functest | vnf | 39:21 | PASS | - | heat_ims | functest | vnf | 34:33 | PASS | - | vyos_vrouter | functest | vnf | 20:20 | PASS | - | juju_epc | functest | vnf | 56:55 | PASS | + | cloudify | functest | vnf | 03:49 | PASS | + | cloudify_ims | functest | vnf | 24:20 | PASS | + | heat_ims | functest | vnf | 32:13 | PASS | + | vyos_vrouter | functest | vnf | 14:55 | PASS | + | juju_epc | functest | vnf | 41:24 | PASS | +----------------------+------------------+--------------+------------------+----------------+ Functest Dockers for Kubernetes deployment @@ -199,7 +204,7 @@ Results shall be displayed as follows:: +-------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 02:27 | PASS | + | k8s_smoke | functest | healthcheck | 01:09 | PASS | +-------------------+------------------+---------------------+------------------+----------------+ Testing smoke suite @@ -213,11 +218,12 @@ Run smoke suite:: Results shall be displayed as follows:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:14 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | xrally_kubernetes | functest | smoke | 22:04 | PASS | + | k8s_conformance | functest | smoke | 173:48 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ Environment variables ===================== diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 8023c4661..511b0b4cb 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -21,75 +21,83 @@ end of each suite and can be described as follow. Healthcheck suite:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:07 | PASS | - | api_check | functest | healthcheck | 07:46 | PASS | - | snaps_health_check | functest | healthcheck | 00:36 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:02 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:28 | PASS | + | singlevm2 | functest | healthcheck | 00:25 | PASS | + | vping_ssh | functest | healthcheck | 00:36 | PASS | + | vping_userdata | functest | healthcheck | 00:34 | PASS | + | cinder_test | functest | healthcheck | 01:03 | PASS | + | tempest_smoke | functest | healthcheck | 05:13 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ Smoke suite:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:57 | PASS | - | vping_userdata | functest | smoke | 00:33 | PASS | - | tempest_smoke_serial | functest | smoke | 13:22 | PASS | - | rally_sanity | functest | smoke | 24:07 | PASS | - | refstack_defcore | functest | smoke | 05:21 | PASS | - | patrole | functest | smoke | 04:29 | PASS | - | snaps_smoke | functest | smoke | 46:54 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - -Components suite:: - - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 48:28 | PASS | - | rally_full | functest | components | 126:02 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +------------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------------+------------------+---------------+------------------+----------------+ + | neutron-tempest-plugin-api | functest | smoke | 09:12 | PASS | + | rally_sanity | functest | smoke | 16:29 | PASS | + | refstack_compute | functest | smoke | 06:25 | PASS | + | refstack_object | functest | smoke | 01:54 | PASS | + | refstack_platform | functest | smoke | 06:52 | PASS | + | tempest_full | functest | smoke | 30:26 | PASS | + | tempest_scenario | functest | smoke | 09:23 | PASS | + | tempest_slow | functest | smoke | 24:42 | PASS | + | patrole | functest | smoke | 02:36 | PASS | + | barbican | functest | smoke | 02:13 | PASS | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | + | octavia | functest | smoke | 00:00 | SKIP | + +------------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 18:49 | PASS | + | vmtp | functest | benchmarking | 15:28 | PASS | + | shaker | functest | benchmarking | 24:04 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ Vnf suite:: - +----------------------+------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:15 | PASS | - | vyos_vrouter | functest | vnf | 17:59 | PASS | - | juju_epc | functest | vnf | 46:44 | PASS | - +----------------------+------------------+--------------+------------------+----------------+ - -Functest Kubernetes test result:: - - +--------------------------------------+------------------------------------------------------------+ - | ENV VAR | VALUE | - +--------------------------------------+------------------------------------------------------------+ - | INSTALLER_TYPE | compass | - | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | - | CI_LOOP | daily | - +--------------------------------------+------------------------------------------------------------+ + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 03:49 | PASS | + | cloudify_ims | functest | vnf | 24:20 | PASS | + | heat_ims | functest | vnf | 32:13 | PASS | + | vyos_vrouter | functest | vnf | 14:55 | PASS | + | juju_epc | functest | vnf | 41:24 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ Kubernetes healthcheck suite:: - +-------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 01:54 | PASS | - +-------------------+------------------+---------------------+------------------+----------------+ + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_smoke | functest | healthcheck | 01:09 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ Kubernetes smoke suite:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:47 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | xrally_kubernetes | functest | smoke | 22:04 | PASS | + | k8s_conformance | functest | smoke | 173:48 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. |