diff options
| -rw-r--r-- | docs/release/release-notes/functest-release.rst | 13 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 72 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 122 |
3 files changed, 111 insertions, 96 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index f3551d2c8..0a443d11a 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -32,21 +32,26 @@ The internal test cases are: * vping_ssh * vping_userdata * cinder_test - * odl * tempest_smoke + * odl * neutron-tempest-plugin-api * rally_sanity * refstack_compute * refstack_object * refstack_platform + * tempest_full + * tempest_scenario + * tempest_slow * patrole + * barbican * neutron_trunk * networking-bgpvpn * networking-sfc - * barbican - * tempest_full - * tempest_scenario + * octavia * rally_full + * rally_jobs + * vmtp + * shaker * cloudify * cloudify_ims * heat_ims diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index a0ef6bf4a..7731ea4ba 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -29,6 +29,9 @@ cat env:: NAMESERVER=XXX # if not 8.8.8.8 EXTERNAL_NETWORK=XXX # if not first network with router:external=True NEW_USER_ROLE=XXX # if not member + SDN_CONTROLLER_IP=XXX # if odl scenario + VOLUME_DEVICE_NAME=XXX # if not vdb + FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios See section on environment variables for details. @@ -41,6 +44,7 @@ cat env_file:: export OS_PROJECT_NAME=XXX export OS_PASSWORD=XXX export OS_IDENTITY_API_VERSION=3 + export OS_REGION_NAME=XXX See section on OpenStack credentials for details. @@ -51,8 +55,7 @@ Create a directory for the different images (attached as a Docker volume):: images/cirros-0.4.0-aarch64-disk.img images/cirros-0.4.0-x86_64-disk.img images/cloudify-docker-manager-community-19.01.24.tar - images/shaker-image-arm64.qcow2 - images/shaker-image.qcow2 + images/shaker-image-1.3.0+stretch.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img @@ -73,17 +76,17 @@ Results shall be displayed as follows:: +--------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:01 | PASS | + | connection_check | functest | healthcheck | 00:03 | PASS | | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | - | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | - | vmready1 | functest | healthcheck | 00:08 | PASS | - | vmready2 | functest | healthcheck | 00:08 | PASS | - | singlevm1 | functest | healthcheck | 00:35 | PASS | - | singlevm2 | functest | healthcheck | 00:38 | PASS | - | vping_ssh | functest | healthcheck | 00:47 | PASS | - | vping_userdata | functest | healthcheck | 00:33 | PASS | - | cinder_test | functest | healthcheck | 01:16 | PASS | - | tempest_smoke | functest | healthcheck | 07:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:07 | PASS | + | vmready1 | functest | healthcheck | 00:09 | PASS | + | vmready2 | functest | healthcheck | 00:09 | PASS | + | singlevm1 | functest | healthcheck | 00:30 | PASS | + | singlevm2 | functest | healthcheck | 00:33 | PASS | + | vping_ssh | functest | healthcheck | 00:42 | PASS | + | vping_userdata | functest | healthcheck | 00:38 | PASS | + | cinder_test | functest | healthcheck | 01:06 | PASS | + | tempest_smoke | functest | healthcheck | 04:03 | PASS | | odl | functest | healthcheck | 00:00 | SKIP | +--------------------------+------------------+---------------------+------------------+----------------+ @@ -104,18 +107,19 @@ Results shall be displayed as follows:: +------------------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +------------------------------------+------------------+---------------+------------------+----------------+ - | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS | - | rally_sanity | functest | smoke | 14:13 | PASS | - | refstack_compute | functest | smoke | 07:24 | PASS | - | refstack_object | functest | smoke | 02:08 | PASS | - | refstack_platform | functest | smoke | 08:32 | PASS | - | tempest_full | functest | smoke | 58:40 | PASS | - | tempest_scenario | functest | smoke | 09:47 | PASS | - | patrole | functest | smoke | 05:34 | PASS | + | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS | + | rally_sanity | functest | smoke | 16:43 | PASS | + | refstack_compute | functest | smoke | 06:48 | PASS | + | refstack_object | functest | smoke | 02:04 | PASS | + | refstack_platform | functest | smoke | 07:53 | PASS | + | tempest_full | functest | smoke | 31:51 | PASS | + | tempest_scenario | functest | smoke | 09:38 | PASS | + | tempest_slow | functest | smoke | 22:21 | PASS | + | patrole | functest | smoke | 02:32 | PASS | + | barbican | functest | smoke | 02:16 | PASS | | neutron_trunk | functest | smoke | 00:00 | SKIP | - | networking-bgpvpn | functest | smoke | 01:43 | PASS | - | networking-sfc | functest | smoke | 04:49 | PASS | - | barbican | functest | smoke | 05:09 | PASS | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | +------------------------------------+------------------+---------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -136,10 +140,10 @@ Results shall be displayed as follows:: +--------------------+------------------+----------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +--------------------+------------------+----------------------+------------------+----------------+ - | rally_full | functest | benchmarking | 121:55 | PASS | - | rally_jobs | functest | benchmarking | 24:21 | PASS | - | vmtp | functest | benchmarking | 00:00 | SKIP | - | shaker | functest | benchmarking | 00:00 | SKIP | + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 17:31 | PASS | + | vmtp | functest | benchmarking | 15:20 | PASS | + | shaker | functest | benchmarking | 22:44 | PASS | +--------------------+------------------+----------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -160,11 +164,11 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify | functest | vnf | 06:42 | PASS | - | cloudify_ims | functest | vnf | 39:21 | PASS | - | heat_ims | functest | vnf | 34:33 | PASS | - | vyos_vrouter | functest | vnf | 20:20 | PASS | - | juju_epc | functest | vnf | 56:55 | PASS | + | cloudify | functest | vnf | 03:35 | PASS | + | cloudify_ims | functest | vnf | 23:26 | PASS | + | heat_ims | functest | vnf | 34:22 | PASS | + | vyos_vrouter | functest | vnf | 13:49 | PASS | + | juju_epc | functest | vnf | 41:49 | PASS | +----------------------+------------------+--------------+------------------+----------------+ Functest Dockers for Kubernetes deployment @@ -199,7 +203,7 @@ Results shall be displayed as follows:: +-------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 02:27 | PASS | + | k8s_smoke | functest | healthcheck | 01:15 | PASS | +-------------------+------------------+---------------------+------------------+----------------+ Testing smoke suite @@ -216,7 +220,7 @@ Results shall be displayed as follows:: +-------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:14 | PASS | + | k8s_conformance | functest | smoke | 135:54 | PASS | +-------------------------+------------------+---------------+------------------+----------------+ Environment variables diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index c4d85f631..59d3d0fba 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -10,7 +10,7 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/xtesting/ci. +/usr/lib/python3.6/site-packages/xtesting/ci. Automated testing ----------------- @@ -21,75 +21,81 @@ end of each suite and can be described as follow. Healthcheck suite:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:07 | PASS | - | api_check | functest | healthcheck | 07:46 | PASS | - | snaps_health_check | functest | healthcheck | 00:36 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:07 | PASS | + | vmready1 | functest | healthcheck | 00:09 | PASS | + | vmready2 | functest | healthcheck | 00:09 | PASS | + | singlevm1 | functest | healthcheck | 00:30 | PASS | + | singlevm2 | functest | healthcheck | 00:33 | PASS | + | vping_ssh | functest | healthcheck | 00:42 | PASS | + | vping_userdata | functest | healthcheck | 00:38 | PASS | + | cinder_test | functest | healthcheck | 01:06 | PASS | + | tempest_smoke | functest | healthcheck | 04:03 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ Smoke suite:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:57 | PASS | - | vping_userdata | functest | smoke | 00:33 | PASS | - | tempest_smoke_serial | functest | smoke | 13:22 | PASS | - | rally_sanity | functest | smoke | 24:07 | PASS | - | refstack_defcore | functest | smoke | 05:21 | PASS | - | patrole | functest | smoke | 04:29 | PASS | - | snaps_smoke | functest | smoke | 46:54 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - -Components suite:: - - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 48:28 | PASS | - | rally_full | functest | components | 126:02 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +------------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------------+------------------+---------------+------------------+----------------+ + | neutron-tempest-plugin-api | functest | smoke | 09:26 | PASS | + | rally_sanity | functest | smoke | 16:43 | PASS | + | refstack_compute | functest | smoke | 06:48 | PASS | + | refstack_object | functest | smoke | 02:04 | PASS | + | refstack_platform | functest | smoke | 07:53 | PASS | + | tempest_full | functest | smoke | 31:51 | PASS | + | tempest_scenario | functest | smoke | 09:38 | PASS | + | tempest_slow | functest | smoke | 22:21 | PASS | + | patrole | functest | smoke | 02:32 | PASS | + | barbican | functest | smoke | 02:16 | PASS | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | + +------------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 92:16 | PASS | + | rally_jobs | functest | benchmarking | 17:31 | PASS | + | vmtp | functest | benchmarking | 15:20 | PASS | + | shaker | functest | benchmarking | 22:44 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ Vnf suite:: - +----------------------+------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:15 | PASS | - | vyos_vrouter | functest | vnf | 17:59 | PASS | - | juju_epc | functest | vnf | 46:44 | PASS | - +----------------------+------------------+--------------+------------------+----------------+ - -Functest Kubernetes test result:: - - +--------------------------------------+------------------------------------------------------------+ - | ENV VAR | VALUE | - +--------------------------------------+------------------------------------------------------------+ - | INSTALLER_TYPE | compass | - | DEPLOY_SCENARIO | k8-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-compass-baremetal-daily-master-75 | - | CI_LOOP | daily | - +--------------------------------------+------------------------------------------------------------+ + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 03:35 | PASS | + | cloudify_ims | functest | vnf | 23:26 | PASS | + | heat_ims | functest | vnf | 34:22 | PASS | + | vyos_vrouter | functest | vnf | 13:49 | PASS | + | juju_epc | functest | vnf | 41:49 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ Kubernetes healthcheck suite:: - +-------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------+------------------+---------------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 01:54 | PASS | - +-------------------+------------------+---------------------+------------------+----------------+ + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_smoke | functest | healthcheck | 01:15 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ Kubernetes smoke suite:: - +-------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:47 | PASS | - +-------------------------+------------------+---------------+------------------+----------------+ + +-------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 135:54 | PASS | + +-------------------------+------------------+---------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. |