diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/release/release-notes/functest-release.rst | 1 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 3 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 133 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/testing/user/userguide/runfunctest.rst | 114 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 6 |
6 files changed, 58 insertions, 200 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index d0c86f918..4938c57e3 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -95,7 +95,6 @@ Software * https://hub.docker.com/r/opnfv/functest-smoke * https://hub.docker.com/r/opnfv/functest-benchmarking * https://hub.docker.com/r/opnfv/functest-features - * https://hub.docker.com/r/opnfv/functest-components * https://hub.docker.com/r/opnfv/functest-vnf Functest Docker images (Kubernetes): diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 9ca010b95..07191dd80 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -53,7 +53,6 @@ to the different tiers: * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ * functest-features: https://hub.docker.com/r/opnfv/functest-features/ - * functest-components: https://hub.docker.com/r/opnfv/functest-components/ * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ @@ -163,8 +162,8 @@ The definition of the tiers has been agreed by the testing working group. The tiers are: * healthcheck * smoke + * benchmarking * features - * components * vnf Functest abstraction classes diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index d3e770afa..ba0b4bf30 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -18,7 +18,6 @@ Docker images are available on the dockerhub: * opnfv/functest-smoke * opnfv/functest-benchmarking * opnfv/functest-features - * opnfv/functest-components * opnfv/functest-vnf @@ -27,9 +26,10 @@ Preparing your environment cat env:: - EXTERNAL_NETWORK=XXX DEPLOY_SCENARIO=XXX # if not os-nosdn-nofeature-noha scenario NAMESERVER=XXX # if not 8.8.8.8 + EXTERNAL_NETWORK=XXX # if not first network with router:external=True + NEW_USER_ROLE=XXX # if not member See section on environment variables for details. @@ -47,20 +47,17 @@ See section on OpenStack credentials for details. Create a directory for the different images (attached as a Docker volume):: - mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/fraser | bash -s -- images && ls -1 images/* + mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/iruya | bash -s -- images && ls -1 images/* - images/CentOS-7-aarch64-GenericCloud.qcow2 - images/CentOS-7-aarch64-GenericCloud.qcow2.xz - images/CentOS-7-x86_64-GenericCloud.qcow2 + images/cirros-0.4.0-aarch64-disk.img images/cirros-0.4.0-x86_64-disk.img - images/cirros-0.4.0-x86_64-lxc.tar.gz - images/cloudify-manager-premium-4.0.1.qcow2 + images/cloudify-docker-manager-community-19.01.24.tar images/shaker-image-arm64.qcow2 - images/shaker-image.qcow + images/shaker-image.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img - images/vyos-1.1.7.img + images/vyos-1.1.8-amd64.qcow2 Testing healthcheck suite ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -74,23 +71,22 @@ Run healthcheck suite:: Results shall be displayed as follows:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:09 | PASS | - | tenantnetwork1 | functest | healthcheck | 00:14 | PASS | - | tenantnetwork2 | functest | healthcheck | 00:11 | PASS | - | vmready1 | functest | healthcheck | 00:19 | PASS | - | vmready2 | functest | healthcheck | 00:16 | PASS | - | singlevm1 | functest | healthcheck | 00:41 | PASS | - | singlevm2 | functest | healthcheck | 00:36 | PASS | - | vping_ssh | functest | healthcheck | 00:46 | PASS | - | vping_userdata | functest | healthcheck | 00:41 | PASS | - | cinder_test | functest | healthcheck | 01:18 | PASS | - | api_check | functest | healthcheck | 10:33 | PASS | - | snaps_health_check | functest | healthcheck | 00:44 | PASS | - | odl | functest | healthcheck | 00:00 | SKIP | - +----------------------------+------------------+---------------------+------------------+----------------+ + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:01 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:06 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:08 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:35 | PASS | + | singlevm2 | functest | healthcheck | 00:38 | PASS | + | vping_ssh | functest | healthcheck | 00:47 | PASS | + | vping_userdata | functest | healthcheck | 00:33 | PASS | + | cinder_test | functest | healthcheck | 01:16 | PASS | + | tempest_smoke | functest | healthcheck | 07:05 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ NOTE: the duration is a reference and it might vary depending on your SUT. @@ -109,17 +105,16 @@ Results shall be displayed as follows:: +------------------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +------------------------------------+------------------+---------------+------------------+----------------+ - | tempest_smoke | functest | smoke | 06:13 | PASS | - | neutron-tempest-plugin-api | functest | smoke | 09:32 | PASS | - | rally_sanity | functest | smoke | 29:34 | PASS | - | rally_jobs | functest | smoke | 24:02 | PASS | - | refstack_defcore | functest | smoke | 13:07 | PASS | - | patrole | functest | smoke | 05:17 | PASS | - | snaps_smoke | functest | smoke | 90:13 | PASS | + | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS | + | rally_sanity | functest | smoke | 14:13 | PASS | + | refstack_defcore | functest | smoke | 09:60 | PASS | + | tempest_full | functest | smoke | 58:40 | PASS | + | tempest_scenario | functest | smoke | 09:47 | PASS | + | patrole | functest | smoke | 05:34 | PASS | | neutron_trunk | functest | smoke | 00:00 | SKIP | - | networking-bgpvpn | functest | smoke | 00:00 | SKIP | - | networking-sfc | functest | smoke | 00:00 | SKIP | - | barbican | functest | smoke | 05:01 | PASS | + | networking-bgpvpn | functest | smoke | 01:43 | PASS | + | networking-sfc | functest | smoke | 04:49 | PASS | + | barbican | functest | smoke | 05:09 | PASS | +------------------------------------+------------------+---------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. @@ -137,12 +132,14 @@ Run benchmarking suite:: Results shall be displayed as follows:: - +-------------------+------------------+----------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------+------------------+----------------------+------------------+----------------+ - | vmtp | functest | benchmarking | 18:43 | PASS | - | shaker | functest | benchmarking | 29:45 | PASS | - +-------------------+------------------+----------------------+------------------+----------------+ + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 121:55 | PASS | + | rally_jobs | functest | benchmarking | 24:21 | PASS | + | vmtp | functest | benchmarking | 00:00 | SKIP | + | shaker | functest | benchmarking | 00:00 | SKIP | + +--------------------+------------------+----------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. See User guide for details. @@ -159,41 +156,19 @@ Run features suite:: Results shall be displayed as follows:: - +-----------------------------+------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-----------------------------+------------------------+------------------+------------------+----------------+ - | doctor-notification | doctor | features | 00:00 | SKIP | - | bgpvpn | sdnvpn | features | 00:00 | SKIP | - | functest-odl-sfc | sfc | features | 00:00 | SKIP | - | barometercollectd | barometer | features | 00:00 | SKIP | - | fds | fastdatastacks | features | 00:00 | SKIP | - | vgpu | functest | features | 00:00 | SKIP | - | stor4nfv_os | stor4nfv | features | 00:00 | SKIP | - +-----------------------------+------------------------+------------------+------------------+----------------+ + +-----------------------------+-------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-----------------------------+-------------------+------------------+------------------+----------------+ + | doctor-notification | doctor | features | 00:00 | SKIP | + | functest-odl-sfc | sfc | features | 00:00 | SKIP | + | barometercollectd | barometer | features | 00:00 | SKIP | + | vgpu | functest | features | 00:00 | SKIP | + | stor4nfv_os | stor4nfv | features | 00:00 | SKIP | + +-----------------------------+-------------------+------------------+------------------+----------------+ Note: if the scenario does not support some tests, they are indicated as SKIP. See User guide for details. -Testing components suite -^^^^^^^^^^^^^^^^^^^^^^^^ - -Run components suite:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-components - -Results shall be displayed as follows:: - - +--------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +--------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full | functest | components | 49:51 | PASS | - | tempest_scenario | functest | components | 18:50 | PASS | - | rally_full | functest | components | 167:13 | PASS | - +--------------------------+------------------+--------------------+------------------+----------------+ - Testing vnf suite ^^^^^^^^^^^^^^^^^ @@ -209,11 +184,11 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify | functest | vnf | 04:05 | PASS | - | cloudify_ims | functest | vnf | 24:07 | PASS | - | heat_ims | functest | vnf | 18:15 | PASS | - | vyos_vrouter | functest | vnf | 15:48 | PASS | - | juju_epc | functest | vnf | 29:38 | PASS | + | cloudify | functest | vnf | 06:42 | PASS | + | cloudify_ims | functest | vnf | 39:21 | PASS | + | heat_ims | functest | vnf | 34:33 | PASS | + | vyos_vrouter | functest | vnf | 20:20 | PASS | + | juju_epc | functest | vnf | 56:55 | PASS | +----------------------+------------------+--------------+------------------+----------------+ Functest Dockers for Kubernetes deployment diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index bfe8c95b2..ecbc7f5fd 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -12,7 +12,6 @@ Functest User Guide intro.rst test_overview.rst test_details.rst - runfunctest.rst test_results.rst reporting.rst troubleshooting.rst diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst deleted file mode 100644 index aea8c02b7..000000000 --- a/docs/testing/user/userguide/runfunctest.rst +++ /dev/null @@ -1,114 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-4.0 - -Executing Functest suites -========================= - -As mentioned in the :ref:`functest-install-guide`, Alpine docker containers -have been introduced in Euphrates. -Tier containers have been created. -Assuming that you pulled the container and your environement is ready, you can -simply run the tiers by typing (e.g. with functest-healthcheck):: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck - -You should get:: - - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:02 | PASS | - | api_check | functest | healthcheck | 03:19 | PASS | - | snaps_health_check | functest | healthcheck | 00:46 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - -You can run functest-healcheck, functest-smoke, functest-features, -functest-components and functest-vnf. - -The result tables show the results by test case, it can be:: - - * PASS - * FAIL - * SKIP: if the scenario/installer does not support the test case - - -Manual run -========== -If you want to run the test step by step, you may add docker option then run -the different commands within the docker. - -Considering the healthcheck example, running functest manaully means:: - - sudo docker run -ti --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck /bin/bash - -The docker prompt shall be returned. Then within the docker run the following -commands:: - - $ source /home/opnfv/functest/conf/env_file - -Tier ----- -Each Alpine container provided on the docker hub matches with a tier. -The following commands are available:: - - # functest tier list - - 0. healthcheck: - ['connection_check', 'api_check', 'snaps_health_check'] - # functest tier show healthcheck - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | - | | | | basic operations in the VIM. | snaps_health_check | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - -To run all the cases of the tier, type:: - - # functest tier run healthcheck - -Testcase --------- -Testcases can be listed, shown and run though the CLI:: - - # functest testcase list - connection_check - api_check - snaps_health_check - # functest testcase show api_check - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | api_check | This test case verifies the retrieval of | 100 | ^((?!lxd).)*$ | - | | OpenStack clients: Keystone, Glance, | | | - | | Neutron and Nova and may perform some | | | - | | simple queries. When the config value of | | | - | | snaps.use_keystone is True, functest | | | - | | must have access to the cloud's private | | | - | | network. | | | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - # functest testcase run connection_check - ... - # functest run all - -You can also type run_tests -t all to run all the tests. - -Note the list of test cases depend on the installer and the scenario. - -Note that the flavors for the SNAPS test cases are able to be configured giving -new metadata values as well as new values for the basic elements of flavor -(i.e. ram, vcpu, disk, ephemeral, swap etc). -The snaps.flavor_extra_specs dict in the config_functest.yaml file could be -used for this purpose. - -Reporting results to the test Database -====================================== -In OPNFV CI we collect all the results from CI. A test API shall be available -as well as a test database `[16]`_. - - -.. _`[16]`: http://testresults.opnfv.org/test/ diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 5755544e6..b0711be82 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -258,8 +258,8 @@ debug information can be found from tempest.log file stored into related Rally deployment folder. Functest offers a possibility to test a customized list of Tempest test cases. -To enable that, add a new entry in docker/components/testcases.yaml on the -"components" container with the following content:: +To enable that, add a new entry in docker/smoke/testcases.yaml on the +"smoke" container with the following content:: - case_name: tempest_custom @@ -292,7 +292,7 @@ This is an example of running a customized list of Tempest tests in Functest:: -v $(pwd)/images:/home/opnfv/functest/images \ -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \ -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ - opnfv/functest-components run_tests -t tempest_custom + opnfv/functest-smoke run_tests -t tempest_custom Rally |