aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
authorCédric Ollivier <cedric.ollivier@orange.com>2019-04-09 20:46:45 +0200
committerCédric Ollivier <cedric.ollivier@orange.com>2019-04-09 20:47:43 +0200
commite108f1ad405e788e0516a55875269adafb338c28 (patch)
treeb232c2f8a427d3759ac0923730e533543ddf308e /docs/testing
parentebc2b41505299846faeb2163d34d820a8ef2bdb2 (diff)
Update tab results
It also removes all refs to components and to the former binaries. Change-Id: I4f7e3308673ba6ba4dd9c28a7a053e3d558c289d Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/developer/devguide/index.rst3
-rw-r--r--docs/testing/user/configguide/configguide.rst133
-rw-r--r--docs/testing/user/userguide/index.rst1
-rw-r--r--docs/testing/user/userguide/runfunctest.rst114
-rw-r--r--docs/testing/user/userguide/troubleshooting.rst6
5 files changed, 58 insertions, 199 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index 9ca010b95..07191dd80 100644
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -53,7 +53,6 @@ to the different tiers:
* functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
* functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
* functest-features: https://hub.docker.com/r/opnfv/functest-features/
- * functest-components: https://hub.docker.com/r/opnfv/functest-components/
* functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
* functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
@@ -163,8 +162,8 @@ The definition of the tiers has been agreed by the testing working group.
The tiers are:
* healthcheck
* smoke
+ * benchmarking
* features
- * components
* vnf
Functest abstraction classes
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index d3e770afa..8ebab03ca 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -18,7 +18,6 @@ Docker images are available on the dockerhub:
* opnfv/functest-smoke
* opnfv/functest-benchmarking
* opnfv/functest-features
- * opnfv/functest-components
* opnfv/functest-vnf
@@ -27,9 +26,10 @@ Preparing your environment
cat env::
- EXTERNAL_NETWORK=XXX
DEPLOY_SCENARIO=XXX # if not os-nosdn-nofeature-noha scenario
NAMESERVER=XXX # if not 8.8.8.8
+ EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+ NEW_USER_ROLE=XXX # if not member
See section on environment variables for details.
@@ -47,20 +47,17 @@ See section on OpenStack credentials for details.
Create a directory for the different images (attached as a Docker volume)::
- mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/fraser | bash -s -- images && ls -1 images/*
+ mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/*
- images/CentOS-7-aarch64-GenericCloud.qcow2
- images/CentOS-7-aarch64-GenericCloud.qcow2.xz
- images/CentOS-7-x86_64-GenericCloud.qcow2
+ images/cirros-0.4.0-aarch64-disk.img
images/cirros-0.4.0-x86_64-disk.img
- images/cirros-0.4.0-x86_64-lxc.tar.gz
- images/cloudify-manager-premium-4.0.1.qcow2
+ images/cloudify-docker-manager-community-19.01.24.tar
images/shaker-image-arm64.qcow2
- images/shaker-image.qcow
+ images/shaker-image.qcow2
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
- images/vyos-1.1.7.img
+ images/vyos-1.1.8-amd64.qcow2
Testing healthcheck suite
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -74,23 +71,22 @@ Run healthcheck suite::
Results shall be displayed as follows::
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:09 | PASS |
- | tenantnetwork1 | functest | healthcheck | 00:14 | PASS |
- | tenantnetwork2 | functest | healthcheck | 00:11 | PASS |
- | vmready1 | functest | healthcheck | 00:19 | PASS |
- | vmready2 | functest | healthcheck | 00:16 | PASS |
- | singlevm1 | functest | healthcheck | 00:41 | PASS |
- | singlevm2 | functest | healthcheck | 00:36 | PASS |
- | vping_ssh | functest | healthcheck | 00:46 | PASS |
- | vping_userdata | functest | healthcheck | 00:41 | PASS |
- | cinder_test | functest | healthcheck | 01:18 | PASS |
- | api_check | functest | healthcheck | 10:33 | PASS |
- | snaps_health_check | functest | healthcheck | 00:44 | PASS |
- | odl | functest | healthcheck | 00:00 | SKIP |
- +----------------------------+------------------+---------------------+------------------+----------------+
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------------+------------------+---------------------+------------------+----------------+
+ | connection_check | functest | healthcheck | 00:01 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:06 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:06 | PASS |
+ | vmready1 | functest | healthcheck | 00:08 | PASS |
+ | vmready2 | functest | healthcheck | 00:08 | PASS |
+ | singlevm1 | functest | healthcheck | 00:35 | PASS |
+ | singlevm2 | functest | healthcheck | 00:38 | PASS |
+ | vping_ssh | functest | healthcheck | 00:47 | PASS |
+ | vping_userdata | functest | healthcheck | 00:33 | PASS |
+ | cinder_test | functest | healthcheck | 01:16 | PASS |
+ | tempest_smoke | functest | healthcheck | 07:05 | PASS |
+ | odl | functest | healthcheck | 00:00 | SKIP |
+ +--------------------------+------------------+---------------------+------------------+----------------+
NOTE: the duration is a reference and it might vary depending on your SUT.
@@ -109,17 +105,16 @@ Results shall be displayed as follows::
+------------------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+------------------------------------+------------------+---------------+------------------+----------------+
- | tempest_smoke | functest | smoke | 06:13 | PASS |
- | neutron-tempest-plugin-api | functest | smoke | 09:32 | PASS |
- | rally_sanity | functest | smoke | 29:34 | PASS |
- | rally_jobs | functest | smoke | 24:02 | PASS |
- | refstack_defcore | functest | smoke | 13:07 | PASS |
- | patrole | functest | smoke | 05:17 | PASS |
- | snaps_smoke | functest | smoke | 90:13 | PASS |
+ | neutron-tempest-plugin-api | functest | smoke | 13:50 | PASS |
+ | rally_sanity | functest | smoke | 14:13 | PASS |
+ | refstack_defcore | functest | smoke | 09:60 | PASS |
+ | tempest_full | functest | smoke | 58:40 | PASS |
+ | tempest_scenario | functest | smoke | 09:47 | PASS |
+ | patrole | functest | smoke | 05:34 | PASS |
| neutron_trunk | functest | smoke | 00:00 | SKIP |
- | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
- | networking-sfc | functest | smoke | 00:00 | SKIP |
- | barbican | functest | smoke | 05:01 | PASS |
+ | networking-bgpvpn | functest | smoke | 01:43 | PASS |
+ | networking-sfc | functest | smoke | 04:49 | PASS |
+ | barbican | functest | smoke | 05:09 | PASS |
+------------------------------------+------------------+---------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
@@ -137,12 +132,14 @@ Run benchmarking suite::
Results shall be displayed as follows::
- +-------------------+------------------+----------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------+------------------+----------------------+------------------+----------------+
- | vmtp | functest | benchmarking | 18:43 | PASS |
- | shaker | functest | benchmarking | 29:45 | PASS |
- +-------------------+------------------+----------------------+------------------+----------------+
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------+------------------+----------------------+------------------+----------------+
+ | rally_full | functest | benchmarking | 121:55 | PASS |
+ | rally_jobs | functest | benchmarking | 24:21 | PASS |
+ | vmtp | functest | benchmarking | 00:00 | SKIP |
+ | shaker | functest | benchmarking | 00:00 | SKIP |
+ +--------------------+------------------+----------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
@@ -159,41 +156,19 @@ Run features suite::
Results shall be displayed as follows::
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | doctor-notification | doctor | features | 00:00 | SKIP |
- | bgpvpn | sdnvpn | features | 00:00 | SKIP |
- | functest-odl-sfc | sfc | features | 00:00 | SKIP |
- | barometercollectd | barometer | features | 00:00 | SKIP |
- | fds | fastdatastacks | features | 00:00 | SKIP |
- | vgpu | functest | features | 00:00 | SKIP |
- | stor4nfv_os | stor4nfv | features | 00:00 | SKIP |
- +-----------------------------+------------------------+------------------+------------------+----------------+
+ +-----------------------------+-------------------+------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-----------------------------+-------------------+------------------+------------------+----------------+
+ | doctor-notification | doctor | features | 00:00 | SKIP |
+ | functest-odl-sfc | sfc | features | 00:00 | SKIP |
+ | barometercollectd | barometer | features | 00:00 | SKIP |
+ | vgpu | functest | features | 00:00 | SKIP |
+ | stor4nfv_os | stor4nfv | features | 00:00 | SKIP |
+ +-----------------------------+-------------------+------------------+------------------+----------------+
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
-Testing components suite
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Run components suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-components
-
-Results shall be displayed as follows::
-
- +--------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +--------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full | functest | components | 49:51 | PASS |
- | tempest_scenario | functest | components | 18:50 | PASS |
- | rally_full | functest | components | 167:13 | PASS |
- +--------------------------+------------------+--------------------+------------------+----------------+
-
Testing vnf suite
^^^^^^^^^^^^^^^^^
@@ -209,11 +184,11 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify | functest | vnf | 04:05 | PASS |
- | cloudify_ims | functest | vnf | 24:07 | PASS |
- | heat_ims | functest | vnf | 18:15 | PASS |
- | vyos_vrouter | functest | vnf | 15:48 | PASS |
- | juju_epc | functest | vnf | 29:38 | PASS |
+ | cloudify | functest | vnf | 06:42 | PASS |
+ | cloudify_ims | functest | vnf | 39:21 | PASS |
+ | heat_ims | functest | vnf | 34:33 | PASS |
+ | vyos_vrouter | functest | vnf | 20:20 | PASS |
+ | juju_epc | functest | vnf | 56:55 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
Functest Dockers for Kubernetes deployment
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
index bfe8c95b2..ecbc7f5fd 100644
--- a/docs/testing/user/userguide/index.rst
+++ b/docs/testing/user/userguide/index.rst
@@ -12,7 +12,6 @@ Functest User Guide
intro.rst
test_overview.rst
test_details.rst
- runfunctest.rst
test_results.rst
reporting.rst
troubleshooting.rst
diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst
deleted file mode 100644
index aea8c02b7..000000000
--- a/docs/testing/user/userguide/runfunctest.rst
+++ /dev/null
@@ -1,114 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-4.0
-
-Executing Functest suites
-=========================
-
-As mentioned in the :ref:`functest-install-guide`, Alpine docker containers
-have been introduced in Euphrates.
-Tier containers have been created.
-Assuming that you pulled the container and your environement is ready, you can
-simply run the tiers by typing (e.g. with functest-healthcheck)::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-healthcheck
-
-You should get::
-
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:02 | PASS |
- | api_check | functest | healthcheck | 03:19 | PASS |
- | snaps_health_check | functest | healthcheck | 00:46 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
-
-You can run functest-healcheck, functest-smoke, functest-features,
-functest-components and functest-vnf.
-
-The result tables show the results by test case, it can be::
-
- * PASS
- * FAIL
- * SKIP: if the scenario/installer does not support the test case
-
-
-Manual run
-==========
-If you want to run the test step by step, you may add docker option then run
-the different commands within the docker.
-
-Considering the healthcheck example, running functest manaully means::
-
- sudo docker run -ti --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-healthcheck /bin/bash
-
-The docker prompt shall be returned. Then within the docker run the following
-commands::
-
- $ source /home/opnfv/functest/conf/env_file
-
-Tier
-----
-Each Alpine container provided on the docker hub matches with a tier.
-The following commands are available::
-
- # functest tier list
- - 0. healthcheck:
- ['connection_check', 'api_check', 'snaps_health_check']
- # functest tier show healthcheck
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
- | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES |
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
- | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check |
- | | | | basic operations in the VIM. | snaps_health_check |
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
-
-To run all the cases of the tier, type::
-
- # functest tier run healthcheck
-
-Testcase
---------
-Testcases can be listed, shown and run though the CLI::
-
- # functest testcase list
- connection_check
- api_check
- snaps_health_check
- # functest testcase show api_check
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY |
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- | api_check | This test case verifies the retrieval of | 100 | ^((?!lxd).)*$ |
- | | OpenStack clients: Keystone, Glance, | | |
- | | Neutron and Nova and may perform some | | |
- | | simple queries. When the config value of | | |
- | | snaps.use_keystone is True, functest | | |
- | | must have access to the cloud's private | | |
- | | network. | | |
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- # functest testcase run connection_check
- ...
- # functest run all
-
-You can also type run_tests -t all to run all the tests.
-
-Note the list of test cases depend on the installer and the scenario.
-
-Note that the flavors for the SNAPS test cases are able to be configured giving
-new metadata values as well as new values for the basic elements of flavor
-(i.e. ram, vcpu, disk, ephemeral, swap etc).
-The snaps.flavor_extra_specs dict in the config_functest.yaml file could be
-used for this purpose.
-
-Reporting results to the test Database
-======================================
-In OPNFV CI we collect all the results from CI. A test API shall be available
-as well as a test database `[16]`_.
-
-
-.. _`[16]`: http://testresults.opnfv.org/test/
diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst
index 5755544e6..b0711be82 100644
--- a/docs/testing/user/userguide/troubleshooting.rst
+++ b/docs/testing/user/userguide/troubleshooting.rst
@@ -258,8 +258,8 @@ debug information can be found from tempest.log file stored into related Rally
deployment folder.
Functest offers a possibility to test a customized list of Tempest test cases.
-To enable that, add a new entry in docker/components/testcases.yaml on the
-"components" container with the following content::
+To enable that, add a new entry in docker/smoke/testcases.yaml on the
+"smoke" container with the following content::
-
case_name: tempest_custom
@@ -292,7 +292,7 @@ This is an example of running a customized list of Tempest tests in Functest::
-v $(pwd)/images:/home/opnfv/functest/images \
-v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \
-v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \
- opnfv/functest-components run_tests -t tempest_custom
+ opnfv/functest-smoke run_tests -t tempest_custom
Rally