summaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/release/release-notes/functest-release.rst98
-rw-r--r--docs/testing/developer/devguide/index.rst43
-rw-r--r--docs/testing/user/configguide/configguide.rst154
3 files changed, 157 insertions, 138 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst
index bb7600d35..67fa3dcbb 100644
--- a/docs/release/release-notes/functest-release.rst
+++ b/docs/release/release-notes/functest-release.rst
@@ -23,22 +23,37 @@ OpenStack
The internal test cases are:
* connection_check
- * api_check
- * snaps_health_check
+ * tenantnetwork1
+ * tenantnetwork2
+ * vmready1
+ * vmready2
+ * singlevm1
+ * singlevm2
* vping_ssh
* vping_userdata
- * tempest_smoke_serial
+ * cinder_test
+ * api_check
+ * snaps_health_check
+ * odl
+ * tempest_smoke
+ * neutron-tempest-plugin-api
* rally_sanity
* refstack_defcore
* patrole
- * odl
* snaps_smoke
* neutron_trunk
+ * networking-bgpvpn
+ * networking-sfc
+ * barbican
* tempest_full
+ * tempest_scenario
* rally_full
+ * cloudify
* cloudify_ims
+ * heat_ims
* vyos_vrouter
* juju_epc
+ * vgpu
The OPNFV projects integrated into Functest framework for automation are:
@@ -47,6 +62,7 @@ The OPNFV projects integrated into Functest framework for automation are:
* odl-sfc
* barometer
* fds
+ * stor4nfv_os
Kubernetes
----------
@@ -80,6 +96,7 @@ Software
* https://hub.docker.com/r/opnfv/functest-healthcheck
* https://hub.docker.com/r/opnfv/functest-smoke
+ * https://hub.docker.com/r/opnfv/functest-benchmarking
* https://hub.docker.com/r/opnfv/functest-features
* https://hub.docker.com/r/opnfv/functest-components
* https://hub.docker.com/r/opnfv/functest-vnf
@@ -95,9 +112,9 @@ Docker tag for master: latest
Documents
---------
- * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
- * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
- * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
+ * Config Guide: https://functest.readthedocs.io/projects/configguide/en/latest/
+ * User Guide: https://functest.readthedocs.io/projects/userguide/en/latest/
+ * Developer Guide: https://functest.readthedocs.io/projects/devguide/en/latest/
* API Docs: http://functest.readthedocs.io/en/latest/
Version change
@@ -106,44 +123,51 @@ Version change
New test cases
--------------
- * neutron_trunk
- * patrole
- * juju_epc
- * k8s_smoke
- * k8s_conformance
- * stor4nfv
- * clover
+ * tenantnetwork1
+ * tenantnetwork2
+ * vmready1
+ * vmready2
+ * singlevm1
+ * singlevm2
+ * cinder_test
+ * neutron-tempest-plugin-api
+ * rally_jobs
+ * networking-bgpvpn
+ * networking-sfc
+ * barbican
+ * vmtp
+ * shaker
+ * tempest_scenario
+ * cloudify
+ * heat_ims
+ * vgpu
Key changes
-----------
- * update test cases and containers to `OpenStack Pike`_
- * move the framework into a separate project: Xtesting_
- * ease running all containers thanks to default values
- * clean interfaces with OPNFV Installers and Features
- * rewrite all vnfs to allow multiple tests in parallel
- * fully support non-default region names and Keystone v3 domains
- * refactor all tempest-based test cases (e.g. refstack_defcore)
- * remove obsolete OpenStack and Functest utils
- * verify all changes via doc8 and yamllint too
- * generate reports for all tempest-based test cases
-
-.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt
-.. _Xtesting: http://xtesting.readthedocs.io/en/latest/
+ * update test cases and containers to `OpenStack master`_ and to
+ `Kubernetes master`_
+ * define new scenarios to ease writing testcases vs OpenStack
+ * isolate all resources created in different tenants
+ * fully remove all OPNFV logics
+ * publish new Jenkins jobs
+ * support VIO (VMware Integrated OpenStack) and arm64 for Kubernetes
+ * reduce Functest Kubernetes image sizes
+ * add tempest_full and tempest_scenario in all daily jobs
+ * include benchmarking tools such as Vmtp ans Shaker
+ * increase functional scope by adding bgpvpn and sfc tempest plugins
+
+.. _`OpenStack master`: https://github.com/openstack/requirements/blob/master/upper-constraints.txt
+.. _`Kubernetes master`: https://github.com/kubernetes/kubernetes
Key benefits
------------
- * the enduser can run all tests by setting only one input (EXTERNAL_NETWORK)
- * the developer can only work on the test suites without diving into CI/CD
- integration
- * both OpenStack and Kubernetes deployments can be verified
- * Functest test cases are trustable as they meet the best coding rules (unit
- tests, coverage, linters, etc.)
- * Functest can be reused in other projects (e.g.
- `ONS: Re-using OPNFV framework tests for LFN projects`_)
-
-.. _`ONS: Re-using OPNFV framework tests for LFN projects`: https://wiki.lfnetworking.org/display/LN/LFN+Developer+Forum+Schedule?preview=/328197/328329/ONS-OPNFV%20framework%20tests%20for%20LFN%20projects.pdf
+ * the enduser can easily build its own toolchains by loading our Jenkins jobs
+ * all developpers can easily verify their changes before merge
+ * our testcases may be run vs VIM in production
+ * all testcases can run in parallel to decrease the overall duration
+ * Functest includes most of the OpenStack gate jobs
Code quality
------------
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index caa1ccb3b..d26a537a2 100644
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -211,35 +211,6 @@ Feature, unit and vnf_base inherit from testcase::
+--------------------+ +---------+ +------------------------+ +-----------------+
-Testcase
---------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
-
-Feature
--------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
-
-Unit
-----
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
-
-VNF
----
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
-
-Robotframework
---------------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html
-
-
-see `[5]`_ to get code samples
-
-
Functest util classes
=====================
@@ -266,7 +237,7 @@ instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils.
TestAPI
=======
Functest is using the Test collection framework and the TestAPI developed by
-the OPNFV community. See `[6]`_ for details.
+the OPNFV community. See `[5]`_ for details.
Reporting
@@ -278,7 +249,7 @@ jinja2 templates `[3]`_.
Dashboard
=========
-Additional dashboarding is managed at the testing group level, see `[7]`_ for
+Additional dashboarding is managed at the testing group level, see `[6]`_ for
details.
@@ -286,7 +257,7 @@ details.
How TOs
=======
-See How to section on Functest wiki `[8]`_
+See How to section on Functest wiki `[7]`_
==========
@@ -301,12 +272,10 @@ _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting
_`[4]`: https://git.opnfv.org/snaps/
-_`[5]` : http://testresults.opnfv.org/functest/framework/index.html
-
-_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
+_`[5]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
-_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
+_`[6]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
-_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
+_`[7]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
IRC support chan: #opnfv-functest
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
index c04689191..09fca19b5 100644
--- a/docs/testing/user/configguide/configguide.rst
+++ b/docs/testing/user/configguide/configguide.rst
@@ -16,19 +16,11 @@ Docker images are available on the dockerhub:
* opnfv/functest-core
* opnfv/functest-healthcheck
* opnfv/functest-smoke
+ * opnfv/functest-benchmarking
* opnfv/functest-features
* opnfv/functest-components
* opnfv/functest-vnf
-The tag "opnfv-6.0.0" is the official release image in Fraser, but you can
-also pull "fraser" tag as it is being maintained by Functest team and might
-include bugfixes.
-
-The Functest docker container environment can -in principle- be also
-used with non-OPNFV official installers (e.g. 'devstack'), with the
-**disclaimer** that support for such environments is outside of the
-scope and responsibility of the OPNFV project.
-
Preparing your environment
^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -62,10 +54,9 @@ Create a directory for the different images (attached as a Docker volume)::
images/CentOS-7-x86_64-GenericCloud.qcow2
images/cirros-0.4.0-x86_64-disk.img
images/cirros-0.4.0-x86_64-lxc.tar.gz
- images/cirros-d161201-aarch64-disk.img
- images/cirros-d161201-aarch64-initramfs
- images/cirros-d161201-aarch64-kernel
images/cloudify-manager-premium-4.0.1.qcow2
+ images/shaker-image-arm64.qcow2
+ images/shaker-image.qcow
images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
@@ -86,11 +77,22 @@ Results shall be displayed as follows::
+----------------------------+------------------+---------------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:07 | PASS |
- | api_check | functest | healthcheck | 07:46 | PASS |
- | snaps_health_check | functest | healthcheck | 00:36 | PASS |
+ | connection_check | functest | healthcheck | 00:09 | PASS |
+ | tenantnetwork1 | functest | healthcheck | 00:14 | PASS |
+ | tenantnetwork2 | functest | healthcheck | 00:11 | PASS |
+ | vmready1 | functest | healthcheck | 00:19 | PASS |
+ | vmready2 | functest | healthcheck | 00:16 | PASS |
+ | singlevm1 | functest | healthcheck | 00:41 | PASS |
+ | singlevm2 | functest | healthcheck | 00:36 | PASS |
+ | vping_ssh | functest | healthcheck | 00:46 | PASS |
+ | vping_userdata | functest | healthcheck | 00:41 | PASS |
+ | cinder_test | functest | healthcheck | 01:18 | PASS |
+ | api_check | functest | healthcheck | 10:33 | PASS |
+ | snaps_health_check | functest | healthcheck | 00:44 | PASS |
+ | odl | functest | healthcheck | 00:00 | SKIP |
+----------------------------+------------------+---------------------+------------------+----------------+
- NOTE: the duration is a reference and it might vary depending on your SUT.
+
+NOTE: the duration is a reference and it might vary depending on your SUT.
Testing smoke suite
^^^^^^^^^^^^^^^^^^^
@@ -104,21 +106,46 @@ Run smoke suite::
Results shall be displayed as follows::
- +------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------+------------------+----------------+
- | vping_ssh | functest | smoke | 00:57 | PASS |
- | vping_userdata | functest | smoke | 00:33 | PASS |
- | tempest_smoke_serial | functest | smoke | 13:22 | PASS |
- | rally_sanity | functest | smoke | 24:07 | PASS |
- | refstack_defcore | functest | smoke | 05:21 | PASS |
- | patrole | functest | smoke | 04:29 | PASS |
- | snaps_smoke | functest | smoke | 46:54 | PASS |
- | odl | functest | smoke | 00:00 | SKIP |
- | neutron_trunk | functest | smoke | 00:00 | SKIP |
- +------------------------------+------------------+---------------+------------------+----------------+
- Note: if the scenario does not support some tests, they are indicated as SKIP.
- See User guide for details.
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+ | tempest_smoke | functest | smoke | 06:13 | PASS |
+ | neutron-tempest-plugin-api | functest | smoke | 09:32 | PASS |
+ | rally_sanity | functest | smoke | 29:34 | PASS |
+ | rally_jobs | functest | smoke | 24:02 | PASS |
+ | refstack_defcore | functest | smoke | 13:07 | PASS |
+ | patrole | functest | smoke | 05:17 | PASS |
+ | snaps_smoke | functest | smoke | 90:13 | PASS |
+ | neutron_trunk | functest | smoke | 00:00 | SKIP |
+ | networking-bgpvpn | functest | smoke | 00:00 | SKIP |
+ | networking-sfc | functest | smoke | 00:00 | SKIP |
+ | barbican | functest | smoke | 05:01 | PASS |
+ +------------------------------------+------------------+---------------+------------------+----------------+
+
+Note: if the scenario does not support some tests, they are indicated as SKIP.
+See User guide for details.
+
+Testing benchmarking suite
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run benchmarking suite::
+
+ sudo docker run --env-file env \
+ -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
+ -v $(pwd)/images:/home/opnfv/functest/images \
+ opnfv/functest-benchmarking
+
+Results shall be displayed as follows::
+
+ +-------------------+------------------+----------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+----------------------+------------------+----------------+
+ | vmtp | functest | benchmarking | 18:43 | PASS |
+ | shaker | functest | benchmarking | 29:45 | PASS |
+ +-------------------+------------------+----------------------+------------------+----------------+
+
+Note: if the scenario does not support some tests, they are indicated as SKIP.
+See User guide for details.
Testing features suite
^^^^^^^^^^^^^^^^^^^^^^
@@ -140,9 +167,12 @@ Results shall be displayed as follows::
| functest-odl-sfc | sfc | features | 00:00 | SKIP |
| barometercollectd | barometer | features | 00:00 | SKIP |
| fds | fastdatastacks | features | 00:00 | SKIP |
+ | vgpu | functest | features | 00:00 | SKIP |
+ | stor4nfv_os | stor4nfv | features | 00:00 | SKIP |
+-----------------------------+------------------------+------------------+------------------+----------------+
- Note: if the scenario does not support some tests, they are indicated as SKIP.
- See User guide for details.
+
+Note: if the scenario does not support some tests, they are indicated as SKIP.
+See User guide for details.
Testing components suite
^^^^^^^^^^^^^^^^^^^^^^^^
@@ -156,12 +186,13 @@ Run components suite::
Results shall be displayed as follows::
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full_parallel | functest | components | 48:28 | PASS |
- | rally_full | functest | components | 126:02 | PASS |
- +-------------------------------+------------------+--------------------+------------------+----------------+
+ +--------------------------+------------------+--------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +--------------------------+------------------+--------------------+------------------+----------------+
+ | tempest_full | functest | components | 49:51 | PASS |
+ | tempest_scenario | functest | components | 18:50 | PASS |
+ | rally_full | functest | components | 167:13 | PASS |
+ +--------------------------+------------------+--------------------+------------------+----------------+
Testing vnf suite
^^^^^^^^^^^^^^^^^
@@ -178,12 +209,13 @@ Results shall be displayed as follows::
+----------------------+------------------+--------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+----------------------+------------------+--------------+------------------+----------------+
- | cloudify_ims | functest | vnf | 28:15 | PASS |
- | vyos_vrouter | functest | vnf | 17:59 | PASS |
- | juju_epc | functest | vnf | 46:44 | PASS |
+ | cloudify | functest | vnf | 04:05 | PASS |
+ | cloudify_ims | functest | vnf | 24:07 | PASS |
+ | heat_ims | functest | vnf | 18:15 | PASS |
+ | vyos_vrouter | functest | vnf | 15:48 | PASS |
+ | juju_epc | functest | vnf | 29:38 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
-
Functest Dockers for Kubernetes deployment
------------------------------------------
Docker images are available on the dockerhub:
@@ -198,14 +230,7 @@ Preparing your environment
cat env::
- INSTALLER_TYPE=XXX
- DEPLOY_SCENARIO=XXX
-
-cat k8s.creds::
-
- export KUBERNETES_PROVIDER=local
- export KUBE_MASTER_URL=XXX
- export KUBE_MASTER_IP=XXX
+ DEPLOY_SCENARIO=k8s-XXX
Testing healthcheck suite
^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -213,7 +238,6 @@ Testing healthcheck suite
Run healthcheck suite::
sudo docker run -it --env-file env \
- -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
-v $(pwd)/config:/root/.kube/config \
opnfv/functest-kubernetes-healthcheck
@@ -222,11 +246,11 @@ volume mapped to ~/.kube/config inside kubernetes container.
Results shall be displayed as follows::
- +-------------------------+------------------+-----------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------+------------------+-----------------+------------------+----------------+
- | k8s_smoke | functest | healthcheck | 01:54 | PASS |
- +-------------------------+------------------+-----------------+------------------+----------------+
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | TEST CASE | PROJECT | TIER | DURATION | RESULT |
+ +-------------------+------------------+---------------------+------------------+----------------+
+ | k8s_smoke | functest | healthcheck | 02:27 | PASS |
+ +-------------------+------------------+---------------------+------------------+----------------+
Testing smoke suite
^^^^^^^^^^^^^^^^^^^
@@ -234,7 +258,6 @@ Testing smoke suite
Run smoke suite::
sudo docker run -it --env-file env \
- -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
-v $(pwd)/config:/root/.kube/config \
opnfv/functest-kubernetes-smoke
@@ -243,7 +266,7 @@ Results shall be displayed as follows::
+-------------------------+------------------+---------------+------------------+----------------+
| TEST CASE | PROJECT | TIER | DURATION | RESULT |
+-------------------------+------------------+---------------+------------------+----------------+
- | k8s_conformance | functest | smoke | 57:47 | PASS |
+ | k8s_conformance | functest | smoke | 57:14 | PASS |
+-------------------------+------------------+---------------+------------------+----------------+
Testing features suite
@@ -252,7 +275,6 @@ Testing features suite
Run features suite::
sudo docker run -it --env-file env \
- -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \
-v $(pwd)/config:/root/.kube/config \
opnfv/functest-kubernetes-features
@@ -269,10 +291,13 @@ Environment variables
=====================
Several environement variables may be specified:
- * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid)
+
* INSTALLER_IP=<Specific IP Address>
* DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode>
-
+ * NAMESERVER=XXX # if not 8.8.8.8
+ * VOLUME_DEVICE_NAME=XXX # if not vdb
+ * EXTERNAL_NETWORK=XXX # if not first network with router:external=True
+ * NEW_USER_ROLE=XXX # if not member
INSTALLER_IP is required by Barometer in order to access the installer node and
the deployment.
@@ -305,8 +330,9 @@ to the Functest Docker Container, using the -e
only relevant to Jenkins CI invoked testing and **should not be used**
when performing manual test scenarios:
- * NODE_NAME = <Test POD Name>
- * BUILD_TAG = <Jenkins Build Tag>
+ * INSTALLER_TYPE=(apex|compass|daisy|fuel)
+ * NODE_NAME=<Test POD Name>
+ * BUILD_TAG=<Jenkins Build Tag>
where: