diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/release/release-notes/functest-release.rst | 98 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 43 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 154 |
3 files changed, 157 insertions, 138 deletions
diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index bb7600d35..67fa3dcbb 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -23,22 +23,37 @@ OpenStack The internal test cases are: * connection_check - * api_check - * snaps_health_check + * tenantnetwork1 + * tenantnetwork2 + * vmready1 + * vmready2 + * singlevm1 + * singlevm2 * vping_ssh * vping_userdata - * tempest_smoke_serial + * cinder_test + * api_check + * snaps_health_check + * odl + * tempest_smoke + * neutron-tempest-plugin-api * rally_sanity * refstack_defcore * patrole - * odl * snaps_smoke * neutron_trunk + * networking-bgpvpn + * networking-sfc + * barbican * tempest_full + * tempest_scenario * rally_full + * cloudify * cloudify_ims + * heat_ims * vyos_vrouter * juju_epc + * vgpu The OPNFV projects integrated into Functest framework for automation are: @@ -47,6 +62,7 @@ The OPNFV projects integrated into Functest framework for automation are: * odl-sfc * barometer * fds + * stor4nfv_os Kubernetes ---------- @@ -80,6 +96,7 @@ Software * https://hub.docker.com/r/opnfv/functest-healthcheck * https://hub.docker.com/r/opnfv/functest-smoke + * https://hub.docker.com/r/opnfv/functest-benchmarking * https://hub.docker.com/r/opnfv/functest-features * https://hub.docker.com/r/opnfv/functest-components * https://hub.docker.com/r/opnfv/functest-vnf @@ -95,9 +112,9 @@ Docker tag for master: latest Documents --------- - * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html - * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html - * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html + * Config Guide: https://functest.readthedocs.io/projects/configguide/en/latest/ + * User Guide: https://functest.readthedocs.io/projects/userguide/en/latest/ + * Developer Guide: https://functest.readthedocs.io/projects/devguide/en/latest/ * API Docs: http://functest.readthedocs.io/en/latest/ Version change @@ -106,44 +123,51 @@ Version change New test cases -------------- - * neutron_trunk - * patrole - * juju_epc - * k8s_smoke - * k8s_conformance - * stor4nfv - * clover + * tenantnetwork1 + * tenantnetwork2 + * vmready1 + * vmready2 + * singlevm1 + * singlevm2 + * cinder_test + * neutron-tempest-plugin-api + * rally_jobs + * networking-bgpvpn + * networking-sfc + * barbican + * vmtp + * shaker + * tempest_scenario + * cloudify + * heat_ims + * vgpu Key changes ----------- - * update test cases and containers to `OpenStack Pike`_ - * move the framework into a separate project: Xtesting_ - * ease running all containers thanks to default values - * clean interfaces with OPNFV Installers and Features - * rewrite all vnfs to allow multiple tests in parallel - * fully support non-default region names and Keystone v3 domains - * refactor all tempest-based test cases (e.g. refstack_defcore) - * remove obsolete OpenStack and Functest utils - * verify all changes via doc8 and yamllint too - * generate reports for all tempest-based test cases - -.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt -.. _Xtesting: http://xtesting.readthedocs.io/en/latest/ + * update test cases and containers to `OpenStack master`_ and to + `Kubernetes master`_ + * define new scenarios to ease writing testcases vs OpenStack + * isolate all resources created in different tenants + * fully remove all OPNFV logics + * publish new Jenkins jobs + * support VIO (VMware Integrated OpenStack) and arm64 for Kubernetes + * reduce Functest Kubernetes image sizes + * add tempest_full and tempest_scenario in all daily jobs + * include benchmarking tools such as Vmtp ans Shaker + * increase functional scope by adding bgpvpn and sfc tempest plugins + +.. _`OpenStack master`: https://github.com/openstack/requirements/blob/master/upper-constraints.txt +.. _`Kubernetes master`: https://github.com/kubernetes/kubernetes Key benefits ------------ - * the enduser can run all tests by setting only one input (EXTERNAL_NETWORK) - * the developer can only work on the test suites without diving into CI/CD - integration - * both OpenStack and Kubernetes deployments can be verified - * Functest test cases are trustable as they meet the best coding rules (unit - tests, coverage, linters, etc.) - * Functest can be reused in other projects (e.g. - `ONS: Re-using OPNFV framework tests for LFN projects`_) - -.. _`ONS: Re-using OPNFV framework tests for LFN projects`: https://wiki.lfnetworking.org/display/LN/LFN+Developer+Forum+Schedule?preview=/328197/328329/ONS-OPNFV%20framework%20tests%20for%20LFN%20projects.pdf + * the enduser can easily build its own toolchains by loading our Jenkins jobs + * all developpers can easily verify their changes before merge + * our testcases may be run vs VIM in production + * all testcases can run in parallel to decrease the overall duration + * Functest includes most of the OpenStack gate jobs Code quality ------------ diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index caa1ccb3b..d26a537a2 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -211,35 +211,6 @@ Feature, unit and vnf_base inherit from testcase:: +--------------------+ +---------+ +------------------------+ +-----------------+ -Testcase --------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html - -Feature -------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html - -Unit ----- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html - -VNF ---- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html - -Robotframework --------------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html - - -see `[5]`_ to get code samples - - Functest util classes ===================== @@ -266,7 +237,7 @@ instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. TestAPI ======= Functest is using the Test collection framework and the TestAPI developed by -the OPNFV community. See `[6]`_ for details. +the OPNFV community. See `[5]`_ for details. Reporting @@ -278,7 +249,7 @@ jinja2 templates `[3]`_. Dashboard ========= -Additional dashboarding is managed at the testing group level, see `[7]`_ for +Additional dashboarding is managed at the testing group level, see `[6]`_ for details. @@ -286,7 +257,7 @@ details. How TOs ======= -See How to section on Functest wiki `[8]`_ +See How to section on Functest wiki `[7]`_ ========== @@ -301,12 +272,10 @@ _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting _`[4]`: https://git.opnfv.org/snaps/ -_`[5]` : http://testresults.opnfv.org/functest/framework/index.html - -_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html +_`[5]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba +_`[6]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba -_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 +_`[7]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 IRC support chan: #opnfv-functest diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index c04689191..09fca19b5 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -16,19 +16,11 @@ Docker images are available on the dockerhub: * opnfv/functest-core * opnfv/functest-healthcheck * opnfv/functest-smoke + * opnfv/functest-benchmarking * opnfv/functest-features * opnfv/functest-components * opnfv/functest-vnf -The tag "opnfv-6.0.0" is the official release image in Fraser, but you can -also pull "fraser" tag as it is being maintained by Functest team and might -include bugfixes. - -The Functest docker container environment can -in principle- be also -used with non-OPNFV official installers (e.g. 'devstack'), with the -**disclaimer** that support for such environments is outside of the -scope and responsibility of the OPNFV project. - Preparing your environment ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -62,10 +54,9 @@ Create a directory for the different images (attached as a Docker volume):: images/CentOS-7-x86_64-GenericCloud.qcow2 images/cirros-0.4.0-x86_64-disk.img images/cirros-0.4.0-x86_64-lxc.tar.gz - images/cirros-d161201-aarch64-disk.img - images/cirros-d161201-aarch64-initramfs - images/cirros-d161201-aarch64-kernel images/cloudify-manager-premium-4.0.1.qcow2 + images/shaker-image-arm64.qcow2 + images/shaker-image.qcow images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img @@ -86,11 +77,22 @@ Results shall be displayed as follows:: +----------------------------+------------------+---------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:07 | PASS | - | api_check | functest | healthcheck | 07:46 | PASS | - | snaps_health_check | functest | healthcheck | 00:36 | PASS | + | connection_check | functest | healthcheck | 00:09 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:14 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:11 | PASS | + | vmready1 | functest | healthcheck | 00:19 | PASS | + | vmready2 | functest | healthcheck | 00:16 | PASS | + | singlevm1 | functest | healthcheck | 00:41 | PASS | + | singlevm2 | functest | healthcheck | 00:36 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:41 | PASS | + | cinder_test | functest | healthcheck | 01:18 | PASS | + | api_check | functest | healthcheck | 10:33 | PASS | + | snaps_health_check | functest | healthcheck | 00:44 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | +----------------------------+------------------+---------------------+------------------+----------------+ - NOTE: the duration is a reference and it might vary depending on your SUT. + +NOTE: the duration is a reference and it might vary depending on your SUT. Testing smoke suite ^^^^^^^^^^^^^^^^^^^ @@ -104,21 +106,46 @@ Run smoke suite:: Results shall be displayed as follows:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:57 | PASS | - | vping_userdata | functest | smoke | 00:33 | PASS | - | tempest_smoke_serial | functest | smoke | 13:22 | PASS | - | rally_sanity | functest | smoke | 24:07 | PASS | - | refstack_defcore | functest | smoke | 05:21 | PASS | - | patrole | functest | smoke | 04:29 | PASS | - | snaps_smoke | functest | smoke | 46:54 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | neutron_trunk | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. + +------------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------------+------------------+---------------+------------------+----------------+ + | tempest_smoke | functest | smoke | 06:13 | PASS | + | neutron-tempest-plugin-api | functest | smoke | 09:32 | PASS | + | rally_sanity | functest | smoke | 29:34 | PASS | + | rally_jobs | functest | smoke | 24:02 | PASS | + | refstack_defcore | functest | smoke | 13:07 | PASS | + | patrole | functest | smoke | 05:17 | PASS | + | snaps_smoke | functest | smoke | 90:13 | PASS | + | neutron_trunk | functest | smoke | 00:00 | SKIP | + | networking-bgpvpn | functest | smoke | 00:00 | SKIP | + | networking-sfc | functest | smoke | 00:00 | SKIP | + | barbican | functest | smoke | 05:01 | PASS | + +------------------------------------+------------------+---------------+------------------+----------------+ + +Note: if the scenario does not support some tests, they are indicated as SKIP. +See User guide for details. + +Testing benchmarking suite +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run benchmarking suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-benchmarking + +Results shall be displayed as follows:: + + +-------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+----------------------+------------------+----------------+ + | vmtp | functest | benchmarking | 18:43 | PASS | + | shaker | functest | benchmarking | 29:45 | PASS | + +-------------------+------------------+----------------------+------------------+----------------+ + +Note: if the scenario does not support some tests, they are indicated as SKIP. +See User guide for details. Testing features suite ^^^^^^^^^^^^^^^^^^^^^^ @@ -140,9 +167,12 @@ Results shall be displayed as follows:: | functest-odl-sfc | sfc | features | 00:00 | SKIP | | barometercollectd | barometer | features | 00:00 | SKIP | | fds | fastdatastacks | features | 00:00 | SKIP | + | vgpu | functest | features | 00:00 | SKIP | + | stor4nfv_os | stor4nfv | features | 00:00 | SKIP | +-----------------------------+------------------------+------------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. + +Note: if the scenario does not support some tests, they are indicated as SKIP. +See User guide for details. Testing components suite ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -156,12 +186,13 @@ Run components suite:: Results shall be displayed as follows:: - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 48:28 | PASS | - | rally_full | functest | components | 126:02 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +--------------------------+------------------+--------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+--------------------+------------------+----------------+ + | tempest_full | functest | components | 49:51 | PASS | + | tempest_scenario | functest | components | 18:50 | PASS | + | rally_full | functest | components | 167:13 | PASS | + +--------------------------+------------------+--------------------+------------------+----------------+ Testing vnf suite ^^^^^^^^^^^^^^^^^ @@ -178,12 +209,13 @@ Results shall be displayed as follows:: +----------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +----------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:15 | PASS | - | vyos_vrouter | functest | vnf | 17:59 | PASS | - | juju_epc | functest | vnf | 46:44 | PASS | + | cloudify | functest | vnf | 04:05 | PASS | + | cloudify_ims | functest | vnf | 24:07 | PASS | + | heat_ims | functest | vnf | 18:15 | PASS | + | vyos_vrouter | functest | vnf | 15:48 | PASS | + | juju_epc | functest | vnf | 29:38 | PASS | +----------------------+------------------+--------------+------------------+----------------+ - Functest Dockers for Kubernetes deployment ------------------------------------------ Docker images are available on the dockerhub: @@ -198,14 +230,7 @@ Preparing your environment cat env:: - INSTALLER_TYPE=XXX - DEPLOY_SCENARIO=XXX - -cat k8s.creds:: - - export KUBERNETES_PROVIDER=local - export KUBE_MASTER_URL=XXX - export KUBE_MASTER_IP=XXX + DEPLOY_SCENARIO=k8s-XXX Testing healthcheck suite ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -213,7 +238,6 @@ Testing healthcheck suite Run healthcheck suite:: sudo docker run -it --env-file env \ - -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/config:/root/.kube/config \ opnfv/functest-kubernetes-healthcheck @@ -222,11 +246,11 @@ volume mapped to ~/.kube/config inside kubernetes container. Results shall be displayed as follows:: - +-------------------------+------------------+-----------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------+------------------+-----------------+------------------+----------------+ - | k8s_smoke | functest | healthcheck | 01:54 | PASS | - +-------------------------+------------------+-----------------+------------------+----------------+ + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_smoke | functest | healthcheck | 02:27 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ Testing smoke suite ^^^^^^^^^^^^^^^^^^^ @@ -234,7 +258,6 @@ Testing smoke suite Run smoke suite:: sudo docker run -it --env-file env \ - -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/config:/root/.kube/config \ opnfv/functest-kubernetes-smoke @@ -243,7 +266,7 @@ Results shall be displayed as follows:: +-------------------------+------------------+---------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------------+------------------+---------------+------------------+----------------+ - | k8s_conformance | functest | smoke | 57:47 | PASS | + | k8s_conformance | functest | smoke | 57:14 | PASS | +-------------------------+------------------+---------------+------------------+----------------+ Testing features suite @@ -252,7 +275,6 @@ Testing features suite Run features suite:: sudo docker run -it --env-file env \ - -v $(pwd)/k8s.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/config:/root/.kube/config \ opnfv/functest-kubernetes-features @@ -269,10 +291,13 @@ Environment variables ===================== Several environement variables may be specified: - * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid) + * INSTALLER_IP=<Specific IP Address> * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode> - + * NAMESERVER=XXX # if not 8.8.8.8 + * VOLUME_DEVICE_NAME=XXX # if not vdb + * EXTERNAL_NETWORK=XXX # if not first network with router:external=True + * NEW_USER_ROLE=XXX # if not member INSTALLER_IP is required by Barometer in order to access the installer node and the deployment. @@ -305,8 +330,9 @@ to the Functest Docker Container, using the -e only relevant to Jenkins CI invoked testing and **should not be used** when performing manual test scenarios: - * NODE_NAME = <Test POD Name> - * BUILD_TAG = <Jenkins Build Tag> + * INSTALLER_TYPE=(apex|compass|daisy|fuel) + * NODE_NAME=<Test POD Name> + * BUILD_TAG=<Jenkins Build Tag> where: |