From 5504724dc69096b36948de9bd07b82f5058242f0 Mon Sep 17 00:00:00 2001 From: Morgan Richomme Date: Tue, 12 Sep 2017 11:39:11 +0200 Subject: Update Functest documentation for Euphrates - Better description of Alpine docker (default artifact) - Reorganization of config and user guide - Upgrade of the developer guide Change-Id: Ie8e10f027bfcdb01c992cd161a1af2d6d6e29536 Signed-off-by: Morgan Richomme --- docs/testing/developer/devguide/index.rst | 86 +-- docs/testing/user/configguide/ci.rst | 50 +- docs/testing/user/configguide/configguide.rst | 844 ++++++++++-------------- docs/testing/user/configguide/index.rst | 36 +- docs/testing/user/configguide/prerequisites.rst | 28 +- docs/testing/user/userguide/index.rst | 26 +- docs/testing/user/userguide/reporting.rst | 2 +- docs/testing/user/userguide/runfunctest.rst | 466 ++++--------- docs/testing/user/userguide/test_details.rst | 71 +- docs/testing/user/userguide/test_overview.rst | 49 +- docs/testing/user/userguide/test_results.rst | 5 +- docs/testing/user/userguide/troubleshooting.rst | 30 +- 12 files changed, 657 insertions(+), 1036 deletions(-) (limited to 'docs/testing') diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 8a8b09cb..1eae33f6 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -1,9 +1,9 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -****************************** -OPNFV FUNCTEST developer guide -****************************** +************************ +Functest Developer Guide +************************ .. toctree:: :numbered: @@ -32,7 +32,7 @@ Introduction ============ Functest is a project dealing with functional testing. -Functest produces its own internal test cases but can also be considered +The project produces its own internal test cases but can also be considered as a framework to support feature and VNF onboarding project testing. Therefore there are many ways to contribute to Functest. You can: @@ -44,8 +44,10 @@ Therefore there are many ways to contribute to Functest. You can: Additional tasks involving Functest but addressing all the test projects may also be mentioned: - * Develop the API / Test collection framework - * Develop dashboards or automatic reporting portals + * The API / Test collection framework + * The dashboards + * The automatic reporting portals + * The testcase catalog This document describes how, as a developer, you may interact with the Functest project. The first section details the main working areas of @@ -69,17 +71,22 @@ Until Danube, Functest produced 2 docker files based on Ubuntu 14.04: * aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/ In Euphrates Alpine containers have been introduced in order to lighten the -container and manage testing slicing, the new containers are created according +container and manage testing slicing. The new containers are created according to the different tiers: * functest-core: https://hub.docker.com/r/opnfv/functest-core/ * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ - * functest-features: TODO - * functest-components: TODO - * functest-vnf: TODO + * functest-features: https://hub.docker.com/r/opnfv/functest-features/ + * functest-components: https://hub.docker.com/r/opnfv/functest-components/ + * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ + * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/ + * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ -Functest can be described as follow:: +Standalone functest dockers are maintained for Euphrates but Alpine containers +are recommended. + +Functest can be described as follow: +----------------------+ | | @@ -87,8 +94,8 @@ Functest can be described as follow:: | | | | Public | | | | Tools | +------------------+ OPNFV | | | Scripts | | | System Under Test | - | | Scenarios | +------------------+ | - | | | | Management | | + | | Scenarios | | | | + | | | | | | | +--------------+ | +-------------------+ | | | Functest Docker | @@ -101,8 +108,8 @@ The internal test cases in Euphrates are: * api_check - * cloudify_ims * connection_check + * snaps_health_check * vping_ssh * vping_userdata * odl @@ -110,9 +117,8 @@ The internal test cases in Euphrates are: * odl-fds * rally_full * rally_sanity - * snaps_health_check - * tempest_full_parallel * tempest_smoke_serial + * tempest_full_parallel * cloudify_ims By internal, we mean that this particular test cases have been developed and/or @@ -124,7 +130,7 @@ just integrated in Functest). The structure of this repository is detailed in `[1]`_. The main internal test cases are in the opnfv_tests subfolder of the -repository, the internal test cases are: +repository, the internal test cases can be grouped by domain: * sdn: odl, odl_netvirt, odl_fds * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_* @@ -145,23 +151,29 @@ The external test cases are: * doctor * domino * fds - * orchestra_ims * parser * promise * refstack_defcore * snaps_smoke * functest-odl-sfc - * vyos_vrouter + * orchestra_clearwaterims + * orchestra_openims + * cloudify_vrouter + * juju_vepc External test cases integrated in previous versions but not released in Euphrates: * copper + * moon * netready * security_scan The code to run these test cases is hosted in the repository of the project. +Please note that orchestra test cases are hosted in Functest repository and not +in orchestra repository. Cloudify_vrouter and juju_vepc code is also hosted in +functest as there are no dedicated projects. Functest framework @@ -174,7 +186,7 @@ a CLI to prepare the environment and run tests. It simplifies the integration of external test suites in CI pipeline and provide commodity tools to collect and display results. -Since Colorado, test categories also known as tiers have been created to +Since Colorado, test categories also known as **tiers** have been created to group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section). @@ -205,7 +217,7 @@ introduced: The goal is to unify the way to run tests in Functest. -Feature, unit and vnf_base inherit from testcase:: +Feature, unit and vnf_base inherit from testcase: +-----------------------------------------+ | | @@ -232,34 +244,34 @@ Feature, unit and vnf_base inherit from testcase:: Testcase -======== +-------- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html Feature -======= +------- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html Unit -==== +---- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html VNF -=== +--- .. raw:: html :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html -see `Functest framework overview`_ to get code samples +see `[5]`_ to get code samples Functest util classes ===================== In order to simplify the creation of test cases, Functest develops also some -functions that can be used by any feature or internal test cases. +functions that are used by internal test cases. Several features are supported such as logger, configuration management and Openstack capabilities (snapshot, clean, tacker,..). These functions can be found under /functest/utils and can be described as @@ -276,14 +288,14 @@ follows:: |-- openstack_tacker.py `-- openstack_utils.py -Please note that it is possible to use snaps utils. SNAPS `[4]`_ is an OPNFV -project providing OpenStack utils. +It is recommended to use the SNAPS-OO library for deploying OpenStack instances. +SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. TestAPI ======= Functest is using the Test collection framework and the TestAPI developed by -the OPNFV community. See `OPNFV Test collection framework`_ for details. +the OPNFV community. See `[6]`_ for details. Reporting @@ -295,15 +307,15 @@ jinja2 templates `[3]`_. Dashboard ========= -Additional dashboarding is managed at the testing group level, see -`OPNFV Testing dashboard`_ +Additional dashboarding is managed at the testing group level, see `[7]`_ for +details. ======= How TOs ======= -See `How to section`_ on Functest wiki +See How to section on Functest wiki `[8]`_ ========== @@ -318,12 +330,12 @@ _`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting _`[4]`: https://git.opnfv.org/snaps/ -_`Functest framework overview` : http://testresults.opnfv.org/functest/framework/index.html +_`[5]` : http://testresults.opnfv.org/functest/framework/index.html -_`OPNFV Test collection framework`: TODO +_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -_`OPNFV Testing dashboard`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba +_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba -_`How to section`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 +_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 IRC support chan: #opnfv-functest diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst index 384bc34e..ada803c5 100644 --- a/docs/testing/user/configguide/ci.rst +++ b/docs/testing/user/configguide/ci.rst @@ -1,50 +1,14 @@ Integration in CI ================= -In CI we use the Docker image and execute the appropriate commands within the +In CI we use the Docker images and execute the appropriate commands within the container from Jenkins. -Docker creation in set-functest-env builder `[3]`_:: - - envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}" - [...] - docker pull opnfv/functest:$DOCKER_TAG >/dev/null - cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash" - echo "Functest: Running docker run command: ${cmd}" - ${cmd} >${redirect} - sleep 5 - container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1) - echo "Container ID=${container_id}" - if [ -z ${container_id} ]; then - echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing." - docker ps -a - exit 1 - fi - echo "Starting the container: docker start ${container_id}" - docker start ${container_id} - sleep 5 - docker ps >${redirect} - if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then - echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..." - exit 1 - fi - - cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start" - echo "Executing command inside the docker: ${cmd}" - docker exec ${container_id} ${cmd} - - -Test execution in functest-all builder `[3]`_:: - - branch=${GIT_BRANCH##*/} - echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}" - cmd="functest testcase run $FUNCTEST_SUITE_NAME" - fi - container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1) - docker exec $container_id $cmd - ret_value=$? - exit $ret_value - -Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi +4 steps have been defined:: + * functest-cleanup: clean existing functest dockers on the jumphost + * set-functest-env-alpine: prepare en environement files + * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf) + * functest-store-results: push logs to artifacts +See `[3]`_ for details. .. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index d19939af..0a260ad0 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -1,346 +1,366 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -Installation and configuration (Ubuntu) -======================================= +Installation and configuration +============================== -The historical docker file is based on Ubuntu. It has been maintained for -Euphrates. +Alpine containers have been introduced in Euphrates. +Alpine allows Functest testing in several very light containers and thanks to +the refactoring on dependency management should allow the creation of light and +fully customized docker files. -Pulling the Docker image ------------------------- -Pull the Functest Docker image ('opnfv/functest') from the public -dockerhub registry under the OPNFV account: [̀`dockerhub`_], with the -following docker command:: - - docker pull opnfv/functest: - -where identifies a release of the Functest docker -container image in the public Dockerhub registry. There are many tags -created automatically by the CI mechanisms, and you must ensure you -pull an image with the **correct tag** to match the OPNFV software -release installed in your environment. All available tagged images can -be seen from location [FunctestDockerTags_]. For example, when running -on the first official release of the OPNFV Danube system platform, -tag "danube.1.0" is needed. For the second and third releases, the tag -"danube.2.0" and "danube.3.0" can be used respectively. -Pulling other tags might cause some problems while running the tests. -Docker images pulled without a tag specifier bear the implicitly -assigned label "latest". If you need to specifically pull the latest -Functest docker image, then omit the tag argument:: - - docker pull opnfv/functest - -After pulling the Docker image, check that it is available with the -following docker command:: - - [functester@jumphost ~]$ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB - opnfv/functest danube.2.0 d2c174a91911 7 minutes ago 1.471 GB - opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB +It is still possible to use the monolithic functest opnfv/functest especially +for tests on Aarch64 architecture. + +Functest Dockers +---------------- +Docker files are available on the dockerhub: + + * opnfv/functest-core + * opnfv/functest-healthcheck + * opnfv/functest-smoke + * opnfv/functest-features + * opnfv/functest-components + * opnfv/functest-vnf + * opnfv/functest-parser + * opnfv/functest-restapi + +By default, we use the docker tag latest, but you may pull a tagged docker +image. The Functest docker container environment can -in principle- be also used with non-OPNFV official installers (e.g. 'devstack'), with the **disclaimer** that support for such environments is outside of the scope and responsibility of the OPNFV project. -Please note that alpine dockers have been introduced in Euphrates. See alpine -section for details. -Accessing the Openstack credentials ------------------------------------ -OpenStack credentials are mandatory and must be provided to Functest. -When running the command "functest env prepare", the framework will -automatically look for the Openstack credentials file -"/home/opnfv/functest/conf/openstack.creds" and will exit with -error if it is not present or it is empty. +Preparing your environment +-------------------------- -There are 2 ways to provide that file: +cat env:: - * by using a Docker volume with -v option when creating the Docker container. - This is referred to in docker documentation as "Bind Mounting". - See the usage of this parameter in the following chapter. - * or creating manually the file '/home/opnfv/functest/conf/openstack.creds' - inside the running container and pasting the credentials in it. Consult - your installer guide for further details. This is however not - instructed in this document. + INSTALLER_TYPE=XXX + INSTALLER_IP=XXX + EXTERNAL_NETWORK=XXX + DEPLOY_SCENARIO=XXX -There is a default environment variable in the Functest container **$creds** -that points to the credentials absolute path to help the user with this task. +See section on environment variables for details. -In proxified environment you may need to change the credentials file. -There are some tips in chapter: `Proxy support`_ +cat openstack.creds:: + + export OS_AUTH_URL=XXX + export OS_USER_DOMAIN_NAME=XXX + export OS_PROJECT_DOMAIN_NAME=XXX + export OS_USERNAME=XXX + export OS_TENANT_NAME=XXX + export OS_PROJECT_NAME=XXX + export OS_PASSWORD=XXX + export OS_VOLUME_API_VERSION=XXX + export OS_IDENTITY_API_VERSION=XXX + export OS_IMAGE_API_VERSION=XXX + +See section on OpenStack credentials for details. -Functest Docker parameters +Create a directory for the different images (included as volume):: + + mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/* + + images/CentOS-7-aarch64-GenericCloud.qcow2 + images/CentOS-7-aarch64-GenericCloud.qcow2.xz + images/CentOS-7-x86_64-GenericCloud.qcow2 + images/cirros-0.3.5-x86_64-disk.img + images/cirros-0.3.5-x86_64-lxc.tar.gz + images/cirros-d161201-aarch64-disk.img + images/cirros-d161201-aarch64-initramfs + images/cirros-d161201-aarch64-kernel + images/cloudify-manager-premium-4.0.1.qcow2 + images/img + images/trusty-server-cloudimg-amd64-disk1.img + images/ubuntu-14.04-server-cloudimg-amd64-disk1.img + images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img + images/ubuntu-16.04-server-cloudimg-amd64-disk1.img + images/vyos-1.1.7.img + + +Testing healthcheck suite -------------------------- -This chapter explains how to run a container for executing functest -test suites. Numbered list below explains some details of the -recommended parameters for invoking docker container - #. It is a good practice to assign a precise container name through - the **--name** option. +Run healthcheck suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-healthcheck + +Results shall be displayed as follows:: + + +----------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:02 | PASS | + | api_check | functest | healthcheck | 04:57 | PASS | + | snaps_health_check | functest | healthcheck | 00:51 | PASS | + +----------------------------+------------------+---------------------+------------------+----------------+ + +Testing smoke suite +------------------- + +Run smoke suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-smoke + +Results shall be displayed as follows:: + + +------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +------------------------------+------------------+---------------+------------------+----------------+ + | vping_ssh | functest | smoke | 01:19 | PASS | + | vping_userdata | functest | smoke | 01:56 | PASS | + | tempest_smoke_serial | functest | smoke | 26:30 | PASS | + | rally_sanity | functest | smoke | 19:42 | PASS | + | refstack_defcore | functest | smoke | 22:00 | PASS | + | snaps_smoke | functest | smoke | 41:14 | PASS | + | odl | functest | smoke | 00:16 | PASS | + | odl_netvirt | functest | smoke | 00:00 | SKIP | + | fds | functest | smoke | 00:00 | SKIP | + +------------------------------+------------------+---------------+------------------+----------------+ + Note: if the scenario does not support some tests, they are indicated as SKIP. + See User guide for details. + +Testing features suite +---------------------- + +Run features suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-features + +Results shall be displayed as follows:: + + +---------------------------+--------------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+--------------------------+------------------+------------------+----------------+ + | promise | promise | features | 00:00 | SKIP | + | bgpvpn | sdnvpn | features | 00:00 | SKIP | + | security_scan | securityscanning | features | 00:00 | SKIP | + | functest-odl-sfc | sfc | features | 00:00 | SKIP | + | domino-multinode | domino | features | 00:00 | SKIP | + | barometercollectd | barometer | features | 00:00 | SKIP | + +---------------------------+--------------------------+------------------+------------------+----------------+ + Note: if the scenario does not support some tests, they are indicated as SKIP. + See User guide for details. + +Testing components suite +------------------------ - #. Assign parameter for installer type:: +Run components suite:: - -e "INSTALLER_TYPE=" - # Use one of following apex, compass, fuel or joid + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-components - #. Functest needs to know the IP of some installers:: +Results shall be displayed as follows:: - -e "INSTALLER_IP=" + +-------------------------------+------------------+--------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+--------------------+------------------+----------------+ + | tempest_full_parallel | functest | components | 102:48 | PASS | + | rally_full | functest | components | 160:58 | PASS | + | tempest_custom | functest | components | 00:00 | SKIP | + +-------------------------------+------------------+--------------------+------------------+----------------+ - These two env variables are useful extract some information - from the deployment. However, for some test cases like - SFC or Barometer they are mandatory since the tests - need to access the installer node and the deployment. +Testing vnf suite +----------------- - #. Credentials for accessing the Openstack. - Most convenient way of passing them to container is by having a - local copy of the credentials file in Jumphost and then using the - **-v** option. In the example we have local file by the name of - "overcloudrc" and we are using that as an argument:: +Run vnf suite:: - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds +sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-vnf - The credentials file needs to exist in the Docker container - under the path: '/home/opnfv/functest/conf/openstack.creds'. +Results shall be displayed as follows:: - **WARNING:** If you are using the Joid installer, you must pass the - credentials using the **-v** option: - -v /var/lib/jenkins/admin-openrc:/home/opnfv/functest/conf/openstack.creds. - See the section `Accessing the Openstack credentials`_ above. + +---------------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------------+------------------+--------------+------------------+----------------+ + | cloudify_ims | functest | vnf | 21:25 | PASS | + | orchestra_openims | functest | vnf | 11:02 | FAIL | + | orchestra_clearwaterims | functest | vnf | 09:13 | FAIL | + | vyos_vrouter | functest | vnf | 00:00 | SKIP | + +---------------------------------+------------------+--------------+------------------+----------------+ - #. Passing deployment scenario - When running Functest against any of the supported OPNFV scenarios, - it is recommended to include also the environment variable - **DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable - is passed with the format:: - -e "DEPLOY_SCENARIO=os---" - where: - os = OpenStack (No other VIM choices currently available) - controller is one of ( nosdn | odl_l2 | odl_l3 ) - nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) - If several features are pertinent then use the underscore - character '_' to separate each feature (e.g. ovs_kvm) - 'nofeature' indicates no NFV feature is deployed - ha_mode (high availability) is one of ( ha | noha ) +Environment variables +===================== - **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are - supported. The name passed in to the Functest Docker container - must match the scenario used when the actual OPNFV platform was - deployed. See release note to see the list of supported scenarios. +Several environement variables may be specified: + * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid|osa) + * INSTALLER_IP= + * DEPLOY_SCENARIO=--- - **NOTE:** The scenario name is mainly used to automatically detect - if a test suite is runnable or not (e.g. it will prevent ONOS test suite - to be run on ODL scenarios). If not set, Functest will try to run the - default test cases that might not include SDN controller or a specific - feature - **NOTE:** A HA scenario means that 3 OpenStack controller nodes are - deployed. It does not necessarily mean that the whole system is HA. See - installer release notes for details. +INSTALLER IP may be required by some test cases like SFC or Barometer in order +to access the installer node and the deployment. +The format for the DEPLOY_SCENARIO env variable can be described as follows: + * vim: (os|k8s) = OpenStack or Kubernetes + * controller is one of ( nosdn | odl ) + * nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) + * ha_mode (high availability) is one of ( ha | noha ) -Putting all above together, when using installer 'fuel' and an invented -INSTALLER_IP of '10.20.0.2', the recommended command to create the -Functest Docker container is as follows:: +If several features are pertinent then use the underscore character '_' to +separate each feature (e.g. ovs_kvm) 'nofeature' indicates no OPNFV feature is +deployed - docker run --name "FunctestContainer" -it \ - -e "INSTALLER_IP=10.20.0.2" \ - -e "INSTALLER_TYPE=fuel" \ - -e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \ - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest /bin/bash +The list of supported scenarios per release/installer is indicated in the +release note. -After the *run* command, a new prompt appears which means that we are inside -the container and ready to move to the next step. +**NOTE:** The scenario name is mainly used to automatically detect +if a test suite is runnable or not (e.g. it will prevent ONOS test suite to be +run on ODL scenarios). If not set, Functest will try to run the default test +cases that might not include SDN controller or a specific +feature -For tips on how to set up container with installer Apex, see chapter -`Apex Installer Tips`_. +**NOTE:** A HA scenario means that 3 OpenStack controller nodes are +deployed. It does not necessarily mean that the whole system is HA. See +installer release notes for details. Finally, three additional environment variables can also be passed in to the Functest Docker Container, using the -e -"=" mechanism. The first two of these are +"=" mechanism. The first two parameters are only relevant to Jenkins CI invoked testing and **should not be used** -when performing manual test scenarios:: - - -e "NODE_NAME=" \ - -e "BUILD_TAG=" \ - -e "CI_DEBUG=" - where: - = Symbolic name of the POD where the tests are run. - Visible in test results files, which are stored - to the database. This option is only used when - tests are activated under Jenkins CI control. - It indicates the POD/hardware where the test has - been run. If not specified, then the POD name is - defined as "Unknown" by default. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - = Symbolic name of the Jenkins Build Job. - Visible in test results files, which are stored - to the database. This option is only set when - tests are activated under Jenkins CI control. - It enables the correlation of test results, - which - are independently pushed to the results database - from different Jenkins jobs. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - = "true" or "false" - Default = "false", if not specified - If "true" is specified, then additional debug trace - text can be sent to the test results file / log files - and also to the standard console output. - -Installer Tips --------------- - -Apex Installer Tips -^^^^^^^^^^^^^^^^^^^ -Some specific tips are useful for the Apex Installer case. If not using -Apex Installer; ignore this section. - -In case of Triple-O based installer (like Apex) the docker container -needs to connect to the installer VM, so it is then required that some -known SSH keys are present in docker container. Since the Jumphost root -SSH keys are already known, easiest way is to use those using the -'Bind mount' method. See below for sample parameter:: - - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa - - NOTE: You need the "sudo" when creating the container to access root - users ssh credentials even the docker command itself might not - require that. - -HINT! In case of Triple-O installers you can find value for the -INSTALLER_IP parameter by executing command and note the returned IP -address:: - - inst=$(sudo virsh list | grep -iEo "undercloud|instack") - sudo virsh domifaddr ${inst} - - NOTE: In releases prior to Colorado, the name 'instack' was - used. Currently the name 'undercloud' is used. - -You can copy the credentials file from the "stack" users home directory -in installer VM to Jumphost. Please check the correct IP from the -command above. In the example below we are using invented IP address -"192.168.122.89":: - - scp stack@192.168.122.89:overcloudrc . - -Here is an example of the full docker command invocation for an Apex -installed system, using latest Functest docker container, for -illustration purposes:: - - sudo docker run -it --name "ApexFuncTestODL" \ - -e "INSTALLER_IP=192.168.122.89" \ - -e "INSTALLER_TYPE=apex" \ - -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \ - -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \ - -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \ - opnfv/functest /bin/bash - -Compass installer local development env usage Tips -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In the compass-functest local test case check and development environment, -in order to get openstack service inside the functest container, some -parameters should be configured during container creation, which are -hard to guess for freshman. This section will provide the guideline, the -parameters values are defaults here, which should be adjusted according -to the settings, the complete steps are given here so as not to appear -too abruptly. - -1, Pull Functest docker image from public dockerhub:: - - docker pull opnfv/functest: - - here can be "brahmaputra.1.0", "colorado.1.0", etc. -Tag omitted means the latest docker image:: - - docker pull opnfv/functest - -2, Functest Docker container creation - -To make a file used for the environment, such as 'functest-docker-env':: - - CINDER_ENDPOINT_TYPE=publicURL - NOVA_ENDPOINT_TYPE=publicURL - OS_ENDPOINT_TYPE=publicURL - OS_INTERFACE=publicURL - OS_USERNAME=admin - OS_PASSWORD='990232e0885da343ac805523322d' - OS_PROJECT_NAME=admin - OS_TENANT_NAME=admin - OS_AUTH_URL=https://192.16.1.222:5000/v3 - OS_NO_CACHE=1 - OS_USER_DOMAIN_NAME=Default - OS_PROJECT_DOMAIN_NAME=Default - OS_REGION_NAME=RegionOne - OS_IDENTITY_API_VERSION=3 - OS_AUTH_VERSION=3 +when performing manual test scenarios: + + * NODE_NAME = + * BUILD_TAG = + * CI_DEBUG = + +where: + + * = Symbolic name of the POD where the tests are run. + Visible in test results files, which are stored + to the database. This option is only used when + tests are activated under Jenkins CI control. + It indicates the POD/hardware where the test has + been run. If not specified, then the POD name is + defined as "Unknown" by default. + DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. + * = Symbolic name of the Jenkins Build Job. + Visible in test results files, which are stored + to the database. This option is only set when + tests are activated under Jenkins CI control. + It enables the correlation of test results, + which + are independently pushed to the results database + from different Jenkins jobs. + DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. + * = "true" or "false" + Default = "false", if not specified + If "true" is specified, then additional debug trace + text can be sent to the test results file / log files + and also to the standard console output. + + +Openstack credentials +===================== +OpenStack credentials are mandatory and must be provided to Functest. +When running the command "functest env prepare", the framework will +automatically look for the Openstack credentials file +"/home/opnfv/functest/conf/openstack.creds" and will exit with +error if it is not present or is empty. +There are 2 ways to provide that file: -Note: please adjust the content according to the environment, such as -'TENANT_ID' maybe used for some special cases. + * by using a Docker volume with -v option when creating the Docker container. + This is referred to in docker documentation as "Bind Mounting". + See the usage of this parameter in the following chapter. + * or creating manually the file '/home/opnfv/functest/conf/openstack.creds' + inside the running container and pasting the credentials in it. Consult + your installer guide for further details. This is however not + instructed in this document. -Then to create the Functest docker:: +There is a default environment variable in the Functest container **$creds** +that points to the credentials absolute path to help the user with this task. - docker run --privileged=true --rm -t \ - --env-file functest-docker-env \ - --name \ - opnfv/functest: /bin/bash +In proxified environment you may need to change the credentials file. +There are some tips in chapter: `Proxy support`_ -3, To attach Functest container +SSL Support +----------- +If you need to connect to a server that is TLS-enabled (the auth URL +begins with "https") and it uses a certificate from a private CA or a +self-signed certificate, then you will need to specify the path to an +appropriate CA certificate to use, to validate the server certificate +with the environment variable OS_CACERT:: + + echo $OS_CACERT + /etc/ssl/certs/ca.crt + +However, this certificate does not exist in the container by default. +It has to be copied manually from the OpenStack deployment. This can be +done in 2 ways: -Before trying to attach the Functest container, the status can be checked by:: + #. Create manually that file and copy the contents from the OpenStack + controller. + #. (Recommended) Add the file using a Docker volume when starting the + container:: - docker ps -a + -v :/etc/ssl/certs/ca.cert -to attach the 'Up' status Functest container and start bash mode:: +You might need to export OS_CACERT environment variable inside the +container:: - docker exec -it bash + export OS_CACERT=/etc/ssl/certs/ca.crt -4, Functest environment preparation and check +Certificate verification can be turned off using OS_INSECURE=true. For +example, Fuel uses self-signed cacerts by default, so an pre step would +be:: -To see the Section below `Preparing the Functest environment`_. + export OS_INSECURE=true Functest docker container directory structure ---------------------------------------------- +============================================= Inside the Functest docker container, the following directory structure should now be in place:: - `-- home - | `-- opnfv - | |-- functest - | | |-- conf - | | |-- data - | | |-- images - | | `-- results - | `-- repos - | |-- doctor + `-- + |- home + | |-- opnfv + | | `- functest + | | |-- conf + | | `-- results + | `-- repos | `-- vnfs - -- src - |-- tempest - |-- vims-test - |-- odl_test - `-- fds - -Underneath the '/home/opnfv/' directory, the Functest docker container + |- src + | |-- tempest + | |-- vims-test + | |-- odl_test + | `-- fds + `- usr + `- lib + `- python2.7 + `- site-packages + `- functest + |-- ... + +Underneath the '/home/opnfv/functest' directory, the Functest docker container includes two main directories: - * The **functest** directory stores configuration files (e.g. the + * The **conf** directory stores configuration files (e.g. the OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), - the **data** directory stores a 'cirros' test image used in some - functional tests and the **results** directory stores some temporary - result log files - * The **repos** directory holds various repositories. The directories - are used for the installation of the needed tooling (e.g. rally) or - for the retrieval of feature projects scenarios (e.g. promise) + * the **results** directory stores some temporary result log files + +src and repos directories are used to host third party code used for the tests. -The structure under the **functest** repository can be described as -follows:: +The functest code is under /usr/lib/python2.7/site-packages/functest +The structure can be described as follows:: |-- INFO |-- LICENSE @@ -465,8 +485,7 @@ follows:: We may distinguish several directories, the first level has 5 directories: -* **api**: This directory is dedicated for the internal Functest API and the API - (framework) documentations. +* **api**: This directory is dedicated to the API (framework) documentations. * **commons**: This directory is dedicated for storage of traffic profile or any other test inputs that could be reused by any test project. @@ -477,7 +496,9 @@ We may distinguish several directories, the first level has 5 directories: * **functest**: This directory contains all the code needed to run functest internal cases and OPNFV onboarded feature or VNF test cases. -Functest directory has 6 directories: +Functest directory has 7 sub-directories: + * **api**: This directory is dedicated for the internal Functest API and the + API (framework) documentations. * **ci**: This directory contains test structure definition files (e.g .yaml) and bash shell/python scripts used to configure and execute Functional tests. The test execution script @@ -495,8 +516,31 @@ Functest directory has 6 directories: own test code. See for an example the Openstack helper utility: 'openstack_utils.py'. -Useful Docker commands ----------------------- + +Logs +==== +By default all the logs are put un /home/opnfv/functest/results/functest.log. +If you want to have more logs in console, you may edit the logging.ini file +manually. +Connect on the docker then edit the file located in +/usr/lib/python2.7/site-packages/functest/ci/logging.ini + +Change wconsole to console in the desired module to get more traces. + + +Configuration +============= + +You may also directly modify the python code or the configuration file (e.g. +testcases.yaml used to declare test constraints) under +/usr/lib/python2.7/site-packages/functest + + +Tips +==== + +Docker +------ When typing **exit** in the container prompt, this will cause exiting the container and probably stopping it. When stopping a running Docker container all the changes will be lost, there is a keyboard shortcut @@ -538,116 +582,6 @@ destroy it:: Check the Docker documentation [`dockerdocs`_] for more information. -Preparing the Functest environment ----------------------------------- -Once the Functest docker container is up and running, the required -Functest environment needs to be prepared. A custom built **functest** -CLI utility is available to perform the needed environment preparation -action. Once the environment is prepared, the **functest** CLI utility -can be used to run different functional tests. The usage of the -**functest** CLI utility to run tests is described further in the -`Functest User Guide`_ - -Prior to commencing the Functest environment preparation, we can check -the initial status of the environment. Issue the **functest env status** -command at the prompt:: - - functest env status - Functest environment is not installed. - - Note: When the Functest environment is prepared, the command will - return the status: "Functest environment ready to run tests." - -To prepare the Functest docker container for test case execution, issue -the **functest env prepare** command at the prompt:: - - functest env prepare - -This script will make sure that the requirements to run the tests are -met and will install the needed libraries and tools by all Functest -test cases. It should be run only once every time the Functest docker -container is started from scratch. If you try to run this command, on -an already prepared environment, you will be prompted whether you really -want to continue or not:: - - functest env prepare - It seems that the environment has been already prepared. - Do you want to do it again? [y|n] - - (Type 'n' to abort the request, or 'y' to repeat the - environment preparation) - - -To list some basic information about an already prepared Functest -docker container environment, issue the **functest env show** at the -prompt:: - - functest env show - +======================================================+ - | Functest Environment info | - +======================================================+ - | INSTALLER: apex, 192.168.122.89 | - | SCENARIO: os-odl_l2-nofeature-ha | - | POD: localhost | - | GIT BRANCH: master | - | GIT HASH: 5bf1647dec6860464eeb082b2875798f0759aa91 | - | DEBUG FLAG: false | - +------------------------------------------------------+ - | STATUS: ready | - +------------------------------------------------------+ - - Where: - - INSTALLER: Displays the INSTALLER_TYPE value - - here = "apex" - and the INSTALLER_IP value - - here = "192.168.122.89" - SCENARIO: Displays the DEPLOY_SCENARIO value - - here = "os-odl_l2-nofeature-ha" - POD: Displays the value passed in NODE_NAME - - here = "localhost" - GIT BRANCH: Displays the git branch of the OPNFV Functest - project repository included in the Functest - Docker Container. - - here = "master" - (In first official colorado release - would be "colorado.1.0") - GIT HASH: Displays the git hash of the OPNFV Functest - project repository included in the Functest - Docker Container. - - here = "5bf1647dec6860464eeb082b2875798f0759aa91" - DEBUG FLAG: Displays the CI_DEBUG value - - here = "false" - - NOTE: In Jenkins CI runs, an additional item "BUILD TAG" - would also be listed. The value is set by Jenkins CI. - -Finally, the **functest** CLI has a **--help** options: - -Some examples:: - - functest --help Usage: functest [OPTIONS] COMMAND [ARGS]... - - Options: - --version Show the version and exit. - -h, --help Show this message and exit. - - Commands: - env - openstack - testcase - tier - - functest env --help - Usage: functest env [OPTIONS] COMMAND [ARGS]... - - Options: - -h, --help Show this message and exit. - - Commands: - prepare Prepares the Functest environment. - show Shows information about the current... - status Checks if the Functest environment is ready... Checking Openstack and credentials ---------------------------------- @@ -673,53 +607,24 @@ This command must show a set of environment variables starting with *OS_*, for example:: OS_REGION_NAME=RegionOne - OS_DEFAULT_DOMAIN=default + OS_USER_DOMAIN_NAME=Default OS_PROJECT_NAME=admin - OS_PASSWORD=admin - OS_AUTH_STRATEGY=keystone - OS_AUTH_URL=http://172.30.10.3:5000/v2.0 + OS_AUTH_VERSION=3 + OS_IDENTITY_API_VERSION=3 + OS_PASSWORD=da54c27ae0d10dfae5297e6f0d6be54ebdb9f58d0f9dfc + OS_AUTH_URL=http://10.1.0.9:5000/v3 OS_USERNAME=admin OS_TENANT_NAME=admin OS_ENDPOINT_TYPE=internalURL - OS_NO_CACHE=true + OS_INTERFACE=internalURL + OS_NO_CACHE=1 + OS_PROJECT_DOMAIN_NAME=Default + If the OpenStack command still does not show anything or complains about connectivity issues, it could be due to an incorrect url given to the OS_AUTH_URL environment variable. Check the deployment settings. -SSL Support ------------ -If you need to connect to a server that is TLS-enabled (the auth URL -begins with "https") and it uses a certificate from a private CA or a -self-signed certificate, then you will need to specify the path to an -appropriate CA certificate to use, to validate the server certificate -with the environment variable OS_CACERT:: - - echo $OS_CACERT - /etc/ssl/certs/ca.crt - -However, this certificate does not exist in the container by default. -It has to be copied manually from the OpenStack deployment. This can be -done in 2 ways: - - #. Create manually that file and copy the contents from the OpenStack - controller. - #. (Recommended) Add the file using a Docker volume when starting the - container:: - - -v :/etc/ssl/certs/ca.cert - -You might need to export OS_CACERT environment variable inside the -container:: - - export OS_CACERT=/etc/ssl/certs/ca.crt - -Certificate verification can be turned off using OS_INSECURE=true. For -example, Fuel uses self-signed cacerts by default, so an pre step would -be:: - - export OS_INSECURE=true - Proxy support ------------- If your Jumphost node is operating behind a http proxy, then there are @@ -732,12 +637,6 @@ succeed: in section: `Docker Installation on CentOS behind http proxy`_ below. - #. Execution of the Functest environment preparation inside the - created docker container Functest needs internet access to - download some resources for some test cases. This might not - work properly if the Jumphost is connecting to internet - through a http Proxy. - If that is the case, make sure the resolv.conf and the needed http_proxy and https_proxy environment variables, as well as the 'no_proxy' environment variable are set correctly:: @@ -769,10 +668,6 @@ http_proxy and https_proxy environment variables, as well as the git config --global http.sslVerify True git config --global http.proxy -Validation check: Before running **'functest env prepare'** CLI command, -make sure you can reach http and https sites from inside the Functest -docker container. - For example, try to use the **nc** command from inside the functest docker container:: @@ -786,12 +681,14 @@ Note: In a Jumphost node based on the CentOS family OS, the **nc** commands might not work. You can use the **curl** command instead. curl http://www.opnfv.org:80 + curl https://www.opnfv.org:443 + ', then the -Functest CLI utility will call **all valid Test Cases**, which belong to the -specified Test Tier, as relevant to scenarios deployed to the SUT environment. -Thus, the Functest CLI utility calculates automatically which tests can be -executed and which cannot, given the environment variable **DEPLOY_SCENARIO**, -which is passed in to the Functest docker container. - -Currently, the Functest CLI command 'functest testcase run ', supports -two possibilities:: - - * Run a single Test Case, specified by a valid choice of - * Run ALL test Test Cases (for all Tiers) by specifying = 'all' - -Functest includes a cleaning mechanism in order to remove all the OpenStack -resources except those present before running any test. The script -*$REPOS_DIR/functest/functest/utils/openstack_snapshot.py* is called once when setting up -the Functest environment (i.e. CLI command 'functest env prepare') to snapshot -all the OpenStack resources (images, networks, volumes, security groups, tenants, -users) so that an eventual cleanup does not remove any of these defaults. - -The script **openstack_clean.py** which is located in -*$REPOS_DIR/functest/functest/utils/* is in charge of cleaning the OpenStack resources -that are not specified in the defaults file generated previously which is stored in -*/home/opnfv/functest/conf/openstack_snapshot.yaml* in the Functest docker container. - -It is important to mention that if there are new OpenStack resources created -manually after the snapshot done before running the tests, they will be removed, -unless you use the special method of invoking the test case with specific -suppression of clean up. (See the `Troubleshooting`_ section). - -The reason to include this cleanup meachanism in Functest is because some -test suites create a lot of resources (users, tenants, networks, volumes etc.) -that are not always properly cleaned, so this function has been set to keep the -system as clean as it was before a full Functest execution. - -Although the Functest CLI provides an easy way to run any test, it is possible to -do a direct call to the desired test script. For example: - - python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py - - -Automated testing ------------------ - -As mentioned previously, the Functest Docker container preparation as well as -invocation of Test Cases can be called within the container from the Jenkins CI -system. There are 3 jobs that automate the whole process. The first job runs all -the tests referenced in the daily loop (i.e. that must been run daily), the second -job runs the tests referenced in the weekly loop (usually long duration tests run -once a week maximum) and the third job allows testing test suite by test suite specifying -the test suite name. The user may also use either of these Jenkins jobs to execute -the desired test suites. - -One of the most challenging task in the Danube release consists -in dealing with lots of scenarios and installers. Thus, when the tests are -automatically started from CI, a basic algorithm has been created in order to -detect whether a given test is runnable or not on the given scenario. -Some Functest test suites cannot be systematically run (e.g. ODL suite can not -be run on an ONOS scenario). The daily/weekly notion has been introduces in -Colorado in order to save CI time and avoid running systematically -long duration tests. It was not used in Colorado due to CI resource shortage. -The mechanism remains however as part of the CI evolution. - -CI provides some useful information passed to the container as environment -variables: - - * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE - * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP - * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with - - * controller = (odl|ocl|nosdn) - * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|ovs_dpdk_bar) - * mode = (ha|noha) - -The constraints per test case are defined in the Functest configuration file -*/usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml*:: - - tiers: - - - name: smoke - order: 1 - ci_loop: '(daily)|(weekly)' - description : >- - Set of basic Functional tests to validate the OpenStack deployment. - testcases: - - - name: vping_ssh - criteria: 'status == "PASS"' - blocking: true - description: >- - This test case verifies: 1) SSH to an instance using floating - IPs over the public network. 2) Connectivity between 2 instances - over a private network. - dependencies: - installer: '' - scenario: '^((?!bgpvpn|odl_l3).)*$' - run: - module: 'functest.opnfv_tests.openstack.vping.vping_ssh' - class: 'VPingSSH' - .... - -We may distinguish 2 levels in the test case description: - * Tier level - * Test case level - -At the tier level, we define the following parameters: - - * ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs - * description: a high level view of the test case - -For a given test case we defined: - * the name of the test case - * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL - * blocking: if set to true, if the test is failed, the execution of the following tests is canceled - * the description of the test case - * the dependencies: a combination of 2 regex on the scenario and the installer name - * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests - -For further details on abstraction classes, see developper guide. - -Additional parameters have been added in the desription in the Database. -The target is to use the configuration stored in the Database and consider the -local file as backup if the Database is not reachable. -The additional fields related to a test case are: - * trust: we introduced this notion to put in place a mechanism of scenario promotion. - * Version: it indicates since which version you can run this test - * domains: the main domain covered by the test suite - * tags: a list of tags related to the test suite - -The order of execution is the one defined in the file if all test cases are selected. - -In CI daily job the tests are executed in the following order: - - 1) healthcheck (blocking) - 2) smoke: both vPings are blocking - 3) Feature project tests cases - -In CI weekly job we add 2 tiers: - - 4) VNFs (vIMS) - 5) Components (Rally and Tempest long duration suites) - -As explained before, at the end of an automated execution, the OpenStack resources -might be eventually removed. -Please note that a system snapshot is taken before any test case execution. - -This testcase.yaml file is used for CI, for the CLI and for the automatic reporting. - - -Executing Functest suites (Alpine) -================================== +Executing Functest suites +========================= As mentioned in the configuration guide `[1]`_, Alpine docker containers have been introduced in Euphrates. @@ -362,9 +29,117 @@ You should get:: You can run functest-healcheck, functest-smoke, functest-features, functest-components and functest-vnf. -Please note that you may also use the CLI for manual tests using Alpine -containers. +The result tables show the results by test case, it can be:: + + * PASS + * FAIL + * SKIP: if the scenario/installer does not support the test case + + +Manual run +========== +If you want to run the test step by step, you may add docker option then run the +different commands within the docker. + +Considering the healthcheck example, running functest manaully means:: + + sudo docker run -ti --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-healthcheck /bin/bash + +The docker prompt shall be returned. Then within the docker run the following +commands:: + + $ source /home/opnfv/functest/conf/openstack.creds + +Prepare environment +------------------- +Prior to commencing the Functest environment preparation, we can check +the initial status of the environment. Issue the **functest env status** +command at the prompt:: + + # functest env status + # Functest environment is not installed. + +To prepare the Functest docker container for test case execution, type:: + + # functest env prepare + # ... + # Functest environment ready to run tests. + +you may also type prepare_env instead of functest env prepare. + +To list some basic information about an already prepared Functest +docker container environment, issue the **functest env show** at the +prompt:: + bash-4.3# functest env show + +------------------------------+---------------------------------+ + | FUNCTEST ENVIRONMENT | VALUE | + +------------------------------+---------------------------------+ + | STATUS | ready | + | SCENARIO | os-nosdn-nofeature-noha | + | DEBUG FLAG | false | + | BUILD_TAG | None | + | INSTALLER | compass | + | POD | huawei-pod1 | + +------------------------------+---------------------------------+ + +See configuration guide for details on Functest environnement variables. + +Tier +---- +Each Alpine container provided on the docker hub matches with a tier. +The following commands are available:: + + # functest tier list + - 0. healthcheck: + ['connection_check', 'api_check', 'snaps_health_check'] + # functest tier show healthcheck + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | + | | | | basic operations in the VIM. | snaps_health_check | + +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ + +To run all the cases of the tier, type:: + + # functest tier run healthcheck + +Testcase +-------- +Testcases can be listed, shown and run though the CLI:: + + # functest testcase list + connection_check + api_check + snaps_health_check + # functest testcase show api_check + +-------------------+--------------------------------------------------+------------------+---------------------------+ + | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | + +-------------------+--------------------------------------------------+------------------+---------------------------+ + | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ | + | | OpenStack clients: Keystone, Glance, | | | + | | Neutron and Nova and may perform some | | | + | | simple queries. When the config value of | | | + | | snaps.use_keystone is True, functest | | | + | | must have access to the cloud's private | | | + | | network. | | | + +-------------------+--------------------------------------------------+------------------+---------------------------+ + # functest testcase run connection_check + ... + # functest run all + +You can also type run_tests -t all to run all the tests. + +Note the list of test cases depend on the installer and the scenario. + +Reporting results to the test Database +====================================== +In OPNFV CI we collect all the results from CI. A test APi shall be available +as well as a test database `[17]`_. Functest internal API ===================== @@ -377,18 +152,23 @@ other test projects. In Euphrates the main method of the APIs are: - * Show environment - * Prepare Environment * Show credentials + * Update openrc file + * Show environment + * Update hosts info for domain name + * Prepare environment * List all testcases * Show a testcase + * Run a testcase * List all tiers * Show a tier * List all testcases within given tier + * Get the result of the specified task + * Get the log of the specified task -The API can be invoked as follows: - http://:5000/api/v1/functest/envs +See `[16]`_ to get examples on how to use the API. -TODO -.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/# +.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html +.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API +.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index f23abc50..1cdd98e5 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -9,7 +9,7 @@ VIM (Virtualized Infrastructure Manager) Healthcheck tests ^^^^^^^^^^^^^^^^^ -In Danube, healthcheck tests have been refactored and rely on SNAPS, an +Since Danube, healthcheck tests have been refactored and rely on SNAPS, an OPNFV middleware project. SNAPS stands for "SDN/NFV Application development Platform and Stack". @@ -195,6 +195,7 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas + * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -207,35 +208,20 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run Defcore testcases ------------------------------------------ +Refstack-client to run OpenStack interop testcases +-------------------------------------------------- Refstack-client `[8]`_ is a command line utility that allows you to execute Tempest test runs based on configurations you specify. -It is the official tool to run Defcore `[9]`_ testcases, -which focuses on testing interoperability between OpenStack clouds. +It is the official tool to run Openstack Interop (previously known as Defcore) +testcases `[9]`_, which focus on testing interoperability between OpenStack +clouds. Refstack-client is integrated in Functest, consumed by Dovetail, which intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. This progress is under the guideline of Compliance Verification Program(CVP). -Defcore testcases -^^^^^^^^^^^^^^^^^^ - -*Danube Release* - -Set of DefCore tempest test cases not flagged and required. -According to `[10]`_, some tests are still flagged due to outstanding bugs -in the Tempest library, particularly tests that require SSH. Refstack developers -are working on correcting these bugs upstream. Please note that although some tests -are flagged because of bugs, there is still an expectation that the capabilities -covered by the tests are available. It only contains Openstack core compute -(no object storage). The approved guidelines (2016.08) are valid for Kilo, -Liberty, Mitaka and Newton releases of OpenStack. -The list can be generated using the Rest API from RefStack project: -https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false - Running methods ^^^^^^^^^^^^^^^ @@ -314,8 +300,8 @@ snaps_smoke This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. When -the config value of snaps.use_keystone is True, Functest must have access +value snaps.use_floating_ips (True|False) to toggle this functionality. +Please note that When the configuration value of snaps.use_keystone is True, Functest must have access the cloud's private network. This suite consists in 38 tests (test duration < 10 minutes) @@ -323,12 +309,6 @@ This suite consists in 38 tests (test duration < 10 minutes) SDN Controllers --------------- -There are currently 3 available controllers: - - * OpenDaylight (ODL) - * ONOS - * OpenContrail (OCL) - OpenDaylight ^^^^^^^^^^^^ @@ -397,7 +377,7 @@ Functest has been supporting several feature projects since Brahpamutra: +-----------------+---------+----------+--------+-----------+ | fds | | | X | X | +-----------------+---------+----------+--------+-----------+ -| moon | | X | | X | +| moon | | X | | | +-----------------+---------+----------+--------+-----------+ | multisite | | X | X | | +-----------------+---------+----------+--------+-----------+ @@ -409,7 +389,7 @@ Functest has been supporting several feature projects since Brahpamutra: +-----------------+---------+----------+--------+-----------+ | orchestra | | | X | X | +-----------------+---------+----------+--------+-----------+ -| parser | | | X | | +| parser | | | X | X | +-----------------+---------+----------+--------+-----------+ | promise | X | X | X | X | +-----------------+---------+----------+--------+-----------+ @@ -459,7 +439,7 @@ orchestrator. parser ^^^^^^ -See parser user guide for details: `[12]`_ +See parser user guide for details. vyos-vrouter @@ -482,13 +462,38 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +cloudify_ims_perf +^^^^^^^^^^^^^^^^^ + +This test case is available but not declared in testcases.yaml. If you want to +run it you need to get the Ixia loader images and have access to an Ixia license +server. + +The declaration of the testcase is simple, connect to your functest-vnf docker, +add the following section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: + + case_name: cloudify_ims_perf + project_name: functest + criteria: 80 + blocking: false + description: >- + Stress tests based on Cloudify. Ixia loader images and access to Ixia + server license. + dependencies: + installer: '' + scenario: 'os-nosdn-nofeature-ha' + run: + module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' + class: 'CloudifyImsPerf' + + .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst .. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html +.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index b9faa24a..2fca325a 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -4,14 +4,16 @@ Overview of the Functest suites =============================== -Functest is the OPNFV project primarily targeting function testing. +Functest is the OPNFV project primarily targeting functional testing. In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 5 main domains: VIM -(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), -Features, VNF (Virtual Network Functions) and MANO stacks. +The current list of test suites can be distributed over 4 main domains: + * VIM (Virtualised Infrastructure Manager) + * Controllers (i.e. SDN Controllers) + * Features + * VNF (Virtual Network Functions) Functest test suites are also distributed in the OPNFV testing categories: healthcheck, smoke, features, components, performance, VNF, Stress tests. @@ -41,20 +43,20 @@ validate the scenario for the release. | | | | VM and then executed via SSH. | | | | | The script will ping another | | | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network. | +| | | | the SUT Private Tenant network | | | +----------------+----------------------------------+ | | | vping_userdata | Uses Ping with given userdata | | | | | to test intra-VM connectivity | | | | | over the SUT Private Tenant | | | | | network. The correct operation | | | | | of the NOVA Metadata service is | -| | | | also verified in this test. | +| | | | also verified in this test | | | +----------------+----------------------------------+ | | | tempest_smoke | Generate and run a relevant | | | | \_serial | Tempest Test Suite in smoke mode.| | | | | The generated test set is | | | | | dependent on the OpenStack | -| | | | deployment environment. | +| | | | deployment environment | | | +----------------+----------------------------------+ | | | rally_sanity | Run a subset of the OpenStack | | | | | Rally Test Suite in smoke mode | @@ -70,11 +72,11 @@ validate the scenario for the release. | | | | See the OpenStack reference test | | | | | suite `[2]`_. The generated | | | | | test set is dependent on the | -| | | | OpenStack deployment environment.| +| | | | OpenStack deployment environment | | | +----------------+----------------------------------+ | | | rally_full | Run the OpenStack testing tool | | | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_. | +| | | | See the Rally documents `[3]`_ | | | +----------------+----------------------------------+ | | | tempest_custom | Allow to run a customized list | | | | | of Tempest cases | @@ -149,7 +151,7 @@ validate the scenario for the release. | | | | regarding compute, network and | | | | | storage. | | | | | See `Promise User Guide`_ for | -| | | | details. | +| | | | details | +-------------+---------------+----------------+----------------------------------+ | VNF | vnf | cloudify_ims | Example of a real VNF deployment | | | | | to show the NFV capabilities of | @@ -166,6 +168,21 @@ validate the scenario for the release. | | | cleawaterims | Openbaton orchestrator | | | +----------------+----------------------------------+ | | | vyos_vrouter | vRouter testing | +| | +----------------+----------------------------------+ +| | | cloudify_ims | Based on cloudify_ims test case | +| | | perf | cloudify_ims_perf substitutes | +| | | | the signaling test suite by an | +| | | | automatic deployment of an Ixia | +| | | | loader and generic SIP stress | +| | | | tests. | +| | | | This work has been initiated | +| | | | during the plugfest and allows | +| | | | realistic load tests on top of | +| | | | cloudify_ims. Please note that | +| | | | this test is available but not | +| | | | declared in testcases.yaml as it | +| | | | requires access to proprietary | +| | | | resources (Ixia loader) | +-------------+---------------+----------------+----------------------------------+ @@ -199,7 +216,7 @@ NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot of a live Functest Dashboard is shown below: -.. figure:: ../../../images/FunctestDashboardDanube.png +.. figure:: ../../../images/FunctestDashboardEuphrates.png :align: center :alt: Functest Dashboard @@ -215,15 +232,15 @@ This parameters as well as possible tags can be used for the Test case catalog. vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure. -Functest considers OPNFV as a black box. As of Danube release the OPNFV -offers a lot of potential combinations: +Functest considers OPNFV as a black box. OPNFV offers a lot of potential +combinations (which may change from one version to another): * 3 controllers (OpenDaylight, ONOS, OpenContrail) * 5 installers (Apex, Compass, Daisy, Fuel, Joid) Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_IP and +deployed features. The system uses the environment variables (INSTALLER_TYPE and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given environment. @@ -233,8 +250,8 @@ clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution -of **ALL** testcases. The Functest CLI is introduced in more detail in the -section `Executing the functest suites`_ of this document. +of **ALL** testcases. The Functest CLI is introduced in more details in next +section. .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 53e4d3a8..6129ef3a 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -7,7 +7,8 @@ Manual testing In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. -If you want additionnal logs, you may configure the logging.ini under /functest:functest/ci +If you want additional logs, you may configure the logging.ini under +/usr/lib/python2.7/site-packages/functest/ci. Automated testing -------------- @@ -47,4 +48,4 @@ Based on the results stored in the result database, a `Functest reporting`_ portal is also automatically updated. This portal provides information on the overall status per scenario and per installer -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html +.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 1c516608..29facc29 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -43,9 +43,9 @@ These test cases can be run inside the container, using new Functest CLI as foll $ functest testcase run vping_userdata The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths: -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py* and -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_userdata.py* +python scripts, located in paths +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py +and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -122,9 +122,10 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/ping.sh* and takes an IP as -a parameter. When the SCP is completed, the test will do an SSH call to that script -inside the second instance. Some problems can happen here:: +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +and takes an IP as a parameter. When the SCP is completed, the test will do a +SSH call to that script inside the second instance. Some problems can happen +here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting @@ -210,9 +211,6 @@ If this text or similar is shown:: it means that the instance failed to read from the metadata service. Contact the Functest or installer teams for more information. -NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests -have been excluded from CI in those scenarios. - Tempest ^^^^^^^ @@ -238,7 +236,7 @@ of the following | execution test cases are | Rally in Functest, does not contain all the needed | | missing | parameters or some parameters are not set properly. | | | The tempest.conf file is located in directory | -| | '/home/opnfv/.rally/verification/verifier- | +| | 'root/.rally/verification/verifier- | | | /for-deployment-' | | | in the Functest Docker container. Use the "rally | | | deployment list" command in order to check the UUID | @@ -266,12 +264,12 @@ Possible scenarios are: * keystone * neutron * nova + * ceilometer * quotas - * requests * vm To know more about what those scenarios are doing, they are defined in directory: -*$REPOS_DIR/functest/functest/opnfv_tests/openstack/rally/scenario* +/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario For more info about Rally scenario definition please refer to the Rally official documentation. `[3]`_ @@ -295,10 +293,6 @@ If any of the other test cases fails, check that Neutron and ODL have been correctly configured to work together. Check Neutron configuration files, accounts, IP addresses etc.). -ONOS -^^^^ -Please refer to the ONOS documentation. `ONOSFW User Guide`_ . - Features -------- @@ -306,7 +300,6 @@ Features Please refer to the dedicated feature user guides for details. - VNF --- @@ -342,6 +335,7 @@ described in the following table: | the VM | the vIMS VNF installation fails | +-----------------------------------+------------------------------------+ - +Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it +is there fore not recommended to run it on a light configuration. .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/# -- cgit 1.2.3-korg