diff options
Diffstat (limited to 'docs/testing')
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 315 | ||||
| -rw-r--r-- | docs/testing/developer/internship/security_group/index.rst | 70 | ||||
| -rw-r--r-- | docs/testing/developer/internship/testapi_evolution/index.rst | 237 | ||||
| -rw-r--r-- | docs/testing/developer/internship/unit_tests/index.rst | 126 | ||||
| -rw-r--r-- | docs/testing/developer/internship/vnf_catalog/index.rst | 170 | ||||
| -rw-r--r-- | docs/testing/user/configguide/ci.rst | 13 | ||||
| -rw-r--r-- | docs/testing/user/configguide/configguide.rst | 727 | ||||
| -rw-r--r-- | docs/testing/user/configguide/index.rst | 130 | ||||
| -rw-r--r-- | docs/testing/user/configguide/prerequisites.rst | 102 | ||||
| -rw-r--r-- | docs/testing/user/userguide/index.rst | 106 | ||||
| -rw-r--r-- | docs/testing/user/userguide/reporting.rst | 90 | ||||
| -rw-r--r-- | docs/testing/user/userguide/runfunctest.rst | 140 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_details.rst | 497 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_overview.rst | 239 | ||||
| -rw-r--r-- | docs/testing/user/userguide/test_results.rst | 53 | ||||
| -rw-r--r-- | docs/testing/user/userguide/troubleshooting.rst | 378 |
16 files changed, 0 insertions, 3393 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst deleted file mode 100644 index 19a1bdcb..00000000 --- a/docs/testing/developer/devguide/index.rst +++ /dev/null @@ -1,315 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. SPDX-License-Identifier: CC-BY-4.0 - -************************ -Functest Developer Guide -************************ - -.. toctree:: - :numbered: - :maxdepth: 2 - -============ -Introduction -============ - -Functest is a project dealing with functional testing. -The project produces its own internal test cases but can also be considered -as a framework to support feature and VNF onboarding project testing. - -Therefore there are many ways to contribute to Functest. You can: - - * Develop new internal test cases - * Integrate the tests from your feature project - * Develop the framework to ease the integration of external test cases - -Additional tasks involving Functest but addressing all the test projects -may also be mentioned: - - * The API / Test collection framework - * The dashboards - * The automatic reporting portals - * The testcase catalog - -This document describes how, as a developer, you may interact with the -Functest project. The first section details the main working areas of -the project. The Second part is a list of "How to" to help you to join -the Functest family whatever your field of interest is. - - -======================== -Functest developer areas -======================== - - -Functest High level architecture -================================ - -Functest is a project delivering test containers dedicated to OPNFV. -It includes the tools, the scripts and the test scenarios. -In Euphrates Alpine containers have been introduced in order to lighten the -container and manage testing slicing. The new containers are created according -to the different tiers: - - * functest-core: https://hub.docker.com/r/opnfv/functest-core/ - * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ - * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ - * functest-features: https://hub.docker.com/r/opnfv/functest-features/ - * functest-components: https://hub.docker.com/r/opnfv/functest-components/ - * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ - * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/ - * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ - -Standalone functest dockers are maintained for Euphrates but Alpine containers -are recommended. - -Functest can be described as follow:: - - +----------------------+ - | | - | +--------------+ | +-------------------+ - | | | | Public | | - | | Tools | +------------------+ OPNFV | - | | Scripts | | | System Under Test | - | | Scenarios | | | | - | | | | | | - | +--------------+ | +-------------------+ - | | - | Functest Docker | - | | - +----------------------+ - -Functest internal test cases -============================ -The internal test cases in Euphrates are: - - - * api_check - * connection_check - * snaps_health_check - * vping_ssh - * vping_userdata - * odl - * odl_netvirt - * rally_full - * rally_sanity - * tempest_smoke_serial - * tempest_full_parallel - * cloudify_ims - -By internal, we mean that this particular test cases have been developed and/or -integrated by functest contributors and the associated code is hosted in the -Functest repository. -An internal case can be fully developed or a simple integration of -upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are -just integrated in Functest). - -The structure of this repository is detailed in `[1]`_. -The main internal test cases are in the opnfv_tests subfolder of the -repository, the internal test cases can be grouped by domain: - - * sdn: odl, odl_netvirt, odl_fds - * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_* - * vnf: cloudify_ims - -If you want to create a new test case you will have to create a new folder under -the testcases directory (See next section for details). - -Functest external test cases -============================ -The external test cases are inherited from other OPNFV projects, especially the -feature projects. - -The external test cases are: - - * barometer - * bgpvpn - * doctor - * domino - * fds - * parser - * promise - * refstack_defcore - * snaps_smoke - * functest-odl-sfc - * orchestra_clearwaterims - * orchestra_openims - * vyos_vrouter - * juju_vepc - -External test cases integrated in previous versions but not released in -Euphrates: - - * copper - * moon - * netready - * security_scan - - -The code to run these test cases is hosted in the repository of the project. -Please note that orchestra test cases are hosted in Functest repository and not -in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in -functest as there are no dedicated projects. - - -Functest framework -================== - -Functest is a framework. - -Historically Functest is released as a docker file, including tools, scripts and -a CLI to prepare the environment and run tests. -It simplifies the integration of external test suites in CI pipeline and provide -commodity tools to collect and display results. - -Since Colorado, test categories also known as **tiers** have been created to -group similar tests, provide consistent sub-lists and at the end optimize -test duration for CI (see How To section). - -The definition of the tiers has been agreed by the testing working group. - -The tiers are: - * healthcheck - * smoke - * features - * components - * vnf - -Functest abstraction classes -============================ - -In order to harmonize test integration, abstraction classes have been -introduced: - - * testcase: base for any test case - * unit: run unit tests as test case - * feature: abstraction for feature project - * vnf: abstraction for vnf onboarding - -The goal is to unify the way to run tests in Functest. - -Feature, unit and vnf_base inherit from testcase:: - - +----------------------------------------------------------------+ - | | - | TestCase | - | | - | - init() | - | - run() | - | - push_to_db() | - | - is_successful() | - | | - +----------------------------------------------------------------+ - | | | | - V V V V - +--------------------+ +---------+ +------------------------+ +-----------------+ - | | | | | | | | - | feature | | unit | | vnf | | robotframework | - | | | | | | | | - | | | | |- prepare() | | | - | - execute() | | | |- deploy_orchestrator() | | | - | BashFeature class | | | |- deploy_vnf() | | | - | | | | |- test_vnf() | | | - | | | | |- clean() | | | - +--------------------+ +---------+ +------------------------+ +-----------------+ - - -Testcase --------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html - -Feature -------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html - -Unit ----- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html - -VNF ---- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html - -Robotframework --------------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html - - -see `[5]`_ to get code samples - - -Functest util classes -===================== - -In order to simplify the creation of test cases, Functest develops also some -functions that are used by internal test cases. -Several features are supported such as logger, configuration management and -Openstack capabilities (tacker,..). -These functions can be found under <repo>/functest/utils and can be described as -follows:: - - functest/utils/ - |-- config.py - |-- constants.py - |-- decorators.py - |-- env.py - |-- functest_utils.py - |-- openstack_tacker.py - `-- openstack_utils.py - -It is recommended to use the SNAPS-OO library for deploying OpenStack instances. -SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. - - -TestAPI -======= -Functest is using the Test collection framework and the TestAPI developed by -the OPNFV community. See `[6]`_ for details. - - -Reporting -========= -A web page is automatically generated every day to display the status based on -jinja2 templates `[3]`_. - - -Dashboard -========= - -Additional dashboarding is managed at the testing group level, see `[7]`_ for -details. - - -======= -How TOs -======= - -See How to section on Functest wiki `[8]`_ - - -========== -References -========== - -_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide - -_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide - -_`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting - -_`[4]`: https://git.opnfv.org/snaps/ - -_`[5]` : http://testresults.opnfv.org/functest/framework/index.html - -_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html - -_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba - -_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 - -IRC support chan: #opnfv-functest diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst deleted file mode 100644 index d1cdbdd8..00000000 --- a/docs/testing/developer/internship/security_group/index.rst +++ /dev/null @@ -1,70 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -================================== -Functest Security group test cases -================================== - -Author: Girish Sukhatankar -mentors: D.Blaisonneau, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - - - -Problem Statement: ------------------- - - - -Curation Phase --------------- - - - - - -Schedule: -========= - - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December - January | ........ | -+--------------------------+------------------------------------------+ -| January - february | ........ | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest - diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst deleted file mode 100644 index 3038d0ac..00000000 --- a/docs/testing/developer/internship/testapi_evolution/index.rst +++ /dev/null @@ -1,237 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -================= -TestAPI evolution -================= - -Author: Sakala Venkata Krishna Rohit -Mentors: S. Feng, J.Lausuch, M.Richomme - -Abstract -======== - -The TestAPI is used by all the test opnfv projects to report results. -It is also used to declare projects, test cases and labs. A major refactoring -has been done in Colorado with the introduction of swagger. The TestAPI is defined in Functest -developer guide. The purpose of this project is to add more features to the TestAPI that automate -the tasks that are done manually now, though there are tasks other than automation. - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ -| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship | -| | | (IIIT Hyderabad) | | -+------------+----------+------------------+------------------------+ - -Overview: -========= - -The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_. -The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme. -The link to the patches submitted is `[2]`_. The internship was successfully completed and the -documentation is as follows. - -Problem Statement: ------------------- - -The problem statement could be divided into pending features that needed to be added into TestAPI -repo. The following were to be accomplished within the internship time frame. - -* **Add verification jenkins job for the TestAPI code** - The purpose of this job is to verify whehter the unit tests are successful or not with the - inclusion of the patchset submitted. - -* **Automatic update of opnfv/testapi docker image** - The docker image of TestAPI is hosted in the opnfv docker hub. To ensure that the TestAPI image - is always updated with the repository, automatic updation of the image is necessary and a job - is triggered whenever a new patch gets merged. - -* **Automation deployment of testresults.opnfv.org/test/ website** - In the same manner as the docker image of TestAPI is updated, the TestAPI website needs to be - in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the - updation of the testresults website. - -* **Generate static documentation of TestAPI calls** - The purpose of this is to give an static/offline view of TestAPI. If someone wants to have a - look at the Restful APIs of TestAPI, he/she does't need to go to the website, he can download - a html page and view it anytime. - -* **Backup MongoDB of TestAPI** - The mongoDB needs to be backed up every week. Till now it was done manually, but due to this - internship, it is now automated using a jenkins job. - -* **Add token based authorization to the TestAPI calls** - The token based authorization was implemented to ensure that only ci_pods could access the - database. Authentication has been added to only delete/put/post requests. - -Curation Phase: ---------------- - -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar -with the TestAPI code and functionality and propose the solutions/tools for the tasks mentioned -above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static -documentaion. - -Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic -deployment task involves accessing a remote server from inside the jenkins build. The deployment -had to be done only after the docker image update is done. For these constraints to satisfy, a -multijob jenkins job was choosen instead of a freestyle job. - -Important Links: ----------------- - -* MongoDB Backup Link - `[4]`_ -* Static Documentation - `[5]`_ -* TestAPI Token addition to ci_pods - `[6]`_ - -Schedule: -========= - -The progress and completion of the tasks is described in the below table. - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| Nov 14th - Dec 31st | Understand TestAPI code and the | -| | requirements. | -+--------------------------+------------------------------------------+ -| Jan 1st - Jan 7th | Add jenkins job to create static | -| | documentation and write build scripts. | -+--------------------------+------------------------------------------+ -| Jan 8th - Jan 21st | Add verification jenkins job for unit | -| | tests. | -+--------------------------+------------------------------------------+ -| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup | -| | | -+--------------------------+------------------------------------------+ -| Jan 29th - Feb 11th | Enable automatic deployment of | -| | testresults.opnfv.org/test/ | -+--------------------------+------------------------------------------+ -| Feb 12th - Feb 17th | Add token based authentication | -| | | -+--------------------------+------------------------------------------+ - -FAQ's -===== - -This section lists the problems that I have faced and the understanding that I have acquired during -the internship. This section may help other developers in solving any errors casused because of the -code written as a part of this internship. - - -TestAPI -------- - -What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If in the setup.cfg, it is defined as - -[files] -data_files = -etc/a.conf = etc/a.conf.sample - -then it ends up installed in the /usr/etc/. With this configuration, it would be installed -correctly within a venv. but when it is defined as - -[files] -data_files = -/etc/a.conf = etc/a.conf.sample - -then it ends up installed on the root of the filesystem instead of properly be installed within the -venv. - -Which attribute does swagger-codegen uses as the title in the generation of document generation ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It uses the nickname of the api call in swagger as the title in the generation of the document -generation. - -Does swagger-codegen take more than one yaml file as input ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml -file, one needs to merge them and give it as an input keeping mind the swagger specs. - - -Jenkins & JJB -------------- - -Which scm macro is used for verification jenkins jobs ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for -verification jenkins job. - -Does the virtualenv created in one build script exists in other build scripts too ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, the virtualenv created in one build script only exists in that build script/shell. - -What parameters are needed for the scm macros ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Project and Branch are the two parameters needed for scm macros. - -What is the directory inside the jenkins build ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give -you all the contents of the directory. - -How to include a bash script in jenkins job yaml file ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An example might be apt here as an answer. - -builders: - - shell: - !include-raw: include-raw001-hello-world.sh - - -How do you make a build server run on a specific machine ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each -server, one can use those parameter macros. -Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses -SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE -defined. - -What job style should be used when there is a situation like one build should trigger other builds -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -or when different build scripts need to be run on different machines ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MultiJob style should be used as it has phases where each phase can be taken as a build scipt and -can have its own parameters by which one can define the SLAVE_LABEL parameter. - -References: -=========== - -_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution - -_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22 - -_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing - -_`[4]` : http://artifacts.opnfv.org/testapibackup.html - -_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html - -_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst deleted file mode 100644 index a117c860..00000000 --- a/docs/testing/developer/internship/unit_tests/index.rst +++ /dev/null @@ -1,126 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -=================== -Functest Unit tests -=================== - -Author: Ashish Kumar -Mentors: H.Yao, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ -| 2017-03-31 | 0.0.2 | Ashish Kumar | During the | -| | | (IIIT Hyderabad) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= -Functest project is developing and integrating functional test cases for OPNFV -and it is part of OPNFV since the beginning. Functest develops its own testcases -and framework. This framework includes several utility libraries. The Project is -growing rapidly with more features, tests added as per requirement. It becomes -the responsibility of every developer to maintain the integrity of code i.e. new -patch should not break the previous functionality of the project. To automate this -process of software development, we should write unit tests and add them to CI so -that when a new patch is ready to merge, we shouldn't allow those which are breaking -previous unit tests or decreasing the coverage. - - - -Problem Statement: ------------------- -The goal of the intership consists in creating unit test suites on Functest code -with good code coverage (>80%) and integrate it in continuous integration in order -to consolidate existing code. - - -Curation Phase --------------- -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get -familiar with the functest code and functionality and explore the solutions for unit -testing in other projects and come up with the strategy for writing unit tests in functest. - -In this phase we decided, -- Coverage should be 80%. There are some functions like __init__, getter, setter and other - private methods for which writing unit test is a tedious job, so we are leaving these methods - for now. -- Do method wise testing for every module. -- Use mock for external or third party services, system calls and other external library calls - which could impact the behaviour of system during the run of unit test. -- Add it in jenkins as passing criteria for patches. -- Write tests in modular way so that it can help to serve as a form of documentation. - - - -Schedule: -========= -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| Nov 14th - Nov 28th | 1. Learn Functest Project Business | -| | 2. Set up the development environment | -| | 3. Run Functest code | -+--------------------------+------------------------------------------+ -| Nov 28th - Dec.9th | 1. Explore Unit Testing Strategy, | -| | 2. Learn about Mock in python | -+--------------------------+------------------------------------------+ -| Dec 12th - Dec 23rd | Implement Unit Tests for CLI | -| | | -+--------------------------+------------------------------------------+ -| Dec 26th - Jan 6th | Implement Unit Tests for Utils | -| | | -+--------------------------+------------------------------------------+ -| Jan 9th - Jan 20th | Implement Unit Tests for CI | -| | | -+--------------------------+------------------------------------------+ -| Jan 23rd - Feb 3rd | Implement Unit Tests for Core | -| | | -+--------------------------+------------------------------------------+ -| Feb 6th - Feb 17th | Implement Unit Tests for | -| | opnfv_tests/openstack/tempest | -+--------------------------+------------------------------------------+ -| Feb 20th - Mar 3rd | Implement Unit Tests for | -| | opnfv_tests/openstack/rally | -+--------------------------+------------------------------------------+ -| Mar 6th - Mar 17th | Implement Unit Tests for | -| | opnfv_tests/vnf/ims | -+--------------------------+------------------------------------------+ -| Mar 20th - Mar 31st | Recheck and Increase Coverage for all | -| | modules > 80% | -+--------------------------+------------------------------------------+ -| Apr 3rd - Apr 14th | Add CI Gating for unit tests | -| | | -+--------------------------+------------------------------------------+ -| Apr 17th - Apr 28th | Use Tox Utility, Documentation | -| | | -+--------------------------+------------------------------------------+ -| Apr 28th - End | Bug Fixing | -| | | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst deleted file mode 100644 index df763339..00000000 --- a/docs/testing/developer/internship/vnf_catalog/index.rst +++ /dev/null @@ -1,170 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -======================= -Open Source VNF Catalog -======================= - -Author: Kumar Rishabh -Mentors: B.Souville, M.Richomme, J.Lausuch - -Abstract -======== - - - -Version hissory -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - -This project aims to create an Open Source catalog for reference and -classification of Virtual Network Functions (VNFs)s available on -Internet. The classification method proposed will be in sync with the -requirements of Telcos active in NFV landscape. The project aims to have -running web platform similar to [1] by the mid of internship (2nd week -of March). By the penultimate month of internship I aim to have fully -functional implementation of an Open Source VNF in functest. - - -Problem Statement: ------------------- - -OPNFV aims to be the reference platform for development, -standardization and integration of Open Source NFV components across -various Open Source Platforms. It mainly deals with the infrastructure -through the Network Function Virtualization Infrastructure (NFVI) and -Virtual Infrastructure manager (VIM). The MANO (Management and -orchestration) stacks have been introduced recently. VNFs are not -directly in OPNFV scope, however VNFs are needed to test and qualify the -infrastructure. In this regard having a common curated Open Source -Reference VNF catalog would be of immense importance to community. - -Since major focus of OPNFV is Telcos, a curated platform targeted from -industry point of view would be very useful. We plan to divide the -entire project into three major phases(with some iterative improvements -and overlaps) - - -Curation Phase --------------- -This phase pertains to studying various Open Source VNFs available and -classification of them based on certain parameters. The parameters that -I currently have in mind are: - * Developer Metrics: These pertain to repo characteristics of VNF under - study - * Usage Statistics - Activity, Number of Commits, stars - * Maturity Statistics - For instance if an NFV community decides code - coverage is important for them, it shows the NFV community is serious - about taking the project forward - * Technical Tagging: These are the tags that pertain to technical - characteristics of a VNF - * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or - SaaS layer or is an hybrid of two/all. - * Generic Use Cases - This in my opinion is the broadest - classification category. For instance a VNF could be built with a - broad idea of powering IOT devices at home or from usage perspective - of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ - * Fields of Application - * Library Status - Whether APIs are standardized, support RESTful - services. - * Dependency Forwarding Graph - This is pretty complex tagging - mechanism. It essentially tries to establish a graph relationship - between the VNFs (elementary VNFs are used in Service Function - Chaining chains such as Firewall, DPI, content enrichment,..). In my - opinion this is useful immensely. This will allow users to go to - platform and ask a question like - “I have this X tech stack to - support, Y and Z are my use cases, which NFVs should I use to support - this. - * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for - the platform. This will allow users to score an NFV on certain - parameters, may be post comments. - -**I plan to use the above three scores and evolve cumulative score which -will be displayed next to each of the NFV on the platform.** - - * Platform building phase - This will involve erecting a Web Platform - which will be similar to this `[1]`_. I am decently familiar with - Django and hence I will write the platform in Django. There are two - action plans that I have in mind right now. Either I can start writing - the platform simultaneously which will help keep track of my progress - or I can write the platform after 1.5 - 2 months into the internship. - Either way I aim to have the Web Platform ready by March 12. - - * Functest VNF implementation phase - This is the last phase that will - involve writing a fully functional implementation of an Open Source VNF - into Functest. I will undertake this after I am 3 months into the - internship. I have a decent familiarity with python and hence I think - it shouldn’t be too difficult. I need to decide how complex the VNFI - should undertake this exercise for (e.g. AAA such as free radius sounds - relatively easy, vCDN is much more challenging). - This will be decided in consent with my mentors. - - - - -Schedule: -========= -I plan to take this project in 6 months time frame as I want to use it -as a chance to read more about NFVs in particular and SDN in general - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December 12 - January 12 | Study the above mentioned metrics | -| | Decide which of them are important for | -| | community (and which are not). | -+--------------------------+------------------------------------------+ -| January 12 - January 27 | Make a database for the above studied | -| | metrics and evolve it further based on | -| | Mentors’ input. + associated API | -+--------------------------+------------------------------------------+ -| January 27 - February 5 | Compile the data collected above and make| -| | it public. Although I can keep everything| -| | public from the beginning too. My | -| | rationale of not making the entire data | -| | public in initial stage as the errors | -| | caused by me could be misleading for | -| | developers. | -+--------------------------+------------------------------------------+ -| February 5 - March 5 | Erect the Web Platform and release it | -| | for restricted group for alpha testing. | -+--------------------------+------------------------------------------+ -| March 5 - March 12 | Make it public. Release it to public for | -| | beta testing. Fix Bugs. | -+--------------------------+------------------------------------------+ -| March 12 - April 12 | Start working on implementation of an | -| | Open Source VNF in Functest. | -+--------------------------+------------------------------------------+ -| April 12 - May 12 | I will decided what to do here based on | -| | discussion with mentors. | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : Openhub: https://www.openhub.net/explore/projects - -.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf - -.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst deleted file mode 100644 index f3901d86..00000000 --- a/docs/testing/user/configguide/ci.rst +++ /dev/null @@ -1,13 +0,0 @@ -Integration in CI -================= -In CI we use the Docker images and execute the appropriate commands within the -container from Jenkins. - -4 steps have been defined:: - * functest-cleanup: clean existing functest dockers on the jumphost - * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf) - * functest-store-results: push logs to artifacts - -See `[3]`_ for details. - -.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst deleted file mode 100644 index f6581081..00000000 --- a/docs/testing/user/configguide/configguide.rst +++ /dev/null @@ -1,727 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. SPDX-License-Identifier: CC-BY-4.0 - -Installation and configuration -============================== - -Alpine containers have been introduced in Euphrates. -Alpine allows Functest testing in several very light containers and thanks to -the refactoring on dependency management should allow the creation of light and -fully customized docker images. - - -Functest Dockers ----------------- -Docker images are available on the dockerhub: - - * opnfv/functest-core - * opnfv/functest-healthcheck - * opnfv/functest-smoke - * opnfv/functest-features - * opnfv/functest-components - * opnfv/functest-vnf - * opnfv/functest-parser - * opnfv/functest-restapi - -The tag "opnfv-5.0.0" is the official release image in Euphrates, but you can also pull "euphrates" -tag as it is being maintained by Functest team and might include bugfixes. - -The Functest docker container environment can -in principle- be also -used with non-OPNFV official installers (e.g. 'devstack'), with the -**disclaimer** that support for such environments is outside of the -scope and responsibility of the OPNFV project. - - -Preparing your environment --------------------------- - -cat env:: - - INSTALLER_TYPE=XXX - EXTERNAL_NETWORK=XXX - DEPLOY_SCENARIO=XXX - -See section on environment variables for details. - -cat env_file:: - - export OS_AUTH_URL=XXX - export OS_USER_DOMAIN_NAME=XXX - export OS_PROJECT_DOMAIN_NAME=XXX - export OS_USERNAME=XXX - export OS_TENANT_NAME=XXX - export OS_PROJECT_NAME=XXX - export OS_PASSWORD=XXX - export OS_VOLUME_API_VERSION=XXX - export OS_IDENTITY_API_VERSION=XXX - export OS_IMAGE_API_VERSION=XXX - -See section on OpenStack credentials for details. - -Create a directory for the different images (attached as a Docker volume):: - - mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/euphrates | bash -s -- images && ls -1 images/* - - images/CentOS-7-aarch64-GenericCloud.qcow2 - images/CentOS-7-aarch64-GenericCloud.qcow2.xz - images/CentOS-7-x86_64-GenericCloud.qcow2 - images/cirros-0.4.0-x86_64-disk.img - images/cirros-0.4.0-x86_64-lxc.tar.gz - images/cirros-d161201-aarch64-disk.img - images/cirros-d161201-aarch64-initramfs - images/cirros-d161201-aarch64-kernel - images/cloudify-manager-premium-4.0.1.qcow2 - images/img - images/trusty-server-cloudimg-amd64-disk1.img - images/ubuntu-14.04-server-cloudimg-amd64-disk1.img - images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img - images/ubuntu-16.04-server-cloudimg-amd64-disk1.img - images/vyos-1.1.7.img - -Testing healthcheck suite --------------------------- - -Run healthcheck suite:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck - -Results shall be displayed as follows:: - - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:06 | PASS | - | api_check | functest | healthcheck | 06:52 | PASS | - | snaps_health_check | functest | healthcheck | 00:42 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - NOTE: the duration is a reference and it might vary depending on your SUT. - -Testing smoke suite -------------------- - -Run smoke suite:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-smoke - -Results shall be displayed as follows:: - - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 00:45 | PASS | - | vping_userdata | functest | smoke | 00:36 | PASS | - | tempest_smoke_serial | functest | smoke | 14:04 | PASS | - | rally_sanity | functest | smoke | 23:59 | PASS | - | refstack_defcore | functest | smoke | 14:37 | FAIL | - | snaps_smoke | functest | smoke | 42:09 | PASS | - | odl | functest | smoke | 00:00 | SKIP | - | odl_netvirt | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. - -Testing features suite ----------------------- - -Run features suite:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-features - -Results shall be displayed as follows:: - - +-----------------------------+------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-----------------------------+------------------------+------------------+------------------+----------------+ - | doctor-notification | doctor | features | 00:00 | SKIP | - | bgpvpn | sdnvpn | features | 00:00 | SKIP | - | functest-odl-sfc | sfc | features | 00:00 | SKIP | - | barometercollectd | barometer | features | 00:00 | SKIP | - | fds | fastdatastacks | features | 00:00 | SKIP | - +-----------------------------+------------------------+------------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. - -Testing components suite ------------------------- - -Run components suite:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-components - -Results shall be displayed as follows:: - - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 52:42 | PASS | - | rally_full | functest | components | 114:22 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ - -Testing vnf suite ------------------ - -Run vnf suite:: - -sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-vnf - -Results shall be displayed as follows:: - - +---------------------------------+-------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +---------------------------------+-------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 28:49 | FAIL | - | vyos_vrouter | functest | vnf | 27:57 | FAIL | - | juju_epc | functest | vnf | 55:03 | PASS | - | orchestra_openims | orchestra | vnf | 00:00 | SKIP | - | orchestra_clearwaterims | orchestra | vnf | 00:00 | SKIP | - +---------------------------------+-------------------+--------------+------------------+----------------+ - - -Environment variables -===================== - -Several environement variables may be specified: - * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid) - * INSTALLER_IP=<Specific IP Address> - * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode> - - -INSTALLER_IP is required by Barometer in order to access the installer node and -the deployment. - -The format for the DEPLOY_SCENARIO env variable can be described as follows: - * vim: (os|k8s) = OpenStack or Kubernetes - * controller is one of ( nosdn | odl ) - * nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature ) - * ha_mode (high availability) is one of ( ha | noha ) - -If several features are pertinent then use the underscore character '_' to -separate each feature (e.g. ovs_kvm). 'nofeature' indicates that no OPNFV -feature is deployed. - -The list of supported scenarios per release/installer is indicated in the -release note. - -**NOTE:** The scenario name is mainly used to automatically detect -if a test suite is runnable or not (e.g. it will prevent ODL test suite to be -run on 'nosdn' scenarios). If not set, Functest will try to run the default test -cases that might not include SDN controller or a specific feature - -**NOTE:** An HA scenario means that 3 OpenStack controller nodes are -deployed. It does not necessarily mean that the whole system is HA. See -installer release notes for details. - -Finally, three additional environment variables can also be passed in -to the Functest Docker Container, using the -e -"<EnvironmentVariable>=<Value>" mechanism. The first two parameters are -only relevant to Jenkins CI invoked testing and **should not be used** -when performing manual test scenarios: - - * NODE_NAME = <Test POD Name> - * BUILD_TAG = <Jenkins Build Tag> - -where: - - * <Test POD Name> = Symbolic name of the POD where the tests are run. - Visible in test results files, which are stored - to the database. This option is only used when - tests are activated under Jenkins CI control. - It indicates the POD/hardware where the test has - been run. If not specified, then the POD name is - defined as "Unknown" by default. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - * <Jenkins Build tag> = Symbolic name of the Jenkins Build Job. - Visible in test results files, which are stored - to the database. This option is only set when - tests are activated under Jenkins CI control. - It enables the correlation of test results, - which are independently pushed to the results database - from different Jenkins jobs. - DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - - -Openstack credentials -===================== -OpenStack credentials are mandatory and must be provided to Functest. -When running the command "functest env prepare", the framework will -automatically look for the Openstack credentials file -"/home/opnfv/functest/conf/env_file" and will exit with -error if it is not present or is empty. - -There are 2 ways to provide that file: - - * by using a Docker volume with -v option when creating the Docker container. - This is referred to in docker documentation as "Bind Mounting". - See the usage of this parameter in the following chapter. - * or creating manually the file '/home/opnfv/functest/conf/env_file' - inside the running container and pasting the credentials in it. Consult - your installer guide for further details. This is however not - instructed in this document. - -In proxified environment you may need to change the credentials file. -There are some tips in chapter: `Proxy support`_ - -SSL Support ------------ -If you need to connect to a server that is TLS-enabled (the auth URL -begins with "https") and it uses a certificate from a private CA or a -self-signed certificate, then you will need to specify the path to an -appropriate CA certificate to use, to validate the server certificate -with the environment variable OS_CACERT:: - - echo $OS_CACERT - /etc/ssl/certs/ca.crt - -However, this certificate does not exist in the container by default. -It has to be copied manually from the OpenStack deployment. This can be -done in 2 ways: - - #. Create manually that file and copy the contents from the OpenStack - controller. - #. (Recommended) Add the file using a Docker volume when starting the - container:: - - -v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert - -You might need to export OS_CACERT environment variable inside the -credentials file:: - - export OS_CACERT=/etc/ssl/certs/ca.crt - -Certificate verification can be turned off using OS_INSECURE=true. For -example, Fuel uses self-signed cacerts by default, so an pre step would -be:: - - export OS_INSECURE=true - -Functest docker container directory structure -============================================= -Inside the Functest docker container, the following directory structure -should now be in place:: - - `-- - |- home - | |-- opnfv - | | `- functest - | | |-- conf - | | `-- results - | `-- repos - | `-- vnfs - |- src - | |-- tempest - | |-- vims-test - | |-- odl_test - | `-- fds - `- usr - `- lib - `- python2.7 - `- site-packages - `- functest - |-- ... - -Underneath the '/home/opnfv/functest' directory, the Functest docker container -includes two main directories: - - * The **conf** directory stores configuration files (e.g. the - OpenStack creds are stored in path '/home/opnfv/functest/conf/env_file'), - * the **results** directory stores some temporary result log files - -src and repos directories are used to host third party code used for the tests. - -The structure of functest repo can be described as follows:: - - |-- INFO - |-- LICENSE - |-- api - | `-- apidoc - |-- build.sh - |-- commons - |-- docker - | |-- components - | |-- core - | |-- features - | |-- healthcheck - | |-- smoke - | |-- vnf - | |-- parser - | |-- restapi - | |-- thirdparty-requirements.txt - |-- docs - | |-- com - | |-- images - | |-- release - | | `-- release-notes - | |-- results - | | testing - | | |-- developer - | | `-- user - | | |-- configguide - | | `-- userguide - `-- functest - |-- api - | |-- base.py - | |-- server.py - | |-- urls.py - | |-- common - | | |-- api_utils.py - | | |-- thread.py - | `-- resources - | `-- v1 - | |-- creds.py - | |-- envs.py - | |-- testcases.py - | |-- tiers.py - | |-- tasks.py - | `-- database - | |-- db.py - | `-- v1 - | |-- handlers.py - | |-- models.py - | `-- swagger - |-- ci - │ |-- check_deployment.py - │ |-- config_aarch64_patch.yaml - │ |-- config_functest.yaml - │ |-- config_patch.yaml - │ |-- download_images.sh - │ |-- logging.ini - │ |-- rally_aarch64_patch.conf - │ |-- run_tests.py - │ |-- testcases.yaml - │ |-- tier_builder.py - │ |-- tier_handler.py - |-- cli - │ |-- cli_base.py - │ |-- commands - │ │ |-- cli_env.py - │ │ |-- cli_os.py - │ │ |-- cli_testcase.py - │ │ |-- cli_tier.py - │ |-- functest-complete.sh - |-- core - │ |-- feature.py - │ |-- robotframework.py - │ |-- testcase.py - │ |-- unit.py - │ |-- vnf.py - |-- energy - │ |-- energy.py - |-- opnfv_tests - │ `-- openstack - │ |-- rally - │ |-- refstack_client - │ |-- snaps - │ |-- tempest - │ |-- vping - │ `-- sdn - │ │ `-- odl - │ `-- vnf - │ |-- ims - │ `-- router - |-- tests - │ `-- unit - │ |-- ci - │ |-- cli - │ |-- core - │ |-- energy - │ |-- features - │ |-- odl - │ |-- openstack - │ |-- opnfv_tests - │ |-- test_utils.py - │ |-- utils - │ `-- vnf - |-- utils - | |-- config.py - | |-- constants.py - | |-- decorators.py - | |-- env.py - | |-- functest_utils.py - | |-- functest_vacation.py - | |-- openstack_clean.py - | |-- openstack_tacker.py - | `-- openstack_utils.py - |-- requirements.txt - |-- setup.cfg - |-- setup.py - |-- test-requirements.txt - |-- tox.ini - |-- upper-constraints.txt - - (Note: All *.pyc files removed from above list for brevity...) - -We may distinguish several directories, the first level has 5 directories: - -* **api**: This directory is dedicated to the API (framework) documentations. -* **commons**: This directory is dedicated for storage of traffic profile or - any other test inputs that could be reused by any test project. -* **docker**: This directory includes the needed files and tools to - build the Functest Docker images. -* **docs**: This directory includes documentation: Release Notes, - User Guide, Configuration Guide and Developer Guide. -* **functest**: This directory contains all the code needed to run - functest internal cases and OPNFV onboarded feature or VNF test cases. - -Functest directory has 7 sub-directories, which is located under -/usr/lib/python2.7/site-packages/functest: - * **api**: This directory is dedicated for the internal Functest API. - * **ci**: This directory contains test structure definition files - (e.g <filename>.yaml) and bash shell/python scripts used to - configure and execute Functional tests. The test execution script - can be executed under the control of Jenkins CI jobs. - * **cli**: This directory holds the python based Functest CLI utility - source code, which is based on the Python 'click' framework. - * **core**: This directory holds the python based Functest core - source code. Three abstraction classes have been created to ease - the integration of internal, feature or vnf cases. - * **opnfv_tests**: This directory includes the scripts required by - Functest internal test cases and other feature projects test cases. - * **tests**: This directory includes the functest unit tests. - * **utils**: this directory holds Python source code for some general - purpose helper utilities, which testers can also re-use in their - own test code. See for an example the Openstack helper utility: - 'openstack_utils.py'. - - -Logs -==== -By default all the logs are put un /home/opnfv/functest/results/functest.log. -If you want to have more logs in console, you may edit the logging.ini file -manually. -Connect on the docker then edit the file located in -/usr/lib/python2.7/site-packages/functest/ci/logging.ini - -Change wconsole to console in the desired module to get more traces. - - -Configuration -============= - -You may also directly modify the python code or the configuration file (e.g. -testcases.yaml used to declare test constraints) under -/usr/lib/python2.7/site-packages/functest - - -Tips -==== - -Docker ------- -When typing **exit** in the container prompt, this will cause exiting -the container and probably stopping it. When stopping a running Docker -container all the changes will be lost, there is a keyboard shortcut -to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To -reconnect to the running container **DO NOT** use the *run* command -again (since it will create a new container), use the *exec* or *attach* -command instead:: - - docker ps # <check the container ID from the output> - docker exec -ti <CONTAINER_ID> /bin/bash - -There are other useful Docker commands that might be needed to manage possible -issues with the containers. - -List the running containers:: - - docker ps - -List all the containers including the stopped ones:: - - docker ps -a - -Start a stopped container named "FunTest":: - - docker start FunTest - -Attach to a running container named "StrikeTwo":: - - docker attach StrikeTwo - -It is useful sometimes to remove a container if there are some problems:: - - docker rm <CONTAINER_ID> - -Use the *-f* option if the container is still running, it will force to -destroy it:: - - docker rm -f <CONTAINER_ID> - -Check the Docker documentation [`dockerdocs`_] for more information. - - -Checking Openstack and credentials ----------------------------------- -It is recommended and fairly straightforward to check that Openstack -and credentials are working as expected. - -Once the credentials are there inside the container, they should be -sourced before running any Openstack commands:: - - source /home/opnfv/functest/conf/env_file - -After this, try to run any OpenStack command to see if you get any -output, for instance:: - - openstack user list - -This will return a list of the actual users in the OpenStack -deployment. In any other case, check that the credentials are sourced:: - - env|grep OS_ - -This command must show a set of environment variables starting with -*OS_*, for example:: - - OS_REGION_NAME=RegionOne - OS_USER_DOMAIN_NAME=Default - OS_PROJECT_NAME=admin - OS_AUTH_VERSION=3 - OS_IDENTITY_API_VERSION=3 - OS_PASSWORD=da54c27ae0d10dfae5297e6f0d6be54ebdb9f58d0f9dfc - OS_AUTH_URL=http://10.1.0.9:5000/v3 - OS_USERNAME=admin - OS_TENANT_NAME=admin - OS_ENDPOINT_TYPE=internalURL - OS_INTERFACE=internalURL - OS_NO_CACHE=1 - OS_PROJECT_DOMAIN_NAME=Default - - -If the OpenStack command still does not show anything or complains -about connectivity issues, it could be due to an incorrect url given to -the OS_AUTH_URL environment variable. Check the deployment settings. - -Proxy support -------------- -If your Jumphost node is operating behind a http proxy, then there are -2 places where some special actions may be needed to make operations -succeed: - - #. Initial installation of docker engine First, try following the - official Docker documentation for Proxy settings. Some issues were - experienced on CentOS 7 based Jumphost. Some tips are documented - in section: `Docker Installation on CentOS behind http proxy`_ - below. - -If that is the case, make sure the resolv.conf and the needed -http_proxy and https_proxy environment variables, as well as the -'no_proxy' environment variable are set correctly:: - - # Make double sure that the 'no_proxy=...' line in the - # 'env_file' file is commented out first. Otherwise, the - # values set into the 'no_proxy' environment variable below will - # be ovewrwritten, each time the command - # 'source ~/functest/conf/env_file' is issued. - - cd ~/functest/conf/ - sed -i 's/export no_proxy/#export no_proxy/' env_file - source ./env_file - - # Next calculate some IP addresses for which http_proxy - # usage should be excluded: - - publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+") - - adminURL_IP=$(openstack catalog show identity | \ - grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+") - - export http_proxy="<your http proxy settings>" - export https_proxy="<your https proxy settings>" - export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP" - - # Ensure that "git" uses the http_proxy - # This may be needed if your firewall forbids SSL based git fetch - git config --global http.sslVerify True - git config --global http.proxy <Your http proxy settings> - -For example, try to use the **nc** command from inside the functest -docker container:: - - nc -v opnfv.org 80 - Connection to opnfv.org 80 port [tcp/http] succeeded! - - nc -v opnfv.org 443 - Connection to opnfv.org 443 port [tcp/https] succeeded! - -Note: In a Jumphost node based on the CentOS family OS, the **nc** -commands might not work. You can use the **curl** command instead. - - curl http://www.opnfv.org:80 - - <HTML><HEAD><meta http-equiv="content-type" - . - . - </BODY></HTML> - - curl https://www.opnfv.org:443 - - <HTML><HEAD><meta http-equiv="content-type" - . - . - </BODY></HTML> - - (Ignore the content. If command returns a valid HTML page, it proves - the connection.) - -Docker Installation on CentOS behind http proxy ------------------------------------------------ -This section is applicable for CentOS family OS on Jumphost which -itself is behind a proxy server. In that case, the instructions below -should be followed **before** installing the docker engine:: - - 1) # Make a directory '/etc/systemd/system/docker.service.d' - # if it does not exist - sudo mkdir /etc/systemd/system/docker.service.d - - 2) # Create a file called 'env.conf' in that directory with - # the following contents: - [Service] - EnvironmentFile=-/etc/sysconfig/docker - - 3) # Set up a file called 'docker' in directory '/etc/sysconfig' - # with the following contents: - HTTP_PROXY="<Your http proxy settings>" - HTTPS_PROXY="<Your https proxy settings>" - http_proxy="${HTTP_PROXY}" - https_proxy="${HTTPS_PROXY}" - - 4) # Reload the daemon - systemctl daemon-reload - - 5) # Sanity check - check the following docker settings: - systemctl show docker | grep -i env - - Expected result: - ---------------- - EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes) - DropInPaths=/etc/systemd/system/docker.service.d/env.conf - -Now follow the instructions in [`Install Docker on CentOS`_] to download -and install the **docker-engine**. The instructions conclude with a -"test pull" of a sample "Hello World" docker container. This should now -work with the above pre-requisite actions. - - -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`dockerdocs`: https://docs.docker.com/ -.. _`Proxy`: https://docs.docker.com/engine/admin/systemd/#http-proxy -.. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/ -.. _`Functest User Guide`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html -.. _`images/CentOS-7-x86_64-GenericCloud.qcow2`: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 -.. _`images/cirros-0.4.0-x86_64-disk.img`: http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -.. _`images/ubuntu-14.04-server-cloudimg-amd64-disk1.img`: https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst deleted file mode 100644 index fd997344..00000000 --- a/docs/testing/user/configguide/index.rst +++ /dev/null @@ -1,130 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -*************************** -Functest Installation Guide -*************************** - -.. toctree:: - :numbered: - :maxdepth: 2 - - -Introduction -============ -This document describes how to install and configure Functest in OPNFV. - -High level architecture ------------------------ - -The high level architecture of Functest within OPNFV can be described as -follows:: - - CIMC/Lights+out management Admin Mgmt/API Public Storage Private - PXE - + + + + + + - | | | | | | - | +----------------------------+ | | | | | - | | | | | | | | - +-----+ Jumphost | | | | | | - | | | | | | | | - | | +--------------------+ | | | | | | - | | | | | | | | | | - | | | Tools | +----------------+ | | | - | | | - Rally | | | | | | | - | | | - Robot | | | | | | | - | | | - RefStack | | | | | | | - | | | | |-------------------------+ | | - | | | Testcases | | | | | | | - | | | - VIM | | | | | | | - | | | | | | | | | | - | | | - SDN Controller | | | | | | | - | | | | | | | | | | - | | | - Features | | | | | | | - | | | | | | | | | | - | | | - VNF | | | | | | | - | | | | | | | | | | - | | +--------------------+ | | | | | | - | | Functest Docker + | | | | | - | | | | | | | | - | | | | | | | | - | | | | | | | | - | +----------------------------+ | | | | | - | | | | | | - | +----------------+ | | | | | - | | 1 | | | | | | - +----+ +--------------+-+ | | | | | - | | | 2 | | | | | | - | | | +--------------+-+ | | | | | - | | | | 3 | | | | | | - | | | | +--------------+-+ | | | | | - | | | | | 4 | | | | | | - | +-+ | | +--------------+-+ | | | | | - | | | | | 5 +-------------+ | | | | - | +-+ | | nodes for | | | | | | - | | | | deploying +---------------------+ | | | - | +-+ | OPNFV | | | | | | - | | | +------------------------------+ | | - | +-+ SUT | | | | | | - | | +--------------------------------------+ | - | | | | | | | | - | | +----------------------------------------------+ - | +----------------+ | | | | | - | | | | | | - + + + + + + - SUT = System Under Test - -Note connectivity to management network is not needed for most of the testcases. -But it may be needed for some specific snaps tests. - -All the libraries and dependencies needed by all of the Functest tools are -pre-installed into the Docker images. This allows running Functest on any -platform. - -The automated mechanisms inside the Functest Docker containers will: - - * Prepare the environment according to the System Under Test (SUT) - * Perform the appropriate functional tests - * Push the test results into the OPNFV test result database (optional) - -The OpenStack credentials file must be provided to the container. - -These Docker images can be integrated into CI or deployed independently. - -Please note that the Functest Docker images have been designed for OPNFV, -however, it would be possible to adapt them to any OpenStack based VIM + -controller environment, since most of the test cases are integrated from -upstream communities. - -The functional test cases are described in the Functest User Guide `[2]`_ - -.. include:: ./prerequisites.rst - -.. include:: ./configguide.rst - -.. include:: ./ci.rst - - -References -========== - -`[1]`_ : Keystone and public end point constraint - -`[2]`_ : Functest User guide - -`[3]`_ : Functest Jenkins jobs - -`[4]`_ : Functest Configuration guide - -`[5]`_ : OPNFV main site - -`[6]`_ : Functest wiki page - -IRC support channel: #opnfv-functest - -.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ -.. _`[2]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[5]`: http://www.opnfv.org -.. _`[6]`: https://wiki.opnfv.org/functest diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst deleted file mode 100644 index 8289803b..00000000 --- a/docs/testing/user/configguide/prerequisites.rst +++ /dev/null @@ -1,102 +0,0 @@ -Prerequisites -============= -The OPNFV deployment is out of the scope of this document but it can be -found in http://docs.opnfv.org. -The OPNFV platform is considered as the SUT in this document. - -Several prerequisites are needed for Functest: - - #. A Jumphost to run Functest on - #. A Docker daemon shall be installed on the Jumphost - #. A public/external network created on the SUT - #. An admin/management network created on the SUT - #. Connectivity from the Jumphost to the SUT public/external network - -Some specific SNAPS tests may require a connectivity from the Jumphost to the -SUT admin/management network but most of the test cases do not. This requirement -can be changed by overriding the 'interface' attribute (OS_INTERFACE) value -to 'public' in the credentials file. Another means to circumvent this issue -would be to change the 'snaps.use_keystone' value from True to False. - -WARNING: Connectivity from Jumphost is essential and it is of paramount -importance to make sure it is working before even considering to install -and run Functest. Make also sure you understand how your networking is -designed to work. - -NOTE: **Jumphost** refers to any server which meets the previous -requirements. Normally it is the same server from where the OPNFV -deployment has been triggered previously, but it could be any server -with proper connectivity to the SUT. - -NOTE: If your Jumphost is operating behind a company http proxy and/or -firewall, please consult first the section `Proxy Support`_, towards -the end of this document. The section details some tips/tricks which -*may* be of help in a proxified environment. - -Docker installation -------------------- -Docker installation and configuration is only needed to be done once -through the life cycle of Jumphost. - -If your Jumphost is based on Ubuntu, SUSE, RHEL or CentOS linux, please -consult the references below for more detailed instructions. The -commands below are offered as a short reference. - -*Tip:* For running docker containers behind the proxy, you need first -some extra configuration which is described in section -`Docker Installation on CentOS behind http proxy`_. You should follow -that section before installing the docker engine. - -Docker installation needs to be done as root user. You may use other -userid's to create and run the actual containers later if so desired. -Log on to your Jumphost as root user and install the Docker Engine -(e.g. for CentOS family):: - - curl -sSL https://get.docker.com/ | sh - systemctl start docker - - *Tip:* If you are working through proxy, please set the https_proxy - environment variable first before executing the curl command. - -Add your user to docker group to be able to run commands without sudo:: - - sudo usermod -aG docker <your_user> - -A reconnection is needed. There are 2 ways for this: - #. Re-login to your account - #. su - <username> - -References - Installing Docker Engine on different Linux Operating Systems: - * Ubuntu_ - * RHEL_ - * CentOS_ - * SUSE_ - -.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/ -.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/ -.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/ -.. _SUSE: https://docs.docker.com/engine/installation/linux/suse/ - -Public/External network on SUT ------------------------------- -Some of the tests against the VIM (Virtual Infrastructure Manager) need -connectivity through an existing public/external network in order to -succeed. This is needed, for example, to create floating IPs to access -VM instances through the public/external network (i.e. from the Docker -container). - -By default, the five OPNFV installers provide a fresh installation with -a public/external network created along with a router. Make sure that -the public/external subnet is reachable from the Jumphost and an external -router exists. - -*Hint:* For the given OPNFV Installer in use, the IP sub-net address -used for the public/external network is usually a planning item and -should thus be known. Ensure you can reach each node in the SUT, from the -Jumphost using the 'ping' command using the respective IP address on the -public/external network for each node in the SUT. The details of how to -determine the needed IP addresses for each node in the SUT may vary according -to the used installer and are therefore ommitted here. - -.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst deleted file mode 100644 index 66dfd3e7..00000000 --- a/docs/testing/user/userguide/index.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _functest-userguide: - -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. SPDX-License-Identifier: CC-BY-4.0 - -******************* -Functest User Guide -******************* - -.. toctree:: - :maxdepth: 2 - - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. - -**IMPORTANT**: It is assumed here that Functest has been properly deployed -following the installation guide procedure `[1]`_. - -.. include:: ./test_overview.rst - -.. include:: ./test_details.rst - -.. include:: ./runfunctest.rst - -.. include:: ./test_results.rst - -.. include:: ./reporting.rst - -.. figure:: ../../../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - -.. include:: ./troubleshooting.rst - - -References -========== - -`[1]`_: Functest configuration guide - -`[2]`_: OpenStack Tempest documentation - -`[3]`_: Rally documentation - -`[4]`_: Functest in depth (Danube) - -`[5]`_: Clearwater vIMS blueprint - -`[6]`_: NIST web site - -`[7]`_: OpenSCAP web site - -`[8]`_: Refstack client - -`[9]`_: Defcore - -`[10]`_: OpenStack interoperability procedure - -`[11]`_: Robot Framework web site - -`[12]`_: Functest User guide - -`[13]`_: SNAPS wiki - -`[14]`_: vRouter - -`[15]`_: Testing OpenStack Tempest part 1 - -`[16]`_: Running Functest through internal REST API - -`[17]`_: OPNFV Test API - -`OPNFV main site`_: OPNFV official web site - -`Functest page`_: Functest wiki page - -IRC support chan: #opnfv-functest - -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf -.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[6]`: https://scap.nist.gov/ -.. _`[7]`: https://github.com/OpenSCAP/openscap -.. _`[8]`: https://github.com/openstack/refstack-client -.. _`[9]`: https://github.com/openstack/defcore -.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO -.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -.. _`OPNFV main site`: http://www.opnfv.org -.. _`Functest page`: https://wiki.opnfv.org/functest -.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html -.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst deleted file mode 100644 index 88f5e865..00000000 --- a/docs/testing/user/userguide/reporting.rst +++ /dev/null @@ -1,90 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Test reporting -============== - -An automatic reporting page has been created in order to provide a consistent -view of the Functest tests on the different scenarios. - -In this page, each scenario is evaluated according to test criteria. - -The results are collected from the centralized database every day and, per -scenario. A score is calculated based on the results from the last 10 days. -This score is the addition of single test scores. Each test case has a success -criteria reflected in the criteria field from the results. - -As an illustration, let's consider the scenario -os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the -scores of all the runnable tests from the categories (tiers, healthcheck, smoke -and features) corresponding to this scenario. - - +---------------------+---------+---------+---------+---------+ - | Test | Apex | Compass | Fuel | Joid | - +=====================+=========+=========+=========+=========+ - | vPing_ssh | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | vPing_userdata | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | rally_sanity | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | odl | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | promise | | | X | X | - +---------------------+---------+---------+---------+---------+ - | doctor | X | | X | | - +---------------------+---------+---------+---------+---------+ - | security_scan | X | | | | - +---------------------+---------+---------+---------+---------+ - | parser | | | X | | - +---------------------+---------+---------+---------+---------+ - | copper | X | | | X | - +---------------------+---------+---------+---------+---------+ - src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version) - -All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature -scenarios. -Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. -There are not considered as runnable on the generic odl_l2 scenario. - - -If no result is available or if all the results are failed, the test case get 0 -point. -If it was successful at least once but not anymore during the 4 runs, the case -get 1 point (it worked once). -If at least 3 of the last 4 runs were successful, the case get 2 points. -If the last 4 runs of the test are successful, the test get 3 points. - -In the example above, the target score for fuel/os-odl_l2-nofeature-ha is -3 x 8 = 24 points and for compass it is 3 x 5 = 15 points . - -The scenario is validated per installer when we got 3 points for all individual -test cases (e.g 24/24 for fuel, 15/15 for compass). - -Please note that complex or long duration tests are not considered yet for the -scoring. In fact the success criteria are not always easy to define and may -require specific hardware configuration. - -Please also note that all the test cases have the same "weight" for the score -calculation whatever the complexity of the test case. Concretely a vping has the -same weith than the 200 tempst tests. -Moreover some installers support more features than others. The more cases your -scenario is dealing with, the most difficult to rich a good scoring. - -Therefore the scoring provides 3 types of indicators: - - * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features - * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS - * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) - -In any case, the scoring is used to give feedback to the other projects and -does not represent an absolute value of the scenario. - -See `reporting page`_ for details. For the status, click on the version, -Functest then the Status menu. - - -.. _`reporting page`: http://testresults.opnfv.org/reporting/ diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst deleted file mode 100644 index d5b95101..00000000 --- a/docs/testing/user/userguide/runfunctest.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - - -Executing Functest suites -========================= - -As mentioned in the configuration guide `[1]`_, Alpine docker containers have -been introduced in Euphrates. -Tier containers have been created. -Assuming that you pulled the container and your environement is ready, you can -simply run the tiers by typing (e.g. with functest-healthcheck):: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck - -You should get:: - - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:02 | PASS | - | api_check | functest | healthcheck | 03:19 | PASS | - | snaps_health_check | functest | healthcheck | 00:46 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - -You can run functest-healcheck, functest-smoke, functest-features, -functest-components and functest-vnf. - -The result tables show the results by test case, it can be:: - - * PASS - * FAIL - * SKIP: if the scenario/installer does not support the test case - - -Manual run -========== -If you want to run the test step by step, you may add docker option then run the -different commands within the docker. - -Considering the healthcheck example, running functest manaully means:: - - sudo docker run -ti --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck /bin/bash - -The docker prompt shall be returned. Then within the docker run the following -commands:: - - $ source /home/opnfv/functest/conf/env_file - -Tier ----- -Each Alpine container provided on the docker hub matches with a tier. -The following commands are available:: - - # functest tier list - - 0. healthcheck: - ['connection_check', 'api_check', 'snaps_health_check'] - # functest tier show healthcheck - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | - | | | | basic operations in the VIM. | snaps_health_check | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - -To run all the cases of the tier, type:: - - # functest tier run healthcheck - -Testcase --------- -Testcases can be listed, shown and run though the CLI:: - - # functest testcase list - connection_check - api_check - snaps_health_check - # functest testcase show api_check - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ | - | | OpenStack clients: Keystone, Glance, | | | - | | Neutron and Nova and may perform some | | | - | | simple queries. When the config value of | | | - | | snaps.use_keystone is True, functest | | | - | | must have access to the cloud's private | | | - | | network. | | | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - # functest testcase run connection_check - ... - # functest run all - -You can also type run_tests -t all to run all the tests. - -Note the list of test cases depend on the installer and the scenario. - - -Reporting results to the test Database -====================================== -In OPNFV CI we collect all the results from CI. A test APi shall be available -as well as a test database `[17]`_. - -Functest internal API -===================== - -An internal API has been introduced in Euphrates. The goal is to trigger -Functest operations through an API in addition of the CLI. -This could be considered as a first step towards a pseudo micro services -approach where the different test projects could expose and consume APIs to the -other test projects. - -In Euphrates the main method of the APIs are: - - * Show credentials - * Update openrc file - * Show environment - * Update hosts info for domain name - * Prepare environment - * List all testcases - * Show a testcase - * Run a testcase - * List all tiers - * Show a tier - * List all testcases within given tier - * Get the result of the specified task - * Get the log of the specified task - -See `[16]`_ to get examples on how to use the API. - - -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst deleted file mode 100644 index 97c4688c..00000000 --- a/docs/testing/user/userguide/test_details.rst +++ /dev/null @@ -1,497 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - - -The different test cases are described in the remaining sections of this document. - -VIM (Virtualized Infrastructure Manager) ----------------------------------------- - -Healthcheck tests -^^^^^^^^^^^^^^^^^ -Since Danube, healthcheck tests have been refactored and rely on SNAPS, an -OPNFV middleware project. - -SNAPS stands for "SDN/NFV Application development Platform and Stack". -SNAPS is an object-oriented OpenStack library packaged with tests that exercise -OpenStack. -More information on SNAPS can be found in  `[13]`_ - -Three tests are declared as healthcheck tests and can be used for gating by the -installer, they cover functionally the tests previously done by healthcheck -test case. - -The tests are: - - - * *connection_check* - * *api_check* - * *snaps_health_check* - -Connection_check consists in 9 test cases (test duration < 5s) checking the -connectivity with Glance, Keystone, Neutron, Nova and the external network. - -Api_check verifies the retrieval of OpenStack clients: Keystone, Glance, -Neutron and Nova and may perform some simple queries. When the config value of -snaps.use_keystone is True, functest must have access to the cloud's private -network. This suite consists in 49 tests (test duration < 2 minutes). - -Snaps_health_check creates a VM with a single port with an IPv4 address that -is assigned by DHCP and then validates the expected IP with the actual. - -Self-obviously, successful completion of the 'healthcheck' testcase is a -necessary pre-requisite for the execution of all other test Tiers. - - -vPing_ssh -^^^^^^^^^ - -Given the script **ping.sh**:: - - #!/bin/sh - ping -c 1 $1 2>&1 >/dev/null - RES=$? - if [ "Z$RES" = "Z0" ] ; then - echo 'vPing OK' - else - echo 'vPing KO' - fi - - -The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: - - vPing_ssh test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +------------------->| | - | Tester | | System | - | | Boot VM2 | Under | - | +------------------->| Test | - | | | | - | | Create floating IP | | - | +------------------->| | - | | | | - | | Assign floating IP | | - | | to VM2 | | - | +------------------->| | - | | | | - | | Establish SSH | | - | | connection to VM2 | | - | | through floating IP| | - | +------------------->| | - | | | | - | | SCP ping.sh to VM2 | | - | +------------------->| | - | | | | - | | VM2 executes | | - | | ping.sh to VM1 | | - | +------------------->| | - | | | | - | | If ping: | | - | | exit OK | | - | | else (timeout): | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -This test can be considered as an "Hello World" example. -It is the first basic use case which **must** work on any deployment. - -vPing_userdata -^^^^^^^^^^^^^^ - -This test case is similar to vPing_ssh but without the use of Floating IPs -and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. -As vPing_ssh, it checks that 2 instances can talk to -each other on a Private Tenant network:: - - vPing_userdata test case - +-------------+ +-------------+ - | | | | - | | Boot VM1 with IP1 | | - | +-------------------->| | - | | | | - | | Boot VM2 with | | - | | ping.sh as userdata | | - | | with IP1 as $1. | | - | +-------------------->| | - | Tester | | System | - | | VM2 executes ping.sh| Under | - | | (ping IP1) | Test | - | +-------------------->| | - | | | | - | | Monitor nova | | - | | console-log VM 2 | | - | | If ping: | | - | | exit OK | | - | | else (timeout) | | - | | exit Failed | | - | | | | - +-------------+ +-------------+ - -When the second VM boots it will execute the script passed as userdata -automatically. The ping will be detected by periodically capturing the output -in the console-log of the second VM. - - -Tempest -^^^^^^^ - -Tempest `[2]`_ is the reference OpenStack Integration test suite. -It is a set of integration tests to be run against a live OpenStack cluster. -Tempest has suites of tests for: - - * OpenStack API validation - * Scenarios - * Other specific tests useful in validating an OpenStack deployment - -Functest uses Rally `[3]`_ to run the Tempest suite. -Rally generates automatically the Tempest configuration file **tempest.conf**. -Before running the actual test cases, -Functest creates the needed resources (user, tenant) and -updates the appropriate parameters into the configuration file. - -When the Tempest suite is executed, each test duration is measured and the full -console output is stored to a *log* file for further analysis. - -The Tempest testcases are distributed across two -Tiers: - - * Smoke Tier - Test Case 'tempest_smoke_serial' - * Components Tier - Test case 'tempest_full_parallel' - -NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. - -The goal of the Tempest test suite is to check the basic functionalities of the -different OpenStack components on an OPNFV fresh installation, using the -corresponding REST API interfaces. - - -Rally bench test suites -^^^^^^^^^^^^^^^^^^^^^^^ - -Rally `[3]`_ is a benchmarking tool that answers the question: - -*How does OpenStack work at scale?* - -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. - -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: - - * authenticate - * cinder - * glance - * heat - * keystone - * neutron - * nova - * quotas - * ceilometer - -A basic SLA (stop test on errors) has been implemented. - -The Rally testcases are distributed across two Tiers: - - * Smoke Tier - Test Case 'rally_sanity' - * Components Tier - Test case 'rally_full' - -NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test -cases. Test case 'rally_full' executes the full defined set of Rally tests. - - -Refstack-client to run OpenStack interop testcases --------------------------------------------------- - -Refstack-client `[8]`_ is a command line utility that allows you to -execute Tempest test runs based on configurations you specify. -It is the official tool to run Openstack Interop (previously known as Defcore) -testcases `[9]`_, which focus on testing interoperability between OpenStack -clouds. - -Refstack-client is integrated in Functest, consumed by Dovetail, which -intends to define and provide a set of OPNFV related validation criteria -that will provide input for the evaluation of the use of OPNFV trademarks. -This progress is under the guideline of Compliance Verification Program(CVP). - -Running methods -^^^^^^^^^^^^^^^ - -Two running methods are provided after refstack-client integrated into -Functest, Functest command line and manually, respectively. - -By default, for Defcore test cases run by Functest command line, -are run followed with automatically generated -configuration file, i.e., refstack_tempest.conf. In some circumstances, -the automatic configuration file may not quite satisfied with the SUT, -Functest also inherits the refstack-client command line and provides a way -for users to set its configuration file according to its own SUT manually. - -*command line* - -Inside the Functest container, first to prepare Functest environment: - -:: - - functest env prepare - -then to run default defcore testcases by using refstack-client: - -:: - - functest testcase run refstack_defcore - -In OPNFV Continuous Integration(CI) system, the command line method is used. - -*manually* - -Prepare the tempest configuration file and the testcases want to run with the SUT, -run the testcases with: - -:: - - ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> - -using help for more information: - -:: - - ./refstack-client --help - ./refstack-client test --help - -Reference tempest configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*command line method* - -When command line method is used, the default tempest configuration file -is generated by Rally. - -*manually* - -When running manually is used, recommended way to generate tempest configuration -file is: - -:: - - cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client - python tempest_conf.py - -a file called tempest.conf is stored in the current path by default, users can do -some adjustment according to the SUT: - -:: - - vim refstack_tempest.conf - -a reference article can be used `[15]`_. - - -snaps_smoke ------------- - -This test case contains tests that setup and destroy environments with VMs with -and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. -Please note that When the configuration value of snaps.use_keystone is True, Functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) - - -SDN Controllers ---------------- - -OpenDaylight -^^^^^^^^^^^^ - -The OpenDaylight (ODL) test suite consists of a set of basic tests inherited -from the ODL project using the Robot `[11]`_ framework. -The suite verifies creation and deletion of networks, subnets and ports with -OpenDaylight and Neutron. - -The list of tests can be described as follows: - - * Basic Restconf test cases - * Connect to Restconf URL - * Check the HTTP code status - - * Neutron Reachability test cases - * Get the complete list of neutron resources (networks, subnets, ports) - - * Neutron Network test cases - * Check OpenStack networks - * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron - * Check that the network has also been successfully created in OpenDaylight - - * Neutron Subnet test cases - * Check OpenStack subnets - * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron - * Check that the subnet has also been successfully created in OpenDaylight - - * Neutron Port test cases - * Check OpenStack Neutron for known ports - * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron - * Check that the new port has also been successfully created in OpenDaylight - - * Delete operations - * Delete the port previously created via OpenStack - * Check that the port has been also successfully deleted in OpenDaylight - * Delete previously subnet created via OpenStack - * Check that the subnet has also been successfully deleted in OpenDaylight - * Delete the network created via OpenStack - * Check that the network has also been successfully deleted in OpenDaylight - -Note: the checks in OpenDaylight are based on the returned HTTP status -code returned by OpenDaylight. - - -Features --------- - -Functest has been supporting several feature projects since Brahpamutra: - - -+-----------------+---------+----------+--------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | -+=================+=========+==========+========+===========+ -| barometer | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| bgpvpn | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| copper | | X | | | -+-----------------+---------+----------+--------+-----------+ -| doctor | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| domino | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| fds | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| moon | | X | | | -+-----------------+---------+----------+--------+-----------+ -| multisite | | X | X | | -+-----------------+---------+----------+--------+-----------+ -| netready | | | X | | -+-----------------+---------+----------+--------+-----------+ -| odl_sfc | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| opera | | | X | | -+-----------------+---------+----------+--------+-----------+ -| orchestra | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| parser | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| promise | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| security_scan | | X | X | | -+-----------------+---------+----------+--------+-----------+ - -Please refer to the dedicated feature user guides for details. - - -VNF ---- - - -cloudify_ims -^^^^^^^^^^^^ -The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an -architectural framework for delivering IP multimedia services. - -vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. - -The goal of this test suite consists of: - - * deploy a VNF orchestrator (Cloudify) - * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this - orchestrator based on a TOSCA blueprint defined in `[5]`_ - * run suite of signaling tests on top of this VNF - -The Clearwater architecture is described as follows: - -.. figure:: ../../../images/clearwater-architecture.png - :align: center - :alt: vIMS architecture - - -cloudify_ims_perf -^^^^^^^^^^^^^^^^^ -This testcase extends the cloudify_ims test case. -The first part is similar but the testing part is different. -The testing part consists in automating a realistic signaling load on the vIMS -using an Ixia loader (proprietary tools) - - You need to have access to an Ixia licence server defined in the configuration - file and have ixia image locally. - -This test case is available but not declared in testcases.yaml. The declaration -of the testcase is simple, connect to your functest-vnf docker, add the following -section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: - - - - case_name: cloudify_ims_perf - project_name: functest - criteria: 80 - blocking: false - description: >- - Stress tests based on Cloudify. Ixia loader images and access to Ixia - server license. - dependencies: - installer: '' - scenario: 'os-nosdn-nofeature-ha' - run: - module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' - class: 'CloudifyImsPerf' - -orchestra_openims -^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of OpenIMS with OpenBaton -orchestrator. - -orchestra_clearwaterims -^^^^^^^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton -orchestrator. - -vyos-vrouter -^^^^^^^^^^^^ -This test case deals with the deployment and the test of vyos vrouter with -Cloudify orchestrator. The test case can do testing for interchangeability of -BGP Protocol using vyos. - -The Workflow is as follows: - * Deploy - Deploy VNF Testing topology by Cloudify using blueprint. - * Configuration - Setting configuration to Target VNF and reference VNF using ssh - * Run - Execution of test command for test item written YAML format file. - Check VNF status and behavior. - * Reporting - Output of report based on result using JSON format. - -The vyos-vrouter architecture is described in `[14]`_ - - -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[8]`: https://github.com/openstack/refstack-client -.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO -.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst deleted file mode 100644 index a22a5067..00000000 --- a/docs/testing/user/userguide/test_overview.rst +++ /dev/null @@ -1,239 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Overview of the Functest suites -=============================== - -Functest is the OPNFV project primarily targeting functional testing. -In the Continuous Integration pipeline, it is launched after an OPNFV fresh -installation to validate and verify the basic functions of the -infrastructure. - -The current list of test suites can be distributed over 4 main domains: - * VIM (Virtualised Infrastructure Manager) - * Controllers (i.e. SDN Controllers) - * Features - * VNF (Virtual Network Functions) - -Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. - -All the Healthcheck and smoke tests of a given scenario must be succesful to -validate the scenario for the release. - -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| | +----------------+----------------------------------+ -| | | snaps_health | basic instance creation, check | -| | | \_check | DHCP | -| +---------------+----------------+----------------------------------+ -| | smoke | vping_ssh | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network | -| | +----------------+----------------------------------+ -| | | vping_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run the SNAPS-OO integration | -| | | | tests | -| | +----------------+----------------------------------+ -| | | refstack | Reference RefStack suite | -| | | \_defcore | tempest selection for NFV | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment | -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_ | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | odl_netvirt | Test Suite for the OpenDaylight | -| | | | SDN Controller when the NetVirt | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. it | -| | | | deals with a fake vRNC. | -| | | | See `Parser User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | fds | Test Suite for the OpenDaylight | -| | | | SDN Controller when the GBP | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | orchestra | OpenIMS deployment using | -| | | \_openims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | orchestra | Cleawater IMS deployment using | -| | | \_cleawaterims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | vyos_vrouter | vRouter testing | -| | +----------------+----------------------------------+ -| | | cloudify_ims | Based on cloudify_ims test case | -| | | perf | cloudify_ims_perf substitutes | -| | | | the signaling test suite by an | -| | | | automatic deployment of an Ixia | -| | | | loader and generic SIP stress | -| | | | tests. | -| | | | This work has been initiated | -| | | | during the plugfest and allows | -| | | | realistic load tests on top of | -| | | | cloudify_ims. Please note that | -| | | | this test is available but not | -| | | | declared in testcases.yaml as it | -| | | | requires access to proprietary | -| | | | resources (Ixia loader) | -+-------------+---------------+----------------+----------------------------------+ - - -As shown in the above table, Functest is structured into different 'domains', -'tiers' and 'test cases'. Each 'test case' usually represents an actual -'Test Suite' comprised -in turn- of several test cases internally. - -Test cases also have an implicit execution order. For example, if the early -'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. - -In Danube, we merged smoke and sdn controller tiers in smoke tier. - -An overview of the Functest Structural Concept is depicted graphically below: - -.. figure:: ../../../images/concepts_mapping_final.png - :align: center - :alt: Functest Concepts Structure - -Some of the test cases are developed by Functest team members, whereas others -are integrated from upstream communities or other OPNFV projects. For example, -`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the -OpenStack integration test suite and Functest is in charge of the selection, -integration and automation of those tests that fit suitably to OPNFV. - -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. - -The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: - -.. figure:: ../../../images/FunctestDashboardEuphrates.png - :align: center - :alt: Functest Dashboard - - -Basic components (VIM, SDN controllers) are tested through their own suites. -Feature projects also provide their own test suites with different ways of -running their tests. - -The notion of domain has been introduced in the description of the test cases -stored in the Database. -This parameters as well as possible tags can be used for the Test case catalog. - -vIMS test case was integrated to demonstrate the capability to deploy a -relatively complex NFV scenario on top of the OPNFV infrastructure. - -Functest considers OPNFV as a black box. OPNFV offers a lot of potential -combinations (which may change from one version to another): - - * 3 controllers (OpenDaylight, ONOS, OpenContrail) - * 5 installers (Apex, Compass, Daisy, Fuel, Joid) - -Most of the tests are runnable by any combination, but some tests might have -restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_TYPE and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. - -A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource -clean-up) and for executing tests. -The Functest CLI organised the testcase into logical Tiers, which contain in -turn one or more testcases. The CLI allows execution of a single specified -testcase, all test cases in a specified Tier, or the special case of execution -of **ALL** testcases. The Functest CLI is introduced in more details in next -section. - -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst deleted file mode 100644 index 3941ba0a..00000000 --- a/docs/testing/user/userguide/test_results.rst +++ /dev/null @@ -1,53 +0,0 @@ -Test results -============ - -Manual testing --------------- - -In manual mode test results are displayed in the console and result files -are put in /home/opnfv/functest/results. - -If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/functest/ci. - -Automated testing --------------- - -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ - -Results are automatically pushed to the test results database, some additional -result files are pushed to OPNFV artifact web sites. - -Based on the results stored in the result database, a `Functest reporting`_ -portal is also automatically updated. This portal provides information on the -overall status per scenario and per installer - -.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst deleted file mode 100644 index 1649ac14..00000000 --- a/docs/testing/user/userguide/troubleshooting.rst +++ /dev/null @@ -1,378 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - -Troubleshooting -=============== - -This section gives some guidelines about how to troubleshoot the test cases -owned by Functest. - -**IMPORTANT**: As in the previous section, the steps defined below must be -executed inside the Functest Docker container and after sourcing the OpenStack -credentials:: - - . $creds - -or:: - - source /home/opnfv/functest/conf/env_file - -VIM ---- - -This section covers the test cases related to the VIM (healthcheck, vping_ssh, -vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, -rally_full). - -vPing common -^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: - - * Create Glance image - * Create Network - * Create Security Group - * Create Instances - -After these actions, the test cases differ and will be explained in their -respective section. - -These test cases can be run inside the container, using new Functest CLI as follows:: - - $ functest testcase run vping_ssh - $ functest testcase run vping_userdata - -The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py -and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py - -Notes: - - #. There is one difference, between the Functest CLI based test case - execution compared to the earlier used Bash shell script, which is - relevant to point out in troubleshooting scenarios: - - The Functest CLI does **not yet** support the option to suppress - clean-up of the generated OpenStack resources, following the execution - of a test case. - - Explanation: After finishing the test execution, the corresponding - script will remove, by default, all created resources in OpenStack - (image, instances, network and security group). When troubleshooting, - it is advisable sometimes to keep those resources in case the test - fails and a manual testing is needed. - - It is actually still possible to invoke test execution, with suppression - of OpenStack resource cleanup, however this requires invocation of a - **specific Python script:** 'run_tests'. - The `OPNFV Functest Developer Guide`_ provides guidance on the use of that - Python script in such troubleshooting cases. - -Some of the common errors that can appear in this test case are:: - - vPing_ssh- ERROR - There has been a problem when creating the neutron network.... - -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: - - neutron net-create net-test - neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ - --gateway 10.6.0.254 net-test 10.6.0.0/24 - neutron router-create test_router - neutron router-interface-add <ROUTER_ID> test_subnet - neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> - -Another related error can occur while creating the Security Groups for the instances:: - - vPing_ssh- ERROR - Failed to create the security group... - -In this case, proceed to create it manually. These are some hints:: - - neutron security-group-create sg-test - neutron security-group-rule-create sg-test --direction ingress --protocol icmp \ - --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \ - --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 - -The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: - - nova list - nova show <INSTANCE_ID> - -It might show some messages about the booting failure. To try that manually:: - - nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test - -This will spawn a VM using the network created previously manually. -In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. - - -vPing_SSH -^^^^^^^^^ -This test case creates a floating IP on the external network and assigns it to -the second instance **opnfv-vping-2**. The purpose of this is to establish -a SSH connection to that instance and SCP a script that will ping the first -instance. This script is located in the repository under -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh -and takes an IP as a parameter. When the SCP is completed, the test will do a -SSH call to that script inside the second instance. Some problems can happen -here:: - - vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting - -If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not -reach the Public/External IP assigned to the instance **opnfv-vping-2**. There -are many possible reasons, and they really depend on the chosen scenario. For -most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known -limitation. - -First, make sure that the instance **opnfv-vping-2** succeeded to get an IP -from the DHCP agent. It can be checked by doing:: - - nova console-log opnfv-vping-2 - -If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. - -If the instance got an IP properly, try to ping manually the VM from the container:: - - nova list - <grab the public IP> - ping <public IP> - -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. - -At this point, if the ping does not work either, try to reproduce the test -manually with the steps described above in the vPing common section with the -addition:: - - neutron floatingip-create <EXT_NET_NAME> - nova floating-ip-associate nova-test <FLOATING_IP> - - -Further troubleshooting is out of scope of this document, as it might be due to -problems with the SDN controller. Contact the installer team members or send an -email to the corresponding OPNFV mailing list for more information. - - - -vPing_userdata -^^^^^^^^^^^^^^ -This test case does not create any floating IP neither establishes an SSH -connection. Instead, it uses nova-metadata service when creating an instance -to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. - -The only known problem here for this test to fail is mainly the lack of support -of cloud-init (nova-metadata service). Check the console of the instance:: - - nova console-log opnfv-vping-2 - -If this text or similar is shown:: - - checking http://169.254.169.254/2009-04-04/instance-id - failed 1/20: up 1.13. request failed - failed 2/20: up 13.18. request failed - failed 3/20: up 25.20. request failed - failed 4/20: up 37.23. request failed - failed 5/20: up 49.25. request failed - failed 6/20: up 61.27. request failed - failed 7/20: up 73.29. request failed - failed 8/20: up 85.32. request failed - failed 9/20: up 97.34. request failed - failed 10/20: up 109.36. request failed - failed 11/20: up 121.38. request failed - failed 12/20: up 133.40. request failed - failed 13/20: up 145.43. request failed - failed 14/20: up 157.45. request failed - failed 15/20: up 169.48. request failed - failed 16/20: up 181.50. request failed - failed 17/20: up 193.52. request failed - failed 18/20: up 205.54. request failed - failed 19/20: up 217.56. request failed - failed 20/20: up 229.58. request failed - failed to read iid from metadata. tried 20 - -it means that the instance failed to read from the metadata service. Contact -the Functest or installer teams for more information. - - -Tempest -^^^^^^^ - -In the upstream OpenStack CI all the Tempest test cases are supposed to pass. -If some test cases fail in an OPNFV deployment, the reason is very probably one -of the following - -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/<component>'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | 'root/.rally/verification/verifier-<UUID> | -| | /for-deployment-<UUID>' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ - - -When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. - -Functest offers a possibility to test a customized list of Tempest test cases. -To enable that, add a new entry in docker/components/testcases.yaml on the "components" container -with the following content:: - - - - case_name: tempest_custom - project_name: functest - criteria: 100 - blocking: false - description: >- - The test case allows running a customized list of tempest - test cases - dependencies: - installer: '' - scenario: '' - run: - module: 'functest.opnfv_tests.openstack.tempest.tempest' - class: 'TempestCustom' - -Also, a list of the Tempest test cases must be provided to the container or modify -the existing one in -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt - -Example of custom list of tests 'my-custom-tempest-tests.txt':: - - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke] - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] - -This is an example of running a customized list of Tempest tests in Functest:: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ - -v $(pwd)/images:/home/opnfv/functest/images \ - -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \ - -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ - opnfv/functest-components run_tests -t tempest_custom - - -Rally -^^^^^ - -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. - -Possible scenarios are: - * authenticate - * glance - * cinder - * heat - * keystone - * neutron - * nova - * ceilometer - * quotas - * vm - -To know more about what those scenarios are doing, they are defined in directory: -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ - -To check any possible problems with Rally, the logs are stored under -*/home/opnfv/functest/results/rally/* in the Functest Docker container. - - -Controllers ------------ - -Opendaylight -^^^^^^^^^^^^ - -If the Basic Restconf test suite fails, check that the ODL controller is -reachable and its Restconf module has been installed. - -If the Neutron Reachability test fails, verify that the modules -implementing Neutron requirements have been properly installed. - -If any of the other test cases fails, check that Neutron and ODL have -been correctly configured to work together. Check Neutron configuration -files, accounts, IP addresses etc.). - - -Features --------- - -Please refer to the dedicated feature user guides for details. - - -VNF ---- - -cloudify_ims -^^^^^^^^^^^^ -vIMS deployment may fail for several reasons, the most frequent ones are -described in the following table: - -+-----------------------------------+------------------------------------+ -| Error | Comments | -+===================================+====================================+ -| Keystone admin API not reachable | Impossible to create vIMS user and | -| | tenant | -+-----------------------------------+------------------------------------+ -| Impossible to retrieve admin role | Impossible to create vIMS user and | -| id | tenant | -+-----------------------------------+------------------------------------+ -| Error when uploading image from | impossible to deploy VNF | -| OpenStack to glance | | -+-----------------------------------+------------------------------------+ -| Cinder quota cannot be updated | Default quotas not sufficient, they| -| | are adapted in the script | -+-----------------------------------+------------------------------------+ -| Impossible to create a volume | VNF cannot be deployed | -+-----------------------------------+------------------------------------+ -| SSH connection issue between the | if vPing test fails, vIMS test will| -| Test Docker container and the VM | fail... | -+-----------------------------------+------------------------------------+ -| No Internet access from the VM | the VMs of the VNF must have an | -| | external access to Internet | -+-----------------------------------+------------------------------------+ -| No access to OpenStack API from | Orchestrator can be installed but | -| the VM | the vIMS VNF installation fails | -+-----------------------------------+------------------------------------+ - -Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it -is there fore not recommended to run it on a light configuration. - -.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html# |