aboutsummaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/developer/devguide/index.rst315
-rw-r--r--docs/testing/developer/internship/security_group/index.rst70
-rw-r--r--docs/testing/developer/internship/testapi_evolution/index.rst237
-rw-r--r--docs/testing/developer/internship/unit_tests/index.rst126
-rw-r--r--docs/testing/developer/internship/vnf_catalog/index.rst170
-rw-r--r--docs/testing/user/configguide/ci.rst13
-rw-r--r--docs/testing/user/configguide/configguide.rst727
-rw-r--r--docs/testing/user/configguide/index.rst130
-rw-r--r--docs/testing/user/configguide/prerequisites.rst102
-rw-r--r--docs/testing/user/userguide/index.rst106
-rw-r--r--docs/testing/user/userguide/reporting.rst90
-rw-r--r--docs/testing/user/userguide/runfunctest.rst140
-rw-r--r--docs/testing/user/userguide/test_details.rst497
-rw-r--r--docs/testing/user/userguide/test_overview.rst239
-rw-r--r--docs/testing/user/userguide/test_results.rst53
-rw-r--r--docs/testing/user/userguide/troubleshooting.rst378
16 files changed, 0 insertions, 3393 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
deleted file mode 100644
index 19a1bdcb..00000000
--- a/docs/testing/developer/devguide/index.rst
+++ /dev/null
@@ -1,315 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. SPDX-License-Identifier: CC-BY-4.0
-
-************************
-Functest Developer Guide
-************************
-
-.. toctree::
- :numbered:
- :maxdepth: 2
-
-============
-Introduction
-============
-
-Functest is a project dealing with functional testing.
-The project produces its own internal test cases but can also be considered
-as a framework to support feature and VNF onboarding project testing.
-
-Therefore there are many ways to contribute to Functest. You can:
-
- * Develop new internal test cases
- * Integrate the tests from your feature project
- * Develop the framework to ease the integration of external test cases
-
-Additional tasks involving Functest but addressing all the test projects
-may also be mentioned:
-
- * The API / Test collection framework
- * The dashboards
- * The automatic reporting portals
- * The testcase catalog
-
-This document describes how, as a developer, you may interact with the
-Functest project. The first section details the main working areas of
-the project. The Second part is a list of "How to" to help you to join
-the Functest family whatever your field of interest is.
-
-
-========================
-Functest developer areas
-========================
-
-
-Functest High level architecture
-================================
-
-Functest is a project delivering test containers dedicated to OPNFV.
-It includes the tools, the scripts and the test scenarios.
-In Euphrates Alpine containers have been introduced in order to lighten the
-container and manage testing slicing. The new containers are created according
-to the different tiers:
-
- * functest-core: https://hub.docker.com/r/opnfv/functest-core/
- * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
- * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
- * functest-features: https://hub.docker.com/r/opnfv/functest-features/
- * functest-components: https://hub.docker.com/r/opnfv/functest-components/
- * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
- * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/
- * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
-
-Standalone functest dockers are maintained for Euphrates but Alpine containers
-are recommended.
-
-Functest can be described as follow::
-
- +----------------------+
- | |
- | +--------------+ | +-------------------+
- | | | | Public | |
- | | Tools | +------------------+ OPNFV |
- | | Scripts | | | System Under Test |
- | | Scenarios | | | |
- | | | | | |
- | +--------------+ | +-------------------+
- | |
- | Functest Docker |
- | |
- +----------------------+
-
-Functest internal test cases
-============================
-The internal test cases in Euphrates are:
-
-
- * api_check
- * connection_check
- * snaps_health_check
- * vping_ssh
- * vping_userdata
- * odl
- * odl_netvirt
- * rally_full
- * rally_sanity
- * tempest_smoke_serial
- * tempest_full_parallel
- * cloudify_ims
-
-By internal, we mean that this particular test cases have been developed and/or
-integrated by functest contributors and the associated code is hosted in the
-Functest repository.
-An internal case can be fully developed or a simple integration of
-upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are
-just integrated in Functest).
-
-The structure of this repository is detailed in `[1]`_.
-The main internal test cases are in the opnfv_tests subfolder of the
-repository, the internal test cases can be grouped by domain:
-
- * sdn: odl, odl_netvirt, odl_fds
- * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*
- * vnf: cloudify_ims
-
-If you want to create a new test case you will have to create a new folder under
-the testcases directory (See next section for details).
-
-Functest external test cases
-============================
-The external test cases are inherited from other OPNFV projects, especially the
-feature projects.
-
-The external test cases are:
-
- * barometer
- * bgpvpn
- * doctor
- * domino
- * fds
- * parser
- * promise
- * refstack_defcore
- * snaps_smoke
- * functest-odl-sfc
- * orchestra_clearwaterims
- * orchestra_openims
- * vyos_vrouter
- * juju_vepc
-
-External test cases integrated in previous versions but not released in
-Euphrates:
-
- * copper
- * moon
- * netready
- * security_scan
-
-
-The code to run these test cases is hosted in the repository of the project.
-Please note that orchestra test cases are hosted in Functest repository and not
-in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in
-functest as there are no dedicated projects.
-
-
-Functest framework
-==================
-
-Functest is a framework.
-
-Historically Functest is released as a docker file, including tools, scripts and
-a CLI to prepare the environment and run tests.
-It simplifies the integration of external test suites in CI pipeline and provide
-commodity tools to collect and display results.
-
-Since Colorado, test categories also known as **tiers** have been created to
-group similar tests, provide consistent sub-lists and at the end optimize
-test duration for CI (see How To section).
-
-The definition of the tiers has been agreed by the testing working group.
-
-The tiers are:
- * healthcheck
- * smoke
- * features
- * components
- * vnf
-
-Functest abstraction classes
-============================
-
-In order to harmonize test integration, abstraction classes have been
-introduced:
-
- * testcase: base for any test case
- * unit: run unit tests as test case
- * feature: abstraction for feature project
- * vnf: abstraction for vnf onboarding
-
-The goal is to unify the way to run tests in Functest.
-
-Feature, unit and vnf_base inherit from testcase::
-
- +----------------------------------------------------------------+
- | |
- | TestCase |
- | |
- | - init() |
- | - run() |
- | - push_to_db() |
- | - is_successful() |
- | |
- +----------------------------------------------------------------+
- | | | |
- V V V V
- +--------------------+ +---------+ +------------------------+ +-----------------+
- | | | | | | | |
- | feature | | unit | | vnf | | robotframework |
- | | | | | | | |
- | | | | |- prepare() | | |
- | - execute() | | | |- deploy_orchestrator() | | |
- | BashFeature class | | | |- deploy_vnf() | | |
- | | | | |- test_vnf() | | |
- | | | | |- clean() | | |
- +--------------------+ +---------+ +------------------------+ +-----------------+
-
-
-Testcase
---------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html
-
-Feature
--------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html
-
-Unit
-----
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html
-
-VNF
----
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html
-
-Robotframework
---------------
-.. raw:: html
- :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html
-
-
-see `[5]`_ to get code samples
-
-
-Functest util classes
-=====================
-
-In order to simplify the creation of test cases, Functest develops also some
-functions that are used by internal test cases.
-Several features are supported such as logger, configuration management and
-Openstack capabilities (tacker,..).
-These functions can be found under <repo>/functest/utils and can be described as
-follows::
-
- functest/utils/
- |-- config.py
- |-- constants.py
- |-- decorators.py
- |-- env.py
- |-- functest_utils.py
- |-- openstack_tacker.py
- `-- openstack_utils.py
-
-It is recommended to use the SNAPS-OO library for deploying OpenStack instances.
-SNAPS `[4]`_ is an OPNFV project providing OpenStack utils.
-
-
-TestAPI
-=======
-Functest is using the Test collection framework and the TestAPI developed by
-the OPNFV community. See `[6]`_ for details.
-
-
-Reporting
-=========
-A web page is automatically generated every day to display the status based on
-jinja2 templates `[3]`_.
-
-
-Dashboard
-=========
-
-Additional dashboarding is managed at the testing group level, see `[7]`_ for
-details.
-
-
-=======
-How TOs
-=======
-
-See How to section on Functest wiki `[8]`_
-
-
-==========
-References
-==========
-
-_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
-
-_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
-
-_`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting
-
-_`[4]`: https://git.opnfv.org/snaps/
-
-_`[5]` : http://testresults.opnfv.org/functest/framework/index.html
-
-_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
-
-_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba
-
-_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932
-
-IRC support chan: #opnfv-functest
diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst
deleted file mode 100644
index d1cdbdd8..00000000
--- a/docs/testing/developer/internship/security_group/index.rst
+++ /dev/null
@@ -1,70 +0,0 @@
-=======
-License
-=======
-
-Functest Docs are licensed under a Creative Commons Attribution 4.0
-International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
-
-==================================
-Functest Security group test cases
-==================================
-
-Author: Girish Sukhatankar
-mentors: D.Blaisonneau, J.Lausuch, M.Richomme
-
-Abstract
-========
-
-
-Version history
-===============
-
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the |
-| | | (Orange) | Internship |
-+------------+----------+------------------+------------------------+
-
-
-Overview:
-=========
-
-
-
-
-Problem Statement:
-------------------
-
-
-
-Curation Phase
---------------
-
-
-
-
-
-Schedule:
-=========
-
-
-
-+--------------------------+------------------------------------------+
-| **Date** | **Comment** |
-| | |
-+--------------------------+------------------------------------------+
-| December - January | ........ |
-+--------------------------+------------------------------------------+
-| January - february | ........ |
-+--------------------------+------------------------------------------+
-
-
-References:
-===========
-
-.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest
-
diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst
deleted file mode 100644
index 3038d0ac..00000000
--- a/docs/testing/developer/internship/testapi_evolution/index.rst
+++ /dev/null
@@ -1,237 +0,0 @@
-=======
-License
-=======
-
-Functest Docs are licensed under a Creative Commons Attribution 4.0
-International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
-
-=================
-TestAPI evolution
-=================
-
-Author: Sakala Venkata Krishna Rohit
-Mentors: S. Feng, J.Lausuch, M.Richomme
-
-Abstract
-========
-
-The TestAPI is used by all the test opnfv projects to report results.
-It is also used to declare projects, test cases and labs. A major refactoring
-has been done in Colorado with the introduction of swagger. The TestAPI is defined in Functest
-developer guide. The purpose of this project is to add more features to the TestAPI that automate
-the tasks that are done manually now, though there are tasks other than automation.
-
-Version history
-===============
-
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the |
-| | | (Orange) | Internship |
-+------------+----------+------------------+------------------------+
-| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship |
-| | | (IIIT Hyderabad) | |
-+------------+----------+------------------+------------------------+
-
-Overview:
-=========
-
-The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_.
-The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme.
-The link to the patches submitted is `[2]`_. The internship was successfully completed and the
-documentation is as follows.
-
-Problem Statement:
-------------------
-
-The problem statement could be divided into pending features that needed to be added into TestAPI
-repo. The following were to be accomplished within the internship time frame.
-
-* **Add verification jenkins job for the TestAPI code**
- The purpose of this job is to verify whehter the unit tests are successful or not with the
- inclusion of the patchset submitted.
-
-* **Automatic update of opnfv/testapi docker image**
- The docker image of TestAPI is hosted in the opnfv docker hub. To ensure that the TestAPI image
- is always updated with the repository, automatic updation of the image is necessary and a job
- is triggered whenever a new patch gets merged.
-
-* **Automation deployment of testresults.opnfv.org/test/ website**
- In the same manner as the docker image of TestAPI is updated, the TestAPI website needs to be
- in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the
- updation of the testresults website.
-
-* **Generate static documentation of TestAPI calls**
- The purpose of this is to give an static/offline view of TestAPI. If someone wants to have a
- look at the Restful APIs of TestAPI, he/she does't need to go to the website, he can download
- a html page and view it anytime.
-
-* **Backup MongoDB of TestAPI**
- The mongoDB needs to be backed up every week. Till now it was done manually, but due to this
- internship, it is now automated using a jenkins job.
-
-* **Add token based authorization to the TestAPI calls**
- The token based authorization was implemented to ensure that only ci_pods could access the
- database. Authentication has been added to only delete/put/post requests.
-
-Curation Phase:
----------------
-
-The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar
-with the TestAPI code and functionality and propose the solutions/tools for the tasks mentioned
-above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static
-documentaion.
-
-Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic
-deployment task involves accessing a remote server from inside the jenkins build. The deployment
-had to be done only after the docker image update is done. For these constraints to satisfy, a
-multijob jenkins job was choosen instead of a freestyle job.
-
-Important Links:
-----------------
-
-* MongoDB Backup Link - `[4]`_
-* Static Documentation - `[5]`_
-* TestAPI Token addition to ci_pods - `[6]`_
-
-Schedule:
-=========
-
-The progress and completion of the tasks is described in the below table.
-
-+--------------------------+------------------------------------------+
-| **Date** | **Comment** |
-| | |
-+--------------------------+------------------------------------------+
-| Nov 14th - Dec 31st | Understand TestAPI code and the |
-| | requirements. |
-+--------------------------+------------------------------------------+
-| Jan 1st - Jan 7th | Add jenkins job to create static |
-| | documentation and write build scripts. |
-+--------------------------+------------------------------------------+
-| Jan 8th - Jan 21st | Add verification jenkins job for unit |
-| | tests. |
-+--------------------------+------------------------------------------+
-| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup |
-| | |
-+--------------------------+------------------------------------------+
-| Jan 29th - Feb 11th | Enable automatic deployment of |
-| | testresults.opnfv.org/test/ |
-+--------------------------+------------------------------------------+
-| Feb 12th - Feb 17th | Add token based authentication |
-| | |
-+--------------------------+------------------------------------------+
-
-FAQ's
-=====
-
-This section lists the problems that I have faced and the understanding that I have acquired during
-the internship. This section may help other developers in solving any errors casused because of the
-code written as a part of this internship.
-
-
-TestAPI
--------
-
-What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If in the setup.cfg, it is defined as
-
-[files]
-data_files =
-etc/a.conf = etc/a.conf.sample
-
-then it ends up installed in the /usr/etc/. With this configuration, it would be installed
-correctly within a venv. but when it is defined as
-
-[files]
-data_files =
-/etc/a.conf = etc/a.conf.sample
-
-then it ends up installed on the root of the filesystem instead of properly be installed within the
-venv.
-
-Which attribute does swagger-codegen uses as the title in the generation of document generation ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-It uses the nickname of the api call in swagger as the title in the generation of the document
-generation.
-
-Does swagger-codegen take more than one yaml file as input ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml
-file, one needs to merge them and give it as an input keeping mind the swagger specs.
-
-
-Jenkins & JJB
--------------
-
-Which scm macro is used for verification jenkins jobs ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for
-verification jenkins job.
-
-Does the virtualenv created in one build script exists in other build scripts too ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-No, the virtualenv created in one build script only exists in that build script/shell.
-
-What parameters are needed for the scm macros ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Project and Branch are the two parameters needed for scm macros.
-
-What is the directory inside the jenkins build ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give
-you all the contents of the directory.
-
-How to include a bash script in jenkins job yaml file ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An example might be apt here as an answer.
-
-builders:
- - shell:
- !include-raw: include-raw001-hello-world.sh
-
-
-How do you make a build server run on a specific machine ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each
-server, one can use those parameter macros.
-Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses
-SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE
-defined.
-
-What job style should be used when there is a situation like one build should trigger other builds
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-or when different build scripts need to be run on different machines ?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-MultiJob style should be used as it has phases where each phase can be taken as a build scipt and
-can have its own parameters by which one can define the SLAVE_LABEL parameter.
-
-References:
-===========
-
-_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution
-
-_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22
-
-_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing
-
-_`[4]` : http://artifacts.opnfv.org/testapibackup.html
-
-_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html
-
-_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization
diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst
deleted file mode 100644
index a117c860..00000000
--- a/docs/testing/developer/internship/unit_tests/index.rst
+++ /dev/null
@@ -1,126 +0,0 @@
-=======
-License
-=======
-
-Functest Docs are licensed under a Creative Commons Attribution 4.0
-International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
-
-===================
-Functest Unit tests
-===================
-
-Author: Ashish Kumar
-Mentors: H.Yao, J.Lausuch, M.Richomme
-
-Abstract
-========
-
-
-Version history
-===============
-
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the |
-| | | (Orange) | Internship |
-+------------+----------+------------------+------------------------+
-| 2017-03-31 | 0.0.2 | Ashish Kumar | During the |
-| | | (IIIT Hyderabad) | Internship |
-+------------+----------+------------------+------------------------+
-
-
-Overview:
-=========
-Functest project is developing and integrating functional test cases for OPNFV
-and it is part of OPNFV since the beginning. Functest develops its own testcases
-and framework. This framework includes several utility libraries. The Project is
-growing rapidly with more features, tests added as per requirement. It becomes
-the responsibility of every developer to maintain the integrity of code i.e. new
-patch should not break the previous functionality of the project. To automate this
-process of software development, we should write unit tests and add them to CI so
-that when a new patch is ready to merge, we shouldn't allow those which are breaking
-previous unit tests or decreasing the coverage.
-
-
-
-Problem Statement:
-------------------
-The goal of the intership consists in creating unit test suites on Functest code
-with good code coverage (>80%) and integrate it in continuous integration in order
-to consolidate existing code.
-
-
-Curation Phase
---------------
-The curation phase was the first 3 to 4 weeks of the internship. This phase was to get
-familiar with the functest code and functionality and explore the solutions for unit
-testing in other projects and come up with the strategy for writing unit tests in functest.
-
-In this phase we decided,
-- Coverage should be 80%. There are some functions like __init__, getter, setter and other
- private methods for which writing unit test is a tedious job, so we are leaving these methods
- for now.
-- Do method wise testing for every module.
-- Use mock for external or third party services, system calls and other external library calls
- which could impact the behaviour of system during the run of unit test.
-- Add it in jenkins as passing criteria for patches.
-- Write tests in modular way so that it can help to serve as a form of documentation.
-
-
-
-Schedule:
-=========
-+--------------------------+------------------------------------------+
-| **Date** | **Comment** |
-| | |
-+--------------------------+------------------------------------------+
-| Nov 14th - Nov 28th | 1. Learn Functest Project Business |
-| | 2. Set up the development environment |
-| | 3. Run Functest code |
-+--------------------------+------------------------------------------+
-| Nov 28th - Dec.9th | 1. Explore Unit Testing Strategy, |
-| | 2. Learn about Mock in python |
-+--------------------------+------------------------------------------+
-| Dec 12th - Dec 23rd | Implement Unit Tests for CLI |
-| | |
-+--------------------------+------------------------------------------+
-| Dec 26th - Jan 6th | Implement Unit Tests for Utils |
-| | |
-+--------------------------+------------------------------------------+
-| Jan 9th - Jan 20th | Implement Unit Tests for CI |
-| | |
-+--------------------------+------------------------------------------+
-| Jan 23rd - Feb 3rd | Implement Unit Tests for Core |
-| | |
-+--------------------------+------------------------------------------+
-| Feb 6th - Feb 17th | Implement Unit Tests for |
-| | opnfv_tests/openstack/tempest |
-+--------------------------+------------------------------------------+
-| Feb 20th - Mar 3rd | Implement Unit Tests for |
-| | opnfv_tests/openstack/rally |
-+--------------------------+------------------------------------------+
-| Mar 6th - Mar 17th | Implement Unit Tests for |
-| | opnfv_tests/vnf/ims |
-+--------------------------+------------------------------------------+
-| Mar 20th - Mar 31st | Recheck and Increase Coverage for all |
-| | modules > 80% |
-+--------------------------+------------------------------------------+
-| Apr 3rd - Apr 14th | Add CI Gating for unit tests |
-| | |
-+--------------------------+------------------------------------------+
-| Apr 17th - Apr 28th | Use Tox Utility, Documentation |
-| | |
-+--------------------------+------------------------------------------+
-| Apr 28th - End | Bug Fixing |
-| | |
-+--------------------------+------------------------------------------+
-
-
-References:
-===========
-
-.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests
diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst
deleted file mode 100644
index df763339..00000000
--- a/docs/testing/developer/internship/vnf_catalog/index.rst
+++ /dev/null
@@ -1,170 +0,0 @@
-=======
-License
-=======
-
-Functest Docs are licensed under a Creative Commons Attribution 4.0
-International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
-
-=======================
-Open Source VNF Catalog
-=======================
-
-Author: Kumar Rishabh
-Mentors: B.Souville, M.Richomme, J.Lausuch
-
-Abstract
-========
-
-
-
-Version hissory
-===============
-
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the |
-| | | (Orange) | Internship |
-+------------+----------+------------------+------------------------+
-
-
-Overview:
-=========
-
-
-This project aims to create an Open Source catalog for reference and
-classification of Virtual Network Functions (VNFs)s available on
-Internet. The classification method proposed will be in sync with the
-requirements of Telcos active in NFV landscape. The project aims to have
-running web platform similar to [1] by the mid of internship (2nd week
-of March). By the penultimate month of internship I aim to have fully
-functional implementation of an Open Source VNF in functest.
-
-
-Problem Statement:
-------------------
-
-OPNFV aims to be the reference platform for development,
-standardization and integration of Open Source NFV components across
-various Open Source Platforms. It mainly deals with the infrastructure
-through the Network Function Virtualization Infrastructure (NFVI) and
-Virtual Infrastructure manager (VIM). The MANO (Management and
-orchestration) stacks have been introduced recently. VNFs are not
-directly in OPNFV scope, however VNFs are needed to test and qualify the
-infrastructure. In this regard having a common curated Open Source
-Reference VNF catalog would be of immense importance to community.
-
-Since major focus of OPNFV is Telcos, a curated platform targeted from
-industry point of view would be very useful. We plan to divide the
-entire project into three major phases(with some iterative improvements
-and overlaps)
-
-
-Curation Phase
---------------
-This phase pertains to studying various Open Source VNFs available and
-classification of them based on certain parameters. The parameters that
-I currently have in mind are:
- * Developer Metrics: These pertain to repo characteristics of VNF under
- study
- * Usage Statistics - Activity, Number of Commits, stars
- * Maturity Statistics - For instance if an NFV community decides code
- coverage is important for them, it shows the NFV community is serious
- about taking the project forward
- * Technical Tagging: These are the tags that pertain to technical
- characteristics of a VNF
- * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or
- SaaS layer or is an hybrid of two/all.
- * Generic Use Cases - This in my opinion is the broadest
- classification category. For instance a VNF could be built with a
- broad idea of powering IOT devices at home or from usage perspective
- of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_
- * Fields of Application
- * Library Status - Whether APIs are standardized, support RESTful
- services.
- * Dependency Forwarding Graph - This is pretty complex tagging
- mechanism. It essentially tries to establish a graph relationship
- between the VNFs (elementary VNFs are used in Service Function
- Chaining chains such as Firewall, DPI, content enrichment,..). In my
- opinion this is useful immensely. This will allow users to go to
- platform and ask a question like - “I have this X tech stack to
- support, Y and Z are my use cases, which NFVs should I use to support
- this.
- * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for
- the platform. This will allow users to score an NFV on certain
- parameters, may be post comments.
-
-**I plan to use the above three scores and evolve cumulative score which
-will be displayed next to each of the NFV on the platform.**
-
- * Platform building phase - This will involve erecting a Web Platform
- which will be similar to this `[1]`_. I am decently familiar with
- Django and hence I will write the platform in Django. There are two
- action plans that I have in mind right now. Either I can start writing
- the platform simultaneously which will help keep track of my progress
- or I can write the platform after 1.5 - 2 months into the internship.
- Either way I aim to have the Web Platform ready by March 12.
-
- * Functest VNF implementation phase - This is the last phase that will
- involve writing a fully functional implementation of an Open Source VNF
- into Functest. I will undertake this after I am 3 months into the
- internship. I have a decent familiarity with python and hence I think
- it shouldn’t be too difficult. I need to decide how complex the VNFI
- should undertake this exercise for (e.g. AAA such as free radius sounds
- relatively easy, vCDN is much more challenging).
- This will be decided in consent with my mentors.
-
-
-
-
-Schedule:
-=========
-I plan to take this project in 6 months time frame as I want to use it
-as a chance to read more about NFVs in particular and SDN in general
-
-
-+--------------------------+------------------------------------------+
-| **Date** | **Comment** |
-| | |
-+--------------------------+------------------------------------------+
-| December 12 - January 12 | Study the above mentioned metrics |
-| | Decide which of them are important for |
-| | community (and which are not). |
-+--------------------------+------------------------------------------+
-| January 12 - January 27 | Make a database for the above studied |
-| | metrics and evolve it further based on |
-| | Mentors’ input. + associated API |
-+--------------------------+------------------------------------------+
-| January 27 - February 5 | Compile the data collected above and make|
-| | it public. Although I can keep everything|
-| | public from the beginning too. My |
-| | rationale of not making the entire data |
-| | public in initial stage as the errors |
-| | caused by me could be misleading for |
-| | developers. |
-+--------------------------+------------------------------------------+
-| February 5 - March 5 | Erect the Web Platform and release it |
-| | for restricted group for alpha testing. |
-+--------------------------+------------------------------------------+
-| March 5 - March 12 | Make it public. Release it to public for |
-| | beta testing. Fix Bugs. |
-+--------------------------+------------------------------------------+
-| March 12 - April 12 | Start working on implementation of an |
-| | Open Source VNF in Functest. |
-+--------------------------+------------------------------------------+
-| April 12 - May 12 | I will decided what to do here based on |
-| | discussion with mentors. |
-+--------------------------+------------------------------------------+
-
-
-References:
-===========
-
-.. _`[1]` : Openhub: https://www.openhub.net/explore/projects
-
-.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf
-
-.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog
diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst
deleted file mode 100644
index f3901d86..00000000
--- a/docs/testing/user/configguide/ci.rst
+++ /dev/null
@@ -1,13 +0,0 @@
-Integration in CI
-=================
-In CI we use the Docker images and execute the appropriate commands within the
-container from Jenkins.
-
-4 steps have been defined::
- * functest-cleanup: clean existing functest dockers on the jumphost
- * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf)
- * functest-store-results: push logs to artifacts
-
-See `[3]`_ for details.
-
-.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
deleted file mode 100644
index f6581081..00000000
--- a/docs/testing/user/configguide/configguide.rst
+++ /dev/null
@@ -1,727 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. SPDX-License-Identifier: CC-BY-4.0
-
-Installation and configuration
-==============================
-
-Alpine containers have been introduced in Euphrates.
-Alpine allows Functest testing in several very light containers and thanks to
-the refactoring on dependency management should allow the creation of light and
-fully customized docker images.
-
-
-Functest Dockers
-----------------
-Docker images are available on the dockerhub:
-
- * opnfv/functest-core
- * opnfv/functest-healthcheck
- * opnfv/functest-smoke
- * opnfv/functest-features
- * opnfv/functest-components
- * opnfv/functest-vnf
- * opnfv/functest-parser
- * opnfv/functest-restapi
-
-The tag "opnfv-5.0.0" is the official release image in Euphrates, but you can also pull "euphrates"
-tag as it is being maintained by Functest team and might include bugfixes.
-
-The Functest docker container environment can -in principle- be also
-used with non-OPNFV official installers (e.g. 'devstack'), with the
-**disclaimer** that support for such environments is outside of the
-scope and responsibility of the OPNFV project.
-
-
-Preparing your environment
---------------------------
-
-cat env::
-
- INSTALLER_TYPE=XXX
- EXTERNAL_NETWORK=XXX
- DEPLOY_SCENARIO=XXX
-
-See section on environment variables for details.
-
-cat env_file::
-
- export OS_AUTH_URL=XXX
- export OS_USER_DOMAIN_NAME=XXX
- export OS_PROJECT_DOMAIN_NAME=XXX
- export OS_USERNAME=XXX
- export OS_TENANT_NAME=XXX
- export OS_PROJECT_NAME=XXX
- export OS_PASSWORD=XXX
- export OS_VOLUME_API_VERSION=XXX
- export OS_IDENTITY_API_VERSION=XXX
- export OS_IMAGE_API_VERSION=XXX
-
-See section on OpenStack credentials for details.
-
-Create a directory for the different images (attached as a Docker volume)::
-
- mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/euphrates | bash -s -- images && ls -1 images/*
-
- images/CentOS-7-aarch64-GenericCloud.qcow2
- images/CentOS-7-aarch64-GenericCloud.qcow2.xz
- images/CentOS-7-x86_64-GenericCloud.qcow2
- images/cirros-0.4.0-x86_64-disk.img
- images/cirros-0.4.0-x86_64-lxc.tar.gz
- images/cirros-d161201-aarch64-disk.img
- images/cirros-d161201-aarch64-initramfs
- images/cirros-d161201-aarch64-kernel
- images/cloudify-manager-premium-4.0.1.qcow2
- images/img
- images/trusty-server-cloudimg-amd64-disk1.img
- images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
- images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
- images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
- images/vyos-1.1.7.img
-
-Testing healthcheck suite
---------------------------
-
-Run healthcheck suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-healthcheck
-
-Results shall be displayed as follows::
-
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:06 | PASS |
- | api_check | functest | healthcheck | 06:52 | PASS |
- | snaps_health_check | functest | healthcheck | 00:42 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
- NOTE: the duration is a reference and it might vary depending on your SUT.
-
-Testing smoke suite
--------------------
-
-Run smoke suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-smoke
-
-Results shall be displayed as follows::
-
- +------------------------------+------------------+---------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------+------------------+----------------+
- | vping_ssh | functest | smoke | 00:45 | PASS |
- | vping_userdata | functest | smoke | 00:36 | PASS |
- | tempest_smoke_serial | functest | smoke | 14:04 | PASS |
- | rally_sanity | functest | smoke | 23:59 | PASS |
- | refstack_defcore | functest | smoke | 14:37 | FAIL |
- | snaps_smoke | functest | smoke | 42:09 | PASS |
- | odl | functest | smoke | 00:00 | SKIP |
- | odl_netvirt | functest | smoke | 00:00 | SKIP |
- +------------------------------+------------------+---------------+------------------+----------------+
- Note: if the scenario does not support some tests, they are indicated as SKIP.
- See User guide for details.
-
-Testing features suite
-----------------------
-
-Run features suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-features
-
-Results shall be displayed as follows::
-
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | doctor-notification | doctor | features | 00:00 | SKIP |
- | bgpvpn | sdnvpn | features | 00:00 | SKIP |
- | functest-odl-sfc | sfc | features | 00:00 | SKIP |
- | barometercollectd | barometer | features | 00:00 | SKIP |
- | fds | fastdatastacks | features | 00:00 | SKIP |
- +-----------------------------+------------------------+------------------+------------------+----------------+
- Note: if the scenario does not support some tests, they are indicated as SKIP.
- See User guide for details.
-
-Testing components suite
-------------------------
-
-Run components suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-components
-
-Results shall be displayed as follows::
-
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-------------------------------+------------------+--------------------+------------------+----------------+
- | tempest_full_parallel | functest | components | 52:42 | PASS |
- | rally_full | functest | components | 114:22 | PASS |
- +-------------------------------+------------------+--------------------+------------------+----------------+
-
-Testing vnf suite
------------------
-
-Run vnf suite::
-
-sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-vnf
-
-Results shall be displayed as follows::
-
- +---------------------------------+-------------------+--------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +---------------------------------+-------------------+--------------+------------------+----------------+
- | cloudify_ims | functest | vnf | 28:49 | FAIL |
- | vyos_vrouter | functest | vnf | 27:57 | FAIL |
- | juju_epc | functest | vnf | 55:03 | PASS |
- | orchestra_openims | orchestra | vnf | 00:00 | SKIP |
- | orchestra_clearwaterims | orchestra | vnf | 00:00 | SKIP |
- +---------------------------------+-------------------+--------------+------------------+----------------+
-
-
-Environment variables
-=====================
-
-Several environement variables may be specified:
- * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid)
- * INSTALLER_IP=<Specific IP Address>
- * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode>
-
-
-INSTALLER_IP is required by Barometer in order to access the installer node and
-the deployment.
-
-The format for the DEPLOY_SCENARIO env variable can be described as follows:
- * vim: (os|k8s) = OpenStack or Kubernetes
- * controller is one of ( nosdn | odl )
- * nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature )
- * ha_mode (high availability) is one of ( ha | noha )
-
-If several features are pertinent then use the underscore character '_' to
-separate each feature (e.g. ovs_kvm). 'nofeature' indicates that no OPNFV
-feature is deployed.
-
-The list of supported scenarios per release/installer is indicated in the
-release note.
-
-**NOTE:** The scenario name is mainly used to automatically detect
-if a test suite is runnable or not (e.g. it will prevent ODL test suite to be
-run on 'nosdn' scenarios). If not set, Functest will try to run the default test
-cases that might not include SDN controller or a specific feature
-
-**NOTE:** An HA scenario means that 3 OpenStack controller nodes are
-deployed. It does not necessarily mean that the whole system is HA. See
-installer release notes for details.
-
-Finally, three additional environment variables can also be passed in
-to the Functest Docker Container, using the -e
-"<EnvironmentVariable>=<Value>" mechanism. The first two parameters are
-only relevant to Jenkins CI invoked testing and **should not be used**
-when performing manual test scenarios:
-
- * NODE_NAME = <Test POD Name>
- * BUILD_TAG = <Jenkins Build Tag>
-
-where:
-
- * <Test POD Name> = Symbolic name of the POD where the tests are run.
- Visible in test results files, which are stored
- to the database. This option is only used when
- tests are activated under Jenkins CI control.
- It indicates the POD/hardware where the test has
- been run. If not specified, then the POD name is
- defined as "Unknown" by default.
- DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
- * <Jenkins Build tag> = Symbolic name of the Jenkins Build Job.
- Visible in test results files, which are stored
- to the database. This option is only set when
- tests are activated under Jenkins CI control.
- It enables the correlation of test results,
- which are independently pushed to the results database
- from different Jenkins jobs.
- DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
-
-
-Openstack credentials
-=====================
-OpenStack credentials are mandatory and must be provided to Functest.
-When running the command "functest env prepare", the framework will
-automatically look for the Openstack credentials file
-"/home/opnfv/functest/conf/env_file" and will exit with
-error if it is not present or is empty.
-
-There are 2 ways to provide that file:
-
- * by using a Docker volume with -v option when creating the Docker container.
- This is referred to in docker documentation as "Bind Mounting".
- See the usage of this parameter in the following chapter.
- * or creating manually the file '/home/opnfv/functest/conf/env_file'
- inside the running container and pasting the credentials in it. Consult
- your installer guide for further details. This is however not
- instructed in this document.
-
-In proxified environment you may need to change the credentials file.
-There are some tips in chapter: `Proxy support`_
-
-SSL Support
------------
-If you need to connect to a server that is TLS-enabled (the auth URL
-begins with "https") and it uses a certificate from a private CA or a
-self-signed certificate, then you will need to specify the path to an
-appropriate CA certificate to use, to validate the server certificate
-with the environment variable OS_CACERT::
-
- echo $OS_CACERT
- /etc/ssl/certs/ca.crt
-
-However, this certificate does not exist in the container by default.
-It has to be copied manually from the OpenStack deployment. This can be
-done in 2 ways:
-
- #. Create manually that file and copy the contents from the OpenStack
- controller.
- #. (Recommended) Add the file using a Docker volume when starting the
- container::
-
- -v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert
-
-You might need to export OS_CACERT environment variable inside the
-credentials file::
-
- export OS_CACERT=/etc/ssl/certs/ca.crt
-
-Certificate verification can be turned off using OS_INSECURE=true. For
-example, Fuel uses self-signed cacerts by default, so an pre step would
-be::
-
- export OS_INSECURE=true
-
-Functest docker container directory structure
-=============================================
-Inside the Functest docker container, the following directory structure
-should now be in place::
-
- `--
- |- home
- | |-- opnfv
- | | `- functest
- | | |-- conf
- | | `-- results
- | `-- repos
- | `-- vnfs
- |- src
- | |-- tempest
- | |-- vims-test
- | |-- odl_test
- | `-- fds
- `- usr
- `- lib
- `- python2.7
- `- site-packages
- `- functest
- |-- ...
-
-Underneath the '/home/opnfv/functest' directory, the Functest docker container
-includes two main directories:
-
- * The **conf** directory stores configuration files (e.g. the
- OpenStack creds are stored in path '/home/opnfv/functest/conf/env_file'),
- * the **results** directory stores some temporary result log files
-
-src and repos directories are used to host third party code used for the tests.
-
-The structure of functest repo can be described as follows::
-
- |-- INFO
- |-- LICENSE
- |-- api
- | `-- apidoc
- |-- build.sh
- |-- commons
- |-- docker
- | |-- components
- | |-- core
- | |-- features
- | |-- healthcheck
- | |-- smoke
- | |-- vnf
- | |-- parser
- | |-- restapi
- | |-- thirdparty-requirements.txt
- |-- docs
- | |-- com
- | |-- images
- | |-- release
- | | `-- release-notes
- | |-- results
- | | testing
- | | |-- developer
- | | `-- user
- | | |-- configguide
- | | `-- userguide
- `-- functest
- |-- api
- | |-- base.py
- | |-- server.py
- | |-- urls.py
- | |-- common
- | | |-- api_utils.py
- | | |-- thread.py
- | `-- resources
- | `-- v1
- | |-- creds.py
- | |-- envs.py
- | |-- testcases.py
- | |-- tiers.py
- | |-- tasks.py
- | `-- database
- | |-- db.py
- | `-- v1
- | |-- handlers.py
- | |-- models.py
- | `-- swagger
- |-- ci
- │ |-- check_deployment.py
- │ |-- config_aarch64_patch.yaml
- │ |-- config_functest.yaml
- │ |-- config_patch.yaml
- │ |-- download_images.sh
- │ |-- logging.ini
- │ |-- rally_aarch64_patch.conf
- │ |-- run_tests.py
- │ |-- testcases.yaml
- │ |-- tier_builder.py
- │ |-- tier_handler.py
- |-- cli
- │ |-- cli_base.py
- │ |-- commands
- │ │ |-- cli_env.py
- │ │ |-- cli_os.py
- │ │ |-- cli_testcase.py
- │ │ |-- cli_tier.py
- │ |-- functest-complete.sh
- |-- core
- │ |-- feature.py
- │ |-- robotframework.py
- │ |-- testcase.py
- │ |-- unit.py
- │ |-- vnf.py
- |-- energy
- │ |-- energy.py
- |-- opnfv_tests
- │ `-- openstack
- │ |-- rally
- │ |-- refstack_client
- │ |-- snaps
- │ |-- tempest
- │ |-- vping
- │ `-- sdn
- │ │ `-- odl
- │ `-- vnf
- │ |-- ims
- │ `-- router
- |-- tests
- │ `-- unit
- │ |-- ci
- │ |-- cli
- │ |-- core
- │ |-- energy
- │ |-- features
- │ |-- odl
- │ |-- openstack
- │ |-- opnfv_tests
- │ |-- test_utils.py
- │ |-- utils
- │ `-- vnf
- |-- utils
- | |-- config.py
- | |-- constants.py
- | |-- decorators.py
- | |-- env.py
- | |-- functest_utils.py
- | |-- functest_vacation.py
- | |-- openstack_clean.py
- | |-- openstack_tacker.py
- | `-- openstack_utils.py
- |-- requirements.txt
- |-- setup.cfg
- |-- setup.py
- |-- test-requirements.txt
- |-- tox.ini
- |-- upper-constraints.txt
-
- (Note: All *.pyc files removed from above list for brevity...)
-
-We may distinguish several directories, the first level has 5 directories:
-
-* **api**: This directory is dedicated to the API (framework) documentations.
-* **commons**: This directory is dedicated for storage of traffic profile or
- any other test inputs that could be reused by any test project.
-* **docker**: This directory includes the needed files and tools to
- build the Functest Docker images.
-* **docs**: This directory includes documentation: Release Notes,
- User Guide, Configuration Guide and Developer Guide.
-* **functest**: This directory contains all the code needed to run
- functest internal cases and OPNFV onboarded feature or VNF test cases.
-
-Functest directory has 7 sub-directories, which is located under
-/usr/lib/python2.7/site-packages/functest:
- * **api**: This directory is dedicated for the internal Functest API.
- * **ci**: This directory contains test structure definition files
- (e.g <filename>.yaml) and bash shell/python scripts used to
- configure and execute Functional tests. The test execution script
- can be executed under the control of Jenkins CI jobs.
- * **cli**: This directory holds the python based Functest CLI utility
- source code, which is based on the Python 'click' framework.
- * **core**: This directory holds the python based Functest core
- source code. Three abstraction classes have been created to ease
- the integration of internal, feature or vnf cases.
- * **opnfv_tests**: This directory includes the scripts required by
- Functest internal test cases and other feature projects test cases.
- * **tests**: This directory includes the functest unit tests.
- * **utils**: this directory holds Python source code for some general
- purpose helper utilities, which testers can also re-use in their
- own test code. See for an example the Openstack helper utility:
- 'openstack_utils.py'.
-
-
-Logs
-====
-By default all the logs are put un /home/opnfv/functest/results/functest.log.
-If you want to have more logs in console, you may edit the logging.ini file
-manually.
-Connect on the docker then edit the file located in
-/usr/lib/python2.7/site-packages/functest/ci/logging.ini
-
-Change wconsole to console in the desired module to get more traces.
-
-
-Configuration
-=============
-
-You may also directly modify the python code or the configuration file (e.g.
-testcases.yaml used to declare test constraints) under
-/usr/lib/python2.7/site-packages/functest
-
-
-Tips
-====
-
-Docker
-------
-When typing **exit** in the container prompt, this will cause exiting
-the container and probably stopping it. When stopping a running Docker
-container all the changes will be lost, there is a keyboard shortcut
-to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To
-reconnect to the running container **DO NOT** use the *run* command
-again (since it will create a new container), use the *exec* or *attach*
-command instead::
-
- docker ps # <check the container ID from the output>
- docker exec -ti <CONTAINER_ID> /bin/bash
-
-There are other useful Docker commands that might be needed to manage possible
-issues with the containers.
-
-List the running containers::
-
- docker ps
-
-List all the containers including the stopped ones::
-
- docker ps -a
-
-Start a stopped container named "FunTest"::
-
- docker start FunTest
-
-Attach to a running container named "StrikeTwo"::
-
- docker attach StrikeTwo
-
-It is useful sometimes to remove a container if there are some problems::
-
- docker rm <CONTAINER_ID>
-
-Use the *-f* option if the container is still running, it will force to
-destroy it::
-
- docker rm -f <CONTAINER_ID>
-
-Check the Docker documentation [`dockerdocs`_] for more information.
-
-
-Checking Openstack and credentials
-----------------------------------
-It is recommended and fairly straightforward to check that Openstack
-and credentials are working as expected.
-
-Once the credentials are there inside the container, they should be
-sourced before running any Openstack commands::
-
- source /home/opnfv/functest/conf/env_file
-
-After this, try to run any OpenStack command to see if you get any
-output, for instance::
-
- openstack user list
-
-This will return a list of the actual users in the OpenStack
-deployment. In any other case, check that the credentials are sourced::
-
- env|grep OS_
-
-This command must show a set of environment variables starting with
-*OS_*, for example::
-
- OS_REGION_NAME=RegionOne
- OS_USER_DOMAIN_NAME=Default
- OS_PROJECT_NAME=admin
- OS_AUTH_VERSION=3
- OS_IDENTITY_API_VERSION=3
- OS_PASSWORD=da54c27ae0d10dfae5297e6f0d6be54ebdb9f58d0f9dfc
- OS_AUTH_URL=http://10.1.0.9:5000/v3
- OS_USERNAME=admin
- OS_TENANT_NAME=admin
- OS_ENDPOINT_TYPE=internalURL
- OS_INTERFACE=internalURL
- OS_NO_CACHE=1
- OS_PROJECT_DOMAIN_NAME=Default
-
-
-If the OpenStack command still does not show anything or complains
-about connectivity issues, it could be due to an incorrect url given to
-the OS_AUTH_URL environment variable. Check the deployment settings.
-
-Proxy support
--------------
-If your Jumphost node is operating behind a http proxy, then there are
-2 places where some special actions may be needed to make operations
-succeed:
-
- #. Initial installation of docker engine First, try following the
- official Docker documentation for Proxy settings. Some issues were
- experienced on CentOS 7 based Jumphost. Some tips are documented
- in section: `Docker Installation on CentOS behind http proxy`_
- below.
-
-If that is the case, make sure the resolv.conf and the needed
-http_proxy and https_proxy environment variables, as well as the
-'no_proxy' environment variable are set correctly::
-
- # Make double sure that the 'no_proxy=...' line in the
- # 'env_file' file is commented out first. Otherwise, the
- # values set into the 'no_proxy' environment variable below will
- # be ovewrwritten, each time the command
- # 'source ~/functest/conf/env_file' is issued.
-
- cd ~/functest/conf/
- sed -i 's/export no_proxy/#export no_proxy/' env_file
- source ./env_file
-
- # Next calculate some IP addresses for which http_proxy
- # usage should be excluded:
-
- publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+")
-
- adminURL_IP=$(openstack catalog show identity | \
- grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+")
-
- export http_proxy="<your http proxy settings>"
- export https_proxy="<your https proxy settings>"
- export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP"
-
- # Ensure that "git" uses the http_proxy
- # This may be needed if your firewall forbids SSL based git fetch
- git config --global http.sslVerify True
- git config --global http.proxy <Your http proxy settings>
-
-For example, try to use the **nc** command from inside the functest
-docker container::
-
- nc -v opnfv.org 80
- Connection to opnfv.org 80 port [tcp/http] succeeded!
-
- nc -v opnfv.org 443
- Connection to opnfv.org 443 port [tcp/https] succeeded!
-
-Note: In a Jumphost node based on the CentOS family OS, the **nc**
-commands might not work. You can use the **curl** command instead.
-
- curl http://www.opnfv.org:80
-
- <HTML><HEAD><meta http-equiv="content-type"
- .
- .
- </BODY></HTML>
-
- curl https://www.opnfv.org:443
-
- <HTML><HEAD><meta http-equiv="content-type"
- .
- .
- </BODY></HTML>
-
- (Ignore the content. If command returns a valid HTML page, it proves
- the connection.)
-
-Docker Installation on CentOS behind http proxy
------------------------------------------------
-This section is applicable for CentOS family OS on Jumphost which
-itself is behind a proxy server. In that case, the instructions below
-should be followed **before** installing the docker engine::
-
- 1) # Make a directory '/etc/systemd/system/docker.service.d'
- # if it does not exist
- sudo mkdir /etc/systemd/system/docker.service.d
-
- 2) # Create a file called 'env.conf' in that directory with
- # the following contents:
- [Service]
- EnvironmentFile=-/etc/sysconfig/docker
-
- 3) # Set up a file called 'docker' in directory '/etc/sysconfig'
- # with the following contents:
- HTTP_PROXY="<Your http proxy settings>"
- HTTPS_PROXY="<Your https proxy settings>"
- http_proxy="${HTTP_PROXY}"
- https_proxy="${HTTPS_PROXY}"
-
- 4) # Reload the daemon
- systemctl daemon-reload
-
- 5) # Sanity check - check the following docker settings:
- systemctl show docker | grep -i env
-
- Expected result:
- ----------------
- EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
- DropInPaths=/etc/systemd/system/docker.service.d/env.conf
-
-Now follow the instructions in [`Install Docker on CentOS`_] to download
-and install the **docker-engine**. The instructions conclude with a
-"test pull" of a sample "Hello World" docker container. This should now
-work with the above pre-requisite actions.
-
-
-.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
-.. _`dockerdocs`: https://docs.docker.com/
-.. _`Proxy`: https://docs.docker.com/engine/admin/systemd/#http-proxy
-.. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/
-.. _`Functest User Guide`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html
-.. _`images/CentOS-7-x86_64-GenericCloud.qcow2`: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
-.. _`images/cirros-0.4.0-x86_64-disk.img`: http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
-.. _`images/ubuntu-14.04-server-cloudimg-amd64-disk1.img`: https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img
diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst
deleted file mode 100644
index fd997344..00000000
--- a/docs/testing/user/configguide/index.rst
+++ /dev/null
@@ -1,130 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-***************************
-Functest Installation Guide
-***************************
-
-.. toctree::
- :numbered:
- :maxdepth: 2
-
-
-Introduction
-============
-This document describes how to install and configure Functest in OPNFV.
-
-High level architecture
------------------------
-
-The high level architecture of Functest within OPNFV can be described as
-follows::
-
- CIMC/Lights+out management Admin Mgmt/API Public Storage Private
- PXE
- + + + + + +
- | | | | | |
- | +----------------------------+ | | | | |
- | | | | | | | |
- +-----+ Jumphost | | | | | |
- | | | | | | | |
- | | +--------------------+ | | | | | |
- | | | | | | | | | |
- | | | Tools | +----------------+ | | |
- | | | - Rally | | | | | | |
- | | | - Robot | | | | | | |
- | | | - RefStack | | | | | | |
- | | | | |-------------------------+ | |
- | | | Testcases | | | | | | |
- | | | - VIM | | | | | | |
- | | | | | | | | | |
- | | | - SDN Controller | | | | | | |
- | | | | | | | | | |
- | | | - Features | | | | | | |
- | | | | | | | | | |
- | | | - VNF | | | | | | |
- | | | | | | | | | |
- | | +--------------------+ | | | | | |
- | | Functest Docker + | | | | |
- | | | | | | | |
- | | | | | | | |
- | | | | | | | |
- | +----------------------------+ | | | | |
- | | | | | |
- | +----------------+ | | | | |
- | | 1 | | | | | |
- +----+ +--------------+-+ | | | | |
- | | | 2 | | | | | |
- | | | +--------------+-+ | | | | |
- | | | | 3 | | | | | |
- | | | | +--------------+-+ | | | | |
- | | | | | 4 | | | | | |
- | +-+ | | +--------------+-+ | | | | |
- | | | | | 5 +-------------+ | | | |
- | +-+ | | nodes for | | | | | |
- | | | | deploying +---------------------+ | | |
- | +-+ | OPNFV | | | | | |
- | | | +------------------------------+ | |
- | +-+ SUT | | | | | |
- | | +--------------------------------------+ |
- | | | | | | | |
- | | +----------------------------------------------+
- | +----------------+ | | | | |
- | | | | | |
- + + + + + +
- SUT = System Under Test
-
-Note connectivity to management network is not needed for most of the testcases.
-But it may be needed for some specific snaps tests.
-
-All the libraries and dependencies needed by all of the Functest tools are
-pre-installed into the Docker images. This allows running Functest on any
-platform.
-
-The automated mechanisms inside the Functest Docker containers will:
-
- * Prepare the environment according to the System Under Test (SUT)
- * Perform the appropriate functional tests
- * Push the test results into the OPNFV test result database (optional)
-
-The OpenStack credentials file must be provided to the container.
-
-These Docker images can be integrated into CI or deployed independently.
-
-Please note that the Functest Docker images have been designed for OPNFV,
-however, it would be possible to adapt them to any OpenStack based VIM +
-controller environment, since most of the test cases are integrated from
-upstream communities.
-
-The functional test cases are described in the Functest User Guide `[2]`_
-
-.. include:: ./prerequisites.rst
-
-.. include:: ./configguide.rst
-
-.. include:: ./ci.rst
-
-
-References
-==========
-
-`[1]`_ : Keystone and public end point constraint
-
-`[2]`_ : Functest User guide
-
-`[3]`_ : Functest Jenkins jobs
-
-`[4]`_ : Functest Configuration guide
-
-`[5]`_ : OPNFV main site
-
-`[6]`_ : Functest wiki page
-
-IRC support channel: #opnfv-functest
-
-.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/
-.. _`[2]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
-.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml
-.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
-.. _`[5]`: http://www.opnfv.org
-.. _`[6]`: https://wiki.opnfv.org/functest
diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst
deleted file mode 100644
index 8289803b..00000000
--- a/docs/testing/user/configguide/prerequisites.rst
+++ /dev/null
@@ -1,102 +0,0 @@
-Prerequisites
-=============
-The OPNFV deployment is out of the scope of this document but it can be
-found in http://docs.opnfv.org.
-The OPNFV platform is considered as the SUT in this document.
-
-Several prerequisites are needed for Functest:
-
- #. A Jumphost to run Functest on
- #. A Docker daemon shall be installed on the Jumphost
- #. A public/external network created on the SUT
- #. An admin/management network created on the SUT
- #. Connectivity from the Jumphost to the SUT public/external network
-
-Some specific SNAPS tests may require a connectivity from the Jumphost to the
-SUT admin/management network but most of the test cases do not. This requirement
-can be changed by overriding the 'interface' attribute (OS_INTERFACE) value
-to 'public' in the credentials file. Another means to circumvent this issue
-would be to change the 'snaps.use_keystone' value from True to False.
-
-WARNING: Connectivity from Jumphost is essential and it is of paramount
-importance to make sure it is working before even considering to install
-and run Functest. Make also sure you understand how your networking is
-designed to work.
-
-NOTE: **Jumphost** refers to any server which meets the previous
-requirements. Normally it is the same server from where the OPNFV
-deployment has been triggered previously, but it could be any server
-with proper connectivity to the SUT.
-
-NOTE: If your Jumphost is operating behind a company http proxy and/or
-firewall, please consult first the section `Proxy Support`_, towards
-the end of this document. The section details some tips/tricks which
-*may* be of help in a proxified environment.
-
-Docker installation
--------------------
-Docker installation and configuration is only needed to be done once
-through the life cycle of Jumphost.
-
-If your Jumphost is based on Ubuntu, SUSE, RHEL or CentOS linux, please
-consult the references below for more detailed instructions. The
-commands below are offered as a short reference.
-
-*Tip:* For running docker containers behind the proxy, you need first
-some extra configuration which is described in section
-`Docker Installation on CentOS behind http proxy`_. You should follow
-that section before installing the docker engine.
-
-Docker installation needs to be done as root user. You may use other
-userid's to create and run the actual containers later if so desired.
-Log on to your Jumphost as root user and install the Docker Engine
-(e.g. for CentOS family)::
-
- curl -sSL https://get.docker.com/ | sh
- systemctl start docker
-
- *Tip:* If you are working through proxy, please set the https_proxy
- environment variable first before executing the curl command.
-
-Add your user to docker group to be able to run commands without sudo::
-
- sudo usermod -aG docker <your_user>
-
-A reconnection is needed. There are 2 ways for this:
- #. Re-login to your account
- #. su - <username>
-
-References - Installing Docker Engine on different Linux Operating Systems:
- * Ubuntu_
- * RHEL_
- * CentOS_
- * SUSE_
-
-.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/
-.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/
-.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/
-.. _SUSE: https://docs.docker.com/engine/installation/linux/suse/
-
-Public/External network on SUT
-------------------------------
-Some of the tests against the VIM (Virtual Infrastructure Manager) need
-connectivity through an existing public/external network in order to
-succeed. This is needed, for example, to create floating IPs to access
-VM instances through the public/external network (i.e. from the Docker
-container).
-
-By default, the five OPNFV installers provide a fresh installation with
-a public/external network created along with a router. Make sure that
-the public/external subnet is reachable from the Jumphost and an external
-router exists.
-
-*Hint:* For the given OPNFV Installer in use, the IP sub-net address
-used for the public/external network is usually a planning item and
-should thus be known. Ensure you can reach each node in the SUT, from the
-Jumphost using the 'ping' command using the respective IP address on the
-public/external network for each node in the SUT. The details of how to
-determine the needed IP addresses for each node in the SUT may vary according
-to the used installer and are therefore ommitted here.
-
-.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/
-.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
deleted file mode 100644
index 66dfd3e7..00000000
--- a/docs/testing/user/userguide/index.rst
+++ /dev/null
@@ -1,106 +0,0 @@
-.. _functest-userguide:
-
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. SPDX-License-Identifier: CC-BY-4.0
-
-*******************
-Functest User Guide
-*******************
-
-.. toctree::
- :maxdepth: 2
-
-
-
-Introduction
-============
-
-The goal of this document is to describe the OPNFV Functest test cases and to
-provide a procedure to execute them.
-
-**IMPORTANT**: It is assumed here that Functest has been properly deployed
-following the installation guide procedure `[1]`_.
-
-.. include:: ./test_overview.rst
-
-.. include:: ./test_details.rst
-
-.. include:: ./runfunctest.rst
-
-.. include:: ./test_results.rst
-
-.. include:: ./reporting.rst
-
-.. figure:: ../../../images/functest-reporting-status.png
- :align: center
- :alt: Functest reporting portal Fuel status page
-
-.. include:: ./troubleshooting.rst
-
-
-References
-==========
-
-`[1]`_: Functest configuration guide
-
-`[2]`_: OpenStack Tempest documentation
-
-`[3]`_: Rally documentation
-
-`[4]`_: Functest in depth (Danube)
-
-`[5]`_: Clearwater vIMS blueprint
-
-`[6]`_: NIST web site
-
-`[7]`_: OpenSCAP web site
-
-`[8]`_: Refstack client
-
-`[9]`_: Defcore
-
-`[10]`_: OpenStack interoperability procedure
-
-`[11]`_: Robot Framework web site
-
-`[12]`_: Functest User guide
-
-`[13]`_: SNAPS wiki
-
-`[14]`_: vRouter
-
-`[15]`_: Testing OpenStack Tempest part 1
-
-`[16]`_: Running Functest through internal REST API
-
-`[17]`_: OPNFV Test API
-
-`OPNFV main site`_: OPNFV official web site
-
-`Functest page`_: Functest wiki page
-
-IRC support chan: #opnfv-functest
-
-.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
-.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
-.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
-.. _`[6]`: https://scap.nist.gov/
-.. _`[7]`: https://github.com/OpenSCAP/openscap
-.. _`[8]`: https://github.com/openstack/refstack-client
-.. _`[9]`: https://github.com/openstack/defcore
-.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
-.. _`[11]`: http://robotframework.org/
-.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
-.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
-.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
-.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
-.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API
-.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
-.. _`OPNFV main site`: http://www.opnfv.org
-.. _`Functest page`: https://wiki.opnfv.org/functest
-.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
-.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
-.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml
-.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html
diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst
deleted file mode 100644
index 88f5e865..00000000
--- a/docs/testing/user/userguide/reporting.rst
+++ /dev/null
@@ -1,90 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-Test reporting
-==============
-
-An automatic reporting page has been created in order to provide a consistent
-view of the Functest tests on the different scenarios.
-
-In this page, each scenario is evaluated according to test criteria.
-
-The results are collected from the centralized database every day and, per
-scenario. A score is calculated based on the results from the last 10 days.
-This score is the addition of single test scores. Each test case has a success
-criteria reflected in the criteria field from the results.
-
-As an illustration, let's consider the scenario
-os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the
-scores of all the runnable tests from the categories (tiers, healthcheck, smoke
-and features) corresponding to this scenario.
-
- +---------------------+---------+---------+---------+---------+
- | Test | Apex | Compass | Fuel | Joid |
- +=====================+=========+=========+=========+=========+
- | vPing_ssh | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | vPing_userdata | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | tempest_smoke_serial| X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | rally_sanity | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | odl | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | promise | | | X | X |
- +---------------------+---------+---------+---------+---------+
- | doctor | X | | X | |
- +---------------------+---------+---------+---------+---------+
- | security_scan | X | | | |
- +---------------------+---------+---------+---------+---------+
- | parser | | | X | |
- +---------------------+---------+---------+---------+---------+
- | copper | X | | | X |
- +---------------------+---------+---------+---------+---------+
- src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version)
-
-All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature
-scenarios.
-Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration
-addons and, as a consequence, specific scenario.
-There are not considered as runnable on the generic odl_l2 scenario.
-
-
-If no result is available or if all the results are failed, the test case get 0
-point.
-If it was successful at least once but not anymore during the 4 runs, the case
-get 1 point (it worked once).
-If at least 3 of the last 4 runs were successful, the case get 2 points.
-If the last 4 runs of the test are successful, the test get 3 points.
-
-In the example above, the target score for fuel/os-odl_l2-nofeature-ha is
-3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .
-
-The scenario is validated per installer when we got 3 points for all individual
-test cases (e.g 24/24 for fuel, 15/15 for compass).
-
-Please note that complex or long duration tests are not considered yet for the
-scoring. In fact the success criteria are not always easy to define and may
-require specific hardware configuration.
-
-Please also note that all the test cases have the same "weight" for the score
-calculation whatever the complexity of the test case. Concretely a vping has the
-same weith than the 200 tempst tests.
-Moreover some installers support more features than others. The more cases your
-scenario is dealing with, the most difficult to rich a good scoring.
-
-Therefore the scoring provides 3 types of indicators:
-
- * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features
- * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS
- * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI)
-
-In any case, the scoring is used to give feedback to the other projects and
-does not represent an absolute value of the scenario.
-
-See `reporting page`_ for details. For the status, click on the version,
-Functest then the Status menu.
-
-
-.. _`reporting page`: http://testresults.opnfv.org/reporting/
diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst
deleted file mode 100644
index d5b95101..00000000
--- a/docs/testing/user/userguide/runfunctest.rst
+++ /dev/null
@@ -1,140 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-
-Executing Functest suites
-=========================
-
-As mentioned in the configuration guide `[1]`_, Alpine docker containers have
-been introduced in Euphrates.
-Tier containers have been created.
-Assuming that you pulled the container and your environement is ready, you can
-simply run the tiers by typing (e.g. with functest-healthcheck)::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-healthcheck
-
-You should get::
-
- +----------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:02 | PASS |
- | api_check | functest | healthcheck | 03:19 | PASS |
- | snaps_health_check | functest | healthcheck | 00:46 | PASS |
- +----------------------------+------------------+---------------------+------------------+----------------+
-
-You can run functest-healcheck, functest-smoke, functest-features,
-functest-components and functest-vnf.
-
-The result tables show the results by test case, it can be::
-
- * PASS
- * FAIL
- * SKIP: if the scenario/installer does not support the test case
-
-
-Manual run
-==========
-If you want to run the test step by step, you may add docker option then run the
-different commands within the docker.
-
-Considering the healthcheck example, running functest manaully means::
-
- sudo docker run -ti --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-healthcheck /bin/bash
-
-The docker prompt shall be returned. Then within the docker run the following
-commands::
-
- $ source /home/opnfv/functest/conf/env_file
-
-Tier
-----
-Each Alpine container provided on the docker hub matches with a tier.
-The following commands are available::
-
- # functest tier list
- - 0. healthcheck:
- ['connection_check', 'api_check', 'snaps_health_check']
- # functest tier show healthcheck
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
- | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES |
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
- | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check |
- | | | | basic operations in the VIM. | snaps_health_check |
- +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+
-
-To run all the cases of the tier, type::
-
- # functest tier run healthcheck
-
-Testcase
---------
-Testcases can be listed, shown and run though the CLI::
-
- # functest testcase list
- connection_check
- api_check
- snaps_health_check
- # functest testcase show api_check
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY |
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ |
- | | OpenStack clients: Keystone, Glance, | | |
- | | Neutron and Nova and may perform some | | |
- | | simple queries. When the config value of | | |
- | | snaps.use_keystone is True, functest | | |
- | | must have access to the cloud's private | | |
- | | network. | | |
- +-------------------+--------------------------------------------------+------------------+---------------------------+
- # functest testcase run connection_check
- ...
- # functest run all
-
-You can also type run_tests -t all to run all the tests.
-
-Note the list of test cases depend on the installer and the scenario.
-
-
-Reporting results to the test Database
-======================================
-In OPNFV CI we collect all the results from CI. A test APi shall be available
-as well as a test database `[17]`_.
-
-Functest internal API
-=====================
-
-An internal API has been introduced in Euphrates. The goal is to trigger
-Functest operations through an API in addition of the CLI.
-This could be considered as a first step towards a pseudo micro services
-approach where the different test projects could expose and consume APIs to the
-other test projects.
-
-In Euphrates the main method of the APIs are:
-
- * Show credentials
- * Update openrc file
- * Show environment
- * Update hosts info for domain name
- * Prepare environment
- * List all testcases
- * Show a testcase
- * Run a testcase
- * List all tiers
- * Show a tier
- * List all testcases within given tier
- * Get the result of the specified task
- * Get the log of the specified task
-
-See `[16]`_ to get examples on how to use the API.
-
-
-.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
-.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API
-.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html
diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst
deleted file mode 100644
index 97c4688c..00000000
--- a/docs/testing/user/userguide/test_details.rst
+++ /dev/null
@@ -1,497 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-
-The different test cases are described in the remaining sections of this document.
-
-VIM (Virtualized Infrastructure Manager)
-----------------------------------------
-
-Healthcheck tests
-^^^^^^^^^^^^^^^^^
-Since Danube, healthcheck tests have been refactored and rely on SNAPS, an
-OPNFV middleware project.
-
-SNAPS stands for "SDN/NFV Application development Platform and Stack".
-SNAPS is an object-oriented OpenStack library packaged with tests that exercise
-OpenStack.
-More information on SNAPS can be found in  `[13]`_
-
-Three tests are declared as healthcheck tests and can be used for gating by the
-installer, they cover functionally the tests previously done by healthcheck
-test case.
-
-The tests are:
-
-
- * *connection_check*
- * *api_check*
- * *snaps_health_check*
-
-Connection_check consists in 9 test cases (test duration < 5s) checking the
-connectivity with Glance, Keystone, Neutron, Nova and the external network.
-
-Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
-Neutron and Nova and may perform some simple queries. When the config value of
-snaps.use_keystone is True, functest must have access to the cloud's private
-network. This suite consists in 49 tests (test duration < 2 minutes).
-
-Snaps_health_check creates a VM with a single port with an IPv4 address that
-is assigned by DHCP and then validates the expected IP with the actual.
-
-Self-obviously, successful completion of the 'healthcheck' testcase is a
-necessary pre-requisite for the execution of all other test Tiers.
-
-
-vPing_ssh
-^^^^^^^^^
-
-Given the script **ping.sh**::
-
- #!/bin/sh
- ping -c 1 $1 2>&1 >/dev/null
- RES=$?
- if [ "Z$RES" = "Z0" ] ; then
- echo 'vPing OK'
- else
- echo 'vPing KO'
- fi
-
-
-The goal of this test is to establish an SSH connection using a floating IP
-on the Public/External network and verify that 2 instances can talk over a Private
-Tenant network::
-
- vPing_ssh test case
- +-------------+ +-------------+
- | | | |
- | | Boot VM1 with IP1 | |
- | +------------------->| |
- | Tester | | System |
- | | Boot VM2 | Under |
- | +------------------->| Test |
- | | | |
- | | Create floating IP | |
- | +------------------->| |
- | | | |
- | | Assign floating IP | |
- | | to VM2 | |
- | +------------------->| |
- | | | |
- | | Establish SSH | |
- | | connection to VM2 | |
- | | through floating IP| |
- | +------------------->| |
- | | | |
- | | SCP ping.sh to VM2 | |
- | +------------------->| |
- | | | |
- | | VM2 executes | |
- | | ping.sh to VM1 | |
- | +------------------->| |
- | | | |
- | | If ping: | |
- | | exit OK | |
- | | else (timeout): | |
- | | exit Failed | |
- | | | |
- +-------------+ +-------------+
-
-This test can be considered as an "Hello World" example.
-It is the first basic use case which **must** work on any deployment.
-
-vPing_userdata
-^^^^^^^^^^^^^^
-
-This test case is similar to vPing_ssh but without the use of Floating IPs
-and the Public/External network to transfer the ping script.
-Instead, it uses Nova metadata service to pass it to the instance at booting time.
-As vPing_ssh, it checks that 2 instances can talk to
-each other on a Private Tenant network::
-
- vPing_userdata test case
- +-------------+ +-------------+
- | | | |
- | | Boot VM1 with IP1 | |
- | +-------------------->| |
- | | | |
- | | Boot VM2 with | |
- | | ping.sh as userdata | |
- | | with IP1 as $1. | |
- | +-------------------->| |
- | Tester | | System |
- | | VM2 executes ping.sh| Under |
- | | (ping IP1) | Test |
- | +-------------------->| |
- | | | |
- | | Monitor nova | |
- | | console-log VM 2 | |
- | | If ping: | |
- | | exit OK | |
- | | else (timeout) | |
- | | exit Failed | |
- | | | |
- +-------------+ +-------------+
-
-When the second VM boots it will execute the script passed as userdata
-automatically. The ping will be detected by periodically capturing the output
-in the console-log of the second VM.
-
-
-Tempest
-^^^^^^^
-
-Tempest `[2]`_ is the reference OpenStack Integration test suite.
-It is a set of integration tests to be run against a live OpenStack cluster.
-Tempest has suites of tests for:
-
- * OpenStack API validation
- * Scenarios
- * Other specific tests useful in validating an OpenStack deployment
-
-Functest uses Rally `[3]`_ to run the Tempest suite.
-Rally generates automatically the Tempest configuration file **tempest.conf**.
-Before running the actual test cases,
-Functest creates the needed resources (user, tenant) and
-updates the appropriate parameters into the configuration file.
-
-When the Tempest suite is executed, each test duration is measured and the full
-console output is stored to a *log* file for further analysis.
-
-The Tempest testcases are distributed across two
-Tiers:
-
- * Smoke Tier - Test Case 'tempest_smoke_serial'
- * Components Tier - Test case 'tempest_full_parallel'
-
-NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
-tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
-executes all defined Tempest tests using several concurrent threads
-(i.e. parallel mode). The number of threads activated corresponds to the number
-of available logical CPUs.
-
-The goal of the Tempest test suite is to check the basic functionalities of the
-different OpenStack components on an OPNFV fresh installation, using the
-corresponding REST API interfaces.
-
-
-Rally bench test suites
-^^^^^^^^^^^^^^^^^^^^^^^
-
-Rally `[3]`_ is a benchmarking tool that answers the question:
-
-*How does OpenStack work at scale?*
-
-The goal of this test suite is to benchmark all the different OpenStack modules and
-get significant figures that could help to define Telco Cloud KPIs.
-
-The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
-
- * authenticate
- * cinder
- * glance
- * heat
- * keystone
- * neutron
- * nova
- * quotas
- * ceilometer
-
-A basic SLA (stop test on errors) has been implemented.
-
-The Rally testcases are distributed across two Tiers:
-
- * Smoke Tier - Test Case 'rally_sanity'
- * Components Tier - Test case 'rally_full'
-
-NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
-cases. Test case 'rally_full' executes the full defined set of Rally tests.
-
-
-Refstack-client to run OpenStack interop testcases
---------------------------------------------------
-
-Refstack-client `[8]`_ is a command line utility that allows you to
-execute Tempest test runs based on configurations you specify.
-It is the official tool to run Openstack Interop (previously known as Defcore)
-testcases `[9]`_, which focus on testing interoperability between OpenStack
-clouds.
-
-Refstack-client is integrated in Functest, consumed by Dovetail, which
-intends to define and provide a set of OPNFV related validation criteria
-that will provide input for the evaluation of the use of OPNFV trademarks.
-This progress is under the guideline of Compliance Verification Program(CVP).
-
-Running methods
-^^^^^^^^^^^^^^^
-
-Two running methods are provided after refstack-client integrated into
-Functest, Functest command line and manually, respectively.
-
-By default, for Defcore test cases run by Functest command line,
-are run followed with automatically generated
-configuration file, i.e., refstack_tempest.conf. In some circumstances,
-the automatic configuration file may not quite satisfied with the SUT,
-Functest also inherits the refstack-client command line and provides a way
-for users to set its configuration file according to its own SUT manually.
-
-*command line*
-
-Inside the Functest container, first to prepare Functest environment:
-
-::
-
- functest env prepare
-
-then to run default defcore testcases by using refstack-client:
-
-::
-
- functest testcase run refstack_defcore
-
-In OPNFV Continuous Integration(CI) system, the command line method is used.
-
-*manually*
-
-Prepare the tempest configuration file and the testcases want to run with the SUT,
-run the testcases with:
-
-::
-
- ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
-
-using help for more information:
-
-::
-
- ./refstack-client --help
- ./refstack-client test --help
-
-Reference tempest configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-*command line method*
-
-When command line method is used, the default tempest configuration file
-is generated by Rally.
-
-*manually*
-
-When running manually is used, recommended way to generate tempest configuration
-file is:
-
-::
-
- cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
- python tempest_conf.py
-
-a file called tempest.conf is stored in the current path by default, users can do
-some adjustment according to the SUT:
-
-::
-
- vim refstack_tempest.conf
-
-a reference article can be used `[15]`_.
-
-
-snaps_smoke
-------------
-
-This test case contains tests that setup and destroy environments with VMs with
-and without Floating IPs with a newly created user and project. Set the config
-value snaps.use_floating_ips (True|False) to toggle this functionality.
-Please note that When the configuration value of snaps.use_keystone is True, Functest must have access
-the cloud's private network.
-This suite consists in 38 tests (test duration < 10 minutes)
-
-
-SDN Controllers
----------------
-
-OpenDaylight
-^^^^^^^^^^^^
-
-The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
-from the ODL project using the Robot `[11]`_ framework.
-The suite verifies creation and deletion of networks, subnets and ports with
-OpenDaylight and Neutron.
-
-The list of tests can be described as follows:
-
- * Basic Restconf test cases
- * Connect to Restconf URL
- * Check the HTTP code status
-
- * Neutron Reachability test cases
- * Get the complete list of neutron resources (networks, subnets, ports)
-
- * Neutron Network test cases
- * Check OpenStack networks
- * Check OpenDaylight networks
- * Create a new network via OpenStack and check the HTTP status code returned by Neutron
- * Check that the network has also been successfully created in OpenDaylight
-
- * Neutron Subnet test cases
- * Check OpenStack subnets
- * Check OpenDaylight subnets
- * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
- * Check that the subnet has also been successfully created in OpenDaylight
-
- * Neutron Port test cases
- * Check OpenStack Neutron for known ports
- * Check OpenDaylight ports
- * Create a new port via OpenStack and check the HTTP status code returned by Neutron
- * Check that the new port has also been successfully created in OpenDaylight
-
- * Delete operations
- * Delete the port previously created via OpenStack
- * Check that the port has been also successfully deleted in OpenDaylight
- * Delete previously subnet created via OpenStack
- * Check that the subnet has also been successfully deleted in OpenDaylight
- * Delete the network created via OpenStack
- * Check that the network has also been successfully deleted in OpenDaylight
-
-Note: the checks in OpenDaylight are based on the returned HTTP status
-code returned by OpenDaylight.
-
-
-Features
---------
-
-Functest has been supporting several feature projects since Brahpamutra:
-
-
-+-----------------+---------+----------+--------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| copper | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| doctor | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| domino | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| fds | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| moon | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| multisite | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-| netready | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| opera | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| orchestra | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| parser | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| promise | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| security_scan | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-
-Please refer to the dedicated feature user guides for details.
-
-
-VNF
----
-
-
-cloudify_ims
-^^^^^^^^^^^^
-The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
-architectural framework for delivering IP multimedia services.
-
-vIMS has been integrated in Functest to demonstrate the capability to deploy a
-relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
-functional VNF allows the test of most of the essential functions needed for a
-NFV platform.
-
-The goal of this test suite consists of:
-
- * deploy a VNF orchestrator (Cloudify)
- * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
- orchestrator based on a TOSCA blueprint defined in `[5]`_
- * run suite of signaling tests on top of this VNF
-
-The Clearwater architecture is described as follows:
-
-.. figure:: ../../../images/clearwater-architecture.png
- :align: center
- :alt: vIMS architecture
-
-
-cloudify_ims_perf
-^^^^^^^^^^^^^^^^^
-This testcase extends the cloudify_ims test case.
-The first part is similar but the testing part is different.
-The testing part consists in automating a realistic signaling load on the vIMS
-using an Ixia loader (proprietary tools)
- - You need to have access to an Ixia licence server defined in the configuration
- file and have ixia image locally.
-
-This test case is available but not declared in testcases.yaml. The declaration
-of the testcase is simple, connect to your functest-vnf docker, add the following
-section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml::
-
- -
- case_name: cloudify_ims_perf
- project_name: functest
- criteria: 80
- blocking: false
- description: >-
- Stress tests based on Cloudify. Ixia loader images and access to Ixia
- server license.
- dependencies:
- installer: ''
- scenario: 'os-nosdn-nofeature-ha'
- run:
- module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf'
- class: 'CloudifyImsPerf'
-
-orchestra_openims
-^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of OpenIMS with OpenBaton
-orchestrator.
-
-orchestra_clearwaterims
-^^^^^^^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton
-orchestrator.
-
-vyos-vrouter
-^^^^^^^^^^^^
-This test case deals with the deployment and the test of vyos vrouter with
-Cloudify orchestrator. The test case can do testing for interchangeability of
-BGP Protocol using vyos.
-
-The Workflow is as follows:
- * Deploy
- Deploy VNF Testing topology by Cloudify using blueprint.
- * Configuration
- Setting configuration to Target VNF and reference VNF using ssh
- * Run
- Execution of test command for test item written YAML format file.
- Check VNF status and behavior.
- * Reporting
- Output of report based on result using JSON format.
-
-The vyos-vrouter architecture is described in `[14]`_
-
-
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
-.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
-.. _`[8]`: https://github.com/openstack/refstack-client
-.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
-.. _`[11]`: http://robotframework.org/
-.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
-.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
-.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
-.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst
deleted file mode 100644
index a22a5067..00000000
--- a/docs/testing/user/userguide/test_overview.rst
+++ /dev/null
@@ -1,239 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-Overview of the Functest suites
-===============================
-
-Functest is the OPNFV project primarily targeting functional testing.
-In the Continuous Integration pipeline, it is launched after an OPNFV fresh
-installation to validate and verify the basic functions of the
-infrastructure.
-
-The current list of test suites can be distributed over 4 main domains:
- * VIM (Virtualised Infrastructure Manager)
- * Controllers (i.e. SDN Controllers)
- * Features
- * VNF (Virtual Network Functions)
-
-Functest test suites are also distributed in the OPNFV testing categories:
-healthcheck, smoke, features, components, performance, VNF, Stress tests.
-
-All the Healthcheck and smoke tests of a given scenario must be succesful to
-validate the scenario for the release.
-
-+-------------+---------------+----------------+----------------------------------+
-| Domain | Tier | Test case | Comments |
-+=============+===============+================+==================================+
-| VIM | healthcheck | connection | Check OpenStack connectivity |
-| | | _check | through SNAPS framework |
-| | +----------------+----------------------------------+
-| | | api_check | Check OpenStack API through |
-| | | | SNAPS framework |
-| | +----------------+----------------------------------+
-| | | snaps_health | basic instance creation, check |
-| | | \_check | DHCP |
-| +---------------+----------------+----------------------------------+
-| | smoke | vping_ssh | NFV "Hello World" using an SSH |
-| | | | connection to a destination VM |
-| | | | over a created floating IP |
-| | | | address on the SUT Public / |
-| | | | External network. Using the SSH |
-| | | | connection a test script is then |
-| | | | copied to the destination |
-| | | | VM and then executed via SSH. |
-| | | | The script will ping another |
-| | | | VM on a specified IP address over|
-| | | | the SUT Private Tenant network |
-| | +----------------+----------------------------------+
-| | | vping_userdata | Uses Ping with given userdata |
-| | | | to test intra-VM connectivity |
-| | | | over the SUT Private Tenant |
-| | | | network. The correct operation |
-| | | | of the NOVA Metadata service is |
-| | | | also verified in this test |
-| | +----------------+----------------------------------+
-| | | tempest_smoke | Generate and run a relevant |
-| | | \_serial | Tempest Test Suite in smoke mode.|
-| | | | The generated test set is |
-| | | | dependent on the OpenStack |
-| | | | deployment environment |
-| | +----------------+----------------------------------+
-| | | rally_sanity | Run a subset of the OpenStack |
-| | | | Rally Test Suite in smoke mode |
-| | +----------------+----------------------------------+
-| | | snaps_smoke | Run the SNAPS-OO integration |
-| | | | tests |
-| | +----------------+----------------------------------+
-| | | refstack | Reference RefStack suite |
-| | | \_defcore | tempest selection for NFV |
-| +---------------+----------------+----------------------------------+
-| | components | tempest_full | Generate and run a full set of |
-| | | \_parallel | the OpenStack Tempest Test Suite.|
-| | | | See the OpenStack reference test |
-| | | | suite `[2]`_. The generated |
-| | | | test set is dependent on the |
-| | | | OpenStack deployment environment |
-| | +----------------+----------------------------------+
-| | | rally_full | Run the OpenStack testing tool |
-| | | | benchmarking OpenStack modules |
-| | | | See the Rally documents `[3]`_ |
-+-------------+---------------+----------------+----------------------------------+
-| Controllers | smoke | odl | Opendaylight Test suite |
-| | | | Limited test suite to check the |
-| | | | basic neutron (Layer 2) |
-| | | | operations mainly based on |
-| | | | upstream testcases. See below |
-| | | | for details |
-| | +----------------+----------------------------------+
-| | | odl_netvirt | Test Suite for the OpenDaylight |
-| | | | SDN Controller when the NetVirt |
-| | | | features are installed. It |
-| | | | integrates some test suites from |
-| | | | upstream using Robot as the test |
-| | | | framework |
-+-------------+---------------+----------------+----------------------------------+
-| Features | features | bgpvpn | Implementation of the OpenStack |
-| | | | bgpvpn API from the SDNVPN |
-| | | | feature project. It allows for |
-| | | | the creation of BGP VPNs. |
-| | | | See `SDNVPN User Guide`_ for |
-| | | | details |
-| | +----------------+----------------------------------+
-| | | doctor | Doctor platform, as of Colorado |
-| | | | release, provides the three |
-| | | | features: |
-| | | | * Immediate Notification |
-| | | | * Consistent resource state |
-| | | | awareness for compute host down |
-| | | | * Valid compute host status |
-| | | | given to VM owner |
-| | | | See `Doctor User Guide`_ for |
-| | | | details |
-| | +----------------+----------------------------------+
-| | | odl-sfc | SFC testing for odl scenarios |
-| | | | See `SFC User Guide`_ for details|
-| | +----------------+----------------------------------+
-| | | parser | Parser is an integration project |
-| | | | which aims to provide |
-| | | | placement/deployment templates |
-| | | | translation for OPNFV platform, |
-| | | | including TOSCA -> HOT, POLICY ->|
-| | | | TOSCA and YANG -> TOSCA. it |
-| | | | deals with a fake vRNC. |
-| | | | See `Parser User Guide`_ for |
-| | | | details |
-| | +----------------+----------------------------------+
-| | | fds | Test Suite for the OpenDaylight |
-| | | | SDN Controller when the GBP |
-| | | | features are installed. It |
-| | | | integrates some test suites from |
-| | | | upstream using Robot as the test |
-| | | | framework |
-+-------------+---------------+----------------+----------------------------------+
-| VNF | vnf | cloudify_ims | Example of a real VNF deployment |
-| | | | to show the NFV capabilities of |
-| | | | the platform. The IP Multimedia |
-| | | | Subsytem is a typical Telco test |
-| | | | case, referenced by ETSI. |
-| | | | It provides a fully functional |
-| | | | VoIP System |
-| | +----------------+----------------------------------+
-| | | orchestra | OpenIMS deployment using |
-| | | \_openims | Openbaton orchestrator |
-| | +----------------+----------------------------------+
-| | | orchestra | Cleawater IMS deployment using |
-| | | \_cleawaterims | Openbaton orchestrator |
-| | +----------------+----------------------------------+
-| | | vyos_vrouter | vRouter testing |
-| | +----------------+----------------------------------+
-| | | cloudify_ims | Based on cloudify_ims test case |
-| | | perf | cloudify_ims_perf substitutes |
-| | | | the signaling test suite by an |
-| | | | automatic deployment of an Ixia |
-| | | | loader and generic SIP stress |
-| | | | tests. |
-| | | | This work has been initiated |
-| | | | during the plugfest and allows |
-| | | | realistic load tests on top of |
-| | | | cloudify_ims. Please note that |
-| | | | this test is available but not |
-| | | | declared in testcases.yaml as it |
-| | | | requires access to proprietary |
-| | | | resources (Ixia loader) |
-+-------------+---------------+----------------+----------------------------------+
-
-
-As shown in the above table, Functest is structured into different 'domains',
-'tiers' and 'test cases'. Each 'test case' usually represents an actual
-'Test Suite' comprised -in turn- of several test cases internally.
-
-Test cases also have an implicit execution order. For example, if the early
-'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
-Tier testcases, there is little point to launch a full testcase execution round.
-
-In Danube, we merged smoke and sdn controller tiers in smoke tier.
-
-An overview of the Functest Structural Concept is depicted graphically below:
-
-.. figure:: ../../../images/concepts_mapping_final.png
- :align: center
- :alt: Functest Concepts Structure
-
-Some of the test cases are developed by Functest team members, whereas others
-are integrated from upstream communities or other OPNFV projects. For example,
-`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
-OpenStack integration test suite and Functest is in charge of the selection,
-integration and automation of those tests that fit suitably to OPNFV.
-
-The Tempest test suite is the default OpenStack smoke test suite but no new test
-cases have been created in OPNFV Functest.
-
-The results produced by the tests run from CI are pushed and collected into a
-NoSQL database. The goal is to populate the database with results from different
-sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
-of a live Functest Dashboard is shown below:
-
-.. figure:: ../../../images/FunctestDashboardEuphrates.png
- :align: center
- :alt: Functest Dashboard
-
-
-Basic components (VIM, SDN controllers) are tested through their own suites.
-Feature projects also provide their own test suites with different ways of
-running their tests.
-
-The notion of domain has been introduced in the description of the test cases
-stored in the Database.
-This parameters as well as possible tags can be used for the Test case catalog.
-
-vIMS test case was integrated to demonstrate the capability to deploy a
-relatively complex NFV scenario on top of the OPNFV infrastructure.
-
-Functest considers OPNFV as a black box. OPNFV offers a lot of potential
-combinations (which may change from one version to another):
-
- * 3 controllers (OpenDaylight, ONOS, OpenContrail)
- * 5 installers (Apex, Compass, Daisy, Fuel, Joid)
-
-Most of the tests are runnable by any combination, but some tests might have
-restrictions imposed by the utilized installers or due to the available
-deployed features. The system uses the environment variables (INSTALLER_TYPE and
-DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
-environment.
-
-A convenience Functest CLI utility is also available to simplify setting up the
-Functest evironment, management of the OpenStack environment (e.g. resource
-clean-up) and for executing tests.
-The Functest CLI organised the testcase into logical Tiers, which contain in
-turn one or more testcases. The CLI allows execution of a single specified
-testcase, all test cases in a specified Tier, or the special case of execution
-of **ALL** testcases. The Functest CLI is introduced in more details in next
-section.
-
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
-.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
-.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
-.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
-.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
-.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst
deleted file mode 100644
index 3941ba0a..00000000
--- a/docs/testing/user/userguide/test_results.rst
+++ /dev/null
@@ -1,53 +0,0 @@
-Test results
-============
-
-Manual testing
---------------
-
-In manual mode test results are displayed in the console and result files
-are put in /home/opnfv/functest/results.
-
-If you want additional logs, you may configure the logging.ini under
-/usr/lib/python2.7/site-packages/functest/ci.
-
-Automated testing
---------------
-
-In automated mode, test results are displayed in jenkins logs, a summary is provided
-at the end of the job and can be described as follow::
-
- +-------------------------+----------------------------------------------------------+
- | ENV VAR | VALUE |
- +-------------------------+----------------------------------------------------------+
- | INSTALLER_TYPE | daisy |
- | DEPLOY_SCENARIO | os-nosdn-nofeature-ha |
- | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 |
- | CI_LOOP | daily |
- +-------------------------+----------------------------------------------------------+
-
- +------------------------------+------------------+---------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +------------------------------+------------------+---------------------+------------------+----------------+
- | connection_check | functest | healthcheck | 00:08 | PASS |
- | api_check | functest | healthcheck | 04:22 | PASS |
- | snaps_health_check | functest | healthcheck | 00:35 | PASS |
- | vping_ssh | functest | smoke | 00:54 | PASS |
- | vping_userdata | functest | smoke | 00:27 | PASS |
- | tempest_smoke_serial | functest | smoke | 19:39 | FAIL |
- | rally_sanity | functest | smoke | 15:16 | PASS |
- | refstack_defcore | functest | smoke | 15:55 | PASS |
- | snaps_smoke | functest | smoke | 26:45 | FAIL |
- | cloudify_ims | functest | vnf | 23:56 | PASS |
- | orchestra_openims | orchestra | vnf | 15:07 | PASS |
- | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS |
- | vyos_vrouter | functest | vnf | 00:00 | SKIP |
- +------------------------------+------------------+---------------------+------------------+----------------+
-
-Results are automatically pushed to the test results database, some additional
-result files are pushed to OPNFV artifact web sites.
-
-Based on the results stored in the result database, a `Functest reporting`_
-portal is also automatically updated. This portal provides information on the
-overall status per scenario and per installer
-
-.. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html
diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst
deleted file mode 100644
index 1649ac14..00000000
--- a/docs/testing/user/userguide/troubleshooting.rst
+++ /dev/null
@@ -1,378 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-Troubleshooting
-===============
-
-This section gives some guidelines about how to troubleshoot the test cases
-owned by Functest.
-
-**IMPORTANT**: As in the previous section, the steps defined below must be
-executed inside the Functest Docker container and after sourcing the OpenStack
-credentials::
-
- . $creds
-
-or::
-
- source /home/opnfv/functest/conf/env_file
-
-VIM
----
-
-This section covers the test cases related to the VIM (healthcheck, vping_ssh,
-vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity,
-rally_full).
-
-vPing common
-^^^^^^^^^^^^
-For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are
-similar:
-
- * Create Glance image
- * Create Network
- * Create Security Group
- * Create Instances
-
-After these actions, the test cases differ and will be explained in their
-respective section.
-
-These test cases can be run inside the container, using new Functest CLI as follows::
-
- $ functest testcase run vping_ssh
- $ functest testcase run vping_userdata
-
-The Functest CLI is designed to route a call to the corresponding internal
-python scripts, located in paths
-/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py
-and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py
-
-Notes:
-
- #. There is one difference, between the Functest CLI based test case
- execution compared to the earlier used Bash shell script, which is
- relevant to point out in troubleshooting scenarios:
-
- The Functest CLI does **not yet** support the option to suppress
- clean-up of the generated OpenStack resources, following the execution
- of a test case.
-
- Explanation: After finishing the test execution, the corresponding
- script will remove, by default, all created resources in OpenStack
- (image, instances, network and security group). When troubleshooting,
- it is advisable sometimes to keep those resources in case the test
- fails and a manual testing is needed.
-
- It is actually still possible to invoke test execution, with suppression
- of OpenStack resource cleanup, however this requires invocation of a
- **specific Python script:** 'run_tests'.
- The `OPNFV Functest Developer Guide`_ provides guidance on the use of that
- Python script in such troubleshooting cases.
-
-Some of the common errors that can appear in this test case are::
-
- vPing_ssh- ERROR - There has been a problem when creating the neutron network....
-
-This means that there has been some problems with Neutron, even before creating the
-instances. Try to create manually a Neutron network and a Subnet to see if that works.
-The debug messages will also help to see when it failed (subnet and router creation).
-Example of Neutron commands (using 10.6.0.0/24 range for example)::
-
- neutron net-create net-test
- neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
- --gateway 10.6.0.254 net-test 10.6.0.0/24
- neutron router-create test_router
- neutron router-interface-add <ROUTER_ID> test_subnet
- neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
-
-Another related error can occur while creating the Security Groups for the instances::
-
- vPing_ssh- ERROR - Failed to create the security group...
-
-In this case, proceed to create it manually. These are some hints::
-
- neutron security-group-create sg-test
- neutron security-group-rule-create sg-test --direction ingress --protocol icmp \
- --remote-ip-prefix 0.0.0.0/0
- neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \
- --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
- neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \
- --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
-
-The next step is to create the instances. The image used is located in
-*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is created
-with the name **functest-vping**. If booting the instances fails (i.e. the status
-is not **ACTIVE**), you can check why it failed by doing::
-
- nova list
- nova show <INSTANCE_ID>
-
-It might show some messages about the booting failure. To try that manually::
-
- nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test
-
-This will spawn a VM using the network created previously manually.
-In all the OPNFV tested scenarios from CI, it never has been a problem with the
-previous actions. Further possible problems are explained in the following sections.
-
-
-vPing_SSH
-^^^^^^^^^
-This test case creates a floating IP on the external network and assigns it to
-the second instance **opnfv-vping-2**. The purpose of this is to establish
-a SSH connection to that instance and SCP a script that will ping the first
-instance. This script is located in the repository under
-/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh
-and takes an IP as a parameter. When the SCP is completed, the test will do a
-SSH call to that script inside the second instance. Some problems can happen
-here::
-
- vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
-
-If this is displayed, stop the test or wait for it to finish, if you have used
-the special method of test invocation with specific supression of OpenStack
-resource clean-up, as explained earler. It means that the Container can not
-reach the Public/External IP assigned to the instance **opnfv-vping-2**. There
-are many possible reasons, and they really depend on the chosen scenario. For
-most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known
-limitation.
-
-First, make sure that the instance **opnfv-vping-2** succeeded to get an IP
-from the DHCP agent. It can be checked by doing::
-
- nova console-log opnfv-vping-2
-
-If the message *Sending discover* and *No lease, failing* is shown, it probably
-means that the Neutron dhcp-agent failed to assign an IP or even that it was not
-responding. At this point it does not make sense to try to ping the floating IP.
-
-If the instance got an IP properly, try to ping manually the VM from the container::
-
- nova list
- <grab the public IP>
- ping <public IP>
-
-If the ping does not return anything, try to ping from the Host where the Docker
-container is running. If that solves the problem, check the iptable rules because
-there might be some rules rejecting ICMP or TCP traffic coming/going from/to the
-container.
-
-At this point, if the ping does not work either, try to reproduce the test
-manually with the steps described above in the vPing common section with the
-addition::
-
- neutron floatingip-create <EXT_NET_NAME>
- nova floating-ip-associate nova-test <FLOATING_IP>
-
-
-Further troubleshooting is out of scope of this document, as it might be due to
-problems with the SDN controller. Contact the installer team members or send an
-email to the corresponding OPNFV mailing list for more information.
-
-
-
-vPing_userdata
-^^^^^^^^^^^^^^
-This test case does not create any floating IP neither establishes an SSH
-connection. Instead, it uses nova-metadata service when creating an instance
-to pass the same script as before (ping.sh) but as 1-line text. This script
-will be executed automatically when the second instance **opnfv-vping-2** is booted.
-
-The only known problem here for this test to fail is mainly the lack of support
-of cloud-init (nova-metadata service). Check the console of the instance::
-
- nova console-log opnfv-vping-2
-
-If this text or similar is shown::
-
- checking http://169.254.169.254/2009-04-04/instance-id
- failed 1/20: up 1.13. request failed
- failed 2/20: up 13.18. request failed
- failed 3/20: up 25.20. request failed
- failed 4/20: up 37.23. request failed
- failed 5/20: up 49.25. request failed
- failed 6/20: up 61.27. request failed
- failed 7/20: up 73.29. request failed
- failed 8/20: up 85.32. request failed
- failed 9/20: up 97.34. request failed
- failed 10/20: up 109.36. request failed
- failed 11/20: up 121.38. request failed
- failed 12/20: up 133.40. request failed
- failed 13/20: up 145.43. request failed
- failed 14/20: up 157.45. request failed
- failed 15/20: up 169.48. request failed
- failed 16/20: up 181.50. request failed
- failed 17/20: up 193.52. request failed
- failed 18/20: up 205.54. request failed
- failed 19/20: up 217.56. request failed
- failed 20/20: up 229.58. request failed
- failed to read iid from metadata. tried 20
-
-it means that the instance failed to read from the metadata service. Contact
-the Functest or installer teams for more information.
-
-
-Tempest
-^^^^^^^
-
-In the upstream OpenStack CI all the Tempest test cases are supposed to pass.
-If some test cases fail in an OPNFV deployment, the reason is very probably one
-of the following
-
-+-----------------------------+-----------------------------------------------------+
-| Error | Details |
-+=============================+=====================================================+
-| Resources required for test | Such resources could be e.g. an external network |
-| case execution are missing | and access to the management subnet (adminURL) from |
-| | the Functest docker container. |
-+-----------------------------+-----------------------------------------------------+
-| OpenStack components or | Check running services in the controller and compute|
-| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).|
-| configured properly | Configuration parameters can be verified from the |
-| | related .conf files located under '/etc/<component>'|
-| | directories. |
-+-----------------------------+-----------------------------------------------------+
-| Some resources required for | The tempest.conf file, automatically generated by |
-| execution test cases are | Rally in Functest, does not contain all the needed |
-| missing | parameters or some parameters are not set properly. |
-| | The tempest.conf file is located in directory |
-| | 'root/.rally/verification/verifier-<UUID> |
-| | /for-deployment-<UUID>' |
-| | in the Functest Docker container. Use the "rally |
-| | deployment list" command in order to check the UUID |
-| | the UUID of the current deployment. |
-+-----------------------------+-----------------------------------------------------+
-
-
-When some Tempest test case fails, captured traceback and possibly also the
-related REST API requests/responses are output to the console. More detailed debug
-information can be found from tempest.log file stored into related Rally deployment
-folder.
-
-Functest offers a possibility to test a customized list of Tempest test cases.
-To enable that, add a new entry in docker/components/testcases.yaml on the "components" container
-with the following content::
-
- -
- case_name: tempest_custom
- project_name: functest
- criteria: 100
- blocking: false
- description: >-
- The test case allows running a customized list of tempest
- test cases
- dependencies:
- installer: ''
- scenario: ''
- run:
- module: 'functest.opnfv_tests.openstack.tempest.tempest'
- class: 'TempestCustom'
-
-Also, a list of the Tempest test cases must be provided to the container or modify
-the existing one in
-/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt
-
-Example of custom list of tests 'my-custom-tempest-tests.txt'::
-
- tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
- tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
-
-This is an example of running a customized list of Tempest tests in Functest::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \
- -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \
- opnfv/functest-components run_tests -t tempest_custom
-
-
-Rally
-^^^^^
-
-The same error causes which were mentioned above for Tempest test cases, may also
-lead to errors in Rally as well.
-
-Possible scenarios are:
- * authenticate
- * glance
- * cinder
- * heat
- * keystone
- * neutron
- * nova
- * ceilometer
- * quotas
- * vm
-
-To know more about what those scenarios are doing, they are defined in directory:
-/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario
-For more info about Rally scenario definition please refer to the Rally official
-documentation. `[3]`_
-
-To check any possible problems with Rally, the logs are stored under
-*/home/opnfv/functest/results/rally/* in the Functest Docker container.
-
-
-Controllers
------------
-
-Opendaylight
-^^^^^^^^^^^^
-
-If the Basic Restconf test suite fails, check that the ODL controller is
-reachable and its Restconf module has been installed.
-
-If the Neutron Reachability test fails, verify that the modules
-implementing Neutron requirements have been properly installed.
-
-If any of the other test cases fails, check that Neutron and ODL have
-been correctly configured to work together. Check Neutron configuration
-files, accounts, IP addresses etc.).
-
-
-Features
---------
-
-Please refer to the dedicated feature user guides for details.
-
-
-VNF
----
-
-cloudify_ims
-^^^^^^^^^^^^
-vIMS deployment may fail for several reasons, the most frequent ones are
-described in the following table:
-
-+-----------------------------------+------------------------------------+
-| Error | Comments |
-+===================================+====================================+
-| Keystone admin API not reachable | Impossible to create vIMS user and |
-| | tenant |
-+-----------------------------------+------------------------------------+
-| Impossible to retrieve admin role | Impossible to create vIMS user and |
-| id | tenant |
-+-----------------------------------+------------------------------------+
-| Error when uploading image from | impossible to deploy VNF |
-| OpenStack to glance | |
-+-----------------------------------+------------------------------------+
-| Cinder quota cannot be updated | Default quotas not sufficient, they|
-| | are adapted in the script |
-+-----------------------------------+------------------------------------+
-| Impossible to create a volume | VNF cannot be deployed |
-+-----------------------------------+------------------------------------+
-| SSH connection issue between the | if vPing test fails, vIMS test will|
-| Test Docker container and the VM | fail... |
-+-----------------------------------+------------------------------------+
-| No Internet access from the VM | the VMs of the VNF must have an |
-| | external access to Internet |
-+-----------------------------------+------------------------------------+
-| No access to OpenStack API from | Orchestrator can be installed but |
-| the VM | the vIMS VNF installation fails |
-+-----------------------------------+------------------------------------+
-
-Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it
-is there fore not recommended to run it on a light configuration.
-
-.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html#