summaryrefslogtreecommitdiffstats
path: root/docs/testing
diff options
context:
space:
mode:
authorjose.lausuch <jose.lausuch@ericsson.com>2017-02-21 15:21:45 +0100
committerjose.lausuch <jose.lausuch@ericsson.com>2017-03-01 10:54:32 +0100
commit83714aaa9a09e7df27e06d5d95dadc89a8e67f9e (patch)
tree8d52ed9b7ab325e795ee9795da2a9bca2715feda /docs/testing
parent8ad45aea10e3e2ec5168eacd53c7f32cf3a17e9b (diff)
Restructure docs
DOCS directory restructured according to: https://wiki.opnfv.org/display/DOC/Documentation+Guide Now: - release - release notes - testing - developer - dev guide (this is to be released on docs.opnfv.org) - internship (docs about intern projects) - user (this is to be released on docs.opnfv.org) - config guide - user guide Change-Id: I1851189601aac3c5989f19f99d779efe23dbf3d1 Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
Diffstat (limited to 'docs/testing')
-rw-r--r--docs/testing/developer/devguide/index.rst993
-rw-r--r--docs/testing/developer/internship/security_group/index.rst70
-rw-r--r--docs/testing/developer/internship/testapi_evolution/index.rst237
-rw-r--r--docs/testing/developer/internship/unit_tests/index.rst70
-rw-r--r--docs/testing/developer/internship/vnf_catalog/index.rst170
-rw-r--r--docs/testing/user/configguide/configguide.rst821
-rw-r--r--docs/testing/user/configguide/index.rst296
-rw-r--r--docs/testing/user/userguide/index.rst548
-rw-r--r--docs/testing/user/userguide/introduction.rst255
-rw-r--r--docs/testing/user/userguide/runfunctest.rst385
-rw-r--r--docs/testing/user/userguide/troubleshooting.rst387
11 files changed, 4232 insertions, 0 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
new file mode 100644
index 000000000..eee013678
--- /dev/null
+++ b/docs/testing/developer/devguide/index.rst
@@ -0,0 +1,993 @@
+******************************
+OPNFV FUNCTEST developer guide
+******************************
+
+.. toctree::
+ :numbered:
+ :maxdepth: 2
+
+
+============
+Introduction
+============
+
+Functest is a project dealing with functional testing.
+Functest produces its own internal test cases but can also be considered
+as a framework to support feature and VNF onboarding project testing.
+Functest developed a test API and defined a test collection framework
+that can be used by any OPNFV project.
+
+Therefore there are many ways to contribute to Functest. You can:
+
+ * Develop new internal test cases
+ * Integrate the tests from your feature project
+ * Develop the framework to ease the integration of external test cases
+ * Develop the API / Test collection framework
+ * Develop dashboards or automatic reporting portals
+
+This document describes how, as a developer, you may interact with the
+Functest project. The first section details the main working areas of
+the project. The Second part is a list of "How to" to help you to join
+the Functest family whatever your field of interest is.
+
+
+========================
+Functest developer areas
+========================
+
+
+Functest High level architecture
+================================
+
+Functest is project delivering a test container dedicated to OPNFV.
+It includes the tools, the scripts and the test scenarios.
+
+Functest can be described as follow::
+
+ +----------------------+
+ | |
+ | +--------------+ | +-------------------+
+ | | | | Public | |
+ | | Tools | +------------------+ OPNFV |
+ | | Scripts | | | System Under Test |
+ | | Scenarios | +------------------+ |
+ | | | | Management | |
+ | +--------------+ | +-------------------+
+ | |
+ | Functest Docker |
+ | |
+ +----------------------+
+
+Functest internal test cases
+============================
+The internal test cases in Danube are:
+
+ * healthcheck
+ * connection_check
+ * api_check
+ * vping_ssh
+ * vping_userdata
+ * odl
+ * snaps_smoke
+ * tempest_smoke_serial
+ * rally_sanity
+ * tempest_full_parallel
+ * rally_full
+ * cloudify_ims
+
+By internal, we mean that this particular test cases have been
+developped and/or integrated by functest contributors and the associated
+code is hosted in the Functest repository.
+An internal case can be fully developped or a simple integration of
+upstream suites (e.g. Tempest/Rally developped in OpenStack are just
+integrated in Functest).
+The structure of this repository is detailed in `[1]`_.
+The main internal test cases are in the opnfv_tests subfolder of the
+repository, the internal test cases are:
+
+ * sdn: odl, onos
+ * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke
+ * vnf: cloudify_ims
+
+If you want to create a new test case you will have to create a new
+folder under the testcases directory.
+
+Functest external test cases
+============================
+The external test cases are inherited from other OPNFV projects,
+especially the feature projects.
+
+The external test cases are:
+
+ * promise
+ * doctor
+ * onos
+ * bgpvpn
+ * copper
+ * security_scan
+ * sfc-odl
+ * sfc-onos
+ * parser
+ * domino
+ * multisite
+ * opera_ims
+ * orchestra_ims
+
+
+The code to run these test cases may be directly in the repository of
+the project. We have also a **features** sub directory under opnfv_tests
+directory that may be used (it can be usefull if you want to reuse
+Functest library).
+
+Functest framework
+==================
+
+Functest can be considered as a framework.
+Functest is release as a docker file, including tools, scripts and a CLI
+to prepare the environement and run tests.
+It simplifies the integration of external test suites in CI pipeline
+and provide commodity tools to collect and display results.
+
+Since Colorado, test categories also known as tiers have been created to
+group similar tests, provide consistant sub-lists and at the end optimize
+test duration for CI (see How To section).
+
+The definition of the tiers has been agreed by the testing working group.
+
+The tiers are:
+ * healthcheck
+ * smoke
+ * features
+ * components
+ * performance
+ * vnf
+ * stress
+
+Functest abstraction classes
+============================
+
+In order to harmonize test integration, 3 abstraction classes have been
+introduced in Danube:
+
+ * testcase_base: base for any test case
+ * feature_base: abstraction for feature project
+ * vnf_base: abstraction for vnf onboarding
+
+The goal is to unify the way to run test from Functest.
+
+feature_base and vnf_base inherit from testcase_base.
+
+ +-----------------------------------------+
+ | |
+ | Testcase_base |
+ | |
+ | - init() |
+ | - run() |
+ | - publish_report() |
+ | - check_criteria() |
+ | |
+ +-----------------------------------------+
+ | |
+ V V
+ +--------------------+ +--------------------------+
+ | | | |
+ | feature_base | | vnf_base |
+ | | | |
+ | - prepare() | | - prepare() |
+ | - post() | | - deploy_orchestrator() |
+ | - parse_results() | | - deploy_vnf() |
+ | | | - test_vnf() |
+ | | | - clean() |
+ | | | - execute() |
+ | | | |
+ +--------------------+ +--------------------------+
+
+
+Functest util classes
+=====================
+
+In order to simplify the creation of test cases, Functest develops some
+functions that can be used by any feature or internal test cases.
+Several features are supported such as logger, configuration management and
+Openstack capabilities (snapshot, clean, tacker,..).
+These functions can be found under <repo>/functest/utils and can be described as
+follows:
+
+functest/utils/
+|-- config.py
+|-- constants.py
+|-- env.py
+|-- functest_constants.py
+|-- functest_logger.py
+|-- functest_utils.py
+|-- openstack_clean.py
+|-- openstack_snapshot.py
+|-- openstack_tacker.py
+`-- openstack_utils.py
+
+Note that for Openstack, keystone v3 is now deployed by default by compass,
+fuel and joid in Danube. All installers still support keysone v2 (deprecated in
+next version).
+
+Test collection framework
+=========================
+
+The OPNFV testing group created a test collection database to collect
+the test results from CI:
+
+
+ http://testresults.opnfv.org/test/swagger/spec.html
+
+ Authentication: opnfv/api@opnfv
+
+Any test project running on any lab integrated in CI can push the
+results to this database.
+This database can be used to see the evolution of the tests and compare
+the results versus the installers, the scenarios or the labs.
+
+
+Overall Architecture
+--------------------
+The Test result management can be summarized as follows::
+
+ +-------------+ +-------------+ +-------------+
+ | | | | | |
+ | Test | | Test | | Test |
+ | Project #1 | | Project #2 | | Project #N |
+ | | | | | |
+ +-------------+ +-------------+ +-------------+
+ | | |
+ V V V
+ +-----------------------------------------+
+ | |
+ | Test Rest API front end |
+ | http://testresults.opnfv.org/test |
+ | |
+ +-----------------------------------------+
+ A |
+ | V
+ | +-------------------------+
+ | | |
+ | | Test Results DB |
+ | | Mongo DB |
+ | | |
+ | +-------------------------+
+ |
+ |
+ +----------------------+
+ | |
+ | test Dashboard |
+ | |
+ +----------------------+
+
+Test API description
+--------------------
+The Test API is used to declare pods, projects, test cases and test
+results. Pods are the pods used to run the tests.
+The results pushed in the database are related to pods, projects and
+cases. If you try to push results of test done on non referenced pod,
+the API will return an error message.
+
+An additional method dashboard has been added to post-process
+the raw results in release Brahmaputra (deprecated in Colorado).
+
+The data model is very basic, 4 objects are created:
+
+ * Pods
+ * Projects
+ * Testcases
+ * Results
+
+Pods::
+
+ {
+ "id": <ID>,
+ "details": <URL description of the POD>,
+ "creation_date": "YYYY-MM-DD HH:MM:SS",
+ "name": <The POD Name>,
+ "mode": <metal or virtual>,
+ "role": <ci-pod or community-pod or single-node>
+ },
+
+Projects::
+
+ {
+ "id": <ID>,
+ "name": <Name of the Project>,
+ "creation_date": "YYYY-MM-DD HH:MM:SS",
+ "description": <Short description>
+ },
+
+Testcases::
+
+ {
+ "id": <ID>,
+ "name":<Name of the test case>,
+ "project_name":<Name of belonged project>,
+ "creation_date": "YYYY-MM-DD HH:MM:SS",
+ "description": <short description>,
+ "url":<URL for longer description>
+ },
+
+Results::
+
+ {
+ "_id": <ID>,
+ "case_name": <Reference to the test case>,
+ "project_name": <Reference to project>,
+ "pod_name": <Reference to POD where the test was executed>,
+ "installer": <Installer Apex or Compass or Fuel or Joid>,
+ "version": <master or Colorado or Brahmaputra>,
+ "start_date": "YYYY-MM-DD HH:MM:SS",
+ "stop_date": "YYYY-MM-DD HH:MM:SS",
+ "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">,
+ "scenario": <Scenario on which the test was executed>,
+ "criteria": <PASS or FAILED>,
+ "trust_indicator": {
+ "current": 0,
+ "histories": []
+ }
+ }
+
+The API can described as follows. For detailed information, please go to
+
+ http://testresults.opnfv.org/test/swagger/spec.html
+
+ Authentication: opnfv/api@opnfv
+
+Version:
+
+ +--------+--------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+==========================+=========================================+
+ | GET | /versions | Get all supported API versions |
+ +--------+--------------------------+-----------------------------------------+
+
+
+Pods:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /api/v1/pods | Get the list of declared Labs (PODs) |
+ +--------+----------------------------+-----------------------------------------+
+ | POST | /api/v1/pods | Declare a new POD |
+ | | | Content-Type: application/json |
+ | | | { |
+ | | | "name": "pod_foo", |
+ | | | "mode": "metal", |
+ | | | "role": "ci-pod", |
+ | | | "details": "it is a ci pod" |
+ | | | } |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /api/v1/pods/{pod_name} | Get a declared POD |
+ +--------+----------------------------+-----------------------------------------+
+
+Projects:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /api/v1/projects | Get the list of declared projects |
+ +--------+----------------------------+-----------------------------------------+
+ | POST | /api/v1/projects | Declare a new test project |
+ | | | Content-Type: application/json |
+ | | | { |
+ | | | "name": "project_foo", |
+ | | | "description": "whatever you want" |
+ | | | } |
+ +--------+----------------------------+-----------------------------------------+
+ | DELETE | /api/v1/projects/{project} | Delete a test project |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /api/v1/projects/{project} | Get details on a {project} |
+ | | | |
+ +--------+----------------------------+-----------------------------------------+
+ | PUT | /api/v1/projects/{project} | Update a test project |
+ | | | |
+ | | | Content-Type: application/json |
+ | | | { |
+ | | | <the field(s) you want to modify> |
+ | | | } |
+ +--------+----------------------------+-----------------------------------------+
+
+
+Testcases:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} |
+ | | cases | |
+ +--------+----------------------------+-----------------------------------------+
+ | POST | /api/v1/projects/{project}/| Add a new test case to {project} |
+ | | cases | Content-Type: application/json |
+ | | | { |
+ | | | "name": "case_foo", |
+ | | | "description": "whatever you want" |
+ | | | "url": "whatever you want" |
+ | | | } |
+ +--------+----------------------------+-----------------------------------------+
+ | DELETE | /api/v1/projects/{project}/| Delete a test case |
+ | | cases/{case} | |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /api/v1/projects/{project}/| Get a declared test case |
+ | | cases/{case} | |
+ +--------+----------------------------+-----------------------------------------+
+ | PUT | /api/v1/projects/{project}?| Modify a test case of {project} |
+ | | cases/{case} | |
+ | | | Content-Type: application/json |
+ | | | { |
+ | | | <the field(s) you want to modify> |
+ | | | } |
+ +--------+----------------------------+-----------------------------------------+
+
+Results:
+
+ +--------+----------------------------+------------------------------------------+
+ | Method | Path | Description |
+ +========+============================+==========================================+
+ | GET | /api/v1/results | Get all the test results |
+ +--------+----------------------------+------------------------------------------+
+ | POST | /api/v1/results | Add a new test results |
+ | | | Content-Type: application/json |
+ | | | { |
+ | | | "project_name": "project_foo", |
+ | | | "scenario": "odl-l2", |
+ | | | "stop_date": "2016-05-28T14:42:58.384Z", |
+ | | | "trust_indicator": 0.5, |
+ | | | "case_name": "vPing", |
+ | | | "build_tag": "", |
+ | | | "version": "Colorado", |
+ | | | "pod_name": "pod_foo", |
+ | | | "criteria": "PASS", |
+ | | | "installer": "fuel", |
+ | | | "start_date": "2016-05-28T14:41:58.384Z",|
+ | | | "details": <your results> |
+ | | | } |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of {case} |
+ | | case={case} | |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of build_tag |
+ | | build_tag={tag} | {tag}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get last {N} records of test results |
+ | | last={N} | |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of scenario |
+ | | scenario={scenario} | {scenario}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of trust_indicator |
+ | | trust_indicator={ind} | {ind}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of last days |
+ | | period={period} | {period}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of {project} |
+ | | project={project} | |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of version |
+ | | version={version} | {version}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of criteria |
+ | | criteria={criteria} | {criteria}. |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | get the results on pod {pod} |
+ | | pod={pod} | |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the test results of installer {inst} |
+ | | installer={inst} | |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results? | Get the results according to combined |
+ | | <query conditions> | query conditions supported above |
+ +--------+----------------------------+------------------------------------------+
+ | GET | /api/v1/results/{result_id}| Get the test result by result_id |
+ +--------+----------------------------+------------------------------------------+
+
+Scenarios:
+
+ +--------+----------------------------+-----------------------------------------+
+ | Method | Path | Description |
+ +========+============================+=========================================+
+ | GET | /api/v1/scenarios | Get the list of declared scenarios |
+ +--------+----------------------------+-----------------------------------------+
+ | POST | /api/v1/scenario | Declare a new scenario |
+ +--------+----------------------------+-----------------------------------------+
+ | GET | /api/v1/scenario? | Get a declared scenario |
+ | | <query conditions> | |
+ +--------+----------------------------+-----------------------------------------+
+
+
+The code of the API is hosted in the releng repository `[6]`_.
+The static documentation of the API can be found at `[17]`_.
+The test API has been dockerized and may be installed locally in your
+lab. See `[15]`_ for details.
+
+The deployment of the test API has been automated.
+A jenkins job manages:
+ * the unit tests of the test api
+ * the creation of a new docker file
+ * the deployment of the new test api
+ * the archive of the old test api
+ * the backup of the Mongo DB
+
+Test API Authorization
+~~~~~~~~~~~~~~~~~~~~~~
+
+PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs
+to be added in the request using a header 'X-Auth-Token' for access to the database.
+
+e.g::
+ headers['X-Auth-Token']
+
+The value of the header i.e the token can be accessed in the jenkins environment variable
+*TestApiToken*. The token value is added as a masked password.
+
+.. code-block:: python
+
+ headers['X-Auth-Token'] = os.environ.get('TestApiToken')
+
+The above example is in Python. Token based authentication has been added so that only ci pods
+jenkins job can have access to the database.
+
+Please note that currently token authorization is implemented but is not yet enabled.
+
+ Automatic reporting
+ ===================
+
+ An automatic reporting page has been created in order to provide a
+ consistant view of the scenarios.
+ In this page, each scenario is evaluated according to test criteria.
+ The code for the automatic reporting is available at `[8]`_.
+
+ The results are collected from the centralized database every day and,
+ per scenario. A score is calculated based on the results from the last
+ 10 days. This score is the addition of single test scores. Each test
+ case has a success criteria reflected in the criteria field from the
+ results.
+
+ Considering an instance of a scenario os-odl_l2-nofeature-ha, the
+ scoring is the addition of the scores of all the runnable tests from the
+ categories (tiers healthcheck, smoke and features)
+ corresponding to this scenario.
+
+
+ +---------------------+---------+---------+---------+---------+
+ | Test | Apex | Compass | Fuel | Joid |
+ +=====================+=========+=========+=========+=========+
+ | vPing_ssh | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | vPing_userdata | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | tempest_smoke_serial| X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | rally_sanity | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | odl | X | X | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | promise | | | X | X |
+ +---------------------+---------+---------+---------+---------+
+ | doctor | X | | X | |
+ +---------------------+---------+---------+---------+---------+
+ | security_scan | X | | | |
+ +---------------------+---------+---------+---------+---------+
+ | parser | | | X | |
+ +---------------------+---------+---------+---------+---------+
+ | copper | X | | | X |
+ +---------------------+---------+---------+---------+---------+
+
+ All the testcases listed in the table are runnable on
+ os-odl_l2-nofeature scenarios.
+ If no result is available or if all the results are failed, the test
+ case get 0 point.
+ If it was succesfull at least once but not anymore during the 4 runs,
+ the case get 1 point (it worked once).
+ If at least 3 of the last 4 runs were successful, the case get 2 points.
+ If the last 4 runs of the test are successful, the test get 3 points.
+
+ In the example above, the target score for fuel/os-odl_l2-nofeature-ha
+ is 3x6 = 18 points.
+
+ The scenario is validated per installer when we got 3 points for all
+ individual test cases (e.g 18/18).
+ Please note that complex or long duration tests are not considered for
+ the scoring. The success criteria are not always easy to define and may
+ require specific hardware configuration. These results however provide
+ a good level of trust on the scenario.
+
+ A web page is automatically generated every day to display the status.
+ This page can be found at `[9]`_. For the status, click on Status menu,
+ you may also get feedback for vims and tempest_smoke_serial test cases.
+
+ Any validated scenario is stored in a local file on the web server. In
+ fact as we are using a sliding windows to get results, it may happen
+ that a successful scenarios is no more run (because considered as
+ stable) and then the number of iterations (4 needed) would not be
+ sufficient to get the green status.
+
+ Please note that other test cases (e.g. sfc_odl, bgpvpn) need also
+ ODL configuration addons and as a consequence specific scenario.
+ There are not considered as runnable on the generic odl_l2 scenario.
+
+Dashboard
+=========
+
+Dashboard is used to provide a consistant view of the results collected
+in CI.
+The results showed on the dashboard are post processed from the Database,
+which only contains raw results.
+
+In Brahmaputra, we created a basic dashboard.
+Since Colorado, it was decided to adopt ELK framework. Mongo DB results
+are extracted to feed Elasticsearch database (`[7]`_).
+
+A script was developed to build elasticsearch data set. This
+script can be found in `[16]`_.
+
+For next versions, it was decided to integrated bitergia dashboard.
+Bitergia already provides a dashboard for code and infrastructure.
+A new Test tab will be added. The dataset will be built by consuming
+the test API.
+
+
+=======
+How TOs
+=======
+
+How Functest works?
+===================
+
+The installation and configuration of the Functest docker image is
+described in `[1]`_.
+
+The procedure to start tests is described in `[2]`_
+
+
+How can I contribute to Functest?
+=================================
+
+If you are already a contributor of any OPNFV project, you can
+contribute to functest. If you are totally new to OPNFV, you must first
+create your Linux Foundation account, then contact us in order to
+declare you in the repository database.
+
+We distinguish 2 levels of contributors:
+
+ * the standard contributor can push patch and vote +1/0/-1 on any Functest patch
+ * The commitor can vote -2/-1/0/+1/+2 and merge
+
+Functest commitors are promoted by the Functest contributors.
+
+
+Where can I find some help to start?
+====================================
+
+This guide is made for you. You can also have a look at the project wiki
+page `[10]`_.
+There are references on documentation, video tutorials, tips...
+
+You can also directly contact us by mail with [Functest] prefix in the
+title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan
+#opnfv-functest.
+
+
+What kind of testing do you do in Functest?
+===========================================
+
+Functest is focusing on Functional testing. The results must be PASS or
+FAIL. We do not deal with performance and/or qualification tests.
+We consider OPNFV as a black box and execute our tests from the jumphost
+according to Pharos reference technical architecture.
+
+Upstream test suites are integrated (Rally/Tempest/ODL/ONOS,...).
+If needed Functest may bootstrap temporarily testing activities if they
+are identified but not covered yet by an existing testing project (e.g
+security_scan before the creation of the security repository)
+
+
+How test constraints are defined?
+=================================
+
+Test constraints are defined according to 2 paramaters:
+
+ * The scenario (DEPLOY_SCENARIO env variable)
+ * The installer (INSTALLER_TYPE env variable)
+
+A scenario is a formal description of the system under test.
+The rules to define a scenario are described in `[4]`_
+
+These 2 constraints are considered to determinate if the test is runnable
+or not (e.g. no need to run onos suite on odl scenario).
+
+In the test declaration for CI, the test owner shall indicate these 2
+constraints. The file testcases.yaml `[5]`_ must be patched in git to
+include new test cases. A more elaborated system based on template is
+planned for next releases
+
+For each dependency, it is possible to define a regex::
+
+ name: promise
+ criteria: 'success_rate == 100%'
+ description: >-
+ Test suite from Promise project.
+ dependencies:
+ installer: '(fuel)|(joid)'
+ scenario: ''
+
+In the example above, it means that promise test will be runnable only
+with joid or fuel installers on any scenario.
+
+The vims criteria means any installer and exclude onos and odl with
+bgpvpn scenarios::
+
+ name: vims
+ criteria: 'status == "PASS"'
+ description: >-
+ This test case deploys an OpenSource vIMS solution from Clearwater
+ using the Cloudify orchestrator. It also runs some signaling traffic.
+ dependencies:
+ installer: ''
+ scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$'
+
+
+How to write and check constaint regex?
+=======================================
+
+Regex are standard regex. You can have a look at `[11]`_
+
+You can also easily test your regex via an online regex checker such as `[12]`_.
+Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put
+your regex in the REGULAR EXPRESSION window, then you can test your rule
+.
+
+
+How to know which test I can run?
+=================================
+
+You can use the API `[13]`_. The static declaration is in git `[5]`_
+
+If you are in a Functest docker container (assuming that the
+environement has been prepared): just use the CLI.
+
+You can get the list per Test cases or by Tier::
+
+ # functest testcase list
+ healthcheck
+ vping_ssh
+ vping_userdata
+ tempest_smoke_serial
+ rally_sanity
+ odl
+ doctor
+ security_scan
+ tempest_full_parallel
+ rally_full
+ vims
+ # functest tier list
+ - 0. healthcheck:
+ ['healthcheck']
+ - 1. smoke:
+ ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
+ - 2. sdn_suites:
+ ['odl']
+ - 3. features:
+ ['doctor', 'security_scan']
+ - 4. openstack:
+ ['tempest_full_parallel', 'rally_full']
+ - 5. vnf:
+ ['vims']
+
+
+How to manually start Functest tests?
+=====================================
+
+Assuming that you are connected on the jumphost and that the system is
+"Pharos compliant", i.e the technical architecture is compatible with
+the one defined in the Pharos project::
+
+ # docker pull opnfv/functest:latest
+ # envs="-e INSTALLER_TYPE=fuel -e INSTALLER_IP=10.20.0.2 -e DEPLOY_SCENARIO=os-odl_l2-nofeature-ha -e CI_DEBUG=true"
+ # sudo docker run --privileged=true -id ${envs} opnfv/functest:latest /bin/bash
+
+
+Then you must connect to the docker container and source the
+credentials::
+
+ # docker ps (copy the id)
+ # docker exec -ti <container_id> bash
+ # source $creds
+
+
+You must first check if the environment is ready::
+
+ # functest env status
+ Functest environment ready to run tests.
+
+
+If not ready, prepare the env by launching::
+
+ # functest env prepare
+ Functest environment ready to run tests.
+
+Once the Functest env is ready, you can use the CLI to start tests.
+
+You can run test cases per test case or per tier:
+ # functest testcase run <case name> or # functest tier run <tier name>
+
+
+e.g::
+
+ # functest testcase run tempest_smoke_serial
+ # functest tier run features
+
+
+If you want to run all the tests you can type::
+
+ # functest testcase run all
+
+
+If you want to run all the tiers (same at the end that running all the
+test cases) you can type::
+
+ # functest tier run all
+
+
+How to declare my tests in Functest?
+====================================
+
+If you want to add new internal test cases, you can submit patch under
+the testcases directory of Functest repository.
+
+For feature test integration, the code can be kept into your own
+repository. The Functest files to be modified are:
+
+ * functest/docker/Dockerfile: get your code in Functest container
+ * functest/ci/testcases.yaml: reference your test and its associated constraints
+
+
+Dockerfile
+----------
+
+This file lists the repositories (internal or external) to be cloned in
+the Functest container. You can also add external packages::
+
+ RUN git clone https://gerrit.opnfv.org/gerrit/<your project> ${REPOS_DIR}/<your project>
+
+testcases.yaml
+--------------
+
+All the test cases that must be run from CI / CLI must be declared in
+ci/testcases.yaml.
+
+This file is used to get the constraints related to the test::
+
+ name: <my_super_test_case>
+ criteria: <not used yet in Colorado, could be > 'PASS', 'rate > 90%'
+ description: >-
+ <the description of your super test suite>
+ dependencies:
+ installer: regex related to installer e.g. 'fuel', '(apex)||(joid)'
+ scenario: regex related to the scenario e.g. 'ovs*no-ha'
+
+
+You must declare your test case in one of the category (tier).
+
+If you are integrating test suites from a feature project, the default
+category is **features**.
+
+
+How to select my list of tests for CI?
+======================================
+
+Functest can be run automatically from CI, a jenkins job is usually
+called after an OPNFV fresh installation.
+By default we try to run all the possible tests (see `[14]` called from
+Functest jenkins job)::
+
+ cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}"
+
+
+Each case can be configured as daily and/or weekly task.
+Weekly tasks are used for long duration or experimental tests.
+Daily tasks correspond to the minimum set of test suites to validate a scenario.
+
+When executing run_tests.py, a check based on the jenkins build tag will
+be considered to detect whether it is a daily and/or a weekly test.
+
+in your CI you can customize the list of test you want to run by case or
+by tier, just change the line::
+
+ cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t <whatever you want> ${flags}"
+
+e.g.::
+
+ cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}"
+
+This command will run all the test cases of the first 2 tiers, i.e.
+healthcheck, connection_check, api_check, vping_ssh, vping_userdata,
+snaps_somke, tempest_smoke_serial and rally_sanity.
+
+
+How to push your results into the Test Database
+===============================================
+
+The test database is used to collect test results. By default it is
+enabled only for CI tests from Production CI pods.
+
+The architecture and associated API is described in previous chapter.
+If you want to push your results from CI, you just have to call the API
+at the end of your script.
+
+You can also reuse a python function defined in functest_utils.py::
+
+ def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
+ """
+ POST results to the Result target DB
+ """
+ url = db_url + "/results"
+ installer = get_installer_type(logger)
+ params = {"project_name": "functest", "case_name": case_name,
+ "pod_name": pod_name, "installer": installer,
+ "version": version, "details": payload}
+
+ headers = {'Content-Type': 'application/json'}
+ try:
+ r = requests.post(url, data=json.dumps(params), headers=headers)
+ if logger:
+ logger.debug(r)
+ return True
+ except Exception, e:
+ print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
+ % (db_url, case_name, pod_name, version, payload), e
+ return False
+
+
+==========
+References
+==========
+
+_`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide
+
+_`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide
+
+_`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard Brahmaputra dashboard
+
+_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
+
+_`[5]`: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
+
+_`[6]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api
+
+_`[7]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts
+
+_`[8]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest
+
+_`[9]`: http://testresults.opnfv.org/reporting/
+
+_`[10]`: https://wiki.opnfv.org/opnfv_functional_testing
+
+_`[11]`: https://docs.python.org/2/howto/regex.html
+
+_`[12]`: https://regex101.com/
+
+_`[13]`: http://testresults.opnfv.org/test/api/v1/projects/functest/cases
+
+_`[14]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-daily.sh
+
+_`[15]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/README.rst
+
+_`[16]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py
+
+_`[17]`: http://artifacts.opnfv.org/releng/docs/testapi.html
+
+OPNFV main site: http://www.opnfv.org
+
+OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing
+
+IRC support chan: #opnfv-functest
+
+_`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
+
+_`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
+
+_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst
new file mode 100644
index 000000000..d1cdbdd8f
--- /dev/null
+++ b/docs/testing/developer/internship/security_group/index.rst
@@ -0,0 +1,70 @@
+=======
+License
+=======
+
+Functest Docs are licensed under a Creative Commons Attribution 4.0
+International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+
+==================================
+Functest Security group test cases
+==================================
+
+Author: Girish Sukhatankar
+mentors: D.Blaisonneau, J.Lausuch, M.Richomme
+
+Abstract
+========
+
+
+Version history
+===============
+
++------------+----------+------------------+------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+------------------------+
+| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the |
+| | | (Orange) | Internship |
++------------+----------+------------------+------------------------+
+
+
+Overview:
+=========
+
+
+
+
+Problem Statement:
+------------------
+
+
+
+Curation Phase
+--------------
+
+
+
+
+
+Schedule:
+=========
+
+
+
++--------------------------+------------------------------------------+
+| **Date** | **Comment** |
+| | |
++--------------------------+------------------------------------------+
+| December - January | ........ |
++--------------------------+------------------------------------------+
+| January - february | ........ |
++--------------------------+------------------------------------------+
+
+
+References:
+===========
+
+.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest
+
diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst
new file mode 100644
index 000000000..6a1cde7df
--- /dev/null
+++ b/docs/testing/developer/internship/testapi_evolution/index.rst
@@ -0,0 +1,237 @@
+=======
+License
+=======
+
+Functest Docs are licensed under a Creative Commons Attribution 4.0
+International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+
+==================
+Test API evolution
+==================
+
+Author: Sakala Venkata Krishna Rohit
+Mentors: S. Feng, J.Lausuch, M.Richomme
+
+Abstract
+========
+
+The testapi is used by all the test opnfv projects to report results.
+It is also used to declare projects, test cases and labs. A major refactoring
+has been done in Colorado with the introduction of swagger. The testapi is defined in Functest
+developer guide. The purpose of this project is to add more features to the testapi that automate
+the tasks that are done manually now, though there are tasks other than automation.
+
+Version history
+===============
+
++------------+----------+------------------+------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+------------------------+
+| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the |
+| | | (Orange) | Internship |
++------------+----------+------------------+------------------------+
+| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship |
+| | | (IIIT Hyderabad) | |
++------------+----------+------------------+------------------------+
+
+Overview:
+=========
+
+The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_.
+The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme.
+The link to the patches submitted is `[2]`_. The internship was successfully completed and the
+documentation is as follows.
+
+Problem Statement:
+------------------
+
+The problem statement could be divided into pending features that needed to be added into testapi
+repo. The following were to be accomplished within the internship time frame.
+
+* **Add verification jenkins job for the testapi code**
+ The purpose of this job is to verify whehter the unit tests are successful or not with the
+ inclusion of the patchset submitted.
+
+* **Automatic update of opnfv/testapi docker image**
+ The docker image of testapi is hosted in the opnfv docker hub. To ensure that the testapi image
+ is always updated with the repository, automatic updation of the image is necessary and a job
+ is triggered whenever a new patch gets merged.
+
+* **Automation deployment of testresults.opnfv.org/test/ website**
+ In the same manner as the docker image of testapi is updated, the testapi website needs to be
+ in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the
+ updation of the testresults website.
+
+* **Generate static documentation of testapi calls**
+ The purpose of this is to give an static/offline view of testapi. If someone wants to have a
+ look at the Restful apis of testapi, he/she does't need to go to the website, he can download
+ a html page and view it anytime.
+
+* **Backup MongoDB of testapi**
+ The mongoDB needs to be backed up every week. Till now it was done manually, but due to this
+ internship, it is now automated using a jenkins job.
+
+* **Add token based authorization to the testapi calls**
+ The token based authorization was implemented to ensure that only ci_pods could access the
+ database. Authentication has been added to only delete/put/post requests.
+
+Curation Phase:
+---------------
+
+The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar
+with the testapi code and functionality and propose the solutions/tools for the tasks mentioned
+above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static
+documentaion.
+
+Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic
+deployment task involves accessing a remote server from inside the jenkins build. The deployment
+had to be done only after the docker image update is done. For these constraints to satisfy, a
+multijob jenkins job was choosen instead of a freestyle job.
+
+Important Links:
+----------------
+
+* MongoDB Backup Link - `[4]`_
+* Static Documentation - `[5]`_
+* TestAPI Token addition to ci_pods - `[6]`_
+
+Schedule:
+=========
+
+The progress and completion of the tasks is described in the below table.
+
++--------------------------+------------------------------------------+
+| **Date** | **Comment** |
+| | |
++--------------------------+------------------------------------------+
+| Nov 14th - Dec 31st | Understand Testapi code and the |
+| | requirements. |
++--------------------------+------------------------------------------+
+| Jan 1st - Jan 7th | Add jenkins job to create static |
+| | documentation and write build scripts. |
++--------------------------+------------------------------------------+
+| Jan 8th - Jan 21st | Add verification jenkins job for unit |
+| | tests. |
++--------------------------+------------------------------------------+
+| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup |
+| | |
++--------------------------+------------------------------------------+
+| Jan 29th - Feb 11th | Enable automatic deployment of |
+| | testresults.opnfv.org/test/ |
++--------------------------+------------------------------------------+
+| Feb 12th - Feb 17th | Add token based authentication |
+| | |
++--------------------------+------------------------------------------+
+
+FAQ's
+=====
+
+This section lists the problems that I have faced and the understanding that I have acquired during
+the internship. This section may help other developers in solving any errors casused because of the
+code written as a part of this internship.
+
+
+Test Api
+--------
+
+What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If in the setup.cfg, it is defined as
+
+[files]
+data_files =
+etc/a.conf = etc/a.conf.sample
+
+then it ends up installed in the /usr/etc/. With this configuration, it would be installed
+correctly within a venv. but when it is defined as
+
+[files]
+data_files =
+/etc/a.conf = etc/a.conf.sample
+
+then it ends up installed on the root of the filesystem instead of properly be installed within the
+venv.
+
+Which attribute does swagger-codegen uses as the title in the generation of document generation ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It uses the nickname of the api call in swagger as the title in the generation of the document
+generation.
+
+Does swagger-codegen take more than one yaml file as input ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml
+file, one needs to merge them and give it as an input keeping mind the swagger specs.
+
+
+Jenkins & JJB
+-------------
+
+Which scm macro is used for verification jenkins jobs ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for
+verification jenkins job.
+
+Does the virtualenv created in one build script exists in other build scripts too ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No, the virtualenv created in one build script only exists in that build script/shell.
+
+What parameters are needed for the scm macros ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Project and Branch are the two parameters needed for scm macros.
+
+What is the directory inside the jenkins build ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give
+you all the contents of the directory.
+
+How to include a bash script in jenkins job yaml file ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An example might be apt here as an answer.
+
+builders:
+ - shell:
+ !include-raw: include-raw001-hello-world.sh
+
+
+How do you make a build server run on a specific machine ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each
+server, one can use those parameter macros.
+Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses
+SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE
+defined.
+
+What job style should be used when there is a situation like one build should trigger other builds
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+or when different build scripts need to be run on different machines ?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+MultiJob style should be used as it has phases where each phase can be taken as a build scipt and
+can have its own parameters by which one can define the SLAVE_LABEL parameter.
+
+References:
+===========
+
+_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution
+
+_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22
+
+_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing
+
+_`[4]` : http://artifacts.opnfv.org/testapibackup.html
+
+_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html
+
+_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization
diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst
new file mode 100644
index 000000000..f969aa72d
--- /dev/null
+++ b/docs/testing/developer/internship/unit_tests/index.rst
@@ -0,0 +1,70 @@
+=======
+License
+=======
+
+Functest Docs are licensed under a Creative Commons Attribution 4.0
+International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+
+===================
+Functest Unit tests
+===================
+
+Author: Ashish Kumar
+Mentors: H.Yao, J.Lausuch, M.Richomme
+
+Abstract
+========
+
+
+Version history
+===============
+
++------------+----------+------------------+------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+------------------------+
+| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the |
+| | | (Orange) | Internship |
++------------+----------+------------------+------------------------+
+
+
+Overview:
+=========
+
+
+
+
+Problem Statement:
+------------------
+
+
+
+Curation Phase
+--------------
+
+
+
+
+
+Schedule:
+=========
+
+
+
++--------------------------+------------------------------------------+
+| **Date** | **Comment** |
+| | |
++--------------------------+------------------------------------------+
+| December - January | ........ |
++--------------------------+------------------------------------------+
+| January - february | ........ |
++--------------------------+------------------------------------------+
+
+
+References:
+===========
+
+.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests
+
diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst
new file mode 100644
index 000000000..df7633391
--- /dev/null
+++ b/docs/testing/developer/internship/vnf_catalog/index.rst
@@ -0,0 +1,170 @@
+=======
+License
+=======
+
+Functest Docs are licensed under a Creative Commons Attribution 4.0
+International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+
+=======================
+Open Source VNF Catalog
+=======================
+
+Author: Kumar Rishabh
+Mentors: B.Souville, M.Richomme, J.Lausuch
+
+Abstract
+========
+
+
+
+Version hissory
+===============
+
++------------+----------+------------------+------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+------------------------+
+| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the |
+| | | (Orange) | Internship |
++------------+----------+------------------+------------------------+
+
+
+Overview:
+=========
+
+
+This project aims to create an Open Source catalog for reference and
+classification of Virtual Network Functions (VNFs)s available on
+Internet. The classification method proposed will be in sync with the
+requirements of Telcos active in NFV landscape. The project aims to have
+running web platform similar to [1] by the mid of internship (2nd week
+of March). By the penultimate month of internship I aim to have fully
+functional implementation of an Open Source VNF in functest.
+
+
+Problem Statement:
+------------------
+
+OPNFV aims to be the reference platform for development,
+standardization and integration of Open Source NFV components across
+various Open Source Platforms. It mainly deals with the infrastructure
+through the Network Function Virtualization Infrastructure (NFVI) and
+Virtual Infrastructure manager (VIM). The MANO (Management and
+orchestration) stacks have been introduced recently. VNFs are not
+directly in OPNFV scope, however VNFs are needed to test and qualify the
+infrastructure. In this regard having a common curated Open Source
+Reference VNF catalog would be of immense importance to community.
+
+Since major focus of OPNFV is Telcos, a curated platform targeted from
+industry point of view would be very useful. We plan to divide the
+entire project into three major phases(with some iterative improvements
+and overlaps)
+
+
+Curation Phase
+--------------
+This phase pertains to studying various Open Source VNFs available and
+classification of them based on certain parameters. The parameters that
+I currently have in mind are:
+ * Developer Metrics: These pertain to repo characteristics of VNF under
+ study
+ * Usage Statistics - Activity, Number of Commits, stars
+ * Maturity Statistics - For instance if an NFV community decides code
+ coverage is important for them, it shows the NFV community is serious
+ about taking the project forward
+ * Technical Tagging: These are the tags that pertain to technical
+ characteristics of a VNF
+ * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or
+ SaaS layer or is an hybrid of two/all.
+ * Generic Use Cases - This in my opinion is the broadest
+ classification category. For instance a VNF could be built with a
+ broad idea of powering IOT devices at home or from usage perspective
+ of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_
+ * Fields of Application
+ * Library Status - Whether APIs are standardized, support RESTful
+ services.
+ * Dependency Forwarding Graph - This is pretty complex tagging
+ mechanism. It essentially tries to establish a graph relationship
+ between the VNFs (elementary VNFs are used in Service Function
+ Chaining chains such as Firewall, DPI, content enrichment,..). In my
+ opinion this is useful immensely. This will allow users to go to
+ platform and ask a question like - “I have this X tech stack to
+ support, Y and Z are my use cases, which NFVs should I use to support
+ this.
+ * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for
+ the platform. This will allow users to score an NFV on certain
+ parameters, may be post comments.
+
+**I plan to use the above three scores and evolve cumulative score which
+will be displayed next to each of the NFV on the platform.**
+
+ * Platform building phase - This will involve erecting a Web Platform
+ which will be similar to this `[1]`_. I am decently familiar with
+ Django and hence I will write the platform in Django. There are two
+ action plans that I have in mind right now. Either I can start writing
+ the platform simultaneously which will help keep track of my progress
+ or I can write the platform after 1.5 - 2 months into the internship.
+ Either way I aim to have the Web Platform ready by March 12.
+
+ * Functest VNF implementation phase - This is the last phase that will
+ involve writing a fully functional implementation of an Open Source VNF
+ into Functest. I will undertake this after I am 3 months into the
+ internship. I have a decent familiarity with python and hence I think
+ it shouldn’t be too difficult. I need to decide how complex the VNFI
+ should undertake this exercise for (e.g. AAA such as free radius sounds
+ relatively easy, vCDN is much more challenging).
+ This will be decided in consent with my mentors.
+
+
+
+
+Schedule:
+=========
+I plan to take this project in 6 months time frame as I want to use it
+as a chance to read more about NFVs in particular and SDN in general
+
+
++--------------------------+------------------------------------------+
+| **Date** | **Comment** |
+| | |
++--------------------------+------------------------------------------+
+| December 12 - January 12 | Study the above mentioned metrics |
+| | Decide which of them are important for |
+| | community (and which are not). |
++--------------------------+------------------------------------------+
+| January 12 - January 27 | Make a database for the above studied |
+| | metrics and evolve it further based on |
+| | Mentors’ input. + associated API |
++--------------------------+------------------------------------------+
+| January 27 - February 5 | Compile the data collected above and make|
+| | it public. Although I can keep everything|
+| | public from the beginning too. My |
+| | rationale of not making the entire data |
+| | public in initial stage as the errors |
+| | caused by me could be misleading for |
+| | developers. |
++--------------------------+------------------------------------------+
+| February 5 - March 5 | Erect the Web Platform and release it |
+| | for restricted group for alpha testing. |
++--------------------------+------------------------------------------+
+| March 5 - March 12 | Make it public. Release it to public for |
+| | beta testing. Fix Bugs. |
++--------------------------+------------------------------------------+
+| March 12 - April 12 | Start working on implementation of an |
+| | Open Source VNF in Functest. |
++--------------------------+------------------------------------------+
+| April 12 - May 12 | I will decided what to do here based on |
+| | discussion with mentors. |
++--------------------------+------------------------------------------+
+
+
+References:
+===========
+
+.. _`[1]` : Openhub: https://www.openhub.net/explore/projects
+
+.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf
+
+.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog
diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst
new file mode 100644
index 000000000..08e089c2b
--- /dev/null
+++ b/docs/testing/user/configguide/configguide.rst
@@ -0,0 +1,821 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+
+Pulling the Docker image
+------------------------
+Pull the Functest Docker image ('opnfv/functest') from the public
+dockerhub registry under the OPNFV account: [dockerhub_], with the
+following docker command::
+
+ docker pull opnfv/functest:<TagIdentifier>
+
+where <TagIdentifier> identifies a release of the Functest docker
+container image in the public dockerhub registry. There are many tags
+created automatically by the CI mechanisms, and you must ensure you
+pull an image with the **correct tag** to match the OPNFV software
+release installed in your environment. All available tagged images can
+be seen from location [FunctestDockerTags_]. For example, when running
+on the first official release of the OPNFV Colorado system platform,
+tag "colorado.1.0" is needed. Pulling other tags might cause some
+problems while running the tests.
+Docker images pulled without a tag specifier bear the implicitly
+assigned label "latest". If you need to specifically pull the latest
+Functest docker image, then omit the tag argument::
+
+ docker pull opnfv/functest
+
+After pulling the Docker image, check that it is available with the
+following docker command::
+
+ [functester@jumphost ~]$ docker images
+ REPOSITORY TAG IMAGE ID CREATED SIZE
+ opnfv/functest latest 8cd6683c32ae 2 weeks ago 1.321 GB
+ opnfv/functest danube.1.0 13fa54a1b238 4 weeks ago 1.29 GB
+ opnfv/functest colorado.1.0 94b78faa94f7 9 weeks ago 968 MB
+
+The Functest docker container environment can -in principle- be also
+used with non-OPNFV official installers (e.g. 'devstack'), with the
+**disclaimer** that support for such environments is outside of the
+scope and responsibility of the OPNFV project.
+
+Accessing the Openstack credentials
+-----------------------------------
+OpenStack credentials are mandatory and can be retrieved in different
+ways. From inside the running Functest docker container the
+"functest env prepare" command will automatically look for the
+Openstack credentials file "/home/opnfv/functest/conf/openstack.creds"
+and retrieve it unless the file already exists. This Functest
+environment preparation step is described later in this document.
+
+WARNING: When the installer type is "joid" you have to have the
+credentials file inside the running container **before** initiating the
+functest environment preparation. For that reason you have to choose
+either one of the options below, since the automated copying does not
+work for "joid".
+
+You can also specifically pass in the needed file prior to running the
+environment preparation either:
+
+ * by using the -v option when creating the Docker container. This is
+ referred to in docker documentation as "Bind Mounting". See the
+ usage of this parameter in the following chapter.
+ * or creating a local file '/home/opnfv/functest/conf/openstack.creds'
+ inside the running container with the credentials in it. Consult
+ your installer guide for further details. This is however not
+ instructed in this document.
+
+NOTE: When the installer type is "fuel" and virtualized deployment
+is used, there you have to explicitly fetch the credentials file
+executing the following sequence
+
+ #. Create a container as described in next chapter but do not
+ "Bind Mount" the credentials
+ #. Log in to container and execute the following command. Replace
+ the IP with installer address after the "-a" parameter::
+
+ $REPOS_DIR/releng/utils/fetch_os_creds.sh \
+ -d /home/opnfv/functest/conf/openstack.creds \
+ -i fuel \
+ -a 10.20.0.2 \
+ -v
+ ( -d specifies the full path to the Openstack credential file
+ -i specifies the INSTALLER_TYPE
+ -a specifies the INSTALLER_IP
+ -v indicates a virtualized environment and takes no arguments )
+
+ #. Continue with your testing, initiate functest environment
+ preparation, run tests etc.
+
+In proxified environment you may need to change the credentials file.
+There are some tips in chapter: `Proxy support`_
+
+Functest Docker parameters
+--------------------------
+This chapter explains how to run a container for executing functest
+test suites. Numbered list below explains some details of the
+recommended parameters for invoking docker container
+
+ #. It is a good practice to assign a precise container name through
+ the **--name** option.
+
+ #. Assign parameter for installer type::
+
+ -e "INSTALLER_TYPE=<type>"
+ # Use one of following apex, compass, fuel or joid
+
+ #. Functest needs to know the IP of some installers::
+
+ -e "INSTALLER_IP=<Specific IP Address>"
+
+ This IP is needed to fetch RC file from deployment, fetch logs, ...
+ If not provided, there is no way to fetch the RC file. It must be
+ provided manually as a volume
+
+ #. Credentials for accessing the Openstack.
+ Most convenient way of passing them to container is by having a
+ local copy of the credentials file in Jumphost and then using the
+ **-v** option. In the example we have local file by the name of
+ "overcloudrc" and we are using that as an argument::
+
+ -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds
+
+ The credentials file needs to exist in the Docker container
+ under the path: '/home/opnfv/functest/conf/openstack.creds'.
+
+ **WARNING:** If you are using the Joid installer, you must pass the
+ credentials using the **-v** option:
+ -v /var/lib/jenkins/admin-openrc:/home/opnfv/functest/conf/openstack.creds.
+ See the section `Accessing the Openstack credentials`_ above.
+
+ #. Passing deployment scenario
+ When running Functest against any of the supported OPNFV scenarios,
+ it is recommended to include also the environment variable
+ **DEPLOY_SCENARIO**. The **DEPLOY_SCENARIO** environment variable
+ is passed with the format::
+
+ -e "DEPLOY_SCENARIO=os-<controller>-<nfv_feature>-<ha_mode>"
+ where:
+ os = OpenStack (No other VIM choices currently available)
+ controller is one of ( nosdn | odl_l2 | odl_l3 | onos | ocl)
+ nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature )
+ If several features are pertinent then use the underscore
+ character '_' to separate each feature (e.g. ovs_kvm)
+ 'nofeature' indicates no NFV feature is deployed
+ ha_mode (high availability) is one of ( ha | noha )
+
+ **NOTE:** Not all possible combinations of "DEPLOY_SCENARIO" are
+ supported. The name passed in to the Functest Docker container
+ must match the scenario used when the actual OPNFV platform was
+ deployed. See release note to see the list of supported scenarios.
+
+ **NOTE:** The scenario name is mainly used to automatically detect
+ if a test suite is runnable or not (e.g. it will prevent ONOS test suite
+ to be run on ODL scenarios). If not set, Functest will try to run the
+ default test cases that might not include SDN controller or a specific
+ feature
+
+ **NOTE:** A HA scenario means that 3 OpenStack controller nodes are
+ deployed. It does not necessarily mean that the whole system is HA. See
+ installer release notes for details.
+
+
+Putting all above together, when using installer 'fuel' and an invented
+INSTALLER_IP of '10.20.0.2', the recommended command to create the
+Functest Docker container is as follows::
+
+ docker run --name "FunctestContainer" -it \
+ -e "INSTALLER_IP=10.20.0.2" \
+ -e "INSTALLER_TYPE=fuel" \
+ -e "DEPLOY_SCENARIO=os-odl_l2-ovs_kvm-ha" \
+ -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \
+ opnfv/functest /bin/bash
+
+After the *run* command, a new prompt appears which means that we are inside
+the container and ready to move to the next step.
+
+For tips on how to set up container with installer Apex, see chapter
+`Apex Installer Tips`_.
+
+Finally, three additional environment variables can also be passed in
+to the Functest Docker Container, using the -e
+"<EnvironmentVariable>=<Value>" mechanism. The first two of these are
+only relevant to Jenkins CI invoked testing and **should not be used**
+when performing manual test scenarios::
+
+ -e "NODE_NAME=<Test POD Name>" \
+ -e "BUILD_TAG=<Jenkins Build Tag>" \
+ -e "CI_DEBUG=<DebugTraceValue>"
+ where:
+ <Test POD Name> = Symbolic name of the POD where the tests are run.
+ Visible in test results files, which are stored
+ to the database. This option is only used when
+ tests are activated under Jenkins CI control.
+ It indicates the POD/hardware where the test has
+ been run. If not specified, then the POD name is
+ defined as "Unknown" by default.
+ DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
+ <Jenkins Build tag> = Symbolic name of the Jenkins Build Job.
+ Visible in test results files, which are stored
+ to the database. This option is only set when
+ tests are activated under Jenkins CI control.
+ It enables the correlation of test results,
+ which
+ are independently pushed to the results datbase
+ from different Jenkins jobs.
+ DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS.
+ <DebugTraceValue> = "true" or "false"
+ Default = "false", if not specified
+ If "true" is specified, then additional debug trace
+ text can be sent to the test results file / log files
+ and also to the standard console output.
+
+Apex Installer Tips
+-------------------
+Some specific tips are useful for the Apex Installer case. If not using
+Apex Installer; ignore this section.
+
+In case of Triple-O based installer (like Apex) the docker container
+needs to connect to the installer VM, so it is then required that some
+known SSH keys are present in docker container. Since the Jumphost root
+SSH keys are already known, easiest way is to use those using the
+'Bind mount' method. See below for sample parameter::
+
+ -v /root/.ssh/id_rsa:/root/.ssh/id_rsa
+
+ NOTE: You need the "sudo" when creating the container to access root
+ users ssh credentials even the docker command itself might not
+ require that.
+
+HINT! In case of Triple-O installers you can find value for the
+INSTALLER_IP parameter by executing command and note the returned IP
+address::
+
+ inst=$(sudo virsh list | grep -iEo "undercloud|instack")
+ sudo virsh domifaddr ${inst}
+
+ NOTE: In releases prior to Colorado, the name 'instack' was
+ used. Currently the name 'undercloud' is used.
+
+You can copy the credentials file from the "stack" users home directory
+in installer VM to Jumphost. Please check the correct IP from the
+command above. In the example below we are using invented IP address
+"192.168.122.89"::
+
+ scp stack@192.168.122.89:overcloudrc .
+
+Here is an example of the full docker command invocation for an Apex
+installed system, using latest Functest docker container, for
+illustration purposes::
+
+ sudo docker run -it --name "ApexFuncTestODL" \
+ -e "INSTALLER_IP=192.168.122.89" \
+ -e "INSTALLER_TYPE=apex" \
+ -e "DEPLOY_SCENARIO=os-odl_l2-nofeature-ha" \
+ -v /root/.ssh/id_rsa:/root/.ssh/id_rsa \
+ -v ~/overcloudrc:/home/opnfv/functest/conf/openstack.creds \
+ opnfv/functest /bin/bash
+
+Compass installer local development env usage Tips
+--------------------------------------------------
+In the compass-functest local test case check and development environment,
+in order to get openstack service inside the functest container, some
+parameters should be configured during container creation, which are
+hard to guess for freshman. This section will provide the guideline, the
+parameters values are defaults here, which should be adjusted according
+to the settings, the complete steps are given here so as not to appear
+too abruptly.
+
+1, Pull Functest docker image from public dockerhub::
+
+ docker pull opnfv/functest:<Tag>
+
+<Tag> here can be "brahmaputra.1.0", "colorado.1.0", etc.
+Tag omitted means the latest docker image::
+
+ docker pull opnfv/functest
+
+2, Functest Docker container creation
+
+To make a file used for the environment, such as 'functest-docker-env'::
+
+ OS_AUTH_URL=http://172.16.1.222:35357/v2.0
+ OS_USERNAME=admin
+ OS_PASSWORD=console
+ OS_TENANT_NAME=admin
+ OS_VOLUME_API_VERSION=2
+ OS_PROJECT_NAME=admin
+ INSTALLER_TYPE=compass
+ INSTALLER_IP=192.168.200.2
+ EXTERNAL_NETWORK=ext-net
+
+Note: please adjust the content according to the environment, such as
+'TENANT_ID' maybe used for some special cases.
+
+Then to create the Functest docker::
+
+ docker run --privileged=true --rm -t \
+ --env-file functest-docker-env \
+ --name <Functest_Container_Name> \
+ opnfv/functest:<Tag> /bin/bash
+
+3, To attach Functest container
+
+Before trying to attach the Functest container, the status can be checked by::
+
+ docker ps -a
+
+to attach the 'Up' status Functest container and start bash mode::
+
+ docker exec -it <Functest_Container_Name> bash
+
+4, Functest environemnt preparation and check
+
+To see the Section below `Preparing the Functest environment`_.
+
+Functest docker container directory structure
+---------------------------------------------
+Inside the Functest docker container, the following directory structure
+should now be in place::
+
+ `-- home
+ `-- opnfv
+ |-- functest
+ | |-- conf
+ | |-- data
+ | `-- results
+ `-- repos
+ |-- bgpvpn
+ |-- copper
+ |-- doctor
+ |-- domino
+ |-- functest
+ |-- kingbird
+ |-- odl_test
+ |-- onos
+ |-- ovno
+ |-- parser
+ |-- promise
+ |-- rally
+ |-- releng
+ |-- sdnvpn
+ |-- securityscanning
+ |-- sfc
+ |-- tempest
+ |-- vims_test
+ `-- vnfs
+
+Underneath the '/home/opnfv/' directory, the Functest docker container
+includes two main directories:
+
+ * The **functest** directory stores configuration files (e.g. the
+ OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'),
+ the **data** directory stores a 'cirros' test image used in some
+ functional tests and the **results** directory stores some temporary
+ result log files
+ * The **repos** directory holds various repositories. The directory
+ '/home/opnfv/repos/functest' is used to prepare the needed Functest
+ environment and to run the tests. The other repository directories
+ are used for the installation of the needed tooling (e.g. rally) or
+ for the retrieval of feature projects scenarios (e.g. promise)
+
+The structure under the **functest** repository can be described as
+follows::
+
+ . |-- INFO
+ |-- LICENSE
+ |-- requirements.txt
+ |-- run_unit_tests.sh
+ |-- setup.py
+ |-- test-requirements.txt
+ |-- commons
+ | |-- ims
+ | |-- mobile
+ | `--traffic-profile-guidelines.rst
+ |-- docker
+ | |-- Dockerfile
+ | |-- config_install_env.sh
+ | `-- docker_remote_api
+ |-- docs
+ | |-- com
+ | |-- configguide
+ | |-- devguide
+ | |-- images
+ | |-- internship
+ | |-- release-notes
+ | |-- results
+ | `--userguide
+ |-- functest
+ |-- __init__.py
+ |-- ci
+ | |-- __init__.py
+ | |-- check_os.sh
+ | |-- config_functest.yaml
+ | |-- config_patch.yaml
+ | |-- exec_test.sh
+ | |-- generate_report.py
+ | |-- prepare_env.py
+ | |-- run_tests.py
+ | |-- testcases.yaml
+ | |-- tier_builder.py
+ | `-- tier_handler.py
+ |-- cli
+ | |-- __init__.py
+ | |-- cli_base.py
+ | |-- commands
+ | |-- functest-complete.sh
+ | `-- setup.py
+ |-- core
+ | |-- __init__.py
+ | |-- feature_base.py
+ | |-- pytest_suite_runner.py
+ | |-- testcase_base.py
+ | |-- vnf_base.py
+ |-- opnfv_tests
+ | |-- __init__.py
+ | |-- features
+ | |-- mano
+ | |-- openstack
+ | |-- sdn
+ | |-- security_scan
+ | `-- vnf
+ |-- tests
+ | |-- __init__.py
+ | `-- unit
+ `-- utils
+ |-- __init__.py
+ |-- config.py
+ |-- constants.py
+ |-- env.py
+ |-- functest_constants.py
+ |-- functest_logger.py
+ |-- functest_utils.py
+ |-- openstack
+ |-- openstack_clean.py
+ |-- openstack_snapshot.py
+ |-- openstack_tacker.py
+ `-- openstack_utils.py
+
+
+ (Note: All *.pyc files removed from above list for brevity...)
+
+We may distinguish several directories, the first level has 4 directories:
+
+* **commons**: This directory is dedicated for storage of traffic
+ profile or any other test inputs that could be reused by any test
+ project.
+* **docker**: This directory includes the needed files and tools to
+ build the Funtest Docker image.
+* **docs**: This directory includes documentation: Release Notes,
+ User Guide, Configuration Guide and Developer Guide. Test results
+ are also located in a sub--directory called 'results'.
+* **functest**: This directory contains all the code needed to run
+ functest internal cases and OPNFV onboarded feature or VNF test cases.
+
+Functest directory has 6 directories:
+ * **ci**: This directory contains test structure definition files
+ (e.g <filename>.yaml) and bash shell/python scripts used to
+ configure and execute Functional tests. The test execution script
+ can be executed under the control of Jenkins CI jobs.
+ * **cli**: This directory holds the python based Functest CLI utility
+ source code, which is based on the Python 'click' framework.
+ * **core**: This directory holds the python based Functest core
+ source code. Three abstraction classes have been created to ease
+ the integration of internal, feature or vnf cases.
+ * **opnfv_tests**: This directory includes the scripts required by
+ Functest internal test cases and other feature projects test cases.
+ * **tests**: This directory includes the functest unit tests
+ * **utils**: this directory holds Python source code for some general
+ purpose helper utilities, which testers can also re-use in their
+ own test code. See for an example the Openstack helper utility:
+ 'openstack_utils.py'.
+
+Useful Docker commands
+----------------------
+When typing **exit** in the container prompt, this will cause exiting
+the container and probably stopping it. When stopping a running Docker
+container all the changes will be lost, there is a keyboard shortcut
+to quit the container without stopping it: <CTRL>-P + <CTRL>-Q. To
+reconnect to the running container **DO NOT** use the *run* command
+again (since it will create a new container), use the *exec* or *attach*
+command instead::
+
+ docker ps # <check the container ID from the output>
+ docker exec -ti <CONTAINER_ID> /bin/bash
+
+There are other useful Docker commands that might be needed to manage possible
+issues with the containers.
+
+List the running containers::
+
+ docker ps
+
+List all the containers including the stopped ones::
+
+ docker ps -a
+
+Start a stopped container named "FunTest"::
+
+ docker start FunTest
+
+Attach to a running container named "StrikeTwo"::
+
+ docker attach StrikeTwo
+
+It is useful sometimes to remove a container if there are some problems::
+
+ docker rm <CONTAINER_ID>
+
+Use the *-f* option if the container is still running, it will force to
+destroy it::
+
+ docker rm -f <CONTAINER_ID>
+
+Check the Docker documentation dockerdocs_ for more information.
+
+Preparing the Functest environment
+----------------------------------
+Once the Functest docker container is up and running, the required
+Functest environment needs to be prepared. A custom built **functest**
+CLI utility is available to perform the needed environment preparation
+action. Once the environment is prepared, the **functest** CLI utility
+can be used to run different functional tests. The usage of the
+**functest** CLI utility to run tests is described further in the
+Functest User Guide `OPNFV_FuncTestUserGuide`_
+
+Prior to commencing the Functest environment preparation, we can check
+the initial status of the environment. Issue the **functest env status**
+command at the prompt::
+
+ functest env status
+ Functest environment is not installed.
+
+ Note: When the Functest environment is prepared, the command will
+ return the status: "Functest environment ready to run tests."
+
+To prepare the Functest docker container for test case execution, issue
+the **functest env prepare** command at the prompt::
+
+ functest env prepare
+
+This script will make sure that the requirements to run the tests are
+met and will install the needed libraries and tools by all Functest
+test cases. It should be run only once every time the Functest docker
+container is started from scratch. If you try to run this command, on
+an already prepared enviroment, you will be prompted whether you really
+want to continue or not::
+
+ functest env prepare
+ It seems that the environment has been already prepared.
+ Do you want to do it again? [y|n]
+
+ (Type 'n' to abort the request, or 'y' to repeat the
+ environment preparation)
+
+
+To list some basic information about an already prepared Functest
+docker container environment, issue the **functest env show** at the
+prompt::
+
+ functest env show
+ +======================================================+
+ | Functest Environment info |
+ +======================================================+
+ | INSTALLER: apex, 192.168.122.89 |
+ | SCENARIO: os-odl_l2-nofeature-ha |
+ | POD: localhost |
+ | GIT BRANCH: master |
+ | GIT HASH: 5bf1647dec6860464eeb082b2875798f0759aa91 |
+ | DEBUG FLAG: false |
+ +------------------------------------------------------+
+ | STATUS: ready |
+ +------------------------------------------------------+
+
+ Where:
+
+ INSTALLER: Displays the INSTALLER_TYPE value
+ - here = "apex"
+ and the INSTALLER_IP value
+ - here = "192.168.122.89"
+ SCENARIO: Displays the DEPLOY_SCENARIO value
+ - here = "os-odl_l2-nofeature-ha"
+ POD: Displays the value passed in NODE_NAME
+ - here = "localhost"
+ GIT BRANCH: Displays the git branch of the OPNFV Functest
+ project repository included in the Functest
+ Docker Container.
+ - here = "master"
+ (In first official colorado release
+ would be "colorado.1.0")
+ GIT HASH: Displays the git hash of the OPNFV Functest
+ project repository included in the Functest
+ Docker Container.
+ - here = "5bf1647dec6860464eeb082b2875798f0759aa91"
+ DEBUG FLAG: Displays the CI_DEBUG value
+ - here = "false"
+
+ NOTE: In Jenkins CI runs, an additional item "BUILD TAG"
+ would also be listed. The valaue is set by Jenkins CI.
+
+Finally, the **functest** CLI has a **--help** options:
+
+Some examples::
+
+ functest --help Usage: functest [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ --version Show the version and exit.
+ -h, --help Show this message and exit.
+
+ Commands:
+ env
+ openstack
+ testcase
+ tier
+
+ functest env --help
+ Usage: functest env [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ -h, --help Show this message and exit.
+
+ Commands:
+ prepare Prepares the Functest environment.
+ show Shows information about the current...
+ status Checks if the Functest environment is ready...
+
+Checking Openstack and credentials
+----------------------------------
+It is recommended and fairly straightforward to check that Openstack
+and credentials are working as expected.
+
+Once the credentials are there inside the container, they should be
+sourced before running any Openstack commands::
+
+ source /home/opnfv/functest/conf/openstack.creds
+
+After this, try to run any OpenStack command to see if you get any
+output, for instance::
+
+ openstack user list
+
+This will return a list of the actual users in the OpenStack
+deployment. In any other case, check that the credentials are sourced::
+
+ env|grep OS_
+
+This command must show a set of environment variables starting with
+*OS_*, for example::
+
+ OS_REGION_NAME=RegionOne
+ OS_DEFAULT_DOMAIN=default
+ OS_PROJECT_NAME=admin
+ OS_PASSWORD=admin
+ OS_AUTH_STRATEGY=keystone
+ OS_AUTH_URL=http://172.30.10.3:5000/v2.0
+ OS_USERNAME=admin
+ OS_TENANT_NAME=admin
+ OS_ENDPOINT_TYPE=internalURL
+ OS_NO_CACHE=true
+
+If the OpenStack command still does not show anything or complains
+about connectivity issues, it could be due to an incorrect url given to
+the OS_AUTH_URL environment variable. Check the deployment settings.
+
+SSL Support
+-----------
+If you need to connect to a server that is TLS-enabled (the auth URL
+begins with "https") and it uses a certificate from a private CA or a
+self-signed certificate, then you will need to specify the path to an
+appropriate CA certificate to use, to validate the server certificate
+with the environment variable OS_CACERT::
+
+ echo $OS_CACERT
+ /etc/ssl/certs/ca.crt
+
+However, this certificate does not exist in the container by default.
+It has to be copied manually from the OpenStack deployment. This can be
+done in 2 ways:
+
+ #. Create manually that file and copy the contents from the OpenStack
+ controller.
+ #. (Recommended) Add the file using a Docker volume when starting the
+ container::
+
+ -v <path_to_your_cert_file>:/etc/ssl/certs/ca.cert
+
+You might need to export OS_CACERT environment variable inside the
+container::
+
+ export OS_CACERT=/etc/ssl/certs/ca.crt
+
+Certificate verification can be turned off using OS_INSECURE=true. For
+example, Fuel uses self-signed cacerts by default, so an pre step would
+be::
+
+ export OS_INSECURE=true
+
+Proxy support
+-------------
+If your Jumphost node is operating behind a http proxy, then there are
+2 places where some special actions may be needed to make operations
+succeed:
+
+ #. Initial installation of docker engine First, try following the
+ official Docker documentation for Proxy settings. Some issues were
+ experienced on CentOS 7 based Jumphost. Some tips are documented
+ in section: `Docker Installation on CentOS behind http proxy`_
+ below.
+
+ #. Execution of the Functest environment preparation inside the
+ created docker container Functest needs internet access to
+ download some resources for some test cases. This might not
+ work properly if the Jumphost is connecting to internet
+ through a http Proxy.
+
+If that is the case, make sure the resolv.conf and the needed
+http_proxy and https_proxy environment variables, as well as the
+'no_proxy' environment variable are set correctly::
+
+ # Make double sure that the 'no_proxy=...' line in the
+ # 'openstack.creds' file is commented out first. Otherwise, the
+ # values set into the 'no_proxy' environment variable below will
+ # be ovewrwritten, each time the command
+ # 'source ~/functest/conf/openstack.creds' is issued.
+
+ cd ~/functest/conf/
+ sed -i 's/export no_proxy/#export no_proxy/' openstack.creds
+ source ./openstack.creds
+
+ # Next calculate some IP addresses for which http_proxy
+ # usage should be excluded:
+
+ publicURL_IP=$(echo $OS_AUTH_URL | grep -Eo "([0-9]+\.){3}[0-9]+")
+
+ adminURL_IP=$(openstack catalog show identity | \
+ grep adminURL | grep -Eo "([0-9]+\.){3}[0-9]+")
+
+ export http_proxy="<your http proxy settings>"
+ export https_proxy="<your https proxy settings>"
+ export no_proxy="127.0.0.1,localhost,$publicURL_IP,$adminURL_IP"
+
+ # Ensure that "git" uses the http_proxy
+ # This may be needed if your firewall forbids SSL based git fetch
+ git config --global http.sslVerify True
+ git config --global http.proxy <Your http proxy settings>
+
+Validation check: Before running **'functest env prepare'** CLI command,
+make sure you can reach http and https sites from inside the Functest
+docker container.
+
+For example, try to use the **nc** command from inside the functest
+docker container::
+
+ nc -v opnfv.org 80
+ Connection to opnfv.org 80 port [tcp/http] succeeded!
+
+ nc -v opnfv.org 443
+ Connection to opnfv.org 443 port [tcp/https] succeeded!
+
+Note: In a Jumphost node based on the CentOS family OS, the **nc**
+commands might not work. You can use the **curl** command instead.
+
+ curl http://www.opnfv.org:80
+ <HTML><HEAD><meta http-equiv="content-type"
+ .
+ .
+ </BODY></HTML>
+
+ curl https://www.opnfv.org:443
+ <HTML><HEAD><meta http-equiv="content-type"
+ .
+ .
+ </BODY></HTML>
+
+ (Ignore the content. If command returns a valid HTML page, it proves
+ the connection.)
+
+Docker Installation on CentOS behind http proxy
+-----------------------------------------------
+This section is applicable for CentOS family OS on Jumphost which
+itself is behind a proxy server. In that case, the instructions below
+should be followed **before** installing the docker engine::
+
+ 1) # Make a directory '/etc/systemd/system/docker.service.d'
+ # if it does not exist
+ sudo mkdir /etc/systemd/system/docker.service.d
+
+ 2) # Create a file called 'env.conf' in that directory with
+ # the following contents:
+ [Service]
+ EnvironmentFile=-/etc/sysconfig/docker
+
+ 3) # Set up a file called 'docker' in directory '/etc/sysconfig'
+ # with the following contents:
+ HTTP_PROXY="<Your http proxy settings>"
+ HTTPS_PROXY="<Your https proxy settings>"
+ http_proxy="${HTTP_PROXY}"
+ https_proxy="${HTTPS_PROXY}"
+
+ 4) # Reload the daemon
+ systemctl daemon-reload
+
+ 5) # Sanity check - check the following docker settings:
+ systemctl show docker | grep -i env
+
+ Expected result:
+ ----------------
+ EnvironmentFile=/etc/sysconfig/docker (ignore_errors=yes)
+ DropInPaths=/etc/systemd/system/docker.service.d/env.conf
+
+Now follow the instructions in [`InstallDockerCentOS`_] to download
+and install the **docker-engine**. The instructions conclude with a
+"test pull" of a sample "Hello World" docker container. This should now
+work with the above pre-requisite actions.
+
+.. _dockerdocs: https://docs.docker.com/
+.. _dockerhub: https://hub.docker.com/r/opnfv/functest/
+.. _Proxy: https://docs.docker.com/engine/admin/systemd/#http-proxy
+.. _FunctestDockerTags: https://hub.docker.com/r/opnfv/functest/tags/
+.. _InstallDockerCentOS: https://docs.docker.com/engine/installation/linux/centos/
+.. _OPNFV_FuncTestUserGuide: http://artifacts.opnfv.org/functest/docs/userguide/index.html
diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst
new file mode 100644
index 000000000..f12739e31
--- /dev/null
+++ b/docs/testing/user/configguide/index.rst
@@ -0,0 +1,296 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+==================================
+OPNFV FUNCTEST Configuration Guide
+==================================
+
+.. toctree::
+ :numbered:
+ :maxdepth: 2
+
+Version history
+===============
+
++------------+----------+------------------+----------------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+----------------------------------+
+| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
+| | | Column Gaynor | |
++------------+----------+------------------+----------------------------------+
+| 2017-01-19 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
+| | | | * update testcase list |
+| | | | * update docker command |
++------------+----------+------------------+----------------------------------+
+
+Introduction
+============
+This document describes how to install and configure Functest in OPNFV.
+The Functest CLI is utilized during the Functest environment preparation
+step. The given example commands should work in both virtual and bare
+metal cases alike.
+
+High level architecture
+-----------------------
+
+The high level architecture of Functest within OPNFV can be described as
+follows::
+
+ CIMC/Lights+out management Admin Mgmt/API Public Storage Private
+ PXE
+ + + + + + +
+ | | | | | |
+ | +----------------------------+ | | | | |
+ | | | | | | | |
+ +-----+ Jumphost | | | | | |
+ | | +--------+ | | | |
+ | | | | | | | |
+ | | +--------------------+ | | | | | |
+ | | | | | | | | | |
+ | | | Tools | +----------------+ | | |
+ | | | - Rally | | | | | | |
+ | | | - Robot | | | | | | |
+ | | | - TestON | | | | | | |
+ | | | | |-------------------------+ | |
+ | | | Testcases | | | | | | |
+ | | | - VIM | | | | | | |
+ | | | -- healthcheck | | | | | | |
+ | | | -- vPing_ssh | | | | | | |
+ | | | -- vPing_userdata | | | | | | |
+ | | | -- SNAPS_cases | | | | | | |
+ | | | -- Tempest_smoke | | | | | | |
+ | | | -- Rally_sanity | | | | | | |
+ | | | -- Tempest_full | | | | | | |
+ | | | -- Rally_full | | | | | | |
+ | | | | | | | | | |
+ | | | - SDN Controller | | | | | | |
+ | | | -- odl | | | | | | |
+ | | | -- onos | | | | | | |
+ | | | | | | | | | |
+ | | | - Features | | | | | | |
+ | | | | | | | | | |
+ | | | - VNF | | | | | | |
+ | | | | | | | | | |
+ | | +--------------------+ | | | | | |
+ | | Functest Docker + | | | | |
+ | | | | | | | |
+ | | | | | | | |
+ | | | | | | | |
+ | +----------------------------+ | | | | |
+ | | | | | |
+ | +----------------+ | | | | |
+ | | 1 | | | | | |
+ +----+ +--------------+-+ | | | | |
+ | | | 2 | | | | | |
+ | | | +--------------+-+ | | | | |
+ | | | | 3 | | | | | |
+ | | | | +--------------+-+ | | | | |
+ | | | | | 4 | | | | | |
+ | +-+ | | +--------------+-+ | | | | |
+ | | | | | 5 +-------------+ | | | |
+ | +-+ | | nodes for | | | | | |
+ | | | | deploying +---------------------+ | | |
+ | +-+ | OPNFV | | | | | |
+ | | | +------------------------------+ | |
+ | +-+ SUT | | | | | |
+ | | +--------------------------------------+ |
+ | | | | | | | |
+ | | +----------------------------------------------+
+ | +----------------+ | | | | |
+ | | | | | |
+ + + + + + +
+ SUT = System Under Test
+
+All the libraries and dependencies needed by all of the Functest tools
+are pre-installed into the Docker image. This allows running Functest
+on any platform on any Operating System.
+
+The automated mechanisms inside the Functest Docker container will:
+
+ * Retrieve OpenStack credentials
+ * Prepare the environment according to the SUT
+ * Perform the appropriate functional tests
+ * Push the test results into the OPNFV test result database
+
+This Docker image can be integrated into CI or deployed independently.
+
+Please note that the Functest Docker container has been designed for
+OPNFV, however, it would be possible to adapt it to any VIM + controller
+environment, since most of the test cases are integrated from upstream
+communities.
+
+The functional test cases are described in the Functest User Guide `[2]`_
+
+
+Prerequisites
+=============
+The OPNFV deployment is out of the scope of this document but it can be
+found in http://docs.opnfv.org.
+The OPNFV platform is considered as the System Under Test (SUT) in this
+document.
+
+Several prerequisites are needed for Functest:
+
+ #. A Jumphost to run Functest on
+ #. A Docker daemon shall be installed on the Jumphost
+ #. A public/external network created on the SUT
+ #. An admin/management network created on the SUT
+ #. Connectivity from the Jumphost to the SUT public/external network
+ #. Connectivity from the Jumphost to the SUT admin/management network
+
+WARNING: Connectivity from Jumphost is essential and it is of paramount
+importance to make sure it is working before even considering to install
+and run Functest. Make also sure you understand how your networking is
+designed to work.
+
+NOTE: **Jumphost** refers to any server which meets the previous
+requirements. Normally it is the same server from where the OPNFV
+deployment has been triggered previously.
+
+NOTE: If your Jumphost is operating behind a company http proxy and/or
+Firewall, please consult first the section `Proxy Support`_, towards
+the end of this document. The section details some tips/tricks which
+*may* be of help in a proxified environment.
+
+Docker installation
+-------------------
+Docker installation and configuration is only needed to be done once
+through the life cycle of Jumphost.
+
+If your Jumphost is based on Ubuntu, RHEL or CentOS linux, please
+consult the references below for more detailed instructions. The
+commands below are offered as a short reference.
+
+*Tip:* For running docker containers behind the proxy, you need first
+some extra configuration which is described in section
+`Docker Installation on CentOS behind http proxy`_. You should follow
+that section before installing the docker engine.
+
+Docker installation needs to be done as root user. You may use other
+userid's to create and run the actual containers later if so desired.
+Log on to your Jumphost as root user and install the Docker Engine
+(e.g. for CentOS family)::
+
+ curl -sSL https://get.docker.com/ | sh
+ systemctl start docker
+
+ *Tip:* If you are working through proxy, please set the https_proxy
+ environment variable first before executing the curl command.
+
+Add your user to docker group to be able to run commands without sudo::
+
+ sudo usermod -aG docker <your_user>
+
+A reconnect is needed. There are 2 ways for this:
+ #. Re-login to your account
+ #. su - <username>
+
+References - Installing Docker Engine on different Linux Operating Systems:
+ * Ubuntu_
+ * RHEL_
+ * CentOS_
+
+.. _Ubuntu: https://docs.docker.com/engine/installation/linux/ubuntulinux/
+.. _RHEL: https://docs.docker.com/engine/installation/linux/rhel/
+.. _CentOS: https://docs.docker.com/engine/installation/linux/centos/
+
+Public/External network on SUT
+------------------------------
+Some of the tests against the VIM (Virtual Infrastructure Manager) need
+connectivity through an existing public/external network in order to
+succeed. This is needed, for example, to create floating IPs to access
+VM instances through the public/external network (i.e. from the Docker
+container).
+
+By default, the four OPNFV installers provide a fresh installation with
+a public/external network created along with a router. Make sure that
+the public/external subnet is reachable from the Jumphost.
+
+*Hint:* For the given OPNFV Installer in use, the IP sub-net address
+used for the public/external network is usually a planning item and
+should thus be known. Consult the OPNFV Configuration guide `[4]`_, and
+ensure you can reach each node in the SUT, from the Jumphost using the
+'ping' command using the respective IP address on the public/external
+network for each node in the SUT. The details of how to determine the
+needed IP addresses for each node in the SUT may vary according to the
+used installer and are therefore ommitted here.
+
+Connectivity to Admin/Management network on SUT
+-----------------------------------------------
+Some of the Functest tools need to have access to the OpenStack
+admin/management network of the controllers `[1]`_.
+
+For this reason, check the connectivity from the Jumphost to all the
+controllers in cluster in the OpenStack admin/management network range.
+
+Installation and configuration
+==============================
+
+.. include:: ./configguide.rst
+
+Integration in CI
+=================
+In CI we use the Docker image and execute the appropriate commands within the
+container from Jenkins.
+
+Docker creation in set-functest-env builder `[3]`_::
+
+ envs="-e INSTALLER_TYPE=${INSTALLER_TYPE} -e INSTALLER_IP=${INSTALLER_IP} -e NODE_NAME=${NODE_NAME}"
+ [...]
+ docker pull opnfv/functest:$DOCKER_TAG >/dev/null
+ cmd="sudo docker run -id ${envs} ${volumes} ${custom_params} ${TESTCASE_OPTIONS} opnfv/functest:${DOCKER_TAG} /bin/bash"
+ echo "Functest: Running docker run command: ${cmd}"
+ ${cmd} >${redirect}
+ sleep 5
+ container_id=$(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | awk '{print $1}' | head -1)
+ echo "Container ID=${container_id}"
+ if [ -z ${container_id} ]; then
+ echo "Cannot find opnfv/functest container ID ${container_id}. Please check if it is existing."
+ docker ps -a
+ exit 1
+ fi
+ echo "Starting the container: docker start ${container_id}"
+ docker start ${container_id}
+ sleep 5
+ docker ps >${redirect}
+ if [ $(docker ps | grep "opnfv/functest:${DOCKER_TAG}" | wc -l) == 0 ]; then
+ echo "The container opnfv/functest with ID=${container_id} has not been properly started. Exiting..."
+ exit 1
+ fi
+
+ cmd="python ${FUNCTEST_REPO_DIR}/functest/ci/prepare_env.py start"
+ echo "Executing command inside the docker: ${cmd}"
+ docker exec ${container_id} ${cmd}
+
+
+Test execution in functest-all builder `[3]`_::
+
+ branch=${GIT_BRANCH##*/}
+ echo "Functest: run $FUNCTEST_SUITE_NAME on branch ${branch}"
+ cmd="functest testcase run $FUNCTEST_SUITE_NAME"
+ fi
+ container_id=$(docker ps -a | grep opnfv/functest | awk '{print $1}' | head -1)
+ docker exec $container_id $cmd
+ ret_value=$?
+ exit $ret_value
+
+Docker clean in functest-cleanup builder `[3]`_ calling docker rm and docker rmi
+
+
+References
+==========
+.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/
+.. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html
+.. _`[3]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-ci-jobs.yml
+.. _`[4]`: http://artifacts.opnfv.org/functest/danube/docs/configguide/index.html
+
+
+OPNFV main site: opnfvmain_.
+
+OPNFV functional test page: opnfvfunctest_.
+
+IRC support channel: #opnfv-functest
+
+.. _opnfvmain: http://www.opnfv.org
+.. _opnfvfunctest: https://wiki.opnfv.org/functest
diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst
new file mode 100644
index 000000000..9436de2b9
--- /dev/null
+++ b/docs/testing/user/userguide/index.rst
@@ -0,0 +1,548 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+=========================
+OPNFV FUNCTEST user guide
+=========================
+
+.. toctree::
+ :maxdepth: 2
+
+Version history
+===============
++------------+----------+------------------+----------------------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++------------+----------+------------------+----------------------------------+
+| 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
+| | | Column Gaynor | |
++------------+----------+------------------+----------------------------------+
+| 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
+| | | | |
+| | | | |
++------------+----------+------------------+----------------------------------+
+
+
+Introduction
+============
+
+The goal of this document is to describe the OPNFV Functest test cases and to
+provide a procedure to execute them. In the OPNFV Danube system release,
+a Functest CLI utility is introduced for easier execution of test procedures.
+
+**IMPORTANT**: It is assumed here that the Functest Docker container is already
+properly deployed and that all instructions described in this guide are to be
+performed from *inside* the deployed Functest Docker container.
+
+.. include:: ./introduction.rst
+
+The different test cases are described in the remaining sections of this document.
+
+VIM (Virtualized Infrastructure Manager)
+----------------------------------------
+
+Healthcheck
+^^^^^^^^^^^
+In Colorado release a new Tier 'healthcheck' with one testcase 'healthcheck'
+was introduced. The healthcheck testcase verifies that some basic IP connectivity
+and essential operations of OpenStack functionality over the command line are
+working correctly.
+
+In particular, the following verifications are performed:
+
+ * DHCP agent functionality for IP address allocation
+ * Openstack Authentication management functionality via the Keystone API
+ * OpenStack Image management functionality via the Glance API
+ * OpenStack Block Storage management functionality via the Cinder API
+ * OpenStack Networking management functionality via the Neutron API
+ * Openstack Compute management functionality via the NOVA API
+
+Self-obviously, successful completion of the 'healthcheck' testcase is a
+necessary pre-requisite for the execution of all other test Tiers.
+
+
+vPing_ssh
+^^^^^^^^^
+
+Given the script **ping.sh**::
+
+ #!/bin/sh
+ while true; do
+ ping -c 1 $1 2>&1 >/dev/null
+ RES=$?
+ if [ "Z$RES" = "Z0" ] ; then
+ echo 'vPing OK'
+ break
+ else
+ echo 'vPing KO'
+ fi
+ sleep 1
+ done
+
+
+The goal of this test is to establish an SSH connection using a floating IP
+on the Public/External network and verify that 2 instances can talk over a Private
+Tenant network::
+
+ vPing_ssh test case
+ +-------------+ +-------------+
+ | | | |
+ | | Boot VM1 with IP1 | |
+ | +------------------->| |
+ | Tester | | System |
+ | | Boot VM2 | Under |
+ | +------------------->| Test |
+ | | | |
+ | | Create floating IP | |
+ | +------------------->| |
+ | | | |
+ | | Assign floating IP | |
+ | | to VM2 | |
+ | +------------------->| |
+ | | | |
+ | | Establish SSH | |
+ | | connection to VM2 | |
+ | | through floating IP| |
+ | +------------------->| |
+ | | | |
+ | | SCP ping.sh to VM2 | |
+ | +------------------->| |
+ | | | |
+ | | VM2 executes | |
+ | | ping.sh to VM1 | |
+ | +------------------->| |
+ | | | |
+ | | If ping: | |
+ | | exit OK | |
+ | | else (timeout): | |
+ | | exit Failed | |
+ | | | |
+ +-------------+ +-------------+
+
+This test can be considered as an "Hello World" example.
+It is the first basic use case which **must** work on any deployment.
+
+vPing_userdata
+^^^^^^^^^^^^^^
+
+This test case is similar to vPing_ssh but without the use of Floating IPs
+and the Public/External network to transfer the ping script.
+Instead, it uses Nova metadata service to pass it to the instance at booting time.
+As vPing_ssh, it checks that 2 instances can talk to
+each other on a Private Tenant network::
+
+ vPing_userdata test case
+ +-------------+ +-------------+
+ | | | |
+ | | Boot VM1 with IP1 | |
+ | +------------------->| |
+ | | | |
+ | | Boot VM2 with | |
+ | | ping.sh as userdata| |
+ | | with IP1 as $1. | |
+ | +------------------->| |
+ | Tester | | System |
+ | | VM2 exeutes ping.sh| Under |
+ | | (ping IP1) | Test |
+ | +------------------->| |
+ | | | |
+ | | Monitor nova | |
+ | | console-log VM 2 | |
+ | | If ping: | |
+ | | exit OK | |
+ | | else (timeout) | |
+ | | exit Failed | |
+ | | | |
+ +-------------+ +-------------+
+
+When the second VM boots it will execute the script passed as userdata
+automatically. The ping will be detected by periodically capturing the output
+in the console-log of the second VM.
+
+
+Tempest
+^^^^^^^
+
+Tempest `[2]`_ is the reference OpenStack Integration test suite.
+It is a set of integration tests to be run against a live OpenStack cluster.
+Tempest has suites of tests for:
+
+ * OpenStack API validation
+ * Scenarios
+ * Other specific tests useful in validating an OpenStack deployment
+
+Functest uses Rally `[3]`_ to run the Tempest suite.
+Rally generates automatically the Tempest configuration file **tempest.conf**.
+Before running the actual test cases,
+Functest creates the needed resources (user, tenant) and
+updates the appropriate parameters into the configuration file.
+
+When the Tempest suite is executed, each test duration is measured and the full
+console output is stored to a *log* file for further analysis.
+
+The Tempest testcases are distributed accross two
+Tiers:
+
+ * Smoke Tier - Test Case 'tempest_smoke_serial'
+ * Components Tier - Test case 'tempest_full_parallel'
+
+NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
+tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
+executes all defined Tempest tests using several concurrent threads
+(i.e. parallel mode). The number of threads activated corresponds to the number
+of available logical CPUs.
+
+The goal of the Tempest test suite is to check the basic functionalities of the
+different OpenStack components on an OPNFV fresh installation, using the
+corresponding REST API interfaces.
+
+
+Rally bench test suites
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Rally `[3]`_ is a benchmarking tool that answers the question:
+
+*How does OpenStack work at scale?*
+
+The goal of this test suite is to benchmark all the different OpenStack modules and
+get significant figures that could help to define Telco Cloud KPIs.
+
+The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
+
+ * authenticate
+ * cinder
+ * glance
+ * heat
+ * keystone
+ * neutron
+ * nova
+ * quotas
+ * requests
+
+A basic SLA (stop test on errors) has been implemented.
+
+The Rally testcases are distributed accross two Tiers:
+
+ * Smoke Tier - Test Case 'rally_sanity'
+ * Components Tier - Test case 'rally_full'
+
+NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
+cases. Test case 'rally_full' executes the full defined set of Rally tests.
+
+SNAPS
+-----
+
+SNAPS stands for "SNA/NFV Application development Platform and Stack".
+This project seeks to develop baseline OpenStack NFV installations. It has been
+developed by Steven Pisarski and provided an object oriented library to perform
+functional and performance tests. It has been declined in several test suites in
+Functest, 2 are part of healthcheck tier, one belongs to smoke tier.
+
+connection check
+^^^^^^^^^^^^^^^^
+Connection_check consists in 9 test cases (test duration < 5s) checking the
+connectivity with Glance, Keystone, Neutron, Nova and the external network.
+
+api_check
+^^^^^^^^^
+This test case verifies the retrieval of OpenStack clients: Keystone, Glance,
+Neutron and Nova and may perform some simple queries. When the config value of
+snaps.use_keystone is True, functest must have access to the cloud's private
+network.
+This suite consists in 49 tests (test duration< 2 minutes)
+
+snaps_smoke
+^^^^^^^^^^^
+This test case contains tests that setup and destroy environments with VMs with
+and without Floating IPs with a newly created user and project. Set the config
+value snaps.use_floating_ips (True|False) to toggle this functionality. When
+the config value of snaps.use_keystone is True, functest must have access
+the cloud's private network.
+This suite consists in 38 tests (test duration < 10 minutes)
+
+More information on SNAPS can be found in  `[13]`_
+
+
+SDN Controllers
+---------------
+
+There are currently 2 available controllers:
+
+ * OpenDaylight (ODL)
+ * ONOS
+
+OpenDaylight
+^^^^^^^^^^^^
+
+The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
+from the ODL project using the Robot `[11]`_ framework.
+The suite verifies creation and deletion of networks, subnets and ports with
+OpenDaylight and Neutron.
+
+The list of tests can be described as follows:
+
+ * Basic Restconf test cases
+ * Connect to Restconf URL
+ * Check the HTTP code status
+
+ * Neutron Reachability test cases
+ * Get the complete list of neutron resources (networks, subnets, ports)
+
+ * Neutron Network test cases
+ * Check OpenStack networks
+ * Check OpenDaylight networks
+ * Create a new network via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the network has also been successfully created in OpenDaylight
+
+ * Neutron Subnet test cases
+ * Check OpenStack subnets
+ * Check OpenDaylight subnets
+ * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the subnet has also been successfully created in OpenDaylight
+
+ * Neutron Port test cases
+ * Check OpenStack Neutron for known ports
+ * Check OpenDaylight ports
+ * Create a new port via OpenStack and check the HTTP status code returned by Neutron
+ * Check that the new port has also been successfully created in OpenDaylight
+
+ * Delete operations
+ * Delete the port previously created via OpenStack
+ * Check that the port has been also succesfully deleted in OpenDaylight
+ * Delete previously subnet created via OpenStack
+ * Check that the subnet has also been successfully deleted in OpenDaylight
+ * Delete the network created via OpenStack
+ * Check that the network has also been succesfully deleted in OpenDaylight
+
+Note: the checks in OpenDaylight are based on the returned HTTP status
+code returned by OpenDaylight.
+
+
+ONOS
+^^^^
+
+TestON Framework is used to test the ONOS SDN controller functions.
+The test cases deal with L2 and L3 functions.
+The ONOS test suite can be run on any ONOS compliant scenario.
+
+The test cases are described as follows:
+
+ * onosfunctest: The main executable file contains the initialization of
+ the docker environment and functions called by FUNCvirNetNB and
+ FUNCvirNetNBL3
+
+ * FUNCvirNetNB
+
+ * Create Network: Post Network data and check it in ONOS
+ * Update Network: Update the Network and compare it in ONOS
+ * Delete Network: Delete the Network and check if it's NULL in ONOS or
+ not
+ * Create Subnet: Post Subnet data and check it in ONOS
+ * Update Subnet: Update the Subnet and compare it in ONOS
+ * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not
+ * Create Port: Post Port data and check it in ONOS
+ * Update Port: Update the Port and compare it in ONOS
+ * Delete Port: Delete the Port and check if it's NULL in ONOS or not
+
+ * FUNCvirNetNBL3
+
+ * Create Router: Post data for create Router and check it in ONOS
+ * Update Router: Update the Router and compare it in ONOS
+ * Delete Router: Delete the Router data and check it in ONOS
+ * Create RouterInterface: Post Router Interface data to an existing Router
+ and check it in ONOS
+ * Delete RouterInterface: Delete the RouterInterface and check the Router
+ * Create FloatingIp: Post data for create FloatingIp and check it in ONOS
+ * Update FloatingIp: Update the FloatingIp and compare it in ONOS
+ * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in
+ ONOS
+ * Create External Gateway: Post data to create an External Gateway for an
+ existing Router and check it in ONOS
+ * Update External Gateway: Update the External Gateway and compare the change
+ * Delete External Gateway: Delete the External Gateway and check that it is
+ 'NULL' in ONOS
+
+
+Features
+--------
+
+Please refer to the dedicated feature user guides for details:
+
+ * bgpvpn: http://artifacts.opnfv.org/sdnvpn/danube/docs/userguide/index.html
+ * copper: http://artifacts.opnfv.org/copper/danube/docs/userguide/index.html
+ * doctor: http://artifacts.opnfv.org/doctor/danube/userguide/index.html
+ * domino: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
+ * multisites: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
+ * onos-sfc: http://artifacts.opnfv.org/onosfw/danube/userguide/index.html
+ * odl-sfc: http://artifacts.opnfv.org/sfc/danube/userguide/index.html
+ * promise: http://artifacts.opnfv.org/danube/colorado/docs/userguide/index.html
+ * security_scan: http://artifacts.opnfv.org/security_scan/colorado/docs/userguide/index.html
+ * TODO
+
+security_scan
+^^^^^^^^^^^^^
+
+Security Scanning, is a project to insure security compliance and vulnerability
+checks, as part of an automated CI / CD platform delivery process.
+
+The project makes use of the existing SCAP format `[6]`_ to perform deep
+scanning of NFVI nodes, to insure they are hardened and free of known CVE
+reported vulnerabilities.
+
+The SCAP content itself, is then consumed and run using an upstream opensource tool
+known as OpenSCAP `[7]`_.
+
+The OPNFV Security Group have developed the code that will called by the OPNFV Jenkins
+build platform, to perform a complete scan. Resulting reports are then copied to the
+OPNFV functest dashboard.
+
+The current work flow is as follows:
+
+ * Jenkins Build Initiated
+ * security_scan.py script is called, and a config file is passed to the script as
+ an argument.
+ * The IP addresses of each NFVi node (compute / control) are gathered
+ * A scan profile is matched to the node type.
+ * The OpenSCAP application is remotely installed onto each target node gathered
+ on step 3, using upstream packaging (rpm and .deb).
+ * A scan is made against each node gathered within step 3.
+ * HTML Reports are downloaded for rendering on a dashboard.
+ * If the config file value 'clean' is set to 'True' then the application installed in
+ step 5 is removed, and all reports created at step 6 are deleted.
+
+Security scan is supported by Apex, TODO....
+
+
+
+VNF
+---
+
+
+vIMS
+^^^^
+The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
+architectural framework for delivering IP multimedia services.
+
+vIMS has been integrated in Functest to demonstrate the capability to deploy a
+relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
+functional VNF allows the test of most of the essential functions needed for a
+NFV platform.
+
+The goal of this test suite consists of:
+
+ * deploy a VNF orchestrator (Cloudify)
+ * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
+ orchestrator based on a TOSCA blueprint defined in `[5]`_
+ * run suite of signaling tests on top of this VNF
+
+The Clearwater architecture is described as follows:
+
+.. figure:: ../images/clearwater-architecture.png
+ :align: center
+ :alt: vIMS architecture
+
+
+parser
+^^^^^^
+
+See parser user guide for details: `[12]`_
+
+
+.. include:: ./runfunctest.rst
+
+
+Test results
+============
+
+Manual testing
+--------------
+
+In manual mode test results are displayed in the console and result files
+are put in /home/opnfv/functest/results.
+
+Automated testing
+--------------
+
+In automated mode, test results are displayed in jenkins logs, a summary is provided
+at the end of the job and can be described as follow::
+
+ +==================================================================================================================================================+
+ | FUNCTEST REPORT |
+ +==================================================================================================================================================+
+ | |
+ | Deployment description: |
+ | INSTALLER: fuel |
+ | SCENARIO: os-odl_l2-nofeature-ha |
+ | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 |
+ | CI LOOP: daily |
+ | |
+ +=========================+===============+============+===============+===========================================================================+
+ | TEST CASE | TIER | DURATION | RESULT | URL |
+ +=========================+===============+============+===============+===========================================================================+
+ | healthcheck | healthcheck | 03:07 | PASS | |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | vping_ssh | smoke | 00:56 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac13d79377c54b278bd4c1 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | vping_userdata | smoke | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac14019377c54b278bd4c2 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | tempest_smoke_serial | smoke | 16:05 | FAIL | http://testresults.opnfv.org/test/api/v1/results/57ac17ca9377c54b278bd4c3 |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | rally_sanity | smoke | 12:19 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1aad9377c54b278bd4cd |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | odl | sdn_suites | 00:24 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ad09377c54b278bd4ce |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+ | promise | features | 00:41 | PASS | http://testresults.opnfv.org/test/api/v1/results/57ac1ae59377c54b278bd4cf |
+ +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
+
+Results are automatically pushed to the test results database, some additional
+result files are pushed to OPNFV artifact web sites.
+
+Based on the results stored in the result database, a `Functest reporting`_
+portal is also automatically updated. This portal provides information on:
+
+ * The overall status per scenario and per installer
+ * Tempest: Tempest test case including reported errors per scenario and installer
+ * vIMS: vIMS details per scenario and installer
+
+.. figure:: ../images/functest-reporting-status.png
+ :align: center
+ :alt: Functest reporting portal Fuel status page
+
+
+Test Dashboard
+==============
+
+Based on results collected in CI, a test dashboard is dynamically generated.
+
+
+.. include:: ./troubleshooting.rst
+
+
+References
+==========
+
+.. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
+.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
+.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
+.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
+.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
+.. _`[6]`: https://scap.nist.gov/
+.. _`[7]`: https://github.com/OpenSCAP/openscap
+.. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml
+.. _`[11]`: http://robotframework.org/
+.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
+.. _`[13]`: TODO URL doc SNAPS
+
+OPNFV main site: opnfvmain_.
+
+OPNFV functional test page: opnfvfunctest_.
+
+IRC support chan: #opnfv-testperf
+
+.. _opnfvmain: http://www.opnfv.org
+.. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
+.. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
+.. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
+.. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
+.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
+.. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
diff --git a/docs/testing/user/userguide/introduction.rst b/docs/testing/user/userguide/introduction.rst
new file mode 100644
index 000000000..4dfe79375
--- /dev/null
+++ b/docs/testing/user/userguide/introduction.rst
@@ -0,0 +1,255 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Overview of the Functest suites
+===============================
+
+Functest is the OPNFV project primarily targeting function testing.
+In the Continuous Integration pipeline, it is launched after an OPNFV fresh
+installation to validate and verify the basic functions of the
+infrastructure.
+
+The current list of test suites can be distributed over 5 main domains: VIM
+(Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
+Features, VNF (Virtual Network Functions) and MANO stacks.
+
+Functest test suites are also distributed in the OPNFV testing categories:
+healthcheck, smoke, features, components, performance, VNF, Stress tests.
+
+All the Healthcheck and smoke tests of a given scenario must be succesful to
+validate the scenario for the release.
+
++-------------+---------------+----------------+----------------------------------+
+| Domain | Tier | Test case | Comments |
++=============+===============+================+==================================+
+| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
+| | +----------------+----------------------------------+
+| | | connection | Check OpenStack connectivity |
+| | | _check | through SNAPS framework |
+| | +----------------+----------------------------------+
+| | | api_check | Check OpenStack API through |
+| | | | SNAPS framework |
+| +---------------+----------------+----------------------------------+
+| | smoke | vPing_SSH | NFV "Hello World" using an SSH |
+| | | | connection to a destination VM |
+| | | | over a created floating IP |
+| | | | address on the SUT Public / |
+| | | | External network. Using the SSH |
+| | | | connection a test script is then |
+| | | | copied to the destination |
+| | | | VM and then executed via SSH. |
+| | | | The script will ping another |
+| | | | VM on a specified IP address over|
+| | | | the SUT Private Tenant network. |
+| | +----------------+----------------------------------+
+| | | vPing_userdata | Uses Ping with given userdata |
+| | | | to test intra-VM connectivity |
+| | | | over the SUT Private Tenant |
+| | | | network. The correct operation |
+| | | | of the NOVA Metadata service is |
+| | | | also verified in this test. |
+| | +----------------+----------------------------------+
+| | | tempest_smoke | Generate and run a relevant |
+| | | \_serial | Tempest Test Suite in smoke mode.|
+| | | | The generated test set is |
+| | | | dependent on the OpenStack |
+| | | | deployment environment. |
+| | +----------------+----------------------------------+
+| | | rally_sanity | Run a subset of the OpenStack |
+| | | | Rally Test Suite in smoke mode |
+| | +----------------+----------------------------------+
+| | | snaps_smoke | Run a subset of the OpenStack |
+| | | | Rally Test Suite in smoke mode |
+| +---------------+----------------+----------------------------------+
+| | components | tempest_full | Generate and run a full set of |
+| | | \_parallel | the OpenStack Tempest Test Suite.|
+| | | | See the OpenStack reference test |
+| | | | suite `[2]`_. The generated |
+| | | | test set is dependent on the |
+| | | | OpenStack deployment environment.|
+| | +----------------+----------------------------------+
+| | | rally_full | Run the OpenStack testing tool |
+| | | | benchmarking OpenStack modules |
+| | | | See the Rally documents `[3]`_. |
++-------------+---------------+----------------+----------------------------------+
+| Controllers | smoke | odl | Opendaylight Test suite |
+| | | | Limited test suite to check the |
+| | | | basic neutron (Layer 2) |
+| | | | operations mainly based on |
+| | | | upstream testcases. See below |
+| | | | for details |
+| | +----------------+----------------------------------+
+| | | onos | Test suite of ONOS L2 and L3 |
+| | | | functions. |
+| | | | See `ONOSFW User Guide`_ for |
+| | | | details. |
++-------------+---------------+----------------+----------------------------------+
+| Features | features | promise | Resource reservation and |
+| | | | management project to identify |
+| | | | NFV related requirements and |
+| | | | realize resource reservation for |
+| | | | future usage by capacity |
+| | | | management of resource pools |
+| | | | regarding compute, network and |
+| | | | storage. |
+| | | | See `Promise User Guide`_ for |
+| | | | details. |
+| | +----------------+----------------------------------+
+| | | doctor | Doctor platform, as of Colorado |
+| | | | release, provides the three |
+| | | | features: |
+| | | | * Immediate Notification |
+| | | | * Consistent resource state |
+| | | | awareness for compute host down |
+| | | | * Valid compute host status |
+| | | | given to VM owner |
+| | | | See `Doctor User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | bgpvpn | Implementation of the OpenStack |
+| | | | bgpvpn API from the SDNVPN |
+| | | | feature project. It allows for |
+| | | | the creation of BGP VPNs. |
+| | | | See `SDNVPN User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | security_scan | Implementation of a simple |
+| | | | security scan. (Currently |
+| | | | available only for the Apex |
+| | | | installer environment) |
+| | +----------------+----------------------------------+
+| | | onos-sfc | SFC testing for onos scenarios |
+| | | | See `ONOSFW User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | odl-sfc | SFC testing for odl scenarios |
+| | | | See `SFC User Guide`_ for details|
+| | +----------------+----------------------------------+
+| | | domino | Domino provides TOSCA template |
+| | | | distribution service for network |
+| | | | service and VNF descriptors |
+| | | | among MANO components e.g., |
+| | | | NFVO, VNFM, VIM, SDN-C, etc., |
+| | | | as well as OSS/BSS functions. |
+| | | | See `Domino User Guide`_ for |
+| | | | details |
+| | +----------------+----------------------------------+
+| | | copper | Copper develops OPNFV platform |
+| | | | support for policy management, |
+| | | | using open source projects such |
+| | | | as OpenStack Congress, focused |
+| | | | on helping ensure that virtual |
+| | | | infrastructure and the apps that |
+| | | | execute on it comply with the |
+| | | | configuration policy intent of |
+| | | | service providers, developers, |
+| | | | and end users. See more detail |
+| | | | in the `Copper User Guide`_. |
+| | +----------------+----------------------------------+
+| | | multisites | Multisites |
+| | | | See `Multisite User Guide`_ for |
+| | | | details |
++-------------+---------------+----------------+----------------------------------+
+| VNF | vnf | cloudify_ims | Example of a real VNF deployment |
+| | | | to show the NFV capabilities of |
+| | | | the platform. The IP Multimedia |
+| | | | Subsytem is a typical Telco test |
+| | | | case, referenced by ETSI. |
+| | | | It provides a fully functional |
+| | | | VoIP System |
+| | +----------------+----------------------------------+
+| | | opera_ims | vIMS deployment using openBaton |
+| | +----------------+----------------------------------+
+| | | orchestra_ims | vIMS deployment using open-O |
++ +---------------+----------------+----------------------------------+
+| | | parser | Parser is an integration project |
+| | | | which aims to provide |
+| | | | placement/deployment templates |
+| | | | translation for OPNFV platform, |
+| | | | including TOSCA -> HOT, POLICY ->|
+| | | | TOSCA and YANG -> TOSCA. |
+| | | | See `Parser User Guide`_ for |
+| | | | details |
++-------------+---------------+----------------+----------------------------------+
+
+
+As shown in the above table, Functest is structured into different 'domains',
+'tiers' and 'test cases'. Each 'test case' usually represents an actual
+'Test Suite' comprised -in turn- of several test cases internally.
+
+Test cases also have an implicit execution order. For example, if the early
+'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
+Tier testcases, there is little point to launch a full testcase execution round.
+
+In Danube, we merged smoke and sdn controller tiers in smoke tier.
+
+An overview of the Functest Structural Concept is depicted graphically below:
+
+.. figure:: ../images/concepts_mapping_final.png
+ :align: center
+ :alt: Functest Concepts Structure
+
+Some of the test cases are developed by Functest team members, whereas others
+are integrated from upstream communities or other OPNFV projects. For example,
+`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
+OpenStack integration test suite and Functest is in charge of the selection,
+integration and automation of those tests that fit suitably to OPNFV.
+
+The Tempest test suite is the default OpenStack smoke test suite but no new test
+cases have been created in OPNFV Functest.
+
+The results produced by the tests run from CI are pushed and collected into a
+NoSQL database. The goal is to populate the database with results from different
+sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
+of a live Functest Dashboard is shown below:
+
+** TODO **
+.. figure:: ../images/FunctestDashboardDanube.png
+ :align: center
+ :alt: Functest Dashboard
+
+
+Basic components (VIM, SDN controllers) are tested through their own suites.
+Feature projects also provide their own test suites with different ways of
+running their tests.
+
+The notion of domain has been introduced in the description of the test cases
+stored in the Database.
+This parameters as well as possible tags can be used for the Test case catalog.
+
+vIMS test case was integrated to demonstrate the capability to deploy a
+relatively complex NFV scenario on top of the OPNFV infrastructure.
+
+Functest considers OPNFV as a black box. As of Danube release the OPNFV
+offers a lot of potential combinations:
+
+ * 3 controllers (OpenDaylight, ONOS, OpenContrail)
+ * 4 installers (Apex, Compass, Fuel, Joid)
+
+Most of the tests are runnable by any combination, but some tests might have
+restrictions imposed by the utilized installers or due to the available
+deployed features. The system uses the environment variables (INSTALLER_IP and
+DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
+environment.
+
+A convenience Functest CLI utility is also available to simplify setting up the
+Functest evironment, management of the OpenStack environment (e.g. resource
+clean-up) and for executing tests.
+The Functest CLI organised the testcase into logical Tiers, which contain in
+turn one or more testcases. The CLI allows execution of a single specified
+testcase, all test cases in a specified Tier, or the special case of execution
+of **ALL** testcases. The Functest CLI is introduced in more detail in the
+section `Executing the functest suites`_ of this document.
+
+.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
+.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
+.. _`Copper User Guide`: http://artifacts.opnfv.org/copper/colorado/docs/userguide/index.html
+.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
+.. _`Promise User Guide`: http://artifacts.opnfv.org/promise/colorado/docs/userguide/index.html
+.. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/colorado/userguide/index.html
+.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
+.. _`Domino User Guide`: http://artifacts.opnfv.org/domino/docs/userguide-single/index.html
+.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
+.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
+.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
+.. _`Multisite User Guide`: http://artifacts.opnfv.org/multisite/docs/userguide/index.html
diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst
new file mode 100644
index 000000000..e7ab84b26
--- /dev/null
+++ b/docs/testing/user/userguide/runfunctest.rst
@@ -0,0 +1,385 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Executing the functest suites
+=============================
+
+Manual testing
+--------------
+
+This section assumes the following:
+ * The Functest Docker container is running
+ * The docker prompt is shown
+ * The Functest environment is ready (Functest CLI command 'functest env prepare'
+ has been executed)
+
+If any of the above steps are missing please refer to the Functest Config Guide
+as they are a prerequisite and all the commands explained in this section **must** be
+performed **inside the container**.
+
+The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
+for the execution of Test Tiers or Test Cases::
+
+ root@22e436918db0:~/repos/functest/ci# functest tier --help
+ Usage: functest tier [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ -h, --help Show this message and exit.
+
+ Commands:
+ get-tests Prints the tests in a tier.
+ list Lists the available tiers.
+ run Executes all the tests within a tier.
+ show Shows information about a tier.
+ root@22e436918db0:~/repos/functest/ci# functest testcase --help
+
+ Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
+
+ Options:
+ -h, --help Show this message and exit.
+
+ Commands:
+ list Lists the available testcases.
+ run Executes a test case.
+ show Shows information about a test case.
+
+More details on the existing Tiers and Test Cases can be seen with the 'list'
+command::
+
+ root@22e436918db0:~/repos/functest/ci# functest tier list
+ - 0. healthcheck:
+ ['healthcheck', 'connection_check', 'api_check',]
+ - 1. smoke:
+ ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl']
+ - 2. features:
+ ['doctor', 'security_scan']
+ - 3. components:
+ ['tempest_full_parallel', 'rally_full']
+ - 4. vnf:
+ ['cloudify_ims']
+
+ and
+
+ root@22e436918db0:~/repos/functest/ci# functest testcase list
+ healthcheck
+ api_check
+ connection_check
+ vping_ssh
+ vping_userdata
+ snaps_smoke
+ tempest_smoke_serial
+ rally_sanity
+ odl
+ doctor
+ security_scan
+ tempest_full_parallel
+ rally_full
+ cloudify_ims
+
+More specific details on specific Tiers or Test Cases can be seen wih the
+'show' command::
+
+ root@22e436918db0:~/repos/functest/ci# functest tier show smoke
+ +======================================================================+
+ | Tier: smoke |
+ +======================================================================+
+ | Order: 1 |
+ | CI Loop: (daily)|(weekly) |
+ | Description: |
+ | Set of basic Functional tests to validate the OpenStack |
+ | deployment. |
+ | Test cases: |
+ | - vping_ssh |
+ | - vping_userdata |
+ | - tempest_smoke_serial |
+ | - rally_sanity |
+ | |
+ +----------------------------------------------------------------------+
+
+ and
+
+ root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
+ +======================================================================+
+ | Testcase: tempest_smoke_serial |
+ +======================================================================+
+ | Description: |
+ | This test case runs the smoke subset of the OpenStack Tempest |
+ | suite. The list of test cases is generated by Tempest |
+ | automatically and depends on the parameters of the OpenStack |
+ | deplopyment. |
+ | Dependencies: |
+ | - Installer: |
+ | - Scenario : |
+ | |
+ +----------------------------------------------------------------------+
+
+
+To execute a Test Tier or Test Case, the 'run' command is used::
+
+ root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
+ Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
+ 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
+ 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
+ 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
+ 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
+ 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
+ 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
+ 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
+ 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
+ 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
+ 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
+ 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
+ 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
+ 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
+ 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
+ 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
+ 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
+ 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
+ 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
+ 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
+ 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
+ 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
+ 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
+ :
+ :
+ 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
+ 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
+ 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+ 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
+ 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+ Version 1 is deprecated, use alternative version 2 instead.
+ WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
+ 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
+ 2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
+ 2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
+ 2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
+ 2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
+ 2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
+ 2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
+ 2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
+ 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
+ 2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
+
+ and
+
+ root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
+ Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
+ 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
+ 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
+ 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
+ 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
+ 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'...
+ 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
+ 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
+ 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
+ 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
+ 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
+ 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
+ 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
+ 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
+ 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
+ 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
+ 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
+ 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
+ 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
+ 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
+ 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
+ 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
+ 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
+ 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
+ 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+ 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
+ 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
+ Version 1 is deprecated, use alternative version 2 instead.
+ :
+ :
+ etc.
+
+To list the test cases which are part of a specific Test Tier, the 'get-tests'
+command is used with 'functest tier'::
+
+ root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
+ Test cases in tier 'healthcheck':
+ ['healthcheck']
+
+
+Please note that for some scenarios some test cases might not be launched.
+For example, the last example displayed only the 'odl' testcase for the given
+environment. In this particular system the deployment does not support the 'onos' SDN
+Controller Test Case; for example.
+
+**Important** If you use the command 'functest tier run <tier_name>', then the
+Functest CLI utility will call **all valid Test Cases**, which belong to the
+specified Test Tier, as relevant to scenarios deployed to the SUT environment.
+Thus, the Functest CLI utility calculates automatically which tests can be
+executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
+which is passed in to the Functest docker container.
+
+Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
+two possibilities::
+
+ * Run a single Test Case, specified by a valid choice of <testcase_name>
+ * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
+
+Functest includes a cleaning mechanism in order to remove all the OpenStack
+resources except those present before running any test. The script
+*$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
+the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
+all the OpenStack resources (images, networks, volumes, security groups, tenants,
+users) so that an eventual cleanup does not remove any of these defaults.
+
+The script **clean_openstack.py** which is located in
+*$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
+in charge of cleaning the OpenStack resources that are not specified in the
+defaults file generated previously which is stored in
+*/home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
+
+It is important to mention that if there are new OpenStack resources created
+manually after preparing the Functest environment, they will be removed, unless
+you use the special method of invoking the test case with specific suppression
+of clean up. (See the `Troubleshooting`_ section).
+
+The reason to include this cleanup meachanism in Functest is because some
+test suites such as Tempest or Rally create a lot of resources (users,
+tenants, networks, volumes etc.) that are not always properly cleaned, so this
+function has been set to keep the system as clean as it was before a
+full Functest execution.
+
+Although the Functest CLI provides an easy way to run any test, it is possible to
+do a direct call to the desired test script. For example:
+
+ python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
+
+
+Automated testing
+-----------------
+
+As mentioned previously, the Functest Docker container preparation as well as
+invocation of Test Cases can be called within the container from the Jenkins CI
+system. There are 3 jobs that automate the whole process. The first job runs all
+the tests referenced in the daily loop (i.e. that must been run daily), the second
+job runs the tests referenced in the weekly loop (usually long duration tests run
+once a week maximum) and the third job allows testing test suite by test suite specifying
+the test suite name. The user may also use either of these Jenkins jobs to execute
+the desired test suites.
+
+One of the most challenging task in the Danube release consists
+in dealing with lots of scenarios and installers. Thus, when the tests are
+automatically started from CI, a basic algorithm has been created in order to
+detect whether a given test is runnable or not on the given scenario.
+Some Functest test suites cannot be systematically run (e.g. ODL suite can not
+be run on an ONOS scenario). The daily/weekly notion has been introduces in
+Colorado in order to save CI time and avoid running systematically
+long duration tests. It was not used in Colorado due to CI resource shortage.
+The mechanism remains however as part of the CI evolution.
+
+CI provides some useful information passed to the container as environment
+variables:
+
+ * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE
+ * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
+ * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
+
+ * controller = (odl|onos|ocl|nosdn)
+ * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites)
+ * mode = (ha|noha)
+
+The constraints per test case are defined in the Functest configuration file
+*/home/opnfv/repos/functest/functest/ci/testcases.yaml*::
+
+ tiers:
+ -
+ name: healthcheck
+ order: 0
+ ci_loop: '(daily)|(weekly)'
+ description : >-
+ First tier to be executed to verify the basic
+ operations in the VIM.
+ testcases:
+ -
+ name: healthcheck
+ criteria: 'status == "PASS"'
+ blocking: true
+ description: >-
+ This test case verifies the basic OpenStack services like
+ Keystone, Glance, Cinder, Neutron and Nova.
+
+ dependencies:
+ installer: ''
+ scenario: ''
+
+ -
+ name: smoke
+ order: 1
+ ci_loop: '(daily)|(weekly)'
+ description : >-
+ Set of basic Functional tests to validate the OpenStack deployment.
+ testcases:
+ -
+ name: vping_ssh
+ criteria: 'status == "PASS"'
+ blocking: true
+ description: >-
+ This test case verifies: 1) SSH to an instance using floating
+ IPs over the public network. 2) Connectivity between 2 instances
+ over a private network.
+ dependencies:
+ installer: ''
+ scenario: '^((?!bgpvpn|odl_l3).)*$'
+ run:
+ module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
+ class: 'VPingSSH'
+ ....
+
+We may distinguish 2 levels in the test case description:
+ * Tier level
+ * Test case level
+
+At the tier level, we define the following parameters:
+
+ * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
+ * description: a high level view of the test case
+
+For a given test case we defined:
+ * the name of the test case
+ * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
+ * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
+ * the description of the test case
+ * the dependencies: a combination of 2 regex on the scenario and the installer name
+ * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
+
+For further details on abstraction classes, see developper guide.
+
+Additional parameters have been added in the desription in the Database.
+The target is to use the configuration stored in the Database and consider the
+local file as backup if the Database is not reachable.
+The additional fields related to a test case are:
+ * trust: we introduced this notion to put in place a mechanism of scenario promotion.
+ * Version: it indicates since which version you can run this test
+ * domains: the main domain covered by the test suite
+ * tags: a list of tags related to the test suite
+
+The order of execution is the one defined in the file if all test cases are selected.
+
+In CI daily job the tests are executed in the following order:
+
+ 1) healthcheck (blocking)
+ 2) smoke: both vPings are blocking
+ 3) Feature project tests cases
+
+In CI weekly job we add 2 tiers:
+
+ 4) VNFs (vIMS)
+ 5) Components (Rally and Tempest long duration suites)
+
+As explained before, at the end of an automated execution, the OpenStack resources
+might be eventually removed.
+Please note that a system snapshot is taken before any test case execution.
+
+This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.
diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst
new file mode 100644
index 000000000..845501916
--- /dev/null
+++ b/docs/testing/user/userguide/troubleshooting.rst
@@ -0,0 +1,387 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Troubleshooting
+===============
+
+This section gives some guidelines about how to troubleshoot the test cases
+owned by Functest.
+
+**IMPORTANT**: As in the previous section, the steps defined below must be
+executed inside the Functest Docker container and after sourcing the OpenStack
+credentials::
+
+ . $creds
+
+or::
+
+ source /home/opnfv/functest/conf/openstack.creds
+
+VIM
+---
+
+This section covers the test cases related to the VIM (healthcheck, vping_ssh,
+vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity,
+rally_full).
+
+vPing common
+^^^^^^^^^^^^
+For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are
+similar:
+
+ * Create Glance image
+ * Create Network
+ * Create Security Group
+ * Create Instances
+
+After these actions, the test cases differ and will be explained in their
+respective section.
+
+These test cases can be run inside the container, using new Functest CLI as follows::
+
+ $ functest testcase run vping_ssh
+ $ functest testcase run vping_userdata
+
+The Functest CLI is designed to route a call to the corresponding internal
+python scripts, located in paths:
+*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_ssh.py* and
+*$REPOS_DIR/functest/functest/opnfv_tests/vPing/CI/libraries/vPing_userdata.py*
+
+Notes:
+
+ #. There is one difference, between the Functest CLI based test case
+ execution compared to the earlier used Bash shell script, which is
+ relevant to point out in troubleshooting scenarios:
+
+ The Functest CLI does **not yet** support the option to suppress
+ clean-up of the generated OpenStack resources, following the execution
+ of a test case.
+
+ Explanation: After finishing the test execution, the corresponding
+ script will remove, by default, all created resources in OpenStack
+ (image, instances, network and security group). When troubleshooting,
+ it is advisable sometimes to keep those resources in case the test
+ fails and a manual testing is needed.
+
+ It is actually still possible to invoke test execution, with suppression
+ of OpenStack resource cleanup, however this requires invocation of a
+ **specific Python script:** '/home/opnfv/repos/functest/ci/run_test.py'.
+ The `OPNFV Functest Developer Guide`_ provides guidance on the use of that
+ Python script in such troubleshooting cases.
+
+Some of the common errors that can appear in this test case are::
+
+ vPing_ssh- ERROR - There has been a problem when creating the neutron network....
+
+This means that there has been some problems with Neutron, even before creating the
+instances. Try to create manually a Neutron network and a Subnet to see if that works.
+The debug messages will also help to see when it failed (subnet and router creation).
+Example of Neutron commands (using 10.6.0.0/24 range for example)::
+
+ neutron net-create net-test
+ neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
+ --gateway 10.6.0.254 net-test 10.6.0.0/24
+ neutron router-create test_router
+ neutron router-interface-add <ROUTER_ID> test_subnet
+ neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
+
+Another related error can occur while creating the Security Groups for the instances::
+
+ vPing_ssh- ERROR - Failed to create the security group...
+
+In this case, proceed to create it manually. These are some hints::
+
+ neutron security-group-create sg-test
+ neutron security-group-rule-create sg-test --direction ingress --protocol icmp \
+ --remote-ip-prefix 0.0.0.0/0
+ neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \
+ --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
+ neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \
+ --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
+
+The next step is to create the instances. The image used is located in
+*/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img* and a Glance image is created
+with the name **functest-vping**. If booting the instances fails (i.e. the status
+is not **ACTIVE**), you can check why it failed by doing::
+
+ nova list
+ nova show <INSTANCE_ID>
+
+It might show some messages about the booting failure. To try that manually::
+
+ nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test
+
+This will spawn a VM using the network created previously manually.
+In all the OPNFV tested scenarios from CI, it never has been a problem with the
+previous actions. Further possible problems are explained in the following sections.
+
+
+vPing_SSH
+^^^^^^^^^
+This test case creates a floating IP on the external network and assigns it to
+the second instance **opnfv-vping-2**. The purpose of this is to establish
+a SSH connection to that instance and SCP a script that will ping the first
+instance. This script is located in the repository under
+*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/ping.sh* and takes an IP as
+a parameter. When the SCP is completed, the test will do an SSH call to that script
+inside the second instance. Some problems can happen here::
+
+ vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
+
+If this is displayed, stop the test or wait for it to finish, if you have used
+the special method of test invocation with specific supression of OpenStack
+resource clean-up, as explained earler. It means that the Container can not
+reach the Public/External IP assigned to the instance **opnfv-vping-2**. There
+are many possible reasons, and they really depend on the chosen scenario. For
+most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known
+limitation.
+
+First, make sure that the instance **opnfv-vping-2** succeeded to get an IP
+from the DHCP agent. It can be checked by doing::
+
+ nova console-log opnfv-vping-2
+
+If the message *Sending discover* and *No lease, failing* is shown, it probably
+means that the Neutron dhcp-agent failed to assign an IP or even that it was not
+responding. At this point it does not make sense to try to ping the floating IP.
+
+If the instance got an IP properly, try to ping manually the VM from the container::
+
+ nova list
+ <grab the public IP>
+ ping <public IP>
+
+If the ping does not return anything, try to ping from the Host where the Docker
+container is running. If that solves the problem, check the iptable rules because
+there might be some rules rejecting ICMP or TCP traffic coming/going from/to the
+container.
+
+At this point, if the ping does not work either, try to reproduce the test
+manually with the steps described above in the vPing common section with the
+addition::
+
+ neutron floatingip-create <EXT_NET_NAME>
+ nova floating-ip-associate nova-test <FLOATING_IP>
+
+
+Further troubleshooting is out of scope of this document, as it might be due to
+problems with the SDN controller. Contact the installer team members or send an
+email to the corresponding OPNFV mailing list for more information.
+
+
+
+vPing_userdata
+^^^^^^^^^^^^^^
+This test case does not create any floating IP neither establishes an SSH
+connection. Instead, it uses nova-metadata service when creating an instance
+to pass the same script as before (ping.sh) but as 1-line text. This script
+will be executed automatically when the second instance **opnfv-vping-2** is booted.
+
+The only known problem here for this test to fail is mainly the lack of support
+of cloud-init (nova-metadata service). Check the console of the instance::
+
+ nova console-log opnfv-vping-2
+
+If this text or similar is shown::
+
+ checking http://169.254.169.254/2009-04-04/instance-id
+ failed 1/20: up 1.13. request failed
+ failed 2/20: up 13.18. request failed
+ failed 3/20: up 25.20. request failed
+ failed 4/20: up 37.23. request failed
+ failed 5/20: up 49.25. request failed
+ failed 6/20: up 61.27. request failed
+ failed 7/20: up 73.29. request failed
+ failed 8/20: up 85.32. request failed
+ failed 9/20: up 97.34. request failed
+ failed 10/20: up 109.36. request failed
+ failed 11/20: up 121.38. request failed
+ failed 12/20: up 133.40. request failed
+ failed 13/20: up 145.43. request failed
+ failed 14/20: up 157.45. request failed
+ failed 15/20: up 169.48. request failed
+ failed 16/20: up 181.50. request failed
+ failed 17/20: up 193.52. request failed
+ failed 18/20: up 205.54. request failed
+ failed 19/20: up 217.56. request failed
+ failed 20/20: up 229.58. request failed
+ failed to read iid from metadata. tried 20
+
+it means that the instance failed to read from the metadata service. Contact
+the Functest or installer teams for more information.
+
+NOTE: Cloud-init in not supported on scenarios dealing with ONOS and the tests
+have been excluded from CI in those scenarios.
+
+
+Tempest
+^^^^^^^
+
+In the upstream OpenStack CI all the Tempest test cases are supposed to pass.
+If some test cases fail in an OPNFV deployment, the reason is very probably one
+of the following
+
++-----------------------------+-----------------------------------------------------+
+| Error | Details |
++=============================+=====================================================+
+| Resources required for test | Such resources could be e.g. an external network |
+| case execution are missing | and access to the management subnet (adminURL) from |
+| | the Functest docker container. |
++-----------------------------+-----------------------------------------------------+
+| OpenStack components or | Check running services in the controller and compute|
+| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).|
+| configured properly | Configuration parameters can be verified from the |
+| | related .conf files located under '/etc/<component>'|
+| | directories. |
++-----------------------------+-----------------------------------------------------+
+| Some resources required for | The tempest.conf file, automatically generated by |
+| execution test cases are | Rally in Functest, does not contain all the needed |
+| missing | parameters or some parameters are not set properly. |
+| | The tempest.conf file is located in directory |
+| | '/home/opnfv/.rally/tempest/for-deployment-<UUID>' |
+| | in the Functest Docker container. Use the "rally |
+| | deployment list" command in order to check the UUID |
+| | the UUID of the current deployment. |
++-----------------------------+-----------------------------------------------------+
+
+
+When some Tempest test case fails, captured traceback and possibly also the
+related REST API requests/responses are output to the console. More detailed debug
+information can be found from tempest.log file stored into related Rally deployment
+folder.
+
+
+Rally
+^^^^^
+
+The same error causes which were mentioned above for Tempest test cases, may also
+lead to errors in Rally as well.
+
+It is possible to run only one Rally scenario, instead of the whole suite.
+To do that, call the alternative python script as follows::
+
+ python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -h
+ usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name
+
+ positional arguments:
+ test_name Module name to be tested. Possible values are : [
+ authenticate | glance | cinder | heat | keystone | neutron |
+ nova | quotas | requests | vm | all ] The 'all' value
+ performs all possible test scenarios
+
+ optional arguments:
+ -h, --help show this help message and exit
+ -d, --debug Debug mode
+ -r, --report Create json result file
+ -s, --smoke Smoke test mode
+ -v, --verbose Print verbose info about the progress
+ -n, --noclean Don't clean the created resources for this test.
+
+For example, to run the Glance scenario with debug information::
+
+ python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/run_rally-cert.py -d glance
+
+Possible scenarios are:
+ * authenticate
+ * glance
+ * cinder
+ * heat
+ * keystone
+ * neutron
+ * nova
+ * quotas
+ * requests
+ * vm
+
+To know more about what those scenarios are doing, they are defined in directory:
+*$REPOS_DIR/functest/functest/opnfv_tests/OpenStack/rally/scenario*
+For more info about Rally scenario definition please refer to the Rally official
+documentation. `[3]`_
+
+If the flag *all* is specified, it will run all the scenarios one by one. Please
+note that this might take some time (~1,5hr), taking around 1 hour alone to
+complete the Nova scenario.
+
+To check any possible problems with Rally, the logs are stored under
+*/home/opnfv/functest/results/rally/* in the Functest Docker container.
+
+
+Controllers
+-----------
+
+Opendaylight
+^^^^^^^^^^^^
+
+If the Basic Restconf test suite fails, check that the ODL controller is
+reachable and its Restconf module has been installed.
+
+If the Neutron Reachability test fails, verify that the modules
+implementing Neutron requirements have been properly installed.
+
+If any of the other test cases fails, check that Neutron and ODL have
+been correctly configured to work together. Check Neutron configuration
+files, accounts, IP addresses etc.).
+
+
+ONOS
+^^^^
+
+Please refer to the ONOS documentation. `ONOSFW User Guide`_ .
+
+
+Features
+--------
+
+Please refer to the dedicated feature user guides for details.
+
+
+security_scan
+^^^^^^^^^^^^^
+
+See OpenSCAP web site: https://www.open-scap.org/
+
+
+
+NFV
+---
+
+cloudify_ims
+^^^^^^^^^^^^
+vIMS deployment may fail for several reasons, the most frequent ones are
+described in the following table:
+
++-----------------------------------+------------------------------------+
+| Error | Comments |
++===================================+====================================+
+| Keystone admin API not reachable | Impossible to create vIMS user and |
+| | tenant |
++-----------------------------------+------------------------------------+
+| Impossible to retrieve admin role | Impossible to create vIMS user and |
+| id | tenant |
++-----------------------------------+------------------------------------+
+| Error when uploading image from | impossible to deploy VNF |
+| OpenStack to glance | |
++-----------------------------------+------------------------------------+
+| Cinder quota cannot be updated | Default quotas not sufficient, they|
+| | are adapted in the script |
++-----------------------------------+------------------------------------+
+| Impossible to create a volume | VNF cannot be deployed |
++-----------------------------------+------------------------------------+
+| SSH connection issue between the | if vPing test fails, vIMS test will|
+| Test Docker container and the VM | fail... |
++-----------------------------------+------------------------------------+
+| No Internet access from the VM | the VMs of the VNF must have an |
+| | external access to Internet |
++-----------------------------------+------------------------------------+
+| No access to OpenStack API from | Orchestrator can be installed but |
+| the VM | the vIMS VNF installation fails |
++-----------------------------------+------------------------------------+
+
+
+parser
+^^^^^^
+
+For now log info is the only way to do trouble shooting
+
+
+.. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/devguide/#