Update documentation for Danube

JIRA: FUNCTEST-559
JIRA: FUNCTEST-560

Change-Id: I2104a9b68bb3985c9f5900ebc30bab89b09b7835
Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>

1 files changed, 259 insertions, 244 deletions

diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst
index ce5dc77be..d5295903c 100644
--- a/docs/testing/developer/devguide/index.rst
+++ b/docs/testing/developer/devguide/index.rst
@@ -1,3 +1,6 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
 ******************************
 OPNFV FUNCTEST developer guide
 ******************************
@@ -62,18 +65,18 @@ Functest internal test cases
 ============================
 The internal test cases in Danube are:
 
- * healthcheck
- * connection_check
+
  * api_check
+ * cloudify_ims
+ * connection_check
  * vping_ssh
  * vping_userdata
  * odl
- * snaps_smoke
- * tempest_smoke_serial
+ * rally_full
  * rally_sanity
+ * snaps_health_check
  * tempest_full_parallel
- * rally_full
- * cloudify_ims
+ * tempest_smoke_serial
 
 By internal, we mean that this particular test cases have been
 developped and/or integrated by functest contributors and the associated
@@ -86,7 +89,7 @@ The main internal test cases are in the opnfv_tests subfolder of the
 repository, the internal test cases are:
 
  * sdn: odl, onos
- * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke
+ * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*, snaps_smoke
  * vnf: cloudify_ims
 
 If you want to create a new test case you will have to create a new
@@ -99,19 +102,23 @@ especially the feature projects.
 
 The external test cases are:
 
- * promise
- * doctor
- * onos
+ * barometer
  * bgpvpn
- * copper
- * security_scan
- * sfc-odl
- * sfc-onos
- * parser
+ * doctor
  * domino
+ * odl-netvirt
+ * onos
+ * fds
  * multisite
- * opera_ims
+ * netready
  * orchestra_ims
+ * parser
+ * promise
+ * refstack_defcore
+ * security_scan
+ * snaps_smoke
+ * sfc-odl
+ * vyos_vrouter
 
 
 The code to run these test cases may be directly in the repository of
@@ -155,7 +162,7 @@ introduced in Danube:
 
 The goal is to unify the way to run test from Functest.
 
-feature_base and vnf_base inherit from testcase_base.
+feature_base and vnf_base inherit from testcase_base::
 
               +-----------------------------------------+
               |                                         |
@@ -174,9 +181,9 @@ feature_base and vnf_base inherit from testcase_base.
          |    feature_base    |   |      vnf_base            |
          |                    |   |                          |
          |  - prepare()       |   |  - prepare()             |
-         |  - post()          |   |  - deploy_orchestrator() |
-         |  - parse_results() |   |  - deploy_vnf()          |
-         |                    |   |  - test_vnf()            |
+         |  - execute()       |   |  - deploy_orchestrator() |
+         |  - post()          |   |  - deploy_vnf()          |
+         |  - parse_results() |   |  - test_vnf()            |
          |                    |   |  - clean()               |
          |                    |   |  - execute()             |
          |                    |   |                          |
@@ -270,232 +277,13 @@ the API will return an error message.
 An additional method dashboard has been added to post-process
 the raw results in release Brahmaputra (deprecated in Colorado).
 
-The data model is very basic, 4 objects are created:
+The data model is very basic, 5 objects are created:
 
   * Pods
   * Projects
   * Testcases
   * Results
-
-Pods::
-
-  {
-    "id": <ID>,
-    "details": <URL description of the POD>,
-    "creation_date": "YYYY-MM-DD HH:MM:SS",
-    "name": <The POD Name>,
-    "mode": <metal or virtual>,
-    "role": <ci-pod or community-pod or single-node>
-  },
-
-Projects::
-
-  {
-    "id": <ID>,
-    "name": <Name of the Project>,
-    "creation_date": "YYYY-MM-DD HH:MM:SS",
-    "description": <Short description>
-  },
-
-Testcases::
-
-  {
-    "id": <ID>,
-    "name":<Name of the test case>,
-    "project_name":<Name of belonged project>,
-    "creation_date": "YYYY-MM-DD HH:MM:SS",
-    "description": <short description>,
-    "url":<URL for longer description>
-  },
-
-Results::
-
-  {
-    "_id": <ID>,
-    "case_name": <Reference to the test case>,
-    "project_name": <Reference to project>,
-    "pod_name": <Reference to POD where the test was executed>,
-    "installer": <Installer Apex or Compass or Fuel or Joid>,
-    "version": <master or Colorado or Brahmaputra>,
-    "start_date": "YYYY-MM-DD HH:MM:SS",
-    "stop_date": "YYYY-MM-DD HH:MM:SS",
-    "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">,
-    "scenario": <Scenario on which the test was executed>,
-    "criteria": <PASS or FAILED>,
-    "trust_indicator": {
-                        "current": 0,
-                        "histories": []
-                       }
-  }
-
-The API can described as follows. For detailed information, please go to
-
- http://testresults.opnfv.org/test/swagger/spec.html
-
- Authentication: opnfv/api@opnfv
-
-Version:
-
- +--------+--------------------------+-----------------------------------------+
- | Method | Path                     | Description                             |
- +========+==========================+=========================================+
- | GET    | /versions                | Get all supported API versions          |
- +--------+--------------------------+-----------------------------------------+
-
-
-Pods:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path                       | Description                             |
- +========+============================+=========================================+
- | GET    | /api/v1/pods               | Get the list of declared Labs (PODs)    |
- +--------+----------------------------+-----------------------------------------+
- | POST   | /api/v1/pods               | Declare a new POD                       |
- |        |                            | Content-Type: application/json          |
- |        |                            | {                                       |
- |        |                            | "name": "pod_foo",                      |
- |        |                            | "mode": "metal",                        |
- |        |                            | "role": "ci-pod",                       |
- |        |                            | "details": "it is a ci pod"             |
- |        |                            | }                                       |
- +--------+----------------------------+-----------------------------------------+
- | GET    | /api/v1/pods/{pod_name}    | Get a declared POD                      |
- +--------+----------------------------+-----------------------------------------+
-
-Projects:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path                       | Description                             |
- +========+============================+=========================================+
- | GET    | /api/v1/projects           | Get the list of declared projects       |
- +--------+----------------------------+-----------------------------------------+
- | POST   | /api/v1/projects           | Declare a new test project              |
- |        |                            | Content-Type: application/json          |
- |        |                            | {                                       |
- |        |                            | "name": "project_foo",                  |
- |        |                            | "description": "whatever you want"      |
- |        |                            | }                                       |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project} | Delete a test project                   |
- +--------+----------------------------+-----------------------------------------+
- | GET    | /api/v1/projects/{project} | Get details on a {project}              |
- |        |                            |                                         |
- +--------+----------------------------+-----------------------------------------+
- | PUT    | /api/v1/projects/{project} | Update a test project                   |
- |        |                            |                                         |
- |        |                            | Content-Type: application/json          |
- |        |                            | {                                       |
- |        |                            | <the field(s) you want to modify>       |
- |        |                            | }                                       |
- +--------+----------------------------+-----------------------------------------+
-
-
-Testcases:
-
- +--------+----------------------------+-----------------------------------------+
- | Method | Path                       | Description                             |
- +========+============================+=========================================+
- | GET    | /api/v1/projects/{project}/| Get the list of testcases of {project}  |
- |        | cases                      |                                         |
- +--------+----------------------------+-----------------------------------------+
- | POST   | /api/v1/projects/{project}/| Add a new test case to {project}        |
- |        | cases                      | Content-Type: application/json          |
- |        |                            | {                                       |
- |        |                            | "name": "case_foo",                     |
- |        |                            | "description": "whatever you want"      |
- |        |                            | "url": "whatever you want"              |
- |        |                            | }                                       |
- +--------+----------------------------+-----------------------------------------+
- | DELETE | /api/v1/projects/{project}/| Delete a test case                      |
- |        | cases/{case}               |                                         |
- +--------+----------------------------+-----------------------------------------+
- | GET    | /api/v1/projects/{project}/| Get a declared test case                |
- |        | cases/{case}               |                                         |
- +--------+----------------------------+-----------------------------------------+
- | PUT    | /api/v1/projects/{project}?| Modify a test case of {project}         |
- |        | cases/{case}               |                                         |
- |        |                            | Content-Type: application/json          |
- |        |                            | {                                       |
- |        |                            | <the field(s) you want to modify>       |
- |        |                            | }                                       |
- +--------+----------------------------+-----------------------------------------+
-
-Results:
-
- +--------+----------------------------+------------------------------------------+
- | Method | Path                       | Description                              |
- +========+============================+==========================================+
- | GET    | /api/v1/results            | Get all the test results                 |
- +--------+----------------------------+------------------------------------------+
- | POST   | /api/v1/results            | Add a new test results                   |
- |        |                            | Content-Type: application/json           |
- |        |                            | {                                        |
- |        |                            | "project_name": "project_foo",           |
- |        |                            | "scenario": "odl-l2",                    |
- |        |                            | "stop_date": "2016-05-28T14:42:58.384Z", |
- |        |                            | "trust_indicator": 0.5,                  |
- |        |                            | "case_name": "vPing",                    |
- |        |                            | "build_tag": "",                         |
- |        |                            | "version": "Colorado",                   |
- |        |                            | "pod_name": "pod_foo",                   |
- |        |                            | "criteria": "PASS",                      |
- |        |                            | "installer": "fuel",                     |
- |        |                            | "start_date": "2016-05-28T14:41:58.384Z",|
- |        |                            | "details": <your results>                |
- |        |                            | }                                        |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of {case}           |
- |        | case={case}                |                                          |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of build_tag        |
- |        | build_tag={tag}            | {tag}.                                   |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get last {N} records of test results     |
- |        | last={N}                   |                                          |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of scenario         |
- |        | scenario={scenario}        | {scenario}.                              |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of trust_indicator  |
- |        | trust_indicator={ind}      | {ind}.                                   |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of last days        |
- |        | period={period}            | {period}.                                |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of {project}        |
- |        | project={project}          |                                          |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of version          |
- |        | version={version}          | {version}.                               |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of criteria         |
- |        | criteria={criteria}        | {criteria}.                              |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | get the results on pod {pod}             |
- |        | pod={pod}                  |                                          |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the test results of installer {inst} |
- |        | installer={inst}           |                                          |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results?           | Get the results according to combined    |
- |        | <query conditions>         | query conditions supported above         |
- +--------+----------------------------+------------------------------------------+
- | GET    | /api/v1/results/{result_id}| Get the test result by result_id         |
- +--------+----------------------------+------------------------------------------+
-
-Scenarios:
-
-  +--------+----------------------------+-----------------------------------------+
-  | Method | Path                       | Description                             |
-  +========+============================+=========================================+
-  | GET    | /api/v1/scenarios          | Get the list of declared scenarios      |
-  +--------+----------------------------+-----------------------------------------+
-  | POST   | /api/v1/scenario           | Declare a new scenario                  |
-  +--------+----------------------------+-----------------------------------------+
-  | GET    | /api/v1/scenario?          | Get a declared scenario                 |
-  |        | <query conditions>         |                                         |
-  +--------+----------------------------+-----------------------------------------+
-
+  * Scenarios
 
 The code of the API is hosted in the releng repository `[6]`_.
 The static documentation of the API can be found at `[17]`_.
@@ -574,6 +362,7 @@ Please note that currently token authorization is implemented but is not yet ena
    +---------------------+---------+---------+---------+---------+
    | copper              |    X    |         |         |    X    |
    +---------------------+---------+---------+---------+---------+
+   src: colorado (see release note for the last matrix version)
 
   All the testcases listed in the table are runnable on
   os-odl_l2-nofeature scenarios.
@@ -736,8 +525,7 @@ Regex are standard regex. You can have a look at  `[11]`_
 
 You can also easily test your regex via an online regex checker such as `[12]`_.
 Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put
-your regex in the REGULAR EXPRESSION window, then you can test your rule
-.
+your regex in the REGULAR EXPRESSION window, then you can test your rule.
 
 
 How to know which test I can run?
@@ -941,6 +729,233 @@ You can also reuse a python function defined in functest_utils.py::
           return False
 
 
+Where can I find the documentation on the test API?
+===================================================
+
+http://artifacts.opnfv.org/releng/docs/testapi.html
+
+
+How to exclude Tempest case from default Tempest smoke suite?
+=============================================================
+
+Tempest default smoke suite deals with 165 test cases.
+Since Colorado the success criteria is 100%, i.e. if 1 test is failed the
+success criteria is not matched for the scenario.
+
+It is necessary to exclude some test cases that are expected to fail due to
+known upstream bugs (see release notes).
+
+A file has been created for such operation: https://git.opnfv.org/cgit/functest/tree/functest/opnfv_tests/openstack/tempest/custom_tests/blacklist.txt.
+
+It can be described as follows::
+
+    -
+        scenarios:
+            - os-odl_l2-bgpvpn-ha
+            - os-odl_l2-bgpvpn-noha
+        installers:
+            - fuel
+            - apex
+        tests:
+            - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers
+            - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
+            - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers
+            - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
+            - tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
+            - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
+            - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
+            - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
+            - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
+
+Please note that each exclusion must be justified. the goal is not to exclude
+test cases because they do not pass. Several scenarios reached the 100% criteria.
+So it is expected in the patch submited to exclude the cases to indicate the
+reasons of the exclusion.
+
+
+How do I know the Functest status of a scenario?
+================================================
+
+A Functest automatic reporting page is generated daily.
+This page is dynamically created through a cron job and is based on the results
+stored in the Test DB.
+You can access this reporting page: http://testresults.opnfv.org/reporting
+
+See https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 for details.
+
+
+I have tests, to which category should I declare them?
+======================================================
+
+CATEGORIES/TIERS description:
+
++----------------+-------------------------------------------------------------+
+| healthcheck    | Simple OpenStack healtcheck tests case that validates the   |
+|                | basic operations in OpenStack                               |
++----------------+-------------------------------------------------------------+
+|   Smoke        | Set of smoke test cases/suites to validate the most common  |
+|                | OpenStack and SDN Controller operations                     |
++----------------+-------------------------------------------------------------+
+|   Features     | Test cases that validate a specific feature on top of OPNFV.|
+|                | Those come from Feature projects and need a bit of support  |
+|                | for integration                                             |
++----------------+-------------------------------------------------------------+
+| Components     | Advanced Openstack tests: Full Tempest, Full Rally          |
++----------------+-------------------------------------------------------------+
+| Performance    | Out of Functest Scope                                       |
++----------------+-------------------------------------------------------------+
+| VNF            | Test cases related to deploy an open source VNF including   |
+|                | an orchestrator                                             |
++----------------+-------------------------------------------------------------+
+
+The main ambiguity could be between features and VNF.
+In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
+feature you introduced.
+We recommend to declare your test in the feature category.
+
+VNF category is really dedicated to test including:
+
+ * creation of resources
+ * deployement of an orchestrator/VNFM
+ * deployment of the VNF
+ * test of the VNFM
+ * free resources
+
+The goal is not to study a particular feature on the infrastructure but to have
+a whole end to end test of a VNF automatically deployed in CI.
+Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
+jobs and use to get a scenario score.
+
+Where are the logs?
+===================
+
+Functest deals with internal and external testcases. Each testcase can generate
+logs.
+
+Since Colorado we introduce the possibility to push the logs to the artifact.
+A new script (https://git.opnfv.org/releng/tree/utils/push-test-logs.sh) has
+been created for CI.
+
+When called, and assuming that the POD is authorized to push the logs to
+artifacts, the script will push all the results or logs locally stored under
+/home/opnfv/functest/results/.
+
+If the POD is not connected to CI, logs are not pushed.
+But in both cases, logs are stored in /home/opnfv/functest/results in the
+container.
+Projects are encouraged to push their logs here.
+
+Since Colorado it is also easy for feature project to integrate this feature by
+adding the log file as output_file parameter when calling execute_command from
+functest_utils library
+
+  ret_val = functest_utils.execute_command(cmd, output_file=log_file)
+
+
+How does Functest deal with VNF onboarding?
+===========================================
+
+VNF onboarding has been introduced in Brahmaputra through the automation of a
+clearwater vIMS deployed thanks to cloudify orchestrator.
+
+This automation has been described at OpenStack summit Barcelona:
+https://youtu.be/Jr4nG74glmY
+
+The goal of Functest consists in testing OPNFV from a functional perspective:
+the NFVI and/or the features developed in OPNFV. Feature test suites are
+provided by the feature project. Functest just simplifies the integration of
+the suite into the CI and gives a consolidated view of the tests per scenario.
+
+Functest does not develop VNFs.
+
+Functest does not test any MANO stack.
+
+OPNFV projects dealing with VNF onboarding
+------------------------------------------
+
+Testing VNF is not the main goal however it gives interesting and realistic
+feedback on OPNFV as a Telco cloud.
+
+Onboarding VNF also allows to test a full stack: orchestrator + VNF.
+
+Functest is VNF and MANO stack agnostic.
+
+An internship has been initiated to reference the Open Source VNF: Intern
+Project Open Source VNF catalog
+
+New projects dealing with orchestrators or VNFs are candidate for Danube.
+
+The 2 projects dealing with orchestration are:
+
+ * orchestra (Openbaton)
+ * opera (Open-O)
+
+The Models project address various goals for promoting availability and
+convergence of information and/or data models related to NFV service/VNF
+management, as being defined in standards (SDOs) and as developed in open
+source projects.
+
+Functest VNF onboarding
+-----------------------
+
+In order to simplify VNF onboarding a new abstraction class has been developed
+in Functest.
+
+This class is based on vnf_base and can be described as follow:
+
+ +------------+             +--------------+
+ | test_base  |------------>|   vnf_base   |
+ +------------+             +--------------+
+                              |_ prepare
+                                  |_ deploy_orchestrator (optional)
+                                      |_ deploy_vnf
+                                        |_ test_vnf
+                                          |_ clean
+
+
+Several methods are declared in vnf_base:
+
+ * prepare
+ * deploy_orchestrator
+ * deploy_vnf
+ * test_vnf
+ * clean
+
+deploy_vnf and test_vnf are mandatory.
+
+prepare will create a user and a project.
+
+How to declare your orchestrator/VNF?
+-------------------------------------
+1) test declaration
+
+You must declare your testcase in the file <Functest repo>/functest/ci/testcases.yaml
+
+2) configuration
+
+You can precise some configuration parameters in config_functest.yaml
+
+3) implement your test
+
+Create your own VnfOnboarding file
+
+you must create your entry point through a python clase as referenced in the
+configuration file
+
+e.g. aaa => creation of the file <Functest repo>/functest/opnfv_tests/vnf/aaa/aaa.py
+
+the class shall inherit vnf_base.
+You must implement the methods deploy_vnf() and test_vnf() and may implement
+deploy_orchestrator()
+
+you can call the code from your repo (but need to add the repo in Functest if
+it is not the case)
+
+4) success criteria
+
+So far we considered the test as PASS if vnf_deploy and test_vnf is PASS
+(see example in aaa).
+
 ==========
 References
 ==========
@@ -989,4 +1004,4 @@ _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_varia
 
 _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
 
-_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
+_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml