Merge "Prepare Brahmaputra 3.0 release"

1 files changed, 42 insertions, 57 deletions

diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst
index 0c1f61f74..65480a23a 100644
--- a/docs/release-notes/functest-release.rst
+++ b/docs/release-notes/functest-release.rst
@@ -1,6 +1,6 @@
-===========================================
-OPNFV Brahmaputra release note for Functest
-===========================================
+==============================================
+OPNFV Brahmaputra3.0 release note for Functest
+==============================================
 
 Abstract
 ========
@@ -28,6 +28,9 @@ Version history
 | 2016-02-25 | 1.0.0    | Morgan Richomme  | Functest for B release |
 |            |          | (Orange)         |                        |
 +------------+----------+------------------+------------------------+
+| 2016-04-27 | 3.0.0    | Morgan Richomme  | Add scenarios          |
+|            |          | (Orange)         |                        |
++------------+----------+------------------+------------------------+
 
 OPNFV Brahmaputra Release
 =========================
@@ -50,6 +53,7 @@ The OPNFV projects integrated into Functest framework for automation are:
  * Promise
  * Doctor
  * ONOSFW
+ * bgpvpn
 
 Release Data
 ============
@@ -96,55 +100,39 @@ Version change
 Feature evolution
 -----------------
 
- - dockerization of Functest
-
- - renaming of vPing into vPing userdata
-
- - Tempest update, use custom list of test cases
-
- - Rally update, use global scenario
-
- - Update jenkins logs
-
- - support of the different scenarios
+ - support new scenarios
 
 New features
 ------------
 
- - introduction of a new vPing test case vPing ssh
-
- - introduction of vIMS test case
+ - minor bug fixes (formating)
 
- - support of Promise
+ - Modification of the configuration to support vPing_userdata on ONOS scenario
 
- - support of Doctor
-
- - support of ONOSFW
-
- - scenario management system
-
- - creation of a Test collection API
-
- - creation of the Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
-
- - creation of Functest dashboard: http://testresults.opnfv.org/dashboard/
+ - Use serial option in Tempest to improve success rate
 
 Scenario Matrix
 ===============
 
-For Brahmaputra, Functest supports the following scenarios:
+For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios:
 
 +----------------+---------+---------+---------+---------+
 |    Scenario    |  Apex   | Compass |  Fuel   |   Joid  |
 +================+=========+=========+=========+=========+
 |   odl_l2       |    X    |    X    |    X    |    X    |
 +----------------+---------+---------+---------+---------+
-|   onos         |         |    X    |         |         |
+|   onos         |         |    X    |    X    |         |
 +----------------+---------+---------+---------+---------+
 |   nosdn        |         |    X    |    X    |         |
 +----------------+---------+---------+---------+---------+
 |   ovs (dpdk)   |         |         |    X    |         |
 +----------------+---------+---------+---------+---------+
+|   kvm          |         |         |    X    |         |
++----------------+---------+---------+---------+---------+
+|   bgpvpn       |    X    |         |    X    |         |
++----------------+---------+---------+---------+---------+
+|   sfc          |         |         |    X    |         |
++----------------+---------+---------+---------+---------+
 
 Functest defines the success criteria when having at least 4 consecutive
 successful runs of a given scenario from the Continuous Integration.
@@ -158,6 +146,7 @@ The success criteria is defined as follows:
  * ODL success rate = 100%
  * ONOSFW success rate = 100%
  * Promise success rate = 100%
+ * Bgpvpn success rate = 100%
  * vIMS: deployement of the orchestrator and the vIMS VNF successful
 
 Other scenarios are currently available but did not meet success criteria for
@@ -167,19 +156,23 @@ release.
 Brahmaputra limitations
 -----------------------
 
-- Fuel and Apex Tempest success rate was below 90% but above 80%. Some of the
-error causes were identified (workers, lack of IP)
+- Fuel and Apex Tempest success rate was below 90% but above 80% on some
+scenarios. Some of the error causes were identified (workers, lack of IP)
 
 - vIMS failed in CI for joid/odl_l2 scenario
 
 - vPing userdata and vIMS excluded from onos scenario
 
-- None of the odl_l3 scenarios has been successful due to vPing ssh issue.
+- vPing_ssh and vIMS excluded from bgpvpn and kvm scenario
 
-- joid/nosdn successful but the complete scenario not run 4 times in a raw
+- None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL
+bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582)
 
 - apex/nosdn never run (not a target scenario) but probably succesful
 
+- vPing SSH and vPing userdata no more run on CI since modification of bgpvpn
+configuration regex.
+
 See known issues section for details
 
 Test and installer/scenario dependencies
@@ -191,10 +184,9 @@ following table details the dependencies of the test cases per scenario.
 +----------------+-------------+-------------+-------------+-------------+
 |  Test cases    |    Apex     |   Compass   |    Fuel     |     Joid    |
 +================+=============+=============+=============+=============+
-|   vPing SSH    | all         | all         | all         | all         |
+| vPing SSH      | all         | all         | all         | all         |
 +----------------+-------------+-------------+-------------+-------------+
-| vPing userdata | all except  | all except  | all except  | all except  |
-|                | ONOS        | ONOS        | ONOS        | ONOS        |
+| vPing userdata | all         | all         | all         | all         |
 +----------------+-------------+-------------+-------------+-------------+
 | Tempest        | all         | all         | all         | all         |
 +----------------+-------------+-------------+-------------+-------------+
@@ -211,6 +203,8 @@ following table details the dependencies of the test cases per scenario.
 +----------------+-------------+-------------+-------------+-------------+
 | Doctor         | all         | no          | no          | no          |
 +----------------+-------------+-------------+-------------+-------------+
+| Bgpvpn         | all         | no          | all         | no          |
++----------------+-------------+-------------+-------------+-------------+
 
 Test results
 ============
@@ -226,18 +220,6 @@ Test results are available in:
 Known issues
 ------------
 
- - nova metadata service not supported in ONOS:
-
-    - it excludes vPing userdata and vIMS test cases
-
- - Rally worker issues:
-
-    - A workaround has been implemented to run Tempest test cases sequentially
-
-    - It may explain why Tempest scenarios (mainly on apex) do not run the 210 tests
-
-    - https://bugs.launchpad.net/testrepository/+bug/1538941
-
  - IPv6 issues in tempest suite:
 
     - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
@@ -248,7 +230,7 @@ Known issues
 
  - Lack of IP addresses available lead to several errors in different test cases
 
- - vIMS:
+ - vIMS (http://testresults.opnfv.org/reporting/vims/):
 
     - the VM needs to have access to OpenStack API.
 
@@ -258,24 +240,27 @@ Known issues
 
     - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
 
+    - case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs)
+
 Open JIRA tickets
 =================
 
 +------------------+-----------------------------------------+
 |   JIRA           |         Description                     |
 +==================+=========================================+
+| FUNCTEST-231     | vPing SSH no more run systematically    |
+|                  | in CI                                   |
++------------------+-----------------------------------------+
+| FUNCTEST-230     | Heat issues in Rally scenarios          |
++------------------+-----------------------------------------+
+| FUNCTEST-229     | Extend reporting to brahmaputra         |
++------------------+-----------------------------------------+
 | FUNCTEST-139     | prepare_env failed due to               |
 |                  | https://pypi.python.org/samples is not  |
 |                  | accessible                              |
 +------------------+-----------------------------------------+
-| FUNCTEST-137     | Tempest success rate below 90 on apex   |
-+------------------+-----------------------------------------+
-| FUNCTEST-136     | Tempest success rate below 90 on fuel   |
-+------------------+-----------------------------------------+
 | FUNCTEST-135     | vPing scenario failed in odl_l3 scenario|
 +------------------+-----------------------------------------+
-| FUNCTEST-124     | odl test suite troubleshooting          |
-+------------------+-----------------------------------------+
 
 Useful links
 ============