aboutsummaryrefslogtreecommitdiffstats
path: root/docs/release-notes/functest-release.rst
diff options
context:
space:
mode:
authorMorgan Richomme <morgan.richomme@orange.com>2016-04-27 15:59:03 +0000
committerGerrit Code Review <gerrit@172.30.200.206>2016-04-27 15:59:03 +0000
commit5506086371536397dc532f2da7ae2bf420c260d6 (patch)
treeebca197f7bf345293e80d416b309b1e0e68a7235 /docs/release-notes/functest-release.rst
parent9c13968cdb32806da738dc520d45270e019f0ad3 (diff)
parent385be28ce7c48928361d1ec70c54cb2c73450f9e (diff)
Merge "Prepare Brahmaputra 3.0 release"
Diffstat (limited to 'docs/release-notes/functest-release.rst')
-rw-r--r--docs/release-notes/functest-release.rst99
1 files changed, 42 insertions, 57 deletions
diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst
index 0c1f61f74..65480a23a 100644
--- a/docs/release-notes/functest-release.rst
+++ b/docs/release-notes/functest-release.rst
@@ -1,6 +1,6 @@
-===========================================
-OPNFV Brahmaputra release note for Functest
-===========================================
+==============================================
+OPNFV Brahmaputra3.0 release note for Functest
+==============================================
Abstract
========
@@ -28,6 +28,9 @@ Version history
| 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release |
| | | (Orange) | |
+------------+----------+------------------+------------------------+
+| 2016-04-27 | 3.0.0 | Morgan Richomme | Add scenarios |
+| | | (Orange) | |
++------------+----------+------------------+------------------------+
OPNFV Brahmaputra Release
=========================
@@ -50,6 +53,7 @@ The OPNFV projects integrated into Functest framework for automation are:
* Promise
* Doctor
* ONOSFW
+ * bgpvpn
Release Data
============
@@ -96,55 +100,39 @@ Version change
Feature evolution
-----------------
- - dockerization of Functest
-
- - renaming of vPing into vPing userdata
-
- - Tempest update, use custom list of test cases
-
- - Rally update, use global scenario
-
- - Update jenkins logs
-
- - support of the different scenarios
+ - support new scenarios
New features
------------
- - introduction of a new vPing test case vPing ssh
-
- - introduction of vIMS test case
+ - minor bug fixes (formating)
- - support of Promise
+ - Modification of the configuration to support vPing_userdata on ONOS scenario
- - support of Doctor
-
- - support of ONOSFW
-
- - scenario management system
-
- - creation of a Test collection API
-
- - creation of the Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
-
- - creation of Functest dashboard: http://testresults.opnfv.org/dashboard/
+ - Use serial option in Tempest to improve success rate
Scenario Matrix
===============
-For Brahmaputra, Functest supports the following scenarios:
+For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios:
+----------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+================+=========+=========+=========+=========+
| odl_l2 | X | X | X | X |
+----------------+---------+---------+---------+---------+
-| onos | | X | | |
+| onos | | X | X | |
+----------------+---------+---------+---------+---------+
| nosdn | | X | X | |
+----------------+---------+---------+---------+---------+
| ovs (dpdk) | | | X | |
+----------------+---------+---------+---------+---------+
+| kvm | | | X | |
++----------------+---------+---------+---------+---------+
+| bgpvpn | X | | X | |
++----------------+---------+---------+---------+---------+
+| sfc | | | X | |
++----------------+---------+---------+---------+---------+
Functest defines the success criteria when having at least 4 consecutive
successful runs of a given scenario from the Continuous Integration.
@@ -158,6 +146,7 @@ The success criteria is defined as follows:
* ODL success rate = 100%
* ONOSFW success rate = 100%
* Promise success rate = 100%
+ * Bgpvpn success rate = 100%
* vIMS: deployement of the orchestrator and the vIMS VNF successful
Other scenarios are currently available but did not meet success criteria for
@@ -167,19 +156,23 @@ release.
Brahmaputra limitations
-----------------------
-- Fuel and Apex Tempest success rate was below 90% but above 80%. Some of the
-error causes were identified (workers, lack of IP)
+- Fuel and Apex Tempest success rate was below 90% but above 80% on some
+scenarios. Some of the error causes were identified (workers, lack of IP)
- vIMS failed in CI for joid/odl_l2 scenario
- vPing userdata and vIMS excluded from onos scenario
-- None of the odl_l3 scenarios has been successful due to vPing ssh issue.
+- vPing_ssh and vIMS excluded from bgpvpn and kvm scenario
-- joid/nosdn successful but the complete scenario not run 4 times in a raw
+- None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL
+bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582)
- apex/nosdn never run (not a target scenario) but probably succesful
+- vPing SSH and vPing userdata no more run on CI since modification of bgpvpn
+configuration regex.
+
See known issues section for details
Test and installer/scenario dependencies
@@ -191,10 +184,9 @@ following table details the dependencies of the test cases per scenario.
+----------------+-------------+-------------+-------------+-------------+
| Test cases | Apex | Compass | Fuel | Joid |
+================+=============+=============+=============+=============+
-| vPing SSH | all | all | all | all |
+| vPing SSH | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
-| vPing userdata | all except | all except | all except | all except |
-| | ONOS | ONOS | ONOS | ONOS |
+| vPing userdata | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
| Tempest | all | all | all | all |
+----------------+-------------+-------------+-------------+-------------+
@@ -211,6 +203,8 @@ following table details the dependencies of the test cases per scenario.
+----------------+-------------+-------------+-------------+-------------+
| Doctor | all | no | no | no |
+----------------+-------------+-------------+-------------+-------------+
+| Bgpvpn | all | no | all | no |
++----------------+-------------+-------------+-------------+-------------+
Test results
============
@@ -226,18 +220,6 @@ Test results are available in:
Known issues
------------
- - nova metadata service not supported in ONOS:
-
- - it excludes vPing userdata and vIMS test cases
-
- - Rally worker issues:
-
- - A workaround has been implemented to run Tempest test cases sequentially
-
- - It may explain why Tempest scenarios (mainly on apex) do not run the 210 tests
-
- - https://bugs.launchpad.net/testrepository/+bug/1538941
-
- IPv6 issues in tempest suite:
- tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
@@ -248,7 +230,7 @@ Known issues
- Lack of IP addresses available lead to several errors in different test cases
- - vIMS:
+ - vIMS (http://testresults.opnfv.org/reporting/vims/):
- the VM needs to have access to OpenStack API.
@@ -258,24 +240,27 @@ Known issues
- That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
+ - case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs)
+
Open JIRA tickets
=================
+------------------+-----------------------------------------+
| JIRA | Description |
+==================+=========================================+
+| FUNCTEST-231 | vPing SSH no more run systematically |
+| | in CI |
++------------------+-----------------------------------------+
+| FUNCTEST-230 | Heat issues in Rally scenarios |
++------------------+-----------------------------------------+
+| FUNCTEST-229 | Extend reporting to brahmaputra |
++------------------+-----------------------------------------+
| FUNCTEST-139 | prepare_env failed due to |
| | https://pypi.python.org/samples is not |
| | accessible |
+------------------+-----------------------------------------+
-| FUNCTEST-137 | Tempest success rate below 90 on apex |
-+------------------+-----------------------------------------+
-| FUNCTEST-136 | Tempest success rate below 90 on fuel |
-+------------------+-----------------------------------------+
| FUNCTEST-135 | vPing scenario failed in odl_l3 scenario|
+------------------+-----------------------------------------+
-| FUNCTEST-124 | odl test suite troubleshooting |
-+------------------+-----------------------------------------+
Useful links
============