Update release Note

JIRA: FUNCTEST-436

Change-Id: Idbb02dd26e9578fc53e77f4b30447ff037f38ef4
Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>

1 files changed, 280 insertions, 41 deletions

diff --git a/docs/release-notes/functest-release.rst b/docs/release-notes/functest-release.rst
index 47efa917..2d87253c 100644
--- a/docs/release-notes/functest-release.rst
+++ b/docs/release-notes/functest-release.rst
@@ -1,12 +1,3 @@
-===========================================
-OPNFV Colorado1.0 release note for Functest
-===========================================
-
-Abstract
-========
-
-This document describes the release note of Functest project.
-
 License
 =======
 
@@ -18,6 +9,16 @@ are licensed under a Creative Commons Attribution 4.0 International License.
 You should have received a copy of the license along with this.
 If not, see <http://creativecommons.org/licenses/by/4.0/>.
 
+===========================================
+OPNFV Colorado1.0 release note for Functest
+===========================================
+
+Abstract
+========
+
+This document describes the release note of Functest project.
+
+
 Version history
 ===============
 
@@ -25,8 +26,8 @@ Version history
 | **Date**   | **Ver.** | **Author**       | **Comment**            |
 |            |          |                  |                        |
 +------------+----------+------------------+------------------------+
-| 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for C release |
-|            |          | (Orange)         |                        |
+| 2016-08-17 | 1.0.0    | Morgan Richomme  | Functest for Colorado  |
+|            |          | (Orange)         | release                |
 +------------+----------+------------------+------------------------+
 
 OPNFV Colorado Release
@@ -103,11 +104,11 @@ Software
 Documents
 ---------
 
- - Installation/configuration guide: * TODO link *
+ - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
 
- - User Guide: * TODO link *
+ - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
 
- - Developer Guide: * TODO link *
+ - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
 
 
 Version change
@@ -138,7 +139,7 @@ New features
 
  - creation of the healthcheck test case
 
- - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, multisite)
+ - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
 
  - integration of new OPNFV feature projects (copper, domino, multisite,
  moon, parser, onos-sfc, odl-sfc, security scan)
@@ -156,43 +157,57 @@ Scenario Matrix
 ===============
 
 For Colorado 1.0, Functest was tested on the following scenarios (if not
-precised, the scenario is a na scenario):
+precised, the scenario is a HA scenario):
 
 +---------------------+---------+---------+---------+---------+
 |    Scenario         |  Apex   | Compass |  Fuel   |   Joid  |
 +=====================+=========+=========+=========+=========+
 |   nosdn             |    X    |    X    |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
+|   nosdn-noha        |         |         |    X    |    X    |
++---------------------+---------+---------+---------+---------+
 |   odl_l2            |    X    |    X    |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
+|   odl_l2-noha       |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
 |   odl_l3            |    X    |    X    |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   onos              |    X    |    X    |    X    |    X    |
+|   odl_l3-noha       |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   ocl               |         |    X    |         |         |
+|   odl_l2-bgpvpn     |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   ovs-noha (dpdk)   |         |         |    X    |         |
+|   odl_l2-bgpvpn-noha|         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   kvm-noha          |         |         |    X    |         |
+|   odl_l2-fdio-noha  |    X    |         |         |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-bgpvpn     |    X    |         |    X    |         |
+|   odl_l2-moon       |         |    X    |         |         |
 +---------------------+---------+---------+---------+---------+
 |   odl_l2-sfc        |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   onos-sfc          |    X    |    X    |    X    |    X    |
+|   odl_l2-sfc-noha   |    X    |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
-|   odl_l2-moon       |         |    X    |         |         |
+|   onos              |         |    X    |    X    |    X    |
++---------------------+---------+---------+---------+---------+
+|   onos-noha         |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   onos-sfc          |         |    X    |    X    |    X    |
 +---------------------+---------+---------+---------+---------+
-|   multisite         |         |         |         |         |
+|   onos-sfc-noha     |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   ovs-noha (dpdk)   |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   kvm               |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   kvm-noha          |         |         |    X    |         |
++---------------------+---------+---------+---------+---------+
+|   multisite         |         |         |    X    |         |
 +---------------------+---------+---------+---------+---------+
 |   lxd               |         |         |         |    X    |
 +---------------------+---------+---------+---------+---------+
+|   lxd-noha          |         |         |         |    X    |
++---------------------+---------+---------+---------+---------+
 
-Functest defines a scenario scoring based on the sum of the unitary test
-cases run in CI.
-The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
-
-In Colorado, the functional tests have been sliced in different
+In Colorado, the functional tests have been sliced in 6 different
 categories:
 
 +----------------+-----------------------------------------------+
@@ -214,16 +229,153 @@ categories:
 |                | VNF deployment and tests (e.g. vIMS)          |
 +----------------+-----------------------------------------------+
 
-For the scenario validation, we consider only the categories healthcheck,
-smoke, sdn_suites and features. These tests are run systematically in
-the CI daily loops.
+For the scenario validation, we consider only the healthcheck, smoke,
+sdn_suites and features categories. These tests are run systematically
+in the CI daily loops.
 
 Success criteria have been defined for these test cases, they can be
-PASS/FAIl or a success rate may be declared (100%, > 90%)
+PASS/FAIL or a success rate may be declared (100%, > 90%)
 All the criteria, as well as the test dependencies are declared in the
 ci/testcases.yaml file.
 
-* TODO scoring table *
+The scoring for the Colorado release per installer can be described as
+follows.
+
+The scoring is an indicator showing how many feature project test suites
+have been integrated on the scenario.
+
+The scoring equals the number of tests * succesful iteration of each
+test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
+
+ e.g.
+ apex/odl_l2-nofeature-ha
+ tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
+ Scoring = 21/21 = 7 * 3
+
+By default, if not precised, the scenarios are HA.
+HA means OpenStack High Availability (main services). For copper test,
+the OpenStack congress module is not HA. See the release notes of the
+installers for details.
+
+
+apex
+----
+
++------------------+---------+---------+-------------------+
+|  Scenario        | Scoring | Success |    Results        |
+|                  |         | rate    |                   |
++==================+=========+=========+===================+
+| nosdn            |  17/18  |   95%   | `apex-res-174`_   |
++------------------+---------+---------+-------------------+
+| odl_l2           |  21/21  |   100%  | `apex-res-175`_   |
++------------------+---------+---------+-------------------+
+| odl_l3           |  15/18  |    83%  | `apex-res-176`_   |
++------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn    |  14/18  |    78%  |     ??            |
++------------------+---------+---------+-------------------+
+| odl_l2-fdio-noha |  11/15  |    73%  |     ??            |
++------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha  |  18/21  |    86%  |     ??            |
++------------------+---------+---------+-------------------+
+
+compass
+-------
+
++------------------+---------+---------+-------------------+
+|  Scenario        | Scoring | Success |  Results          |
+|                  |         | rate    |                   |
++==================+=========+=========+===================+
+| nosdn            |  12/12  |   100%  | `compass-res-55`_ |
++------------------+---------+---------+-------------------+
+| odl_l2           |  15/15  |   100%  | `compass-res-59`_ |
++------------------+---------+---------+-------------------+
+| odl_l3           |  9/12   |    75%  | `compass-res-73`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-moon      |  9/18   |      %  |    ??             |
++------------------+---------+---------+-------------------+
+| onos-ha          |  15/15  |   100%  | `compass-res-77`_ |
++------------------+---------+---------+-------------------+
+| onos-sfc-ha      |  16/18  |   100%  | `compass-res-76`_ |
++------------------+---------+---------+-------------------+
+
+Note: all the Compass tests for Colorado have been executed on virtual
+environment. Bare metal resources were used for Master branch.
+
+
+fuel
+----
+
++---------------------+---------+---------+-------------------+
+|  Scenario           | Scoring | Success |  Results          |
+|                     |         | rate    |                   |
++=====================+=========+=========+===================+
+| nosdn               |  18/18  |  100%   | `fuel-res-129`_   |
++---------------------+---------+---------+-------------------+
+| nosdn-noha          |  15/15  |  100%   | `fuel-res-154`_   |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm           |  18/18  |  100%   | `fuel-res-128`_   |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm-noha      |  15/15  |  100%   | `fuel-res-161`_   |
++---------------------+---------+---------+-------------------+
+| nosdn-ovs-noha      |  15/15  |  100%   | `fuel-res-162`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc          |  21/21  |  100%   |                   |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha     |  16/18  |   88%   |                   |
++---------------------+---------+---------+-------------------+
+| odl_l2              |  21/21  |  100%   |  `fuel-res-123`_  |
++---------------------+---------+---------+-------------------+
+| odl_l2-noha         |  17/18  |   94%   | `fuel-res-155`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn       |  17/18  |   94%   | `fuel-res-119`_   |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha  |  15/15  |  100%   | `fuel-res-160`_   |
++---------------------+---------+---------+-------------------+
+| odl_l3              |  15/18  |   67%   | `fuel-res-115`_   |
++---------------------+---------+---------+-------------------+
+| odl_l3-noha         |  12/15  |   80%   | `fuel-res-164`_   |
++---------------------+---------+---------+-------------------+
+| onos                |  20/21  |   95%   | `fuel-res-117`_   |
++---------------------+---------+---------+-------------------+
+| onos-noha           |  18/18  |  100%   | `fuel-res-166`_   |
++---------------------+---------+---------+-------------------+
+| onos-sfc            |  24/24  |  100%   | fuel-res-124`_    |
++---------------------+---------+---------+-------------------+
+| onos-sfc-noha       |  21/21  |  100%   | `fuel-res-129`_   |
++---------------------+---------+---------+-------------------+
+| multisite           |  N.R    |  100%   | `fuel-res-8`_     |
++---------------------+---------+---------+-------------------+
+
+joid
+----
+
++---------------------+---------+---------+-----------------+
+|  Scenario           | Scoring | Success |  Results        |
+|                     |         | rate    |                 |
++=====================+=========+=========+=================+
+| nosdn               |  18/18  |  100%   | `joid-res-102`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-noha          |  17/18  |   95%   | `joid-res-93`_  |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd           |  12/12  |  100%   | `joid-res-104`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd-noha      |  12/12  |  100%   | `joid-res-91`_  |
++---------------------+---------+---------+-----------------+
+| odl_l2              |  19/21  |   91%   | `joid-res-103`_ |
++---------------------+---------+---------+-----------------+
+| onos                |  21/21  |  100%   | `joid-res-99`_  |
++---------------------+---------+---------+-----------------+
+| onos-sfc            |  24/24  |  100%   | `joid-res-97`_  |
++---------------------+---------+---------+-----------------+
+
+It is highly recommended to install a json viewer in your browser
+(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
+
+You can get additional details through test logs on http://artifacts.opnfv.org/.
+As no search engine is available on the OPNFV artifact web site you must
+retrieve the pod identifier on which the tests have been executed (see
+field pod in any of the results) then click on the selected POD and look
+for the date of the test you are interested in.
 
 The reporting pages can be found at:
 
@@ -238,10 +390,6 @@ Colorado known restrictions/issues
 +-----------+-----------+----------------------------------------------+
 | Installer | Scenario  |  Issue                                       |
 +===========+===========+==============================================+
-| any       | onos-*    | vPing userdata and Tempest cases related to  |
-|           |           | metadata service excluded from onos scenarios|
-|           |           | https://gerrit.opnfv.org/gerrit/#/c/18729/   |
-+-----------+-----------+----------------------------------------------+
 | any       | odl_l3-*  | Tempest cases related to using floating IP   |
 |           |           | addresses fail because of a known ODL bug.   |
 |           |           | vPing_ssh test case is excluded for the same |
@@ -261,6 +409,11 @@ Colorado known restrictions/issues
 |           |           | Rally sanity in Colorado 2.0 with the        |
 |           |           | adoption of ODL Boron release.               |
 +-----------+-----------+----------------------------------------------+
+| apex      | *-fdio    | Due to late integration, fdio decided to     |
+|           |           | focus on mandatory tests and exclude feature |
+|           |           | tests (copper, doctor, security_scan) from   |
+|           |           | its scenarios                                |
++-----------+-----------+----------------------------------------------+
 | compass   | moon      | First ODL test FAILS because ODL/Openstack   |
 |           |           | federation done in moon is partial. Only     |
 |           |           | MD-SAL is federated (not AD-SAL)             |
@@ -319,12 +472,28 @@ Open JIRA tickets
 +------------------+-----------------------------------------------+
 |   JIRA           |         Description                           |
 +==================+===============================================+
+| `FUNCTEST-419`_  |  do not try to Remove docker                  |
+|                  |  image opnfv/functest:<none>                  |
+|                  |  reported by joid on Intel POD                |
+|                  |  may impact CI                                |
+|                  |  not reproducible                             |
++------------------+-----------------------------------------------+
+| `FUNCTEST-446`_  |  Cleanup ODL-SFC output in Functest execution |
+|                  |  Impact on odl_l2-sfc scenarios               |
 +------------------+-----------------------------------------------+
+| `FUNCTEST-450`_  |  Functest is Failing to get the token using   |
+|                  |  keystone client                              |
 +------------------+-----------------------------------------------+
+| `FUNCTEST-454`_  |  Cleanup failures when using HA networks in   |
+|                  |  Neutron                                      |
 +------------------+-----------------------------------------------+
+| `FUNCTEST-460`_  |  Wrong image format used in rally cases       |
 +------------------+-----------------------------------------------+
+| `FUNCTEST-462`_  |  OLD test fails after forcing the clone       |
+|                  |  release/beryllium-sr3 branch                 |
 +------------------+-----------------------------------------------+
 
+
 Useful links
 ============
 
@@ -344,10 +513,80 @@ Useful links
 
  - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
 
- - Functest Colorado user guide: * TODO *
+ - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
+
+ - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
+
+ - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
+
+.. _`FUNCTEST-419` : https://jira.opnfv.org/browse/FUNCTEST-419
+
+.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
+
+.. _`FUNCTEST-450` : https://jira.opnfv.org/browse/FUNCTEST-450
+
+.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
+
+.. _`FUNCTEST-460` : https://jira.opnfv.org/browse/FUNCTEST-460
+
+.. _`FUNCTEST-462` : https://jira.opnfv.org/browse/FUNCTEST-462
+
+.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
+
+.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
+
+.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
+
+.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
+
+.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
+
+.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
+
+.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
+
+.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
+
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+
+.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+
+.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
+
+.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
+
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
 
- - Functest installation/configuration guide: * TODO *
+.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
 
- - Functest developer guide: * TODO *
+.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
 
+.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
 
+.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103