summaryrefslogtreecommitdiffstats
path: root/docs/release/release-notes/release-notes.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/release/release-notes/release-notes.rst')
-rw-r--r--docs/release/release-notes/release-notes.rst300
1 files changed, 166 insertions, 134 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst
index 8df0776df..d89f9ed24 100644
--- a/docs/release/release-notes/release-notes.rst
+++ b/docs/release/release-notes/release-notes.rst
@@ -2,7 +2,7 @@
License
=======
-OPNFV Colorado release note for Yardstick Docs
+OPNFV Danube release note for Yardstick Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
@@ -12,7 +12,7 @@ experimental framework are opensource software, licensed under the terms of the
Apache License, Version 2.0.
=========================================
-OPNFV Colorado Release Note for Yardstick
+OPNFV Danube Release Note for Yardstick
=========================================
.. toctree::
@@ -38,13 +38,13 @@ Version History
| *Date* | *Version* | *Comment* |
| | | |
+----------------+--------------------+---------------------------------+
-| Dec 5th, 2016 | 3.0 | Yardstick for Colorado release |
+| | 3.0 | Yardstick for Danube release |
| | | |
+----------------+--------------------+---------------------------------+
-| Oct 27th, 2016 | 2.0 | Yardstick for Colorado release |
+| May 4th, 2017 | 2.0 | Yardstick for Danube release |
| | | |
+----------------+--------------------+---------------------------------+
-| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
+| Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
| | | |
+----------------+--------------------+---------------------------------+
@@ -61,17 +61,17 @@ The *Yardstick* framework is *installer*, *infrastructure* and *application*
independent.
-OPNFV Colorado Release
+OPNFV Danube Release
======================
-This Colorado release provides *Yardstick* as a framework for NFVI testing
+This Danube release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
* User Guide
- * Code Documentation
+ * Developer Guide
* Release notes (this document)
@@ -90,9 +90,9 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Yardstick test cases yaml files
-* Yardstick pliug-in configration yaml files, plug-in install/remove scripts
+* Yardstick plug-in configration yaml files, plug-in install/remove scripts
-For Colorado release, the *Yardstick framework* is used for the following
+For Danube release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
@@ -103,6 +103,10 @@ testing:
* Storage
+* OPNFV platform network service benchmarking(NSB)
+
+ * NSB
+
* Test cases for the following OPNFV Projects:
* High Availability
@@ -135,19 +139,19 @@ Release Data
| **Project** | Yardstick |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/colorado.3.0 |
+| **Repo/tag** | yardstick/Danube.2.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | colorado.3.0 |
+| **Yardstick Docker image tag** | Danube.2.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado |
+| **Release designation** | Danube |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | December 5th, 2016 |
+| **Release date** | May 4th, 2017 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Colorado release 3.0 |
+| **Purpose of the delivery** | OPNFV Danube release 2.0 |
| | |
+--------------------------------------+--------------------------------------+
@@ -158,35 +162,16 @@ Deliverables
Documents
---------
- - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html
+ - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
- - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html
+ - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
Software Deliverables
---------------------
-**Yardstick framework source code <colorado.3.0>**
-+--------------------------------------+--------------------------------------+
-| **Project** | Yardstick |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | December 5th, 2016 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Colorado release |
-| | |
-+--------------------------------------+--------------------------------------+
+ - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.2.0)
**Contexts**
@@ -201,6 +186,9 @@ Software Deliverables
| *Node* | Models Baremetal, Controller, Compute |
| | |
+---------------------+-------------------------------------------------------+
+| *Standalone* | Models VM running on Non-Managed NFVi |
+| | |
++---------------------+-------------------------------------------------------+
**Runners**
@@ -295,6 +283,8 @@ Software Deliverables
| | |
| | * netutilization |
| | |
+| | * nstat |
+| | |
+---------------------+-------------------------------------------------------+
| *Parser* | Tosca2Heat |
| | |
@@ -307,6 +297,9 @@ Software Deliverables
| *StorPerf* | storperf |
| | |
+---------------------+-------------------------------------------------------+
+| *NSB* | vPE thoughput test case |
+| | |
++---------------------+-------------------------------------------------------+
**API to Other Frameworks**
@@ -398,6 +391,8 @@ Delivered Test cases
* OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
+ * OPNFV_YARDSTICK_TCO76 - Network frame error rate
+
* Test Cases for OPNFV HA Project:
* OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
@@ -451,15 +446,17 @@ Delivered Test cases
* Test Cases for Virtual Traffic Classifier:
* OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
-Benchmarking Test
+
+* Benchmarking Test:
* OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
-Benchmarking in presence of noisy neighbors Test
+
+* Benchmarking in presence of noisy neighbors Test:
* OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
* OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
-presence of noisy neighbors Test
+ presence of noisy neighbors Test
Version Change
@@ -468,91 +465,123 @@ Version Change
Module Version Changes
----------------------
-This is the second tracked release of Yardstick. It is based on following
+This is the fourth tracked release of Yardstick. It is based on following
upstream versions:
-- ONOS Goldeneye
+- ONOS Ibis
-- OpenStack Mitaka
+- OpenStack Newton
-- OpenDaylight Beryllium
+- OpenDaylight Boron
Document Version Changes
------------------------
-This is the second tracked version of the Yardstick framework in OPNFV.
+This is the fourth tracked version of the Yardstick framework in OPNFV.
It includes the following documentation updates:
-- Yardstick User Guide: added yardstick plugin chapter; added Store Other
-Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
+- Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
+ add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
+ add "Yardstick user interface" chapter; Update Yardstick installation chapter;
-- Yardstick Code Documentation: no changes
+- Yardstick Developer Guide
- Yardstick Release Notes for Yardstick: this document
-- Test Results report for Colorado testing with Yardstick: updated listed of
-verified scenarios and limitations
-
Feature additions
-----------------
- - Yardstick plugin
- - Yardstick reporting
- - StorPerf Integration
+
+- Yardstick RESTful API support
+
+- Introduce Network service benchmarking
+
+- Introduce stress testing with Bottlenecks team
+
+- Yardstick framework improvement:
+
+ - Parellel test cases execution support
+
+ - yardstick report CLI
+
+ - Node context support openstack configuration via Ansible
+
+ - Https support
+
+- Python 3 support
Scenario Matrix
===============
-For Colorado 3.0, Yardstick was tested on the following scenarios:
+For Danube 2.0, Yardstick was tested on the following scenarios:
+-------------------------+---------+---------+---------+---------+
| Scenario | Apex | Compass | Fuel | Joid |
+=========================+=========+=========+=========+=========+
-| os-nosdn-nofeature-noha | | | | X |
+| os-nosdn-nofeature-noha | | | X | X |
+-------------------------+---------+---------+---------+---------+
-| os-nosdn-nofeature-ha | X | | X | X |
+| os-nosdn-nofeature-ha | X | X | X | X |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-ha | X | X | X | X |
+| os-odl_l2-nofeature-ha | | X | X | X |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-noha| | X | | |
+| os-odl_l2-nofeature-noha| | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-ha | X | | X | |
+| os-odl_l3-nofeature-ha | X | X | X | |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-ha | | X | | |
+| os-odl_l3-nofeature-noha| | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-onos-sfc-ha | X | | X | X |
+| os-onos-sfc-ha | | | | |
+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-ha | X | | X | X |
+| os-onos-nofeature-ha | | X | | X |
+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-noha | | X | | |
+| os-onos-nofeature-noha | | | | |
+-------------------------+---------+---------+---------+---------+
| os-odl_l2-sfc-ha | | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-sfc-noha | X | X | | |
+| os-odl_l2-sfc-noha | | | X | |
+-------------------------+---------+---------+---------+---------+
| os-odl_l2-bgpvpn-ha | X | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-bgpvpn-noha | | X | | |
+| os-odl_l2-bgpvpn-noha | | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-ha | | | X | |
+| os-nosdn-kvm-ha | X | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-noha | | X | | |
+| os-nosdn-kvm-noha | | | X | |
+-------------------------+---------+---------+---------+---------+
| os-nosdn-ovs-ha | | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-nosdn-ovs-noha | X | | X | |
+| os-nosdn-ovs-noha | | | X | |
+-------------------------+---------+---------+---------+---------+
-| os-ocl-nofeature-ha | | | | |
+| os-ocl-nofeature-ha | | X | | |
+-------------------------+---------+---------+---------+---------+
| os-nosdn-lxd-ha | | | | X |
+-------------------------+---------+---------+---------+---------+
| os-nosdn-lxd-noha | | | | X |
+-------------------------+---------+---------+---------+---------+
+| os-nosdn-fdio-ha | X | | | |
++-------------------------+---------+---------+---------+---------+
| os-odl_l2-fdio-noha | X | | | |
+-------------------------+---------+---------+---------+---------+
-
+| os-odl-gluon-noha | X | | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-openo-ha | | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| -noha | | | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm_ovs_dpdk-ha| | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| _bar-ha | | | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| _bar-noha | | | | |
++-------------------------+---------+---------+---------+---------+
+| opnfv_os-ovn-nofeature- | X | | | |
+| noha_daily | | | | |
++-------------------------+---------+---------+---------+---------+
Test results
============
@@ -563,22 +592,18 @@ Test results are available in:
The reporting pages can be found at:
- * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html
-
-You can get additional details through test logs on http://artifacts.opnfv.org/.
-As no search engine is available on the OPNFV artifact web site you must
-retrieve the pod identifier on which the tests have been executed (see
-field pod in any of the results) then click on the selected POD and look
-for the date of the test you are interested in.
+ * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
+ * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
+ * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
+ * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
Known Issues/Faults
------------
+
- Floating IP not supported in bgpvpn scenario
- - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario
+
+ - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
.. note:: The faults not related to *Yardstick* framework, addressing scenarios
which were not fully verified, are listed in the OPNFV installer's release
@@ -588,59 +613,68 @@ Known Issues/Faults
Corrected Faults
----------------
-Colorado.3.0:
+Danube.2.0:
+----------------------------+------------------------------------------------+
| **JIRA REFERENCE** | **SLOGAN** |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-239 | Define process for working with Yardstick |
-| | Grafana dashboard. |
+| JIRA: YARDSTICK-608 | Set work directory in Yardstick container |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. |
-| | |
+| JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure |
+| | situation |
+----------------------------+------------------------------------------------+
-
-
-Colorado.2.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
+| JIRA: YARDSTICK-620 | Yardstick virtualenv support |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for |
-| | nova-lxd scenario. |
+| JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the |
-| | installers. |
-| | |
+| JIRA: YARDSTICK-622 | Bugfix: take test case modification into |
+| | effect in load_images.sh |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on |
-| | vRouter. |
-| | |
+| JIRA: YARDSTICK-623 | change openrc file path to |
+| | /etc/yardstick/openstack.creds |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. |
-| | |
+| JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test |
+| | suite |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept |
+| | --task-args options |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not |
+| | support other installer |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick |
-| | reporting. |
+| JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for |
-| | nova-lxd scenario. |
+| JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA |
+| | Test Cases |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci |
| | |
+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and |
-| | yardstick-img-lxd-modify in environment |
-| | varibales. |
+| JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test |
+| | frameworks |
++----------------------------+------------------------------------------------+
+
+Danube.1.0:
+
++----------------------------+------------------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
| | |
+----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
+| | 'openstack -h' |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
+| | are dynamic |
++----------------------------+------------------------------------------------+
-Colorado 3.0 known restrictions/issues
-==================================
+Danube 2.0 known restrictions/issues
+====================================
+-----------+-----------+----------------------------------------------+
| Installer | Scenario | Issue |
+===========+===========+==============================================+
@@ -649,22 +683,38 @@ Colorado 3.0 known restrictions/issues
+-----------+-----------+----------------------------------------------+
| any | odl_l3-* | Some test cases related to using floating IP |
| | | addresses fail because of a known ODL bug. |
-| | | https://jira.opnfv.org/browse/APEX-112 |
+| | | |
++-----------+-----------+----------------------------------------------+
+| compass | odl_l2-* | In some test cases, VM instance will failed |
+| | | raising network interfaces. |
+| | | |
+-----------+-----------+----------------------------------------------+
Open JIRA tickets
=================
++----------------------------+------------------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
+| | image |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
+| | (which actually pingable) |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
+| | |
++----------------------------+------------------------------------------------+
+
Useful links
============
- wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
- - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
-
- - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
+ - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
- Yardstick repo: https://git.opnfv.org/cgit/yardstick
@@ -673,21 +723,3 @@ Useful links
- Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
- Yardstick IRC chanel: #opnfv-yardstick
-
-.. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239
-
-.. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325
-
-.. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358
-
-.. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359
-
-.. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363
-
-.. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367
-
-.. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371
-
-.. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372
-
-.. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373