diff options
Diffstat (limited to 'docs/release')
| -rw-r--r-- | docs/release/release-notes/index.rst | 2 | ||||
| -rw-r--r-- | docs/release/release-notes/release-notes.rst | 300 | ||||
| -rw-r--r-- | docs/release/results/index.rst | 2 |
3 files changed, 170 insertions, 134 deletions
diff --git a/docs/release/release-notes/index.rst b/docs/release/release-notes/index.rst index c9cadc539..11a508ca6 100644 --- a/docs/release/release-notes/index.rst +++ b/docs/release/release-notes/index.rst @@ -1,3 +1,5 @@ +.. _yardstick-releasenotes: + .. This work is licensed under a Creative Commons Attribution 4.0 International .. License. .. http://creativecommons.org/licenses/by/4.0 diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index 8df0776df..d89f9ed24 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -2,7 +2,7 @@ License ======= -OPNFV Colorado release note for Yardstick Docs +OPNFV Danube release note for Yardstick Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>. @@ -12,7 +12,7 @@ experimental framework are opensource software, licensed under the terms of the Apache License, Version 2.0. ========================================= -OPNFV Colorado Release Note for Yardstick +OPNFV Danube Release Note for Yardstick ========================================= .. toctree:: @@ -38,13 +38,13 @@ Version History | *Date* | *Version* | *Comment* | | | | | +----------------+--------------------+---------------------------------+ -| Dec 5th, 2016 | 3.0 | Yardstick for Colorado release | +| | 3.0 | Yardstick for Danube release | | | | | +----------------+--------------------+---------------------------------+ -| Oct 27th, 2016 | 2.0 | Yardstick for Colorado release | +| May 4th, 2017 | 2.0 | Yardstick for Danube release | | | | | +----------------+--------------------+---------------------------------+ -| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release | +| Mar 31st, 2017 | 1.0 | Yardstick for Danube release | | | | | +----------------+--------------------+---------------------------------+ @@ -61,17 +61,17 @@ The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. -OPNFV Colorado Release +OPNFV Danube Release ====================== -This Colorado release provides *Yardstick* as a framework for NFVI testing +This Danube release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx * User Guide - * Code Documentation + * Developer Guide * Release notes (this document) @@ -90,9 +90,9 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Yardstick test cases yaml files -* Yardstick pliug-in configration yaml files, plug-in install/remove scripts +* Yardstick plug-in configration yaml files, plug-in install/remove scripts -For Colorado release, the *Yardstick framework* is used for the following +For Danube release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: @@ -103,6 +103,10 @@ testing: * Storage +* OPNFV platform network service benchmarking(NSB) + + * NSB + * Test cases for the following OPNFV Projects: * High Availability @@ -135,19 +139,19 @@ Release Data | **Project** | Yardstick | | | | +--------------------------------------+--------------------------------------+ -| **Repo/tag** | yardstick/colorado.3.0 | +| **Repo/tag** | yardstick/Danube.2.0 | | | | +--------------------------------------+--------------------------------------+ -| **Yardstick Docker image tag** | colorado.3.0 | +| **Yardstick Docker image tag** | Danube.2.0 | | | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Colorado | +| **Release designation** | Danube | | | | +--------------------------------------+--------------------------------------+ -| **Release date** | December 5th, 2016 | +| **Release date** | May 4th, 2017 | | | | +--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | OPNFV Colorado release 3.0 | +| **Purpose of the delivery** | OPNFV Danube release 2.0 | | | | +--------------------------------------+--------------------------------------+ @@ -158,35 +162,16 @@ Deliverables Documents --------- - - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html + - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html - - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html + - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html Software Deliverables --------------------- -**Yardstick framework source code <colorado.3.0>** -+--------------------------------------+--------------------------------------+ -| **Project** | Yardstick | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | yardstick/colorado.3.0 | -| | | -+--------------------------------------+--------------------------------------+ -| **Yardstick Docker image tag** | colorado.3.0 | -| | | -+--------------------------------------+--------------------------------------+ -| **Release designation** | Colorado | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | December 5th, 2016 | -| | | -+--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | OPNFV Colorado release | -| | | -+--------------------------------------+--------------------------------------+ + - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.2.0) **Contexts** @@ -201,6 +186,9 @@ Software Deliverables | *Node* | Models Baremetal, Controller, Compute | | | | +---------------------+-------------------------------------------------------+ +| *Standalone* | Models VM running on Non-Managed NFVi | +| | | ++---------------------+-------------------------------------------------------+ **Runners** @@ -295,6 +283,8 @@ Software Deliverables | | | | | * netutilization | | | | +| | * nstat | +| | | +---------------------+-------------------------------------------------------+ | *Parser* | Tosca2Heat | | | | @@ -307,6 +297,9 @@ Software Deliverables | *StorPerf* | storperf | | | | +---------------------+-------------------------------------------------------+ +| *NSB* | vPE thoughput test case | +| | | ++---------------------+-------------------------------------------------------+ **API to Other Frameworks** @@ -398,6 +391,8 @@ Delivered Test cases * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale + * OPNFV_YARDSTICK_TCO76 - Network frame error rate + * Test Cases for OPNFV HA Project: * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down @@ -451,15 +446,17 @@ Delivered Test cases * Test Cases for Virtual Traffic Classifier: * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput -Benchmarking Test + +* Benchmarking Test: * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput -Benchmarking in presence of noisy neighbors Test + +* Benchmarking in presence of noisy neighbors Test: * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in -presence of noisy neighbors Test + presence of noisy neighbors Test Version Change @@ -468,91 +465,123 @@ Version Change Module Version Changes ---------------------- -This is the second tracked release of Yardstick. It is based on following +This is the fourth tracked release of Yardstick. It is based on following upstream versions: -- ONOS Goldeneye +- ONOS Ibis -- OpenStack Mitaka +- OpenStack Newton -- OpenDaylight Beryllium +- OpenDaylight Boron Document Version Changes ------------------------ -This is the second tracked version of the Yardstick framework in OPNFV. +This is the fourth tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: -- Yardstick User Guide: added yardstick plugin chapter; added Store Other -Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter. +- Yardstick User Guide: add "network service benchmarking(NSB)" chapter; + add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter; + add "Yardstick user interface" chapter; Update Yardstick installation chapter; -- Yardstick Code Documentation: no changes +- Yardstick Developer Guide - Yardstick Release Notes for Yardstick: this document -- Test Results report for Colorado testing with Yardstick: updated listed of -verified scenarios and limitations - Feature additions ----------------- - - Yardstick plugin - - Yardstick reporting - - StorPerf Integration + +- Yardstick RESTful API support + +- Introduce Network service benchmarking + +- Introduce stress testing with Bottlenecks team + +- Yardstick framework improvement: + + - Parellel test cases execution support + + - yardstick report CLI + + - Node context support openstack configuration via Ansible + + - Https support + +- Python 3 support Scenario Matrix =============== -For Colorado 3.0, Yardstick was tested on the following scenarios: +For Danube 2.0, Yardstick was tested on the following scenarios: +-------------------------+---------+---------+---------+---------+ | Scenario | Apex | Compass | Fuel | Joid | +=========================+=========+=========+=========+=========+ -| os-nosdn-nofeature-noha | | | | X | +| os-nosdn-nofeature-noha | | | X | X | +-------------------------+---------+---------+---------+---------+ -| os-nosdn-nofeature-ha | X | | X | X | +| os-nosdn-nofeature-ha | X | X | X | X | +-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-ha | X | X | X | X | +| os-odl_l2-nofeature-ha | | X | X | X | +-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-noha| | X | | | +| os-odl_l2-nofeature-noha| | | X | | +-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-ha | X | | X | | +| os-odl_l3-nofeature-ha | X | X | X | | +-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-ha | | X | | | +| os-odl_l3-nofeature-noha| | | X | | +-------------------------+---------+---------+---------+---------+ -| os-onos-sfc-ha | X | | X | X | +| os-onos-sfc-ha | | | | | +-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-ha | X | | X | X | +| os-onos-nofeature-ha | | X | | X | +-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-noha | | X | | | +| os-onos-nofeature-noha | | | | | +-------------------------+---------+---------+---------+---------+ | os-odl_l2-sfc-ha | | | X | | +-------------------------+---------+---------+---------+---------+ -| os-odl_l2-sfc-noha | X | X | | | +| os-odl_l2-sfc-noha | | | X | | +-------------------------+---------+---------+---------+---------+ | os-odl_l2-bgpvpn-ha | X | | X | | +-------------------------+---------+---------+---------+---------+ -| os-odl_l2-bgpvpn-noha | | X | | | +| os-odl_l2-bgpvpn-noha | | | X | | +-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-ha | | | X | | +| os-nosdn-kvm-ha | X | | X | | +-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-noha | | X | | | +| os-nosdn-kvm-noha | | | X | | +-------------------------+---------+---------+---------+---------+ | os-nosdn-ovs-ha | | | X | | +-------------------------+---------+---------+---------+---------+ -| os-nosdn-ovs-noha | X | | X | | +| os-nosdn-ovs-noha | | | X | | +-------------------------+---------+---------+---------+---------+ -| os-ocl-nofeature-ha | | | | | +| os-ocl-nofeature-ha | | X | | | +-------------------------+---------+---------+---------+---------+ | os-nosdn-lxd-ha | | | | X | +-------------------------+---------+---------+---------+---------+ | os-nosdn-lxd-noha | | | | X | +-------------------------+---------+---------+---------+---------+ +| os-nosdn-fdio-ha | X | | | | ++-------------------------+---------+---------+---------+---------+ | os-odl_l2-fdio-noha | X | | | | +-------------------------+---------+---------+---------+---------+ - +| os-odl-gluon-noha | X | | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-openo-ha | | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| -noha | | | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm_ovs_dpdk-ha| | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| _bar-ha | | | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| _bar-noha | | | | | ++-------------------------+---------+---------+---------+---------+ +| opnfv_os-ovn-nofeature- | X | | | | +| noha_daily | | | | | ++-------------------------+---------+---------+---------+---------+ Test results ============ @@ -563,22 +592,18 @@ Test results are available in: The reporting pages can be found at: - * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html - * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html - * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html - * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html - -You can get additional details through test logs on http://artifacts.opnfv.org/. -As no search engine is available on the OPNFV artifact web site you must -retrieve the pod identifier on which the tests have been executed (see -field pod in any of the results) then click on the selected POD and look -for the date of the test you are interested in. + * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html + * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html + * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html + * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html Known Issues/Faults ------------ + - Floating IP not supported in bgpvpn scenario - - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario + + - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario .. note:: The faults not related to *Yardstick* framework, addressing scenarios which were not fully verified, are listed in the OPNFV installer's release @@ -588,59 +613,68 @@ Known Issues/Faults Corrected Faults ---------------- -Colorado.3.0: +Danube.2.0: +----------------------------+------------------------------------------------+ | **JIRA REFERENCE** | **SLOGAN** | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-239 | Define process for working with Yardstick | -| | Grafana dashboard. | +| JIRA: YARDSTICK-608 | Set work directory in Yardstick container | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. | -| | | +| JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure | +| | situation | +----------------------------+------------------------------------------------+ - - -Colorado.2.0: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **SLOGAN** | +| JIRA: YARDSTICK-620 | Yardstick virtualenv support | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for | -| | nova-lxd scenario. | +| JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the | -| | installers. | -| | | +| JIRA: YARDSTICK-622 | Bugfix: take test case modification into | +| | effect in load_images.sh | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on | -| | vRouter. | -| | | +| JIRA: YARDSTICK-623 | change openrc file path to | +| | /etc/yardstick/openstack.creds | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. | -| | | +| JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test | +| | suite | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept | +| | --task-args options | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not | +| | support other installer | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick | -| | reporting. | +| JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for | -| | nova-lxd scenario. | +| JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA | +| | Test Cases | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci | | | | +----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and | -| | yardstick-img-lxd-modify in environment | -| | varibales. | +| JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test | +| | frameworks | ++----------------------------+------------------------------------------------+ + +Danube.1.0: + ++----------------------------+------------------------------------------------+ +| **JIRA REFERENCE** | **SLOGAN** | | | | +----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using | +| | 'openstack -h' | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they | +| | are dynamic | ++----------------------------+------------------------------------------------+ -Colorado 3.0 known restrictions/issues -================================== +Danube 2.0 known restrictions/issues +==================================== +-----------+-----------+----------------------------------------------+ | Installer | Scenario | Issue | +===========+===========+==============================================+ @@ -649,22 +683,38 @@ Colorado 3.0 known restrictions/issues +-----------+-----------+----------------------------------------------+ | any | odl_l3-* | Some test cases related to using floating IP | | | | addresses fail because of a known ODL bug. | -| | | https://jira.opnfv.org/browse/APEX-112 | +| | | | ++-----------+-----------+----------------------------------------------+ +| compass | odl_l2-* | In some test cases, VM instance will failed | +| | | raising network interfaces. | +| | | | +-----------+-----------+----------------------------------------------+ Open JIRA tickets ================= ++----------------------------+------------------------------------------------+ +| **JIRA REFERENCE** | **SLOGAN** | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 | +| | image | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-603 | Timeout waiting for floating ip | +| | (which actually pingable) | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-412 | IPv6 test case should add support for newton | +| | | ++----------------------------+------------------------------------------------+ + Useful links ============ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning - - - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado + - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning - Yardstick repo: https://git.opnfv.org/cgit/yardstick @@ -673,21 +723,3 @@ Useful links - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ - Yardstick IRC chanel: #opnfv-yardstick - -.. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239 - -.. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325 - -.. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358 - -.. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359 - -.. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363 - -.. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367 - -.. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371 - -.. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372 - -.. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373 diff --git a/docs/release/results/index.rst b/docs/release/results/index.rst index 2b67f1b22..0560152e0 100644 --- a/docs/release/results/index.rst +++ b/docs/release/results/index.rst @@ -1,3 +1,5 @@ +.. _yardstick-results: + .. This work is licensed under a Creative Commons Attribution 4.0 International .. License. .. http://creativecommons.org/licenses/by/4.0 |