diff options
Diffstat (limited to 'docs/release/release-notes.rst')
| -rw-r--r-- | docs/release/release-notes.rst | 534 |
1 files changed, 373 insertions, 161 deletions
diff --git a/docs/release/release-notes.rst b/docs/release/release-notes.rst index bc58b2134..8df0776df 100644 --- a/docs/release/release-notes.rst +++ b/docs/release/release-notes.rst @@ -1,12 +1,19 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB and others. +======= +License +======= + +OPNFV Colorado release note for Yardstick Docs +are licensed under a Creative Commons Attribution 4.0 International License. +You should have received a copy of the license along with this. +If not, see <http://creativecommons.org/licenses/by/4.0/>. +The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake* +experimental framework are opensource software, licensed under the terms of the +Apache License, Version 2.0. -============================================ -OPNFV Brahmaputra Release Note for Yardstick -============================================ +========================================= +OPNFV Colorado Release Note for Yardstick +========================================= .. toctree:: :maxdepth: 2 @@ -21,33 +28,25 @@ OPNFV Brahmaputra Release Note for Yardstick Abstract ======== -This document compiles the release notes for the OPNFV Brahmaputra release -for Yardstick framework as well as Yardstick_ Project deliverables. - -License -======= - -The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake* -experimental framework are opensource software, licensed under the terms of the -Apache License, Version 2.0. +This document describes the release note of Yardstick project. Version History =============== -+---------------+--------------------+---------------------------------+ -| *Date* | *Version* | *Comment* | -| | | | -+---------------+--------------------+---------------------------------+ -| Apr 27th,2016 | 3.0 | Brahmaputra release | -| | | | -+---------------+--------------------+---------------------------------+ -| Mar 30th,2016 | 2.0 | Brahmaputra release | -| | | | -+---------------+--------------------+---------------------------------+ -| Feb 25th,2016 | 1.0 | Brahmaputra release | -| | | | -+---------------+--------------------+---------------------------------+ ++----------------+--------------------+---------------------------------+ +| *Date* | *Version* | *Comment* | +| | | | ++----------------+--------------------+---------------------------------+ +| Dec 5th, 2016 | 3.0 | Yardstick for Colorado release | +| | | | ++----------------+--------------------+---------------------------------+ +| Oct 27th, 2016 | 2.0 | Yardstick for Colorado release | +| | | | ++----------------+--------------------+---------------------------------+ +| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release | +| | | | ++----------------+--------------------+---------------------------------+ Important Notes @@ -62,10 +61,10 @@ The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. -Summary -======= +OPNFV Colorado Release +====================== -This Brahmaputra release provides *Yardstick* as a framework for NFVI testing +This Colorado release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx @@ -84,14 +83,16 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Automated Yardstick test results visualization - * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as + * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as backend * Yardstick framework source code * Yardstick test cases yaml files -For Brahmaputra release, the *Yardstick framework* is used for the following +* Yardstick pliug-in configration yaml files, plug-in install/remove scripts + +For Colorado release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: @@ -112,7 +113,9 @@ testing: * Parser -* Test cases added in Brahmaputra2.0: + * StorPerf + + * VSperf * virtual Traffic Classifier @@ -132,165 +135,60 @@ Release Data | **Project** | Yardstick | | | | +--------------------------------------+--------------------------------------+ -| **Repo/tag** | yardstick/brahmaputra.3.0 | +| **Repo/tag** | yardstick/colorado.3.0 | | | | +--------------------------------------+--------------------------------------+ -| **Yardstick Docker image tag** | brahmaputra.3.0 | +| **Yardstick Docker image tag** | colorado.3.0 | | | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Brahmaputra | +| **Release designation** | Colorado | | | | +--------------------------------------+--------------------------------------+ -| **Release date** | Apr 27th, 2016 | +| **Release date** | December 5th, 2016 | | | | +--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | OPNFV Brahmaputra release | +| **Purpose of the delivery** | OPNFV Colorado release 3.0 | | | | +--------------------------------------+--------------------------------------+ -Version Change --------------- - -Module Version Changes -~~~~~~~~~~~~~~~~~~~~~~ - -This is the third tracked release of Yardstick. It is based on following -upstream versions: - -- OpenStack Liberty - -- OpenDaylight Beryllium - - -Document Version Changes -~~~~~~~~~~~~~~~~~~~~~~~~ - -This is the third tracked version of the Yardstick framework in OPNFV. -It includes the following documentation updates: - -- Yardstick User Guide: corrected faulty links - -- Yardstick Code Documentation: no changes - -- Yardstick Release Notes for Yardstick: this document - -- Test Results report for Brahmaputra testing with Yardstick: updated listed of -verified scenarios and limitations - -Documentation updates on the second tracked version: - -- Yardstick User Guide: added software architecture chapter - -- Yardstick Code Documentation: no changes - -- Yardstick Release Notes for Yardstick: this document - -- Test Results report for Brahmaputra testing with Yardstick: added test cases -and results for virtual Traffic Classifier - - -Reason for Version ------------------- - -Feature additions -~~~~~~~~~~~~~~~~~ - -No new features. - -Brahmaputra.2.0: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **SLOGAN** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-227 | Heat HTTPS SSL support. | -| | | -+----------------------------+------------------------------------------------+ - - -Corrected Faults -~~~~~~~~~~~~~~~~ - -No corrected faults. - -Brahmaputra.2.0: +Deliverables +============ -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **SLOGAN** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: - | Change copyrights for base scenario, runners, | -| | dispatchers, cover. | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: - | Update setup.py and dependencies | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: - | Add missing dependencies to docker file | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: - | Fix Heat template for noisy neighbors deploy | -| | | -+----------------------------+------------------------------------------------+ +Documents +--------- -Known Faults -~~~~~~~~~~~~ + - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html + - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **SLOGAN** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-175 | Running test suite, if a test cases running | -| | failed, the test is stopped. | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-176 | Fix plotter bug since Output format has been | -| | changed. | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-216 | ArgsAlreadyParsedError: arguments already | -| | parsed: cannot register CLI option. | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-231 | Installation instructions on Wiki not accurate | -| | | -+----------------------------+------------------------------------------------+ - -.. note:: The faults not related to *Yardstick* framework, addressing scenarios - which were not fully verified, are listed in the OPNFV installer's release - notes. - - -Deliverables ------------- Software Deliverables -~~~~~~~~~~~~~~~~~~~~~ +--------------------- -**Yardstick framework source code <brahmaputra.3.0>** +**Yardstick framework source code <colorado.3.0>** +--------------------------------------+--------------------------------------+ | **Project** | Yardstick | | | | +--------------------------------------+--------------------------------------+ -| **Repo/tag** | yardstick/brahmaputra.3.0 | +| **Repo/tag** | yardstick/colorado.3.0 | | | | +--------------------------------------+--------------------------------------+ -| **Yardstick Docker image tag** | brahmaputra.3.0 | +| **Yardstick Docker image tag** | colorado.3.0 | | | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Brahmaputra | +| **Release designation** | Colorado | | | | +--------------------------------------+--------------------------------------+ -| **Release date** | Apr 27th, 2016 | +| **Release date** | December 5th, 2016 | | | | +--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | OPNFV Brahmaputra release | +| **Purpose of the delivery** | OPNFV Colorado release | | | | +--------------------------------------+--------------------------------------+ + **Contexts** +---------------------+-------------------------------------------------------+ @@ -326,6 +224,7 @@ Software Deliverables | | | +---------------------+-------------------------------------------------------+ + **Scenarios** +---------------------+-------------------------------------------------------+ @@ -353,15 +252,27 @@ Software Deliverables | | | | | * lmbench | | | | +| | * lmbench_cache | +| | | | | * perf | | | | | | * unixbench | | | | +| | * ramspeed | +| | | +| | * cachestat | +| | | +| | * memeoryload | +| | | +| | * computecapacity | +| | | +---------------------+-------------------------------------------------------+ | *Networking* | * iperf3 | | | | | | * netperf | | | | +| | * netperf_node | +| | | | | * ping | | | | | | * ping6 | @@ -380,13 +291,23 @@ Software Deliverables | | | | | * vtc throughput in the presence of noisy neighbors | | | | +| | * networkcapacity | +| | | +| | * netutilization | +| | | +---------------------+-------------------------------------------------------+ | *Parser* | Tosca2Heat | | | | +---------------------+-------------------------------------------------------+ | *Storage* | fio | | | | +| | storagecapacity | +| | | +---------------------+-------------------------------------------------------+ +| *StorPerf* | storperf | +| | | ++---------------------+-------------------------------------------------------+ + **API to Other Frameworks** @@ -401,6 +322,7 @@ Software Deliverables | | | +---------------------+-------------------------------------------------------+ + **Test Results Output** +-----------------------------+-----------------------------------------------+ @@ -413,13 +335,13 @@ Software Deliverables | http | Post data to html. | | | | +-----------------------------+-----------------------------------------------+ -| influxdb | Post data to influxdB. | +| influxdb | Post data to influxDB. | | | | +-----------------------------+-----------------------------------------------+ Delivered Test cases -~~~~~~~~~~~~~~~~~~~~ +-------------------- * Generic NFVI test cases @@ -427,6 +349,8 @@ Delivered Test cases * OPNFV_YARDSTICK_TCOO2 - NW Latency + * OPNFV_YARDSTICK_TCOO4 - Cache Utilization + * OPNFV_YARDSTICK_TCOO5 - Storage Performance * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test @@ -448,6 +372,31 @@ Delivered Test cases * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended Test + * OPNFV_YARDSTICK_TCO42 - Network Performance + + * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes + + * OPNFV_YARDSTICK_TCO44 - Memory Utilization + + * OPNFV_YARDSTICK_TCO55 - Compute Capacity + + * OPNFV_YARDSTICK_TCO61 - Network Utilization + + * OPNFV_YARDSTICK_TCO63 - Storage Capacity + + * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth + + * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet + Loss + + * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss + + * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet + Loss + + * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes + + * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale * Test Cases for OPNFV HA Project: @@ -455,6 +404,34 @@ Delivered Test cases * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down + * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron + server + + * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone + + * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance + api + + * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder + api + + * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift + proxy + + * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High + Availability + + * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High + Availability + + * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High + Availability + + * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High + Availability + + * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability + * Test Case for OPNFV IPv6 Project: * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity @@ -467,6 +444,10 @@ Delivered Test cases * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca +* Test Case for OPNFV StorPerf Project: + + * OPNFV_YARDSTICK_TCO74 - Storperf + * Test Cases for Virtual Traffic Classifier: * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput @@ -479,3 +460,234 @@ Benchmarking in presence of noisy neighbors Test * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in presence of noisy neighbors Test + + +Version Change +============== + +Module Version Changes +---------------------- + +This is the second tracked release of Yardstick. It is based on following +upstream versions: + +- ONOS Goldeneye + +- OpenStack Mitaka + +- OpenDaylight Beryllium + + +Document Version Changes +------------------------ + +This is the second tracked version of the Yardstick framework in OPNFV. +It includes the following documentation updates: + +- Yardstick User Guide: added yardstick plugin chapter; added Store Other +Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter. + +- Yardstick Code Documentation: no changes + +- Yardstick Release Notes for Yardstick: this document + +- Test Results report for Colorado testing with Yardstick: updated listed of +verified scenarios and limitations + + +Feature additions +----------------- + - Yardstick plugin + - Yardstick reporting + - StorPerf Integration + + +Scenario Matrix +=============== + +For Colorado 3.0, Yardstick was tested on the following scenarios: + ++-------------------------+---------+---------+---------+---------+ +| Scenario | Apex | Compass | Fuel | Joid | ++=========================+=========+=========+=========+=========+ +| os-nosdn-nofeature-noha | | | | X | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-nofeature-ha | X | | X | X | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-nofeature-ha | X | X | X | X | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-nofeature-noha| | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l3-nofeature-ha | X | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l3-nofeature-ha | | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-onos-sfc-ha | X | | X | X | ++-------------------------+---------+---------+---------+---------+ +| os-onos-nofeature-ha | X | | X | X | ++-------------------------+---------+---------+---------+---------+ +| os-onos-nofeature-noha | | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-sfc-ha | | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-sfc-noha | X | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-bgpvpn-ha | X | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-bgpvpn-noha | | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm-ha | | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-kvm-noha | | X | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-ovs-ha | | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-ovs-noha | X | | X | | ++-------------------------+---------+---------+---------+---------+ +| os-ocl-nofeature-ha | | | | | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-lxd-ha | | | | X | ++-------------------------+---------+---------+---------+---------+ +| os-nosdn-lxd-noha | | | | X | ++-------------------------+---------+---------+---------+---------+ +| os-odl_l2-fdio-noha | X | | | | ++-------------------------+---------+---------+---------+---------+ + + +Test results +============ + +Test results are available in: + + - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/ + +The reporting pages can be found at: + + * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html + * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html + * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html + * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html + +You can get additional details through test logs on http://artifacts.opnfv.org/. +As no search engine is available on the OPNFV artifact web site you must +retrieve the pod identifier on which the tests have been executed (see +field pod in any of the results) then click on the selected POD and look +for the date of the test you are interested in. + + +Known Issues/Faults +------------ + - Floating IP not supported in bgpvpn scenario + - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario + +.. note:: The faults not related to *Yardstick* framework, addressing scenarios + which were not fully verified, are listed in the OPNFV installer's release + notes. + + +Corrected Faults +---------------- + +Colorado.3.0: + ++----------------------------+------------------------------------------------+ +| **JIRA REFERENCE** | **SLOGAN** | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-239 | Define process for working with Yardstick | +| | Grafana dashboard. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. | +| | | ++----------------------------+------------------------------------------------+ + + +Colorado.2.0: + ++----------------------------+------------------------------------------------+ +| **JIRA REFERENCE** | **SLOGAN** | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for | +| | nova-lxd scenario. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the | +| | installers. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on | +| | vRouter. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick | +| | reporting. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for | +| | nova-lxd scenario. | +| | | ++----------------------------+------------------------------------------------+ +| JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and | +| | yardstick-img-lxd-modify in environment | +| | varibales. | +| | | ++----------------------------+------------------------------------------------+ + + +Colorado 3.0 known restrictions/issues +================================== ++-----------+-----------+----------------------------------------------+ +| Installer | Scenario | Issue | ++===========+===========+==============================================+ +| any | *-bgpvpn | Floating ips not supported. Some Test cases | +| | | related to floating ips are excluded. | ++-----------+-----------+----------------------------------------------+ +| any | odl_l3-* | Some test cases related to using floating IP | +| | | addresses fail because of a known ODL bug. | +| | | https://jira.opnfv.org/browse/APEX-112 | ++-----------+-----------+----------------------------------------------+ + + +Open JIRA tickets +================= + + +Useful links +============ + + - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick + + - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning + + - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado + + - Yardstick repo: https://git.opnfv.org/cgit/yardstick + + - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick + + - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ + + - Yardstick IRC chanel: #opnfv-yardstick + +.. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239 + +.. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325 + +.. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358 + +.. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359 + +.. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363 + +.. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367 + +.. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371 + +.. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372 + +.. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373 |