diff options
Diffstat (limited to 'docs/release/release-notes/release-notes.rst')
| -rw-r--r-- | docs/release/release-notes/release-notes.rst | 435 |
1 files changed, 105 insertions, 330 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index 4ebf0eceb..b3f3e1aed 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -1,72 +1,61 @@ -License -======= - -OPNFV Euphrates release note for Yardstick Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 The *Yardstick framework*, the *Yardstick test cases* are open-source software, licensed under the terms of the Apache License, Version 2.0. -OPNFV Euphrates Release Note for Yardstick -========================================== +======================= +Yardstick Release Notes +======================= .. toctree:: :maxdepth: 2 -.. _Yardstick: https://wiki.opnfv.org/yardstick +.. _Yardstick: https://wiki.opnfv.org/display/yardstick -.. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main +.. _Dashboard: http://testresults.opnfv.org/grafana/ -.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf +.. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf Abstract --------- +======== -This document describes the release note of Yardstick project. +This document compiles the release notes for the Iruya release of OPNFV Yardstick. Version History ---------------- +=============== +-------------------+-----------+---------------------------------+ | *Date* | *Version* | *Comment* | | | | | +-------------------+-----------+---------------------------------+ -| December 15, 2017 | 5.1.0 | Yardstick for Euphrates release | -| | | | -+-------------------+-----------+---------------------------------+ -| October 20, 2017 | 5.0.0 | Yardstick for Euphrates release | +| Jan 10, 2020 | 9.0.0 | Yardstick for Iruya release | | | | | +-------------------+-----------+---------------------------------+ Important Notes ---------------- +=============== The software delivered in the OPNFV Yardstick_ Project, comprising the -*Yardstick framework*, the *Yardstick test cases* and the experimental -framework *Apex Lake* is a realization of the methodology in ETSI-ISG -NFV-TST001_. +*Yardstick framework*, and the *Yardstick test cases* is a realization of +the methodology in ETSI-ISG NFV-TST001_. The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. -OPNFV Euphrates Release ------------------------ +OPNFV Iruya Release +==================== -This Euphrates release provides *Yardstick* as a framework for NFVI testing +This Iruya release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx * User Guide - * Developer Guide - * Release notes (this document) - * Results * Automated Yardstick test suite (daily, weekly) @@ -84,39 +73,29 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Yardstick plug-in configuration yaml files, plug-in install/remove scripts -For Euphrates release, the *Yardstick framework* is used for the following +For Iruya release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: * Compute - * Network - * Storage -* OPNFV platform network service benchmarking(NSB) +* OPNFV platform network service benchmarking (NSB) * NSB * Test cases for the following OPNFV Projects: * Container4NFV - * High Availability - * IPv6 - * KVM - * Parser - * StorPerf - * VSperf - * virtual Traffic Classifier - The *Yardstick framework* is developed in the OPNFV community, by the Yardstick_ team. @@ -126,49 +105,47 @@ Yardstick_ team. Release Data ------------- +============ +--------------------------------+-----------------------+ | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ -| **Repo/tag** | yardstick/opnfv-5.1.0 | +| **Repo/tag** | yardstick/opnfv-9.0.0 | | | | +--------------------------------+-----------------------+ -| **Yardstick Docker image tag** | opnfv-5.1.0 | +| **Yardstick Docker image tag** | opnfv-9.0.0 | | | | +--------------------------------+-----------------------+ -| **Release designation** | Euphrates | +| **Release designation** | Iruya 9.0 | | | | +--------------------------------+-----------------------+ -| **Release date** | December 15, 2017 | +| **Release date** | Jan 10, 2020 | | | | +--------------------------------+-----------------------+ -| **Purpose of the delivery** | OPNFV Euphrates 5.1.0 | +| **Purpose of the delivery** | OPNFV Iruya 9.0.0 | | | | +--------------------------------+-----------------------+ Deliverables ------------- +============ Documents -^^^^^^^^^ +--------- - - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html + - User Guide: :ref:`<yardstick:userguide>` - - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html + - Developer Guide: :ref:`<yardstick:devguide>` Software Deliverables -^^^^^^^^^^^^^^^^^^^^^ - +--------------------- - - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.1.0) + - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-9.0.0) - -New Contexts -############ +List of Contexts +^^^^^^^^^^^^^^^^ +--------------+-------------------------------------------+ | **Context** | **Description** | @@ -188,31 +165,42 @@ New Contexts +--------------+-------------------------------------------+ -New Runners -########### - -+--------------+-------------------------------------------------------+ -| **Runner** | **Description** | -| | | -+--------------+-------------------------------------------------------+ -| *Arithmetic* | Steps every run arithmetically according to specified | -| | input value | -| | | -+--------------+-------------------------------------------------------+ -| *Duration* | Runs for a specified period of time | -| | | -+--------------+-------------------------------------------------------+ -| *Iteration* | Runs for a specified number of iterations | -| | | -+--------------+-------------------------------------------------------+ -| *Sequence* | Selects input value to a scenario from an input file | -| | and runs all entries sequentially | -| | | -+--------------+-------------------------------------------------------+ - - -New Scenarios -############# +List of Runners +^^^^^^^^^^^^^^^ + ++----------------+-------------------------------------------------------+ +| **Runner** | **Description** | +| | | ++----------------+-------------------------------------------------------+ +| *Arithmetic* | Steps every run arithmetically according to specified | +| | input value | +| | | ++----------------+-------------------------------------------------------+ +| *Duration* | Runs for a specified period of time | +| | | ++----------------+-------------------------------------------------------+ +| *Iteration* | Runs for a specified number of iterations | +| | | ++----------------+-------------------------------------------------------+ +| *IterationIPC* | Runs a configurable number of times before it | +| | returns. Each iteration has a configurable timeout. | +| | | ++----------------+-------------------------------------------------------+ +| *Sequence* | Selects input value to a scenario from an input file | +| | and runs all entries sequentially | +| | | ++----------------+-------------------------------------------------------+ +| *Dynamictp* | A runner that searches for the max throughput with | +| | binary search | +| | | ++----------------+-------------------------------------------------------+ +| *Search* | A runner that runs a specific time before it returns | +| | | ++----------------+-------------------------------------------------------+ + + +List of Scenarios +^^^^^^^^^^^^^^^^^ +----------------+-----------------------------------------------------+ | **Category** | **Delivered** | @@ -234,339 +222,126 @@ New Scenarios | | | +----------------+-----------------------------------------------------+ | *Compute* | * cpuload | -| | | | | * cyclictest | -| | | | | * lmbench | -| | | | | * lmbench_cache | -| | | | | * perf | -| | | | | * unixbench | -| | | | | * ramspeed | -| | | | | * cachestat | -| | | | | * memeoryload | -| | | | | * computecapacity | -| | | | | * SpecCPU2006 | | | | +----------------+-----------------------------------------------------+ | *Networking* | * iperf3 | -| | | | | * netperf | -| | | | | * netperf_node | -| | | | | * ping | -| | | | | * ping6 | -| | | | | * pktgen | -| | | | | * sfc | -| | | | | * sfc with tacker | -| | | -| | * vtc instantion validation | -| | | -| | * vtc instantion validation with noisy neighbors | -| | | -| | * vtc throughput | -| | | -| | * vtc throughput in the presence of noisy neighbors | -| | | | | * networkcapacity | -| | | | | * netutilization | -| | | | | * nstat | -| | | | | * pktgenDPDK | | | | +----------------+-----------------------------------------------------+ | *Parser* | Tosca2Heat | | | | +----------------+-----------------------------------------------------+ -| *Storage* | fio | -| | | -| | bonnie++ | -| | | -| | storagecapacity | +| *Storage* | * fio | +| | * bonnie++ | +| | * storagecapacity | | | | +----------------+-----------------------------------------------------+ | *StorPerf* | storperf | | | | +----------------+-----------------------------------------------------+ -| *NSB* | vPE thoughput test case | +| *NSB* | vFW thoughput test case | | | | +----------------+-----------------------------------------------------+ - New Test cases -^^^^^^^^^^^^^^ - -* Generic NFVI test cases - - * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006 - - * OPNFV_YARDSTICK_TCO79 - Bonnie++ - -* Kubernetes Test cases +-------------- - * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER +opnfv_yardstick_tc015: Processing speed with impact on energy consumption +and CPU load. - * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM +The purpose of TC015 is to evaluate the IaaS compute performance with +regards to CPU processing speed with its impact on the energy consumption. +It measures score of single cpu running and parallel running. Energy +consumption and cpu load are monitored while the cpu test is running. +The purpose is also to be able to spot the trends. Test results, graphs +and similar shall be stored for comparison reasons and product evolution +understanding between different OPNFV versions and/or configurations, +different server types. Version Change --------------- +============== Module Version Changes -^^^^^^^^^^^^^^^^^^^^^^ +---------------------- -This is the fifth tracked release of Yardstick. It is based on following +This is the seventh tracked release of Yardstick. It is based on following upstream versions: -- OpenStack Ocata - -- OpenDayLight Nitrogen - -- ONOS Junco +- OpenStack Stein Document Version Changes -^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------ -This is the fifth tracked version of the Yardstick framework in OPNFV. +This is the seventh tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: -- Yardstick User Guide: add "network service benchmarking(NSB)" chapter; - add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter; - add "Yardstick user interface" chapter; Update Yardstick installation chapter; - +- Yardstick User Guide: - Yardstick Developer Guide - - Yardstick Release Notes for Yardstick: this document Feature additions -^^^^^^^^^^^^^^^^^ - -- Yardstick RESTful API support - -- Network service benchmarking - -- Stress testing with Bottlenecks team - -- Yardstick framework improvement: - - - yardstick report CLI - - - Node context support OpenStack configuration via Ansible - - - Https support - - - Kubernetes context type - -- Yardstick container local GUI - -- Python 3 support +----------------- Scenario Matrix ---------------- - -For Euphrates 5.0.0, Yardstick was tested on the following scenarios: - -+--------------------------+------+---------+------+------+ -| Scenario | Apex | Compass | Fuel | Joid | -+==========================+======+=========+======+======+ -| os-nosdn-nofeature-noha | | | X | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-nofeature-ha | X | X | X | X | -+--------------------------+------+---------+------+------+ -| os-odl_l2-nofeature-ha | | X | X | X | -+--------------------------+------+---------+------+------+ -| os-odl_l2-nofeature-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l3-nofeature-ha | X | X | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l3-nofeature-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-onos-sfc-ha | | | | | -+--------------------------+------+---------+------+------+ -| os-onos-nofeature-ha | | X | | X | -+--------------------------+------+---------+------+------+ -| os-onos-nofeature-noha | | | | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-sfc-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-sfc-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-bgpvpn-ha | X | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-bgpvpn-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm-ha | X | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-ovs-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-ovs-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-ocl-nofeature-ha | | X | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-lxd-ha | | | | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-lxd-noha | | | | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-fdio-ha | X | | | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-fdio-noha | X | | | | -+--------------------------+------+---------+------+------+ -| os-odl-gluon-noha | X | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-openo-ha | | X | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| -noha | | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-ha | | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-noha | | | | | -+--------------------------+------+---------+------+------+ -| opnfv_os-ovn-nofeature- | X | | | | -| noha_daily | | | | | -+--------------------------+------+---------+------+------+ +=============== + Test results ------------- +============ Test results are available in: - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/ -The reporting pages can be found at: - -+---------------+-------------------------------------------------------------------------------------+ -| apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html | -+---------------+-------------------------------------------------------------------------------------+ -| compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html | -+---------------+-------------------------------------------------------------------------------------+ -| fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html | -+---------------+-------------------------------------------------------------------------------------+ -| fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html | -+---------------+-------------------------------------------------------------------------------------+ -| joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html | -+---------------+-------------------------------------------------------------------------------------+ Known Issues/Faults -^^^^^^^^^^^^^^^^^^^ +------------------- Corrected Faults -^^^^^^^^^^^^^^^^ +---------------- + -Euphrates 5.1.0: - -+---------------------+-------------------------------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-841 | Fix various NSB license issues | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-73 | How To Work with Test Cases | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-500 | VNF testing documentation | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-826 | Allow overriding Heat IP addresses to match traffic generator profile | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-828 | Refactor doc/testing/user/userguide "Yardstick Installation" | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-830 | build_yardstick_image Ansible mount module doesn't work on Ubuntu 14.04 | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-833 | ansible_common transform password into lower case | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-847 | tc006, tc079, tc082 miss grafana dashboard in local deployment | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-849 | kill process do not accurately kill the process like "nova-api" | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-850 | tc023 miss description and tc050-58 wrong description | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-852 | tc078 cpu2006 fails in some situation | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-854 | yardstick docker lack of trex_client | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-867 | testcase tc078 have no data stored or dashboard to show results | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-871 | Remove img_modify_playbook assignation in build_yardstick_image.yml | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-829 | "nsb_setup.sh" doesn't parse the controller IP correctly | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-839 | NSB Prox BM test cases to be fixed for incorporating scale-up | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-840 | NSB Prox test documentation of vPE and LW-AFTR test cases | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-848 | NSB "Prox" : Cleanup duplicated traffic profile | -+---------------------+-------------------------------------------------------------------------+ - - - - -Euphrates 5.0.0: - -+---------------------+--------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+---------------------+--------------------------------------------+ -| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using | -| | 'openstack -h' | -+---------------------+--------------------------------------------+ -| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they | -| | are dynamic | -+---------------------+--------------------------------------------+ - - -Euphratess 5.0.0 known restrictions/issues ------------------------------------------- -+-----------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+===========+===========+==============================================+ -| any | \*-bgpvpn | Floating ips not supported. Some Test cases | -| | | related to floating ips are excluded. | -+-----------+-----------+----------------------------------------------+ -| any | odl_l3-\* | Some test cases related to using floating IP | -| | | addresses fail because of a known ODL bug. | -| | | | -+-----------+-----------+----------------------------------------------+ -| compass | odl_l2-\* | In some test cases, VM instance will failed | -| | | raising network interfaces. | -| | | | -+-----------+-----------+----------------------------------------------+ +Iruya 9.0.0 known restrictions/issues +====================================== Useful links ------------- +============ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning + - wiki Yardstick Iruya release planning page: https://wiki.opnfv.org/display/yardstick/Release+Iruya - - Yardstick repo: https://git.opnfv.org/cgit/yardstick + - Yardstick repo: https://git.opnfv.org/yardstick - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ - - Yardstick IRC chanel: #opnfv-yardstick + - Yardstick IRC channel: #opnfv-yardstick |