diff options
Diffstat (limited to 'docs/release')
| -rw-r--r-- | docs/release/release-notes/release-notes.rst | 436 | ||||
| -rw-r--r-- | docs/release/results/euphrates_fraser_comparison.rst | 10 | ||||
| -rw-r--r-- | docs/release/results/images/tc014_pod_fraser.png (renamed from docs/release/results/images/tc014_pod_fraseer.png) | bin | 29513 -> 29513 bytes | |||
| -rw-r--r-- | docs/release/results/index.rst | 5 | ||||
| -rw-r--r-- | docs/release/results/overview.rst | 9 | ||||
| -rw-r--r-- | docs/release/results/results.rst | 17 | ||||
| -rw-r--r-- | docs/release/results/yardstick-opnfv-vtc.rst | 248 |
7 files changed, 165 insertions, 560 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index daa4b8187..4b3f12bcf 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -1,33 +1,27 @@ -======= -License -======= - -OPNFV Fraser release note for Yardstick Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 The *Yardstick framework*, the *Yardstick test cases* are open-source software, licensed under the terms of the Apache License, Version 2.0. -======================================= -OPNFV Fraser Release Note for Yardstick -======================================= +======================= +Yardstick Release Notes +======================= .. toctree:: :maxdepth: 2 -.. _Yardstick: https://wiki.opnfv.org/yardstick +.. _Yardstick: https://wiki.opnfv.org/display/yardstick -.. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main +.. _Dashboard: http://testresults.opnfv.org/grafana/ -.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf +.. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf Abstract ======== -This document describes the release note of Yardstick project. +This document compiles the release notes for the Gambia release of OPNFV Yardstick. Version History @@ -36,16 +30,7 @@ Version History | *Date* | *Version* | *Comment* | | | | | +-------------------+-----------+---------------------------------+ -| Jul 2, 2018 | 6.2.1 | Yardstick for Fraser release | -| | | | -+-------------------+-----------+---------------------------------+ -| Jun 29, 2018 | 6.2.0 | Yardstick for Fraser release | -| | | | -+-------------------+-----------+---------------------------------+ -| May 25, 2018 | 6.1.0 | Yardstick for Fraser release | -| | | | -+-------------------+-----------+---------------------------------+ -| April 27, 2018 | 6.0.0 | Yardstick for Fraser release | +| November 9, 2018 | 7.0.0 | Yardstick for Gambia release | | | | | +-------------------+-----------+---------------------------------+ @@ -60,10 +45,10 @@ the methodology in ETSI-ISG NFV-TST001_. The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. -OPNFV Fraser Release +OPNFV Gambia Release ==================== -This Fraser release provides *Yardstick* as a framework for NFVI testing +This Gambia release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx @@ -88,7 +73,7 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Yardstick plug-in configuration yaml files, plug-in install/remove scripts -For Fraser release, the *Yardstick framework* is used for the following +For Gambia release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: @@ -126,19 +111,19 @@ Release Data | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ -| **Repo/tag** | yardstick/opnfv-6.2.0 | +| **Repo/tag** | yardstick/opnfv-7.0.0 | | | | +--------------------------------+-----------------------+ -| **Yardstick Docker image tag** | opnfv-6.2.0 | +| **Yardstick Docker image tag** | opnfv-7.0.0 | | | | +--------------------------------+-----------------------+ -| **Release designation** | Fraser | +| **Release designation** | Gambia 7.0 | | | | +--------------------------------+-----------------------+ -| **Release date** | Jun 29, 2018 | +| **Release date** | November 9, 2018 | | | | +--------------------------------+-----------------------+ -| **Purpose of the delivery** | OPNFV Fraser 6.2.0 | +| **Purpose of the delivery** | OPNFV Gambia 7.0.0 | | | | +--------------------------------+-----------------------+ @@ -149,15 +134,15 @@ Deliverables Documents --------- - - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html + - User Guide: :ref:`<yardstick:userguide>` - - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html + - Developer Guide: :ref:`<yardstick:devguide>` Software Deliverables --------------------- - - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.2.0) + - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0) List of Contexts ^^^^^^^^^^^^^^^^ @@ -183,33 +168,37 @@ List of Contexts List of Runners ^^^^^^^^^^^^^^^ -.. note:: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search". - -+---------------+-------------------------------------------------------+ -| **Runner** | **Description** | -| | | -+---------------+-------------------------------------------------------+ -| *Arithmetic* | Steps every run arithmetically according to specified | -| | input value | -| | | -+---------------+-------------------------------------------------------+ -| *Duration* | Runs for a specified period of time | -| | | -+---------------+-------------------------------------------------------+ -| *Iteration* | Runs for a specified number of iterations | -| | | -+---------------+-------------------------------------------------------+ -| *Sequence* | Selects input value to a scenario from an input file | -| | and runs all entries sequentially | -| | | -+---------------+-------------------------------------------------------+ -| **Dynamictp** | A runner that searches for the max throughput with | -| | binary search | -| | | -+---------------+-------------------------------------------------------+ -| **Search** | A runner that runs a specific time before it returns | -| | | -+---------------+-------------------------------------------------------+ +.. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC". + ++----------------+-------------------------------------------------------+ +| **Runner** | **Description** | +| | | ++----------------+-------------------------------------------------------+ +| *Arithmetic* | Steps every run arithmetically according to specified | +| | input value | +| | | ++----------------+-------------------------------------------------------+ +| *Duration* | Runs for a specified period of time | +| | | ++----------------+-------------------------------------------------------+ +| *Iteration* | Runs for a specified number of iterations | +| | | ++----------------+-------------------------------------------------------+ +| *IterationIPC* | Runs a configurable number of times before it | +| | returns. Each iteration has a configurable timeout. | +| | | ++----------------+-------------------------------------------------------+ +| *Sequence* | Selects input value to a scenario from an input file | +| | and runs all entries sequentially | +| | | ++----------------+-------------------------------------------------------+ +| *Dynamictp* | A runner that searches for the max throughput with | +| | binary search | +| | | ++----------------+-------------------------------------------------------+ +| *Search* | A runner that runs a specific time before it returns | +| | | ++----------------+-------------------------------------------------------+ List of Scenarios @@ -280,19 +269,15 @@ List of Scenarios New Test cases -------------- -.. note:: Yardstick Fraser 6.1.0 added two new test cases, "TC092" and "TC093". +.. note:: Yardstick Gambia 7.0.0 adds no new test cases. * Generic NFVI test cases - * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM + * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM * HA Test cases - * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration - * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance - * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api - * OPNFV_YARDSTICK_TC092 - SDN Controller resilience in HA configuration - * OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration + * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration Version Change @@ -301,22 +286,23 @@ Version Change Module Version Changes ---------------------- -This is the sixth tracked release of Yardstick. It is based on following +This is the seventh tracked release of Yardstick. It is based on following upstream versions: -- OpenStack Pike +- OpenStack Queens - OpenDayLight Oxygen Document Version Changes ------------------------ -This is the sixth tracked version of the Yardstick framework in OPNFV. +This is the seventh tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: -- Yardstick User Guide: add "network service benchmarking(NSB)" chapter; - add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter; - add "Yardstick user interface" chapter; Update Yardstick installation chapter; +- Yardstick User Guide: + + - Remove vTC chapter; + - Yardstick Developer Guide - Yardstick Release Notes for Yardstick: this document @@ -324,49 +310,49 @@ It includes the following documentation updates: Feature additions ----------------- -- Plugin-based test cases support Heat context -- SR-IOV support for the Heat context -- Support using existing network in Heat context -- Support running test cases with existing VNFs/without destroying VNF in Heat context -- Add vFW scale-up template -- Improvements of unit tests and gating -- GUI improvement about passing parameters +- Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml).... +- Spirent support. +- vEPC testcases. +- Agnostic VNF tests cases for reuse of standard RFC-2544 test case. +- PROX enhancements and the addition of Standalone test case using SRIOV and + OVS-DPDK. +- Ixia enhancements for vBNG and PPPoE traffic. +- Improvements of unit tests and gating. +- Add DPDK pktgen traffic generator. +- Kubernetes context enhancements. +- Kubernetes sample test cases of fio and lmbench added. Scenario Matrix =============== -For Fraser 6.0.0, Yardstick was tested on the following scenarios: - -+-------------------------+------+---------+----------+------+------+-------+ -| Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy | -+=========================+======+=========+==========+======+======+=======+ -| os-nosdn-nofeature-noha | X | X | | | X | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-nofeature-ha | X | X | X | X | X | X | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-bar-noha | X | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-bar-ha | X | | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-odl-bgpvpn-ha | X | | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-calipso-noha | X | | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-kvm-ha | | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-odl_l3-nofeature-ha | | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-odl-sfc-ha | | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| os-odl-nofeature-ha | | | | X | | X | -+-------------------------+------+---------+----------+------+------+-------+ -| os-nosdn-ovs-ha | | | | X | | | -+-------------------------+------+---------+----------+------+------+-------+ -| k8-nosdn-nofeature-ha | | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ -| k8-nosdn-stor4nfv-noha | | X | | | | | -+-------------------------+------+---------+----------+------+------+-------+ +For Gambia 7.0.0, Yardstick was tested on the following scenarios: + ++-------------------------+------+---------+----------+------+ +| Scenario | Apex | Compass | Fuel-arm | Fuel | ++=========================+======+=========+==========+======+ +| os-nosdn-nofeature-noha | X | | | | ++-------------------------+------+---------+----------+------+ +| os-nosdn-nofeature-ha | X | | | | ++-------------------------+------+---------+----------+------+ +| os-odl-bgpvpn-noha | X | | | | ++-------------------------+------+---------+----------+------+ +| os-nosdn-calipso-noha | X | | | | ++-------------------------+------+---------+----------+------+ +| os-nosdn-kvm-ha | | X | | | ++-------------------------+------+---------+----------+------+ +| os-odl-nofeature-ha | | | X | X | ++-------------------------+------+---------+----------+------+ +| os-odl-sfc-noha | X | | | | ++-------------------------+------+---------+----------+------+ +| os-nosdn-ovs-ha | | | | X | ++-------------------------+------+---------+----------+------+ +| k8-nosdn-nofeature-ha | | X | | | ++-------------------------+------+---------+----------+------+ +| k8-nosdn-stor4nfv-noha | | X | | | ++-------------------------+------+---------+----------+------+ +| k8-nosdn-stor4nfv-ha | | X | | | ++-------------------------+------+---------+----------+------+ Test results @@ -376,19 +362,6 @@ Test results are available in: - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/ -The reporting pages can be found at: - -+---------------+----------------------------------------------------------------------------------+ -| apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html | -+---------------+----------------------------------------------------------------------------------+ -| compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html | -+---------------+----------------------------------------------------------------------------------+ -| fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html | -+---------------+----------------------------------------------------------------------------------+ -| fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html | -+---------------+----------------------------------------------------------------------------------+ -| joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html | -+---------------+----------------------------------------------------------------------------------+ Known Issues/Faults ------------------- @@ -397,219 +370,70 @@ Known Issues/Faults Corrected Faults ---------------- -Fraser 6.2.1: +Gambia 7.0.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ -| YARDSTICK-1147 | Fix ansible scripts for running in container | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1157 | Bug Fix: correct the file path to build docker file | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1276 | Bugfix: docker build failed | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1280 | Bugfix: uwsgi config file yardstick.ini output error | -+--------------------+--------------------------------------------------------------------------+ - -Fraser 6.2.0: - -+--------------------+--------------------------------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -+====================+==========================================================================+ -| YARDSTICK-1246 | Update pmd/lcore mask for OVS-DPDK context | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-837 | Move tests: unit/network_services/{lib/,collector/,*.py} | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1144 | Correctly set PYTHONPATH in Dockerfile | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1205 | Set "cmd2" library to version 0.8.6 | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1204 | Bump oslo.messaging version to 5.36.0 | +| YARDSTICK-1137 | Fix CLI argument handling in nsb_setup.sh | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1210 | Remove __init__ method overriding in HeatContextTestCase | +| YARDSTICK-1220 | Get stats for multiple port simultaneously | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1189 | Error when adding SR-IOV interfaces in SR-IOV context | +| YARDSTICK-1260 | Added missing functionality to start VM and access it using SSH keys | +| | in Standalone contexts. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1214 | Remove AnsibleCommon class method mock | +| YARDSTICK-1298 | Allows for in-line overriding/modification of traffic profile variables | +| | from the testcase file. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1159 | Add --hwlb options as a command line argument for SampleVNF | +| YARDSTICK-1368 | Updated existing test cases in Yardstick to minimize changes done | +| | manually to run standalone tests for Trex. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1203 | Add scale out TCs with availability zone support | +| YARDSTICK-1389 | Add status filed for RFC2544 TC iterations | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1167 | Do not start collectd twice when SampleVNF is running on Baremetal | +| YARDSTICK-1395 | Update 'configure_uwsgi' role to work in baremetal/container modes. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1188 | Add "host_name_separator" variable to Context class | +| YARDSTICK-1402 | Change IP assignment for VM to static for standalone context | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1112 | MQ startup process refactor | +| YARDSTICK-1404 | CPU Utilization for VNF and traffic generator are now graphed on Grafana | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1229 | Cleanup BaseMonitor unit tests | +| YARDSTICK-1411 | Fix Yardstick Docker image ARM build | +--------------------+--------------------------------------------------------------------------+ -| - | Configure ACL via static file | +| YARDSTICK-1414 | Update the pinned sampleVNF version to use a commit instead of a branch | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1191 | Use TRex release v2.41 to support both x86 and aarch64 | +| YARDSTICK-1418 | NSB PROX NFVi test now stops after reaching expected precision | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1106 | Add IxNetwork API Python Binding package | +| YARDSTICK-1457 | Fix influxdb "field type conflict" error | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1224 | Cleanup TestYardstickNSCli class | +| YARDSTICK-1458 | Update Grafana to display "real-time" data instead of historical data. | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1225 | Remove print out of logger exception in TestUtils | +| YARDSTICK-1462 | NSB: Add OvS 2.8.1 support in SA context | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1194 | Add "duration" parameter to test case definition | +| YARDSTICK-1492 | Make OvS server to listen on TCP | +--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1209 | Remove instantiated contexts in "test_task" | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1192 | Standalone XML machine type is not longer valid | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1197 | Refactor RFC2455 TRex traffic profile injection | -+--------------------+--------------------------------------------------------------------------+ -| - | Fix "os.path" mock problems during tests | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1218 | Refactor "utils.parse_ini_file" testing | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1179 | Start nginx and uwsgi servicies only in not container mode | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1177 | Install dependencies: bare-metal, standalone | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1126 | Migrate install.sh script to ansible | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1146 | Fix nsb_setup.sh script | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1247 | NSB setup inventory name changed | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1116 | Changed IxNextgen library load in IXIA RFC2544 traffic generator call. | -+--------------------+--------------------------------------------------------------------------+ -| - | Corrected scale-up command line arguments | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-878 | OpenStack client replacement | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1222 | Bugfix: HA kill process recovery has a conflict | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1139 | Add "os_cloud_config" as a new context flag parameter | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1255 | Extended Context class with get_physical_nodes functionality | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1244 | NSB NFVi BNG test fails to run - stops after one step | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1219 | Decrease Sampling interval | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1101 | NSB NFVi PROX BNG losing many packets | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1217 | Fix NSB NfVi support for 25 and 40Gbps | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1185 | NSB Topology fix for Prox 4 port test case | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-966 | Convert SLA asserts to raises | +| YARDSTICK-1493 | The RX queues number is hard-codded and cannot be changed | +--------------------+--------------------------------------------------------------------------+ -Fraser 6.1.0: - -+--------------------+--------------------------------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -+====================+==========================================================================+ -| YARDSTICK-995 | Test case spec for SDN Virtual Switch resilience | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1097 | Add pod.yaml file for APEX installer | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1122 | Remove unused code in SampleVNF | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1125 | Update samples/test_suite.yaml | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1132 | Document for Euphrates test case results | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1138 | Support Restart Operation | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1142 | start_service script fails to start openvswitch service in centos distro | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1165 | Bugfix: openrc api dump should be safe_dump | -+--------------------+--------------------------------------------------------------------------+ - -Fraser 6.0.0: - -+--------------------+--------------------------------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -+====================+==========================================================================+ -| YARDSTICK-831 | tc053 kill haproxy wrong | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-842 | load image fails when there's cirros image exist | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-875 | Check if multiverse repository is available in Ubuntu | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-893 | Fix proxy env handling and ansible multinode support | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-899 | Variable local_iface_name is read before it is set | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-994 | NSB set-up build script for baremetal broken | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1012 | constants: fix pylint warnings for OSError | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1014 | Remove unused args in | -| | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1016 | Allow vm to access outside world through default gateway | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1047 | Error in string format in HeatTemplateError message | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1056 | yardstick report command print error when run test case | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1059 | Reduce the log level if TRex client is no connected | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1073 | Error when retrieving "options" section in "scenario" | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1102 | Don't hide exception traceback from Task.start() | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1107 | bad exception traceback print due to atexit_handler | -+--------------------+--------------------------------------------------------------------------+ -| YARDSTICK-1120 | HA test case tc050 should start monitor before attack | -+--------------------+--------------------------------------------------------------------------+ -Fraser 6.0.0 known restrictions/issues +Gambia 7.0.0 known restrictions/issues ====================================== -+-----------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+===========+===========+==============================================+ -| | | | -+-----------+-----------+----------------------------------------------+ ++-----------+-----------------------+------------------+ +| Installer | Scenario | Issue | ++===========+=======================+==================+ +| apex | os-ovn-nofeature-ha | YARDSTICK-1517 | ++-----------+-----------------------+------------------+ Useful links ============ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser + - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia - - Yardstick repo: https://git.opnfv.org/cgit/yardstick + - Yardstick repo: https://git.opnfv.org/yardstick - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ - - Yardstick IRC chanel: #opnfv-yardstick + - Yardstick IRC channel: #opnfv-yardstick diff --git a/docs/release/results/euphrates_fraser_comparison.rst b/docs/release/results/euphrates_fraser_comparison.rst index 53dfb994f..1dd328bb7 100644 --- a/docs/release/results/euphrates_fraser_comparison.rst +++ b/docs/release/results/euphrates_fraser_comparison.rst @@ -2,7 +2,15 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 -======================================================= +.. + Convention for heading levels in Yardstick: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + Avoid deeper levels because they do not render well. + Test results analysis for Euphrates and Fraser releases ======================================================= diff --git a/docs/release/results/images/tc014_pod_fraseer.png b/docs/release/results/images/tc014_pod_fraser.png Binary files differindex 697201d76..697201d76 100644 --- a/docs/release/results/images/tc014_pod_fraseer.png +++ b/docs/release/results/images/tc014_pod_fraser.png diff --git a/docs/release/results/index.rst b/docs/release/results/index.rst index 63445fd1a..30cf62284 100644 --- a/docs/release/results/index.rst +++ b/docs/release/results/index.rst @@ -15,3 +15,8 @@ Yardstick test results .. include:: ./overview.rst .. include:: ./results.rst .. include:: ./euphrates_fraser_comparison.rst + +.. include:: ./yardstick-opnfv-ha.rst +.. include:: ./yardstick-opnfv-kvm.rst +.. include:: ./yardstick-opnfv-parser.rst + diff --git a/docs/release/results/overview.rst b/docs/release/results/overview.rst index 9fd74797c..b5e6a43a6 100644 --- a/docs/release/results/overview.rst +++ b/docs/release/results/overview.rst @@ -3,6 +3,15 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Ericsson AB and others. +.. + Convention for heading levels in Yardstick: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + Avoid deeper levels because they do not render well. + Yardstick test tesult document overview ======================================= diff --git a/docs/release/results/results.rst b/docs/release/results/results.rst index 0ed92f867..f0c20360b 100644 --- a/docs/release/results/results.rst +++ b/docs/release/results/results.rst @@ -2,8 +2,17 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 +.. + Convention for heading levels in Yardstick: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + Avoid deeper levels because they do not render well. + Results listed by test cases -========================== +---------------------------- .. _TOM: https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results @@ -14,7 +23,7 @@ the determined state of the specific test case as executed in the Fraser release process. All test date are analyzed using TOM_ tool. Scenario Results -================ +---------------- .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main .. _Jenkins: https://build.opnfv.org/ci/view/yardstick/ @@ -42,7 +51,7 @@ Test results of executed tests are avilable in Dashboard_ and logs in Jenkins_. Test results for Fraser release are collected from April 10, 2018 to May 13, 2018. Feature Test Results -==================== +-------------------- The following features were verified by Yardstick test cases: @@ -54,8 +63,6 @@ The following features were verified by Yardstick test cases: * Parser - * Virtual Traffic Classifier (see :doc:`yardstick-opnfv-vtc`) - * StorPerf .. note:: The test cases for IPv6 and Parser Projects are included in the diff --git a/docs/release/results/yardstick-opnfv-vtc.rst b/docs/release/results/yardstick-opnfv-vtc.rst deleted file mode 100644 index 059b5491f..000000000 --- a/docs/release/results/yardstick-opnfv-vtc.rst +++ /dev/null @@ -1,248 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - -.. _Dashboard006: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc006 -.. _Dashboard007: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc007 -.. _Dashboard020: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc020 -.. _Dashboard021: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc021 -.. _DashboardVTC: http://testresults.opnfv.org/grafana/dashboard/db/vtc-dashboard -==================================== -Test Results for yardstick-opnfv-vtc -==================================== - -.. toctree:: - :maxdepth: 2 - - -Details -======= - -.. after this doc is filled, remove all comments and include the scenario in -.. results.rst by removing the comment on the file name. - - -Overview of test results ------------------------- - -.. general on metrics collected, number of iterations - -The virtual Traffic Classifier (vtc) Scenario supported by Yardstick is used by 4 Test Cases: - -- TC006 -- TC007 -- TC020 -- TC021 - - -* TC006 - -TC006 is the Virtual Traffic Classifier Data Plane Throughput Benchmarking Test. -It collects measures about the end-to-end throughput supported by the -virtual Traffic Classifier (vTC). -Results of the test are shown in the Dashboard006_ -The throughput is expressed as percentage of the available bandwidth on the NIC. - - -* TC007 - -TC007 is the Virtual Traffic Classifier Data Plane Throughput Benchmarking in presence of -noisy neighbors Test. -It collects measures about the end-to-end throughput supported by the -virtual Traffic Classifier when a user-defined number of noisy neighbors is deployed. -Results of the test are shown in the Dashboard007_ -The throughput is expressed as percentage of the available bandwidth on the NIC. - - -* TC020 - -TC020 is the Virtual Traffic Classifier Instantiation Test. -It verifies that a newly instantiated vTC is alive and functional and its instantiation -is correctly supported by the underlying infrastructure. -Results of the test are shown in the Dashboard020_ - - -* TC021 - -TC021 is the Virtual Traffic Classifier Instantiation in presence of noisy neighbors Test. -It verifies that a newly instantiated vTC is alive and functional and its instantiation -is correctly supported by the underlying infrastructure when noisy neighbors are present. -Results of the test are shown in the Dashboard021_ - -* Generic - -In the Generic scenario the Virtual Traffic Classifier is running on a standard Openstack -setup and traffic is being replayed from a neighbor VM. The traffic sent contains -various protocols and applications, and the VTC identifies them and exports the data. -Results of the test are shown in the DashboardVTC. - -Detailed test results ---------------------- - -* TC006 - -The results for TC006 have been obtained using the following test case -configuration: - -- Context: Dummy -- Scenario: vtc_throughput -- Network Techology: SR-IOV -- vTC Flavor: m1.large - - -* TC007 - -The results for TC007 have been obtained using the following test case -configuration: - -- Context: Dummy -- Scenario: vtc_throughput_noisy -- Network Techology: SR-IOV -- vTC Flavor: m1.large -- Number of noisy neighbors: 2 -- Number of cores per neighbor: 2 -- Amount of RAM per neighbor: 1G - - -* TC020 - -The results for TC020 have been obtained using the following test case -configuration: - -The results listed in previous section have been obtained using the following -test case configuration: - -- Context: Dummy -- Scenario: vtc_instantiation_validation -- Network Techology: SR-IOV -- vTC Flavor: m1.large - - -* TC021 - -The results listed in previous section have been obtained using the following -test case configuration: - -- Context: Dummy -- Scenario: vtc_instantiation_validation -- Network Techology: SR-IOV -- vTC Flavor: m1.large -- Number of noisy neighbors: 2 -- Number of cores per neighbor: 2 -- Amount of RAM per neighbor: 1G - - -For all the test cases, the user can specify different values for the parameters. - -* Generic - -The results listed in the previous section have been obtained, using a -standard Openstack setup. -The user can replay his/her own traffic and see the corresponding results. - -Rationale for decisions ------------------------ - -* TC006 - -The result of the test is a number between 0 and 100 which represents the percentage of bandwidth -available on the NIC that corresponds to the supported throughput by the vTC. - - -* TC007 - -The result of the test is a number between 0 and 100 which represents the percentage of bandwidth -available on the NIC that corresponds to the supported throughput by the vTC. - -* TC020 - -The execution of the test is done as described in the following: - -- The vTC is deployed on the OpenStack testbed; -- Some traffic is sent to the vTC; -- The vTC changes the header of the packets and sends them back to the packet generator; -- The packet generator checks that all the packets are received correctly and have been changed -correctly by the vTC. - -The test is declared as PASSED if all the packets are correcly received by the packet generator -and they have been modified by the virtual Traffic Classifier as required. - - -* TC021 - -The execution of the test is done as described in the following: - -- The vTC is deployed on the OpenStack testbed; -- The noisy neighbors are deployed as requested by the user; -- Some traffic is sent to the vTC; -- The vTC change the header of the packets and sends them back to the packet generator; -- The packet generator checks that all the packets are received correctly and have been changed -correctly by the vTC - -The test is declared as PASSED if all the packets are correcly received by the packet generator -and they have been modified by the virtual Traffic Classifier as required. - -* Generic - -The execution of the test consists of the following actions: - -- The vTC is deployed on the OpenStack testbed; -- The traffic generator VM is deployed on the Openstack Testbed; -- Traffic data are relevant to the network setup; -- Traffic is sent to the vTC; - - - -Conclusions and recommendations -------------------------------- - -* TC006 - -The obtained results show that the virtual Traffic Classifier can support up to 4 Gbps -(40% of the available bandwidth) correspond to the expected behaviour of the virtual -Traffic Classifier. -Using the configuration with SR-IOV and large flavor, the expected throughput should -generally be in the range between 3 and 4 Gbps. - - -* TC007 - -These results correspond to the configuration in which the virtual Traffic Classifier uses SR-IOV -Virtual Functions and the flavor is set to large for the virtual machine. -The throughput is in the range between 2.5 Gbps and 3.7 Gbps. -This shows that the effect of 2 noisy neighbors reduces the throughput of -the service between 10 and 20%. -Increasing number of neihbours would have a higher impact on the performance. - - -* TC020 - -The obtained results correspond to the expected behaviour of the virtual Traffic Classifier. -Using the configuration with SR-IOV and large flavor, the expected result is that the vTC is -correctly instantiated, it is able to receive and send packets using SR-IOV technology -and to forward packets back to the packet generator changing the TCP/IP header as required. - - -* TC021 - -The obtained results correspond to the expected behaviour of the virtual Traffic Classifier. -Using the configuration with SR-IOV and large flavor, the expected result is that the vTC is -correctly instantiated, it is able to receive and send packets using SR-IOV technology -and to forward packets back to the packet generator changing the TCP/IP header as required, -also in presence of noisy neighbors. - -* Generic - -The obtained results correspond to the expected behaviour of the virtual Traffic Classifier. -Using the aforementioned configuration the expected application protocols are identified -and their traffic statistics are demonstrated in the DashboardVTC, a group of popular -applications is selected to demonstrate the sound operation of the vTC. -The demonstrated application protocols are: -- HTTP -- Skype -- Bittorrent -- Youtube -- Dropbox -- Twitter -- Viber -- iCloud |
