aboutsummaryrefslogtreecommitdiffstats
path: root/docs/release
diff options
context:
space:
mode:
Diffstat (limited to 'docs/release')
-rw-r--r--docs/release/release-notes/release-notes.rst428
-rw-r--r--docs/release/results/euphrates_fraser_comparison.rst10
-rw-r--r--docs/release/results/images/tc014_pod_fraser.png (renamed from docs/release/results/images/tc014_pod_fraseer.png)bin29513 -> 29513 bytes
-rw-r--r--docs/release/results/index.rst5
-rw-r--r--docs/release/results/overview.rst9
-rw-r--r--docs/release/results/results.rst17
-rw-r--r--docs/release/results/yardstick-opnfv-vtc.rst248
7 files changed, 151 insertions, 566 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst
index daa4b8187..914daa3a4 100644
--- a/docs/release/release-notes/release-notes.rst
+++ b/docs/release/release-notes/release-notes.rst
@@ -1,33 +1,27 @@
-=======
-License
-=======
-
-OPNFV Fraser release note for Yardstick Docs
-are licensed under a Creative Commons Attribution 4.0 International License.
-You should have received a copy of the license along with this.
-If not, see <http://creativecommons.org/licenses/by/4.0/>.
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
The *Yardstick framework*, the *Yardstick test cases* are open-source software,
licensed under the terms of the Apache License, Version 2.0.
-=======================================
-OPNFV Fraser Release Note for Yardstick
-=======================================
+=======================
+Yardstick Release Notes
+=======================
.. toctree::
:maxdepth: 2
-.. _Yardstick: https://wiki.opnfv.org/yardstick
+.. _Yardstick: https://wiki.opnfv.org/display/yardstick
-.. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
+.. _Dashboard: http://testresults.opnfv.org/grafana/
-.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
+.. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
Abstract
========
-This document describes the release note of Yardstick project.
+This document compiles the release notes for the Gambia release of OPNFV Yardstick.
Version History
@@ -36,16 +30,10 @@ Version History
| *Date* | *Version* | *Comment* |
| | | |
+-------------------+-----------+---------------------------------+
-| Jul 2, 2018 | 6.2.1 | Yardstick for Fraser release |
-| | | |
-+-------------------+-----------+---------------------------------+
-| Jun 29, 2018 | 6.2.0 | Yardstick for Fraser release |
+| November 9, 2018 | 7.0.0 | Yardstick for Gambia release |
| | | |
+-------------------+-----------+---------------------------------+
-| May 25, 2018 | 6.1.0 | Yardstick for Fraser release |
-| | | |
-+-------------------+-----------+---------------------------------+
-| April 27, 2018 | 6.0.0 | Yardstick for Fraser release |
+| December 14, 2018 | 7.1.0 | Yardstick for Gambia release |
| | | |
+-------------------+-----------+---------------------------------+
@@ -60,10 +48,10 @@ the methodology in ETSI-ISG NFV-TST001_.
The *Yardstick* framework is *installer*, *infrastructure* and *application*
independent.
-OPNFV Fraser Release
+OPNFV Gambia Release
====================
-This Fraser release provides *Yardstick* as a framework for NFVI testing
+This Gambia release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
@@ -88,7 +76,7 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Yardstick plug-in configuration yaml files, plug-in install/remove scripts
-For Fraser release, the *Yardstick framework* is used for the following
+For Gambia release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
@@ -126,19 +114,19 @@ Release Data
| **Project** | Yardstick |
| | |
+--------------------------------+-----------------------+
-| **Repo/tag** | yardstick/opnfv-6.2.0 |
+| **Repo/tag** | yardstick/opnfv-7.1.0 |
| | |
+--------------------------------+-----------------------+
-| **Yardstick Docker image tag** | opnfv-6.2.0 |
+| **Yardstick Docker image tag** | opnfv-7.1.0 |
| | |
+--------------------------------+-----------------------+
-| **Release designation** | Fraser |
+| **Release designation** | Gambia 7.1 |
| | |
+--------------------------------+-----------------------+
-| **Release date** | Jun 29, 2018 |
+| **Release date** | December 14, 2018 |
| | |
+--------------------------------+-----------------------+
-| **Purpose of the delivery** | OPNFV Fraser 6.2.0 |
+| **Purpose of the delivery** | OPNFV Gambia 7.1.0 |
| | |
+--------------------------------+-----------------------+
@@ -149,15 +137,15 @@ Deliverables
Documents
---------
- - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html
+ - User Guide: :ref:`<yardstick:userguide>`
- - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html
+ - Developer Guide: :ref:`<yardstick:devguide>`
Software Deliverables
---------------------
- - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.2.0)
+ - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
List of Contexts
^^^^^^^^^^^^^^^^
@@ -183,33 +171,37 @@ List of Contexts
List of Runners
^^^^^^^^^^^^^^^
-.. note:: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search".
-
-+---------------+-------------------------------------------------------+
-| **Runner** | **Description** |
-| | |
-+---------------+-------------------------------------------------------+
-| *Arithmetic* | Steps every run arithmetically according to specified |
-| | input value |
-| | |
-+---------------+-------------------------------------------------------+
-| *Duration* | Runs for a specified period of time |
-| | |
-+---------------+-------------------------------------------------------+
-| *Iteration* | Runs for a specified number of iterations |
-| | |
-+---------------+-------------------------------------------------------+
-| *Sequence* | Selects input value to a scenario from an input file |
-| | and runs all entries sequentially |
-| | |
-+---------------+-------------------------------------------------------+
-| **Dynamictp** | A runner that searches for the max throughput with |
-| | binary search |
-| | |
-+---------------+-------------------------------------------------------+
-| **Search** | A runner that runs a specific time before it returns |
-| | |
-+---------------+-------------------------------------------------------+
+.. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
+
++----------------+-------------------------------------------------------+
+| **Runner** | **Description** |
+| | |
++----------------+-------------------------------------------------------+
+| *Arithmetic* | Steps every run arithmetically according to specified |
+| | input value |
+| | |
++----------------+-------------------------------------------------------+
+| *Duration* | Runs for a specified period of time |
+| | |
++----------------+-------------------------------------------------------+
+| *Iteration* | Runs for a specified number of iterations |
+| | |
++----------------+-------------------------------------------------------+
+| *IterationIPC* | Runs a configurable number of times before it |
+| | returns. Each iteration has a configurable timeout. |
+| | |
++----------------+-------------------------------------------------------+
+| *Sequence* | Selects input value to a scenario from an input file |
+| | and runs all entries sequentially |
+| | |
++----------------+-------------------------------------------------------+
+| *Dynamictp* | A runner that searches for the max throughput with |
+| | binary search |
+| | |
++----------------+-------------------------------------------------------+
+| *Search* | A runner that runs a specific time before it returns |
+| | |
++----------------+-------------------------------------------------------+
List of Scenarios
@@ -280,19 +272,15 @@ List of Scenarios
New Test cases
--------------
-.. note:: Yardstick Fraser 6.1.0 added two new test cases, "TC092" and "TC093".
+.. note:: Yardstick Gambia 7.1.0 adds no new test cases.
* Generic NFVI test cases
- * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
+ * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
* HA Test cases
- * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration
- * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance
- * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api
- * OPNFV_YARDSTICK_TC092 - SDN Controller resilience in HA configuration
- * OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
+ * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
Version Change
@@ -301,22 +289,23 @@ Version Change
Module Version Changes
----------------------
-This is the sixth tracked release of Yardstick. It is based on following
+This is the seventh tracked release of Yardstick. It is based on following
upstream versions:
-- OpenStack Pike
+- OpenStack Queens
- OpenDayLight Oxygen
Document Version Changes
------------------------
-This is the sixth tracked version of the Yardstick framework in OPNFV.
+This is the seventh tracked version of the Yardstick framework in OPNFV.
It includes the following documentation updates:
-- Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
- add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
- add "Yardstick user interface" chapter; Update Yardstick installation chapter;
+- Yardstick User Guide:
+
+ - Remove vTC chapter;
+
- Yardstick Developer Guide
- Yardstick Release Notes for Yardstick: this document
@@ -324,49 +313,49 @@ It includes the following documentation updates:
Feature additions
-----------------
-- Plugin-based test cases support Heat context
-- SR-IOV support for the Heat context
-- Support using existing network in Heat context
-- Support running test cases with existing VNFs/without destroying VNF in Heat context
-- Add vFW scale-up template
-- Improvements of unit tests and gating
-- GUI improvement about passing parameters
+- Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
+- Spirent support.
+- vEPC testcases.
+- Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
+- PROX enhancements and the addition of Standalone test case using SRIOV and
+ OVS-DPDK.
+- Ixia enhancements for vBNG and PPPoE traffic.
+- Improvements of unit tests and gating.
+- Add DPDK pktgen traffic generator.
+- Kubernetes context enhancements.
+- Kubernetes sample test cases of fio and lmbench added.
Scenario Matrix
===============
-For Fraser 6.0.0, Yardstick was tested on the following scenarios:
-
-+-------------------------+------+---------+----------+------+------+-------+
-| Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
-+=========================+======+=========+==========+======+======+=======+
-| os-nosdn-nofeature-noha | X | X | | | X | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-nofeature-ha | X | X | X | X | X | X |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-bar-noha | X | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-bar-ha | X | | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-odl-bgpvpn-ha | X | | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-calipso-noha | X | | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-kvm-ha | | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-odl_l3-nofeature-ha | | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-odl-sfc-ha | | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-odl-nofeature-ha | | | | X | | X |
-+-------------------------+------+---------+----------+------+------+-------+
-| os-nosdn-ovs-ha | | | | X | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| k8-nosdn-nofeature-ha | | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
-| k8-nosdn-stor4nfv-noha | | X | | | | |
-+-------------------------+------+---------+----------+------+------+-------+
+For Gambia 7.1.0, Yardstick was tested on the following scenarios:
+
++-------------------------+------+---------+----------+------+
+| Scenario | Apex | Compass | Fuel-arm | Fuel |
++=========================+======+=========+==========+======+
+| os-nosdn-nofeature-noha | X | | | |
++-------------------------+------+---------+----------+------+
+| os-nosdn-nofeature-ha | X | | | |
++-------------------------+------+---------+----------+------+
+| os-odl-bgpvpn-noha | X | | | |
++-------------------------+------+---------+----------+------+
+| os-nosdn-calipso-noha | X | | | |
++-------------------------+------+---------+----------+------+
+| os-nosdn-kvm-ha | | X | | |
++-------------------------+------+---------+----------+------+
+| os-odl-nofeature-ha | | | X | X |
++-------------------------+------+---------+----------+------+
+| os-odl-sfc-noha | X | | | |
++-------------------------+------+---------+----------+------+
+| os-nosdn-ovs-ha | | | | X |
++-------------------------+------+---------+----------+------+
+| k8-nosdn-nofeature-ha | | X | | |
++-------------------------+------+---------+----------+------+
+| k8-nosdn-stor4nfv-noha | | X | | |
++-------------------------+------+---------+----------+------+
+| k8-nosdn-stor4nfv-ha | | X | | |
++-------------------------+------+---------+----------+------+
Test results
@@ -376,19 +365,6 @@ Test results are available in:
- jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
-The reporting pages can be found at:
-
-+---------------+----------------------------------------------------------------------------------+
-| apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html |
-+---------------+----------------------------------------------------------------------------------+
-| compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html |
-+---------------+----------------------------------------------------------------------------------+
-| fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html |
-+---------------+----------------------------------------------------------------------------------+
-| fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html |
-+---------------+----------------------------------------------------------------------------------+
-| joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html |
-+---------------+----------------------------------------------------------------------------------+
Known Issues/Faults
-------------------
@@ -397,219 +373,47 @@ Known Issues/Faults
Corrected Faults
----------------
-Fraser 6.2.1:
+Gambia 7.1.0:
+--------------------+--------------------------------------------------------------------------+
| **JIRA REFERENCE** | **DESCRIPTION** |
+====================+==========================================================================+
-| YARDSTICK-1147 | Fix ansible scripts for running in container |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1157 | Bug Fix: correct the file path to build docker file |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1276 | Bugfix: docker build failed |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1280 | Bugfix: uwsgi config file yardstick.ini output error |
-+--------------------+--------------------------------------------------------------------------+
-
-Fraser 6.2.0:
-
-+--------------------+--------------------------------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-+====================+==========================================================================+
-| YARDSTICK-1246 | Update pmd/lcore mask for OVS-DPDK context |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-837 | Move tests: unit/network_services/{lib/,collector/,*.py} |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1144 | Correctly set PYTHONPATH in Dockerfile |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1205 | Set "cmd2" library to version 0.8.6 |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1204 | Bump oslo.messaging version to 5.36.0 |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1210 | Remove __init__ method overriding in HeatContextTestCase |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1189 | Error when adding SR-IOV interfaces in SR-IOV context |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1214 | Remove AnsibleCommon class method mock |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1159 | Add --hwlb options as a command line argument for SampleVNF |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1203 | Add scale out TCs with availability zone support |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1167 | Do not start collectd twice when SampleVNF is running on Baremetal |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1188 | Add "host_name_separator" variable to Context class |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1112 | MQ startup process refactor |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1229 | Cleanup BaseMonitor unit tests |
-+--------------------+--------------------------------------------------------------------------+
-| - | Configure ACL via static file |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1191 | Use TRex release v2.41 to support both x86 and aarch64 |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1106 | Add IxNetwork API Python Binding package |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1224 | Cleanup TestYardstickNSCli class |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1225 | Remove print out of logger exception in TestUtils |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1194 | Add "duration" parameter to test case definition |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1209 | Remove instantiated contexts in "test_task" |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1192 | Standalone XML machine type is not longer valid |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1197 | Refactor RFC2455 TRex traffic profile injection |
-+--------------------+--------------------------------------------------------------------------+
-| - | Fix "os.path" mock problems during tests |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1218 | Refactor "utils.parse_ini_file" testing |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1179 | Start nginx and uwsgi servicies only in not container mode |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1177 | Install dependencies: bare-metal, standalone |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1126 | Migrate install.sh script to ansible |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1146 | Fix nsb_setup.sh script |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1247 | NSB setup inventory name changed |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1116 | Changed IxNextgen library load in IXIA RFC2544 traffic generator call. |
-+--------------------+--------------------------------------------------------------------------+
-| - | Corrected scale-up command line arguments |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-878 | OpenStack client replacement |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1222 | Bugfix: HA kill process recovery has a conflict |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1139 | Add "os_cloud_config" as a new context flag parameter |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1255 | Extended Context class with get_physical_nodes functionality |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1244 | NSB NFVi BNG test fails to run - stops after one step |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1219 | Decrease Sampling interval |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1101 | NSB NFVi PROX BNG losing many packets |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1217 | Fix NSB NfVi support for 25 and 40Gbps |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1185 | NSB Topology fix for Prox 4 port test case |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-966 | Convert SLA asserts to raises |
-+--------------------+--------------------------------------------------------------------------+
-
-Fraser 6.1.0:
-
-+--------------------+--------------------------------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-+====================+==========================================================================+
-| YARDSTICK-995 | Test case spec for SDN Virtual Switch resilience |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1097 | Add pod.yaml file for APEX installer |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1122 | Remove unused code in SampleVNF |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1125 | Update samples/test_suite.yaml |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1132 | Document for Euphrates test case results |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1138 | Support Restart Operation |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1142 | start_service script fails to start openvswitch service in centos distro |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1165 | Bugfix: openrc api dump should be safe_dump |
-+--------------------+--------------------------------------------------------------------------+
-
-Fraser 6.0.0:
-
-+--------------------+--------------------------------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-+====================+==========================================================================+
-| YARDSTICK-831 | tc053 kill haproxy wrong |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-842 | load image fails when there's cirros image exist |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-875 | Check if multiverse repository is available in Ubuntu |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-893 | Fix proxy env handling and ansible multinode support |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-899 | Variable local_iface_name is read before it is set |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-994 | NSB set-up build script for baremetal broken |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1012 | constants: fix pylint warnings for OSError |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1014 | Remove unused args in |
-| | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1016 | Allow vm to access outside world through default gateway |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana |
-+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml |
+| YARDSTICK-1241 | Update NSB PROX devguide. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1047 | Error in string format in HeatTemplateError message |
+| YARDSTICK-1458 | NSB NFVi PROX Should report realtime port activity not historical data. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1056 | yardstick report command print error when run test case |
+| YARDSTICK-1471 | Add Testcase Prox Standalone SRIOV. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1059 | Reduce the log level if TRex client is no connected |
+| YARDSTICK-1475 | Adding Testcase for Prox Stanalone OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1073 | Error when retrieving "options" section in "scenario" |
+| YARDSTICK-1500 | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error |
+| YARDSTICK-1517 | Missing opnfv "os-ovn-nofeature-ha" scenario test suite. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml |
+| YARDSTICK-l526 | Run testcase 074 result overridden by job status. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1102 | Don't hide exception traceback from Task.start() |
+| YARDSTICK-1547 | Adding scale up test case for l3fwd OvS-DPDK. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1107 | bad exception traceback print due to atexit_handler |
+| YARDSTICK-1560 | Fix pip environment. |
+--------------------+--------------------------------------------------------------------------+
-| YARDSTICK-1120 | HA test case tc050 should start monitor before attack |
+| YARDSTICK-1561 | L3FWD Gradana Dashboards Out-of-date and incorrect. |
+--------------------+--------------------------------------------------------------------------+
-Fraser 6.0.0 known restrictions/issues
+Gambia 7.1.0 known restrictions/issues
======================================
-+-----------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
-+===========+===========+==============================================+
-| | | |
-+-----------+-----------+----------------------------------------------+
Useful links
============
- wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
- - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser
+ - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
- - Yardstick repo: https://git.opnfv.org/cgit/yardstick
+ - Yardstick repo: https://git.opnfv.org/yardstick
- Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
- Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
- - Yardstick IRC chanel: #opnfv-yardstick
+ - Yardstick IRC channel: #opnfv-yardstick
diff --git a/docs/release/results/euphrates_fraser_comparison.rst b/docs/release/results/euphrates_fraser_comparison.rst
index 53dfb994f..1dd328bb7 100644
--- a/docs/release/results/euphrates_fraser_comparison.rst
+++ b/docs/release/results/euphrates_fraser_comparison.rst
@@ -2,7 +2,15 @@
.. License.
.. http://creativecommons.org/licenses/by/4.0
-=======================================================
+..
+ Convention for heading levels in Yardstick:
+ ======= Heading 0 (reserved for the title in a document)
+ ------- Heading 1
+ ^^^^^^^ Heading 2
+ +++++++ Heading 3
+ ''''''' Heading 4
+ Avoid deeper levels because they do not render well.
+
Test results analysis for Euphrates and Fraser releases
=======================================================
diff --git a/docs/release/results/images/tc014_pod_fraseer.png b/docs/release/results/images/tc014_pod_fraser.png
index 697201d76..697201d76 100644
--- a/docs/release/results/images/tc014_pod_fraseer.png
+++ b/docs/release/results/images/tc014_pod_fraser.png
Binary files differ
diff --git a/docs/release/results/index.rst b/docs/release/results/index.rst
index 63445fd1a..30cf62284 100644
--- a/docs/release/results/index.rst
+++ b/docs/release/results/index.rst
@@ -15,3 +15,8 @@ Yardstick test results
.. include:: ./overview.rst
.. include:: ./results.rst
.. include:: ./euphrates_fraser_comparison.rst
+
+.. include:: ./yardstick-opnfv-ha.rst
+.. include:: ./yardstick-opnfv-kvm.rst
+.. include:: ./yardstick-opnfv-parser.rst
+
diff --git a/docs/release/results/overview.rst b/docs/release/results/overview.rst
index 9fd74797c..b5e6a43a6 100644
--- a/docs/release/results/overview.rst
+++ b/docs/release/results/overview.rst
@@ -3,6 +3,15 @@
.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, Ericsson AB and others.
+..
+ Convention for heading levels in Yardstick:
+ ======= Heading 0 (reserved for the title in a document)
+ ------- Heading 1
+ ^^^^^^^ Heading 2
+ +++++++ Heading 3
+ ''''''' Heading 4
+ Avoid deeper levels because they do not render well.
+
Yardstick test tesult document overview
=======================================
diff --git a/docs/release/results/results.rst b/docs/release/results/results.rst
index 0ed92f867..f0c20360b 100644
--- a/docs/release/results/results.rst
+++ b/docs/release/results/results.rst
@@ -2,8 +2,17 @@
.. License.
.. http://creativecommons.org/licenses/by/4.0
+..
+ Convention for heading levels in Yardstick:
+ ======= Heading 0 (reserved for the title in a document)
+ ------- Heading 1
+ ^^^^^^^ Heading 2
+ +++++++ Heading 3
+ ''''''' Heading 4
+ Avoid deeper levels because they do not render well.
+
Results listed by test cases
-==========================
+----------------------------
.. _TOM: https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results
@@ -14,7 +23,7 @@ the determined state of the specific test case as executed in the Fraser release
process. All test date are analyzed using TOM_ tool.
Scenario Results
-================
+----------------
.. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
.. _Jenkins: https://build.opnfv.org/ci/view/yardstick/
@@ -42,7 +51,7 @@ Test results of executed tests are avilable in Dashboard_ and logs in Jenkins_.
Test results for Fraser release are collected from April 10, 2018 to May 13, 2018.
Feature Test Results
-====================
+--------------------
The following features were verified by Yardstick test cases:
@@ -54,8 +63,6 @@ The following features were verified by Yardstick test cases:
* Parser
- * Virtual Traffic Classifier (see :doc:`yardstick-opnfv-vtc`)
-
* StorPerf
.. note:: The test cases for IPv6 and Parser Projects are included in the
diff --git a/docs/release/results/yardstick-opnfv-vtc.rst b/docs/release/results/yardstick-opnfv-vtc.rst
deleted file mode 100644
index 059b5491f..000000000
--- a/docs/release/results/yardstick-opnfv-vtc.rst
+++ /dev/null
@@ -1,248 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International
-.. License.
-.. http://creativecommons.org/licenses/by/4.0
-
-.. _Dashboard006: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc006
-.. _Dashboard007: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc007
-.. _Dashboard020: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc020
-.. _Dashboard021: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-tc021
-.. _DashboardVTC: http://testresults.opnfv.org/grafana/dashboard/db/vtc-dashboard
-====================================
-Test Results for yardstick-opnfv-vtc
-====================================
-
-.. toctree::
- :maxdepth: 2
-
-
-Details
-=======
-
-.. after this doc is filled, remove all comments and include the scenario in
-.. results.rst by removing the comment on the file name.
-
-
-Overview of test results
-------------------------
-
-.. general on metrics collected, number of iterations
-
-The virtual Traffic Classifier (vtc) Scenario supported by Yardstick is used by 4 Test Cases:
-
-- TC006
-- TC007
-- TC020
-- TC021
-
-
-* TC006
-
-TC006 is the Virtual Traffic Classifier Data Plane Throughput Benchmarking Test.
-It collects measures about the end-to-end throughput supported by the
-virtual Traffic Classifier (vTC).
-Results of the test are shown in the Dashboard006_
-The throughput is expressed as percentage of the available bandwidth on the NIC.
-
-
-* TC007
-
-TC007 is the Virtual Traffic Classifier Data Plane Throughput Benchmarking in presence of
-noisy neighbors Test.
-It collects measures about the end-to-end throughput supported by the
-virtual Traffic Classifier when a user-defined number of noisy neighbors is deployed.
-Results of the test are shown in the Dashboard007_
-The throughput is expressed as percentage of the available bandwidth on the NIC.
-
-
-* TC020
-
-TC020 is the Virtual Traffic Classifier Instantiation Test.
-It verifies that a newly instantiated vTC is alive and functional and its instantiation
-is correctly supported by the underlying infrastructure.
-Results of the test are shown in the Dashboard020_
-
-
-* TC021
-
-TC021 is the Virtual Traffic Classifier Instantiation in presence of noisy neighbors Test.
-It verifies that a newly instantiated vTC is alive and functional and its instantiation
-is correctly supported by the underlying infrastructure when noisy neighbors are present.
-Results of the test are shown in the Dashboard021_
-
-* Generic
-
-In the Generic scenario the Virtual Traffic Classifier is running on a standard Openstack
-setup and traffic is being replayed from a neighbor VM. The traffic sent contains
-various protocols and applications, and the VTC identifies them and exports the data.
-Results of the test are shown in the DashboardVTC.
-
-Detailed test results
----------------------
-
-* TC006
-
-The results for TC006 have been obtained using the following test case
-configuration:
-
-- Context: Dummy
-- Scenario: vtc_throughput
-- Network Techology: SR-IOV
-- vTC Flavor: m1.large
-
-
-* TC007
-
-The results for TC007 have been obtained using the following test case
-configuration:
-
-- Context: Dummy
-- Scenario: vtc_throughput_noisy
-- Network Techology: SR-IOV
-- vTC Flavor: m1.large
-- Number of noisy neighbors: 2
-- Number of cores per neighbor: 2
-- Amount of RAM per neighbor: 1G
-
-
-* TC020
-
-The results for TC020 have been obtained using the following test case
-configuration:
-
-The results listed in previous section have been obtained using the following
-test case configuration:
-
-- Context: Dummy
-- Scenario: vtc_instantiation_validation
-- Network Techology: SR-IOV
-- vTC Flavor: m1.large
-
-
-* TC021
-
-The results listed in previous section have been obtained using the following
-test case configuration:
-
-- Context: Dummy
-- Scenario: vtc_instantiation_validation
-- Network Techology: SR-IOV
-- vTC Flavor: m1.large
-- Number of noisy neighbors: 2
-- Number of cores per neighbor: 2
-- Amount of RAM per neighbor: 1G
-
-
-For all the test cases, the user can specify different values for the parameters.
-
-* Generic
-
-The results listed in the previous section have been obtained, using a
-standard Openstack setup.
-The user can replay his/her own traffic and see the corresponding results.
-
-Rationale for decisions
------------------------
-
-* TC006
-
-The result of the test is a number between 0 and 100 which represents the percentage of bandwidth
-available on the NIC that corresponds to the supported throughput by the vTC.
-
-
-* TC007
-
-The result of the test is a number between 0 and 100 which represents the percentage of bandwidth
-available on the NIC that corresponds to the supported throughput by the vTC.
-
-* TC020
-
-The execution of the test is done as described in the following:
-
-- The vTC is deployed on the OpenStack testbed;
-- Some traffic is sent to the vTC;
-- The vTC changes the header of the packets and sends them back to the packet generator;
-- The packet generator checks that all the packets are received correctly and have been changed
-correctly by the vTC.
-
-The test is declared as PASSED if all the packets are correcly received by the packet generator
-and they have been modified by the virtual Traffic Classifier as required.
-
-
-* TC021
-
-The execution of the test is done as described in the following:
-
-- The vTC is deployed on the OpenStack testbed;
-- The noisy neighbors are deployed as requested by the user;
-- Some traffic is sent to the vTC;
-- The vTC change the header of the packets and sends them back to the packet generator;
-- The packet generator checks that all the packets are received correctly and have been changed
-correctly by the vTC
-
-The test is declared as PASSED if all the packets are correcly received by the packet generator
-and they have been modified by the virtual Traffic Classifier as required.
-
-* Generic
-
-The execution of the test consists of the following actions:
-
-- The vTC is deployed on the OpenStack testbed;
-- The traffic generator VM is deployed on the Openstack Testbed;
-- Traffic data are relevant to the network setup;
-- Traffic is sent to the vTC;
-
-
-
-Conclusions and recommendations
--------------------------------
-
-* TC006
-
-The obtained results show that the virtual Traffic Classifier can support up to 4 Gbps
-(40% of the available bandwidth) correspond to the expected behaviour of the virtual
-Traffic Classifier.
-Using the configuration with SR-IOV and large flavor, the expected throughput should
-generally be in the range between 3 and 4 Gbps.
-
-
-* TC007
-
-These results correspond to the configuration in which the virtual Traffic Classifier uses SR-IOV
-Virtual Functions and the flavor is set to large for the virtual machine.
-The throughput is in the range between 2.5 Gbps and 3.7 Gbps.
-This shows that the effect of 2 noisy neighbors reduces the throughput of
-the service between 10 and 20%.
-Increasing number of neihbours would have a higher impact on the performance.
-
-
-* TC020
-
-The obtained results correspond to the expected behaviour of the virtual Traffic Classifier.
-Using the configuration with SR-IOV and large flavor, the expected result is that the vTC is
-correctly instantiated, it is able to receive and send packets using SR-IOV technology
-and to forward packets back to the packet generator changing the TCP/IP header as required.
-
-
-* TC021
-
-The obtained results correspond to the expected behaviour of the virtual Traffic Classifier.
-Using the configuration with SR-IOV and large flavor, the expected result is that the vTC is
-correctly instantiated, it is able to receive and send packets using SR-IOV technology
-and to forward packets back to the packet generator changing the TCP/IP header as required,
-also in presence of noisy neighbors.
-
-* Generic
-
-The obtained results correspond to the expected behaviour of the virtual Traffic Classifier.
-Using the aforementioned configuration the expected application protocols are identified
-and their traffic statistics are demonstrated in the DashboardVTC, a group of popular
-applications is selected to demonstrate the sound operation of the vTC.
-The demonstrated application protocols are:
-- HTTP
-- Skype
-- Bittorrent
-- Youtube
-- Dropbox
-- Twitter
-- Viber
-- iCloud