======= License ======= OPNFV Fraser release note for Yardstick Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see . The *Yardstick framework*, the *Yardstick test cases* are open-source software, licensed under the terms of the Apache License, Version 2.0. ======================================= OPNFV Fraser Release Note for Yardstick ======================================= .. toctree:: :maxdepth: 2 .. _Yardstick: https://wiki.opnfv.org/display/yardstick .. _Dashboard: http://testresults.opnfv.org/grafana/ .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf Abstract ======== This document describes the release note of Yardstick project. Version History =============== +-------------------+-----------+---------------------------------+ | *Date* | *Version* | *Comment* | | | | | +-------------------+-----------+---------------------------------+ | Jul 2, 2018 | 6.2.1 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ | Jun 29, 2018 | 6.2.0 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ | May 25, 2018 | 6.1.0 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ | April 27, 2018 | 6.0.0 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ Important Notes =============== The software delivered in the OPNFV Yardstick_ Project, comprising the *Yardstick framework*, and the *Yardstick test cases* is a realization of the methodology in ETSI-ISG NFV-TST001_. The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. OPNFV Fraser Release ==================== This Fraser release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx * User Guide * Developer Guide * Release notes (this document) * Results * Automated Yardstick test suite (daily, weekly) * Jenkins Jobs for OPNFV community labs * Automated Yardstick test results visualization * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as backend * Yardstick framework source code * Yardstick test cases yaml files * Yardstick plug-in configuration yaml files, plug-in install/remove scripts For Fraser release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: * Compute * Network * Storage * OPNFV platform network service benchmarking (NSB) * NSB * Test cases for the following OPNFV Projects: * Container4NFV * High Availability * IPv6 * KVM * Parser * StorPerf * VSperf The *Yardstick framework* is developed in the OPNFV community, by the Yardstick_ team. .. note:: The test case description template used for the Yardstick test cases is based on the document ETSI-ISG NFV-TST001_; the results report template used for the Yardstick results is based on the IEEE Std 829-2008. Release Data ============ +--------------------------------+-----------------------+ | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ | **Repo/tag** | yardstick/opnfv-6.2.0 | | | | +--------------------------------+-----------------------+ | **Yardstick Docker image tag** | opnfv-6.2.0 | | | | +--------------------------------+-----------------------+ | **Release designation** | Fraser | | | | +--------------------------------+-----------------------+ | **Release date** | Jun 29, 2018 | | | | +--------------------------------+-----------------------+ | **Purpose of the delivery** | OPNFV Fraser 6.2.0 | | | | +--------------------------------+-----------------------+ Deliverables ============ Documents --------- - User Guide: :ref:`` - Developer Guide: :ref:`` Software Deliverables --------------------- - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.2.0) List of Contexts ^^^^^^^^^^^^^^^^ +--------------+-------------------------------------------+ | **Context** | **Description** | | | | +--------------+-------------------------------------------+ | *Heat* | Models orchestration using OpenStack Heat | | | | +--------------+-------------------------------------------+ | *Node* | Models Baremetal, Controller, Compute | | | | +--------------+-------------------------------------------+ | *Standalone* | Models VM running on Non-Managed NFVi | | | | +--------------+-------------------------------------------+ | *Kubernetes* | Models VM running on Non-Managed NFVi | | | | +--------------+-------------------------------------------+ List of Runners ^^^^^^^^^^^^^^^ .. note:: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search". +---------------+-------------------------------------------------------+ | **Runner** | **Description** | | | | +---------------+-------------------------------------------------------+ | *Arithmetic* | Steps every run arithmetically according to specified | | | input value | | | | +---------------+-------------------------------------------------------+ | *Duration* | Runs for a specified period of time | | | | +---------------+-------------------------------------------------------+ | *Iteration* | Runs for a specified number of iterations | | | | +---------------+-------------------------------------------------------+ | *Sequence* | Selects input value to a scenario from an input file | | | and runs all entries sequentially | | | | +---------------+-------------------------------------------------------+ | **Dynamictp** | A runner that searches for the max throughput with | | | binary search | | | | +---------------+-------------------------------------------------------+ | **Search** | A runner that runs a specific time before it returns | | | | +---------------+-------------------------------------------------------+ List of Scenarios ^^^^^^^^^^^^^^^^^ +----------------+-----------------------------------------------------+ | **Category** | **Delivered** | | | | +----------------+-----------------------------------------------------+ | *Availability* | Attacker: | | | | | | * baremetal, process | | | | | | HA tools: | | | | | | * check host, openstack, process, service | | | * kill process | | | * start/stop service | | | | | | Monitor: | | | | | | * command, process | | | | +----------------+-----------------------------------------------------+ | *Compute* | * cpuload | | | * cyclictest | | | * lmbench | | | * lmbench_cache | | | * perf | | | * unixbench | | | * ramspeed | | | * cachestat | | | * memeoryload | | | * computecapacity | | | * SpecCPU2006 | | | | +----------------+-----------------------------------------------------+ | *Networking* | * iperf3 | | | * netperf | | | * netperf_node | | | * ping | | | * ping6 | | | * pktgen | | | * sfc | | | * sfc with tacker | | | * networkcapacity | | | * netutilization | | | * nstat | | | * pktgenDPDK | | | | +----------------+-----------------------------------------------------+ | *Parser* | Tosca2Heat | | | | +----------------+-----------------------------------------------------+ | *Storage* | * fio | | | * bonnie++ | | | * storagecapacity | | | | +----------------+-----------------------------------------------------+ | *StorPerf* | storperf | | | | +----------------+-----------------------------------------------------+ | *NSB* | vFW thoughput test case | | | | +----------------+-----------------------------------------------------+ New Test cases -------------- .. note:: Yardstick Fraser 6.1.0 added two new test cases, "TC092" and "TC093". * Generic NFVI test cases * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM * HA Test cases * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api * OPNFV_YARDSTICK_TC092 - SDN Controller resilience in HA configuration * OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration Version Change ============== Module Version Changes ---------------------- This is the sixth tracked release of Yardstick. It is based on following upstream versions: - OpenStack Pike - OpenDayLight Oxygen Document Version Changes ------------------------ This is the sixth tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: - Yardstick User Guide: add "network service benchmarking(NSB)" chapter; add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter; add "Yardstick user interface" chapter; Update Yardstick installation chapter; - Yardstick Developer Guide - Yardstick Release Notes for Yardstick: this document Feature additions ----------------- - Plugin-based test cases support Heat context - SR-IOV support for the Heat context - Support using existing network in Heat context - Support running test cases with existing VNFs/without destroying VNF in Heat context - Add vFW scale-up template - Improvements of unit tests and gating - GUI improvement about passing parameters Scenario Matrix =============== For Fraser 6.0.0, Yardstick was tested on the following scenarios: +-------------------------+------+---------+----------+------+------+-------+ | Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy | +=========================+======+=========+==========+======+======+=======+ | os-nosdn-nofeature-noha | X | X | | | X | | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-nofeature-ha | X | X | X | X | X | X | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-bar-noha | X | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-bar-ha | X | | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-odl-bgpvpn-ha | X | | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-calipso-noha | X | | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-kvm-ha | | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-odl_l3-nofeature-ha | | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-odl-sfc-ha | | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ | os-odl-nofeature-ha | | | | X | | X | +-------------------------+------+---------+----------+------+------+-------+ | os-nosdn-ovs-ha | | | | X | | | +-------------------------+------+---------+----------+------+------+-------+ | k8-nosdn-nofeature-ha | | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ | k8-nosdn-stor4nfv-noha | | X | | | | | +-------------------------+------+---------+----------+------+------+-------+ Test results ============ Test results are available in: - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/ The reporting pages can be found at: +---------------+----------------------------------------------------------------------------------+ | apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html | +---------------+----------------------------------------------------------------------------------+ | compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html | +---------------+----------------------------------------------------------------------------------+ | fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html | +---------------+----------------------------------------------------------------------------------+ | fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html | +---------------+----------------------------------------------------------------------------------+ | joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html | +---------------+----------------------------------------------------------------------------------+ Known Issues/Faults ------------------- Corrected Faults ---------------- Fraser 6.2.1: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ | YARDSTICK-1147 | Fix ansible scripts for running in container | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1157 | Bug Fix: correct the file path to build docker file | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1276 | Bugfix: docker build failed | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1280 | Bugfix: uwsgi config file yardstick.ini output error | +--------------------+--------------------------------------------------------------------------+ Fraser 6.2.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ | YARDSTICK-1246 | Update pmd/lcore mask for OVS-DPDK context | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-837 | Move tests: unit/network_services/{lib/,collector/,*.py} | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1144 | Correctly set PYTHONPATH in Dockerfile | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1205 | Set "cmd2" library to version 0.8.6 | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1204 | Bump oslo.messaging version to 5.36.0 | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1210 | Remove __init__ method overriding in HeatContextTestCase | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1189 | Error when adding SR-IOV interfaces in SR-IOV context | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1214 | Remove AnsibleCommon class method mock | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1159 | Add --hwlb options as a command line argument for SampleVNF | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1203 | Add scale out TCs with availability zone support | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1167 | Do not start collectd twice when SampleVNF is running on Baremetal | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1188 | Add "host_name_separator" variable to Context class | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1112 | MQ startup process refactor | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1229 | Cleanup BaseMonitor unit tests | +--------------------+--------------------------------------------------------------------------+ | - | Configure ACL via static file | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1191 | Use TRex release v2.41 to support both x86 and aarch64 | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1106 | Add IxNetwork API Python Binding package | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1224 | Cleanup TestYardstickNSCli class | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1225 | Remove print out of logger exception in TestUtils | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1194 | Add "duration" parameter to test case definition | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1209 | Remove instantiated contexts in "test_task" | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1192 | Standalone XML machine type is not longer valid | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1197 | Refactor RFC2455 TRex traffic profile injection | +--------------------+--------------------------------------------------------------------------+ | - | Fix "os.path" mock problems during tests | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1218 | Refactor "utils.parse_ini_file" testing | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1179 | Start nginx and uwsgi servicies only in not container mode | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1177 | Install dependencies: bare-metal, standalone | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1126 | Migrate install.sh script to ansible | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1146 | Fix nsb_setup.sh script | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1247 | NSB setup inventory name changed | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1116 | Changed IxNextgen library load in IXIA RFC2544 traffic generator call. | +--------------------+--------------------------------------------------------------------------+ | - | Corrected scale-up command line arguments | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-878 | OpenStack client replacement | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1222 | Bugfix: HA kill process recovery has a conflict | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1139 | Add "os_cloud_config" as a new context flag parameter | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1255 | Extended Context class with get_physical_nodes functionality | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1244 | NSB NFVi BNG test fails to run - stops after one step | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1219 | Decrease Sampling interval | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1101 | NSB NFVi PROX BNG losing many packets | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1217 | Fix NSB NfVi support for 25 and 40Gbps | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1185 | NSB Topology fix for Prox 4 port test case | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-966 | Convert SLA asserts to raises | +--------------------+--------------------------------------------------------------------------+ Fraser 6.1.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ | YARDSTICK-995 | Test case spec for SDN Virtual Switch resilience | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1097 | Add pod.yaml file for APEX installer | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1122 | Remove unused code in SampleVNF | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1125 | Update samples/test_suite.yaml | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1132 | Document for Euphrates test case results | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1138 | Support Restart Operation | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1142 | start_service script fails to start openvswitch service in centos distro | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1165 | Bugfix: openrc api dump should be safe_dump | +--------------------+--------------------------------------------------------------------------+ Fraser 6.0.0: +--------------------+--------------------------------------------------------------------------+ | **JIRA REFERENCE** | **DESCRIPTION** | +====================+==========================================================================+ | YARDSTICK-831 | tc053 kill haproxy wrong | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-842 | load image fails when there's cirros image exist | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-875 | Check if multiverse repository is available in Ubuntu | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-893 | Fix proxy env handling and ansible multinode support | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-899 | Variable local_iface_name is read before it is set | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-994 | NSB set-up build script for baremetal broken | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1012 | constants: fix pylint warnings for OSError | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1014 | Remove unused args in | | | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1016 | Allow vm to access outside world through default gateway | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1047 | Error in string format in HeatTemplateError message | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1056 | yardstick report command print error when run test case | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1059 | Reduce the log level if TRex client is no connected | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1073 | Error when retrieving "options" section in "scenario" | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1102 | Don't hide exception traceback from Task.start() | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1107 | bad exception traceback print due to atexit_handler | +--------------------+--------------------------------------------------------------------------+ | YARDSTICK-1120 | HA test case tc050 should start monitor before attack | +--------------------+--------------------------------------------------------------------------+ Fraser 6.0.0 known restrictions/issues ====================================== +-----------+-----------+----------------------------------------------+ | Installer | Scenario | Issue | +===========+===========+==============================================+ | | | | +-----------+-----------+----------------------------------------------+ Useful links ============ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser - Yardstick repo: https://git.opnfv.org/yardstick - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/ - Yardstick IRC chanel: #opnfv-yardstick