diff options
Diffstat (limited to 'docs')
77 files changed, 4966 insertions, 3927 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index 4ebf0eceb..7ea2616e4 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -1,7 +1,8 @@ +======= License ======= -OPNFV Euphrates release note for Yardstick Docs +OPNFV Fraser release note for Yardstick Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>. @@ -9,8 +10,9 @@ If not, see <http://creativecommons.org/licenses/by/4.0/>. The *Yardstick framework*, the *Yardstick test cases* are open-source software, licensed under the terms of the Apache License, Version 2.0. -OPNFV Euphrates Release Note for Yardstick -========================================== +======================================= +OPNFV Fraser Release Note for Yardstick +======================================= .. toctree:: :maxdepth: 2 @@ -23,50 +25,46 @@ OPNFV Euphrates Release Note for Yardstick Abstract --------- +======== This document describes the release note of Yardstick project. Version History ---------------- +=============== +-------------------+-----------+---------------------------------+ | *Date* | *Version* | *Comment* | | | | | +-------------------+-----------+---------------------------------+ -| December 15, 2017 | 5.1.0 | Yardstick for Euphrates release | +| May 25, 2018 | 6.1.0 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ -| October 20, 2017 | 5.0.0 | Yardstick for Euphrates release | +| April 27, 2018 | 6.0.0 | Yardstick for Fraser release | | | | | +-------------------+-----------+---------------------------------+ Important Notes ---------------- +=============== The software delivered in the OPNFV Yardstick_ Project, comprising the -*Yardstick framework*, the *Yardstick test cases* and the experimental -framework *Apex Lake* is a realization of the methodology in ETSI-ISG -NFV-TST001_. +*Yardstick framework*, and the *Yardstick test cases* is a realization of +the methodology in ETSI-ISG NFV-TST001_. The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. -OPNFV Euphrates Release ------------------------ +OPNFV Fraser Release +==================== -This Euphrates release provides *Yardstick* as a framework for NFVI testing +This Fraser release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx * User Guide - * Developer Guide - * Release notes (this document) - * Results * Automated Yardstick test suite (daily, weekly) @@ -84,39 +82,29 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Yardstick plug-in configuration yaml files, plug-in install/remove scripts -For Euphrates release, the *Yardstick framework* is used for the following +For Fraser release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: * Compute - * Network - * Storage -* OPNFV platform network service benchmarking(NSB) +* OPNFV platform network service benchmarking (NSB) * NSB * Test cases for the following OPNFV Projects: * Container4NFV - * High Availability - * IPv6 - * KVM - * Parser - * StorPerf - * VSperf - * virtual Traffic Classifier - The *Yardstick framework* is developed in the OPNFV community, by the Yardstick_ team. @@ -126,49 +114,47 @@ Yardstick_ team. Release Data ------------- +============ +--------------------------------+-----------------------+ | **Project** | Yardstick | | | | +--------------------------------+-----------------------+ -| **Repo/tag** | yardstick/opnfv-5.1.0 | +| **Repo/tag** | yardstick/opnfv-6.1.0 | | | | +--------------------------------+-----------------------+ -| **Yardstick Docker image tag** | opnfv-5.1.0 | +| **Yardstick Docker image tag** | opnfv-6.1.0 | | | | +--------------------------------+-----------------------+ -| **Release designation** | Euphrates | +| **Release designation** | Fraser | | | | +--------------------------------+-----------------------+ -| **Release date** | December 15, 2017 | +| **Release date** | May 25, 2018 | | | | +--------------------------------+-----------------------+ -| **Purpose of the delivery** | OPNFV Euphrates 5.1.0 | +| **Purpose of the delivery** | OPNFV Fraser 6.1.0 | | | | +--------------------------------+-----------------------+ Deliverables ------------- +============ Documents -^^^^^^^^^ +--------- - - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html + - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html - - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html + - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html Software Deliverables -^^^^^^^^^^^^^^^^^^^^^ - +--------------------- - - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.1.0) + - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.1.0) - -New Contexts -############ +List of Contexts +^^^^^^^^^^^^^^^^ +--------------+-------------------------------------------+ | **Context** | **Description** | @@ -188,31 +174,40 @@ New Contexts +--------------+-------------------------------------------+ -New Runners -########### - -+--------------+-------------------------------------------------------+ -| **Runner** | **Description** | -| | | -+--------------+-------------------------------------------------------+ -| *Arithmetic* | Steps every run arithmetically according to specified | -| | input value | -| | | -+--------------+-------------------------------------------------------+ -| *Duration* | Runs for a specified period of time | -| | | -+--------------+-------------------------------------------------------+ -| *Iteration* | Runs for a specified number of iterations | -| | | -+--------------+-------------------------------------------------------+ -| *Sequence* | Selects input value to a scenario from an input file | -| | and runs all entries sequentially | -| | | -+--------------+-------------------------------------------------------+ - - -New Scenarios -############# +List of Runners +^^^^^^^^^^^^^^^ + +.. note:: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search". + ++---------------+-------------------------------------------------------+ +| **Runner** | **Description** | +| | | ++---------------+-------------------------------------------------------+ +| *Arithmetic* | Steps every run arithmetically according to specified | +| | input value | +| | | ++---------------+-------------------------------------------------------+ +| *Duration* | Runs for a specified period of time | +| | | ++---------------+-------------------------------------------------------+ +| *Iteration* | Runs for a specified number of iterations | +| | | ++---------------+-------------------------------------------------------+ +| *Sequence* | Selects input value to a scenario from an input file | +| | and runs all entries sequentially | +| | | ++---------------+-------------------------------------------------------+ +| **Dynamictp** | A runner that searches for the max throughput with | +| | binary search | +| | | ++---------------+-------------------------------------------------------+ +| **Search** | A runner that runs a specific time before it returns | +| | | ++---------------+-------------------------------------------------------+ + + +List of Scenarios +^^^^^^^^^^^^^^^^^ +----------------+-----------------------------------------------------+ | **Category** | **Delivered** | @@ -234,224 +229,142 @@ New Scenarios | | | +----------------+-----------------------------------------------------+ | *Compute* | * cpuload | -| | | | | * cyclictest | -| | | | | * lmbench | -| | | | | * lmbench_cache | -| | | | | * perf | -| | | | | * unixbench | -| | | | | * ramspeed | -| | | | | * cachestat | -| | | | | * memeoryload | -| | | | | * computecapacity | -| | | | | * SpecCPU2006 | | | | +----------------+-----------------------------------------------------+ | *Networking* | * iperf3 | -| | | | | * netperf | -| | | | | * netperf_node | -| | | | | * ping | -| | | | | * ping6 | -| | | | | * pktgen | -| | | | | * sfc | -| | | | | * sfc with tacker | -| | | -| | * vtc instantion validation | -| | | -| | * vtc instantion validation with noisy neighbors | -| | | -| | * vtc throughput | -| | | -| | * vtc throughput in the presence of noisy neighbors | -| | | | | * networkcapacity | -| | | | | * netutilization | -| | | | | * nstat | -| | | | | * pktgenDPDK | | | | +----------------+-----------------------------------------------------+ | *Parser* | Tosca2Heat | | | | +----------------+-----------------------------------------------------+ -| *Storage* | fio | -| | | -| | bonnie++ | -| | | -| | storagecapacity | +| *Storage* | * fio | +| | * bonnie++ | +| | * storagecapacity | | | | +----------------+-----------------------------------------------------+ | *StorPerf* | storperf | | | | +----------------+-----------------------------------------------------+ -| *NSB* | vPE thoughput test case | +| *NSB* | vFW thoughput test case | | | | +----------------+-----------------------------------------------------+ - New Test cases -^^^^^^^^^^^^^^ - -* Generic NFVI test cases +-------------- - * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006 +.. note:: Yardstick Fraser 6.1.0 added two new test cases, "TC092" and "TC093". - * OPNFV_YARDSTICK_TCO79 - Bonnie++ +* Generic NFVI test cases -* Kubernetes Test cases + * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM - * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER +* HA Test cases - * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM + * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration + * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance + * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api + * OPNFV_YARDSTICK_TC092 - SDN Controller resilience in HA configuration + * OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration Version Change --------------- +============== Module Version Changes -^^^^^^^^^^^^^^^^^^^^^^ +---------------------- -This is the fifth tracked release of Yardstick. It is based on following +This is the sixth tracked release of Yardstick. It is based on following upstream versions: -- OpenStack Ocata - -- OpenDayLight Nitrogen - -- ONOS Junco +- OpenStack Pike +- OpenDayLight Oxygen Document Version Changes -^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------ -This is the fifth tracked version of the Yardstick framework in OPNFV. +This is the sixth tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: - Yardstick User Guide: add "network service benchmarking(NSB)" chapter; add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter; add "Yardstick user interface" chapter; Update Yardstick installation chapter; - - Yardstick Developer Guide - - Yardstick Release Notes for Yardstick: this document Feature additions -^^^^^^^^^^^^^^^^^ - -- Yardstick RESTful API support - -- Network service benchmarking - -- Stress testing with Bottlenecks team - -- Yardstick framework improvement: - - - yardstick report CLI - - - Node context support OpenStack configuration via Ansible - - - Https support +----------------- - - Kubernetes context type - -- Yardstick container local GUI - -- Python 3 support +- Plugin-based test cases support Heat context +- SR-IOV support for the Heat context +- Support using existing network in Heat context +- Support running test cases with existing VNFs/without destroying VNF in Heat context +- Add vFW scale-up template +- Improvements of unit tests and gating +- GUI improvement about passing parameters Scenario Matrix ---------------- - -For Euphrates 5.0.0, Yardstick was tested on the following scenarios: - -+--------------------------+------+---------+------+------+ -| Scenario | Apex | Compass | Fuel | Joid | -+==========================+======+=========+======+======+ -| os-nosdn-nofeature-noha | | | X | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-nofeature-ha | X | X | X | X | -+--------------------------+------+---------+------+------+ -| os-odl_l2-nofeature-ha | | X | X | X | -+--------------------------+------+---------+------+------+ -| os-odl_l2-nofeature-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l3-nofeature-ha | X | X | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l3-nofeature-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-onos-sfc-ha | | | | | -+--------------------------+------+---------+------+------+ -| os-onos-nofeature-ha | | X | | X | -+--------------------------+------+---------+------+------+ -| os-onos-nofeature-noha | | | | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-sfc-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-sfc-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-bgpvpn-ha | X | | X | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-bgpvpn-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm-ha | X | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-ovs-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-ovs-noha | | | X | | -+--------------------------+------+---------+------+------+ -| os-ocl-nofeature-ha | | X | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-lxd-ha | | | | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-lxd-noha | | | | X | -+--------------------------+------+---------+------+------+ -| os-nosdn-fdio-ha | X | | | | -+--------------------------+------+---------+------+------+ -| os-odl_l2-fdio-noha | X | | | | -+--------------------------+------+---------+------+------+ -| os-odl-gluon-noha | X | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-openo-ha | | X | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| -noha | | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk-ha | | | X | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-ha | | | | | -+--------------------------+------+---------+------+------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-noha | | | | | -+--------------------------+------+---------+------+------+ -| opnfv_os-ovn-nofeature- | X | | | | -| noha_daily | | | | | -+--------------------------+------+---------+------+------+ +=============== + +For Fraser 6.0.0, Yardstick was tested on the following scenarios: + ++-------------------------+------+---------+----------+------+------+-------+ +| Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy | ++=========================+======+=========+==========+======+======+=======+ +| os-nosdn-nofeature-noha | X | X | | | X | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-nofeature-ha | X | X | X | X | X | X | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-bar-noha | X | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-bar-ha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-bgpvpn-ha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-calipso-noha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-kvm-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl_l3-nofeature-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-sfc-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-nofeature-ha | | | | X | | X | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-ovs-ha | | | | X | | | ++-------------------------+------+---------+----------+------+------+-------+ +| k8-nosdn-nofeature-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| k8-nosdn-stor4nfv-noha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ + Test results ------------- +============ Test results are available in: @@ -459,109 +372,129 @@ Test results are available in: The reporting pages can be found at: -+---------------+-------------------------------------------------------------------------------------+ -| apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html | -+---------------+-------------------------------------------------------------------------------------+ -| compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html | -+---------------+-------------------------------------------------------------------------------------+ -| fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html | -+---------------+-------------------------------------------------------------------------------------+ -| fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html | -+---------------+-------------------------------------------------------------------------------------+ -| joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html | -+---------------+-------------------------------------------------------------------------------------+ ++---------------+----------------------------------------------------------------------------------+ +| apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html | ++---------------+----------------------------------------------------------------------------------+ +| compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html | ++---------------+----------------------------------------------------------------------------------+ +| fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html | ++---------------+----------------------------------------------------------------------------------+ +| fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html | ++---------------+----------------------------------------------------------------------------------+ +| joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html | ++---------------+----------------------------------------------------------------------------------+ Known Issues/Faults -^^^^^^^^^^^^^^^^^^^ +------------------- Corrected Faults -^^^^^^^^^^^^^^^^ +---------------- + +Fraser 6.1.0: + ++--------------------+--------------------------------------------------------------------------+ +| **JIRA REFERENCE** | **DESCRIPTION** | ++====================+==========================================================================+ +| YARDSTICK-995 | Test case spec for SDN Virtual Switch resilience | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1097 | Add pod.yaml file for APEX installer | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1122 | Remove unused code in SampleVNF | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1125 | Update samples/test_suite.yaml | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1132 | Document for Euphrates test case results | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1138 | Support Restart Operation | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1142 | start_service script fails to start openvswitch service in centos distro | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1165 | Bugfix: openrc api dump should be safe_dump | ++--------------------+--------------------------------------------------------------------------+ + +Fraser 6.0.0: + ++--------------------+--------------------------------------------------------------------------+ +| **JIRA REFERENCE** | **DESCRIPTION** | ++====================+==========================================================================+ +| YARDSTICK-831 | tc053 kill haproxy wrong | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-842 | load image fails when there's cirros image exist | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-875 | Check if multiverse repository is available in Ubuntu | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-893 | Fix proxy env handling and ansible multinode support | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-899 | Variable local_iface_name is read before it is set | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-994 | NSB set-up build script for baremetal broken | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1012 | constants: fix pylint warnings for OSError | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1014 | Remove unused args in | +| | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1016 | Allow vm to access outside world through default gateway | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1047 | Error in string format in HeatTemplateError message | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1056 | yardstick report command print error when run test case | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1059 | Reduce the log level if TRex client is no connected | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1073 | Error when retrieving "options" section in "scenario" | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1102 | Don't hide exception traceback from Task.start() | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1107 | bad exception traceback print due to atexit_handler | ++--------------------+--------------------------------------------------------------------------+ +| YARDSTICK-1120 | HA test case tc050 should start monitor before attack | ++--------------------+--------------------------------------------------------------------------+ + +Fraser 6.0.0 known restrictions/issues +====================================== -Euphrates 5.1.0: - -+---------------------+-------------------------------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-841 | Fix various NSB license issues | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-73 | How To Work with Test Cases | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-500 | VNF testing documentation | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-826 | Allow overriding Heat IP addresses to match traffic generator profile | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-828 | Refactor doc/testing/user/userguide "Yardstick Installation" | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-830 | build_yardstick_image Ansible mount module doesn't work on Ubuntu 14.04 | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-833 | ansible_common transform password into lower case | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-847 | tc006, tc079, tc082 miss grafana dashboard in local deployment | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-849 | kill process do not accurately kill the process like "nova-api" | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-850 | tc023 miss description and tc050-58 wrong description | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-852 | tc078 cpu2006 fails in some situation | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-854 | yardstick docker lack of trex_client | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-867 | testcase tc078 have no data stored or dashboard to show results | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-871 | Remove img_modify_playbook assignation in build_yardstick_image.yml | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-829 | "nsb_setup.sh" doesn't parse the controller IP correctly | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-839 | NSB Prox BM test cases to be fixed for incorporating scale-up | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-840 | NSB Prox test documentation of vPE and LW-AFTR test cases | -+---------------------+-------------------------------------------------------------------------+ -| JIRA: YARDSTICK-848 | NSB "Prox" : Cleanup duplicated traffic profile | -+---------------------+-------------------------------------------------------------------------+ - - - - -Euphrates 5.0.0: - -+---------------------+--------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+---------------------+--------------------------------------------+ -| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using | -| | 'openstack -h' | -+---------------------+--------------------------------------------+ -| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they | -| | are dynamic | -+---------------------+--------------------------------------------+ - - -Euphratess 5.0.0 known restrictions/issues ------------------------------------------- +-----------+-----------+----------------------------------------------+ | Installer | Scenario | Issue | +===========+===========+==============================================+ -| any | \*-bgpvpn | Floating ips not supported. Some Test cases | -| | | related to floating ips are excluded. | -+-----------+-----------+----------------------------------------------+ -| any | odl_l3-\* | Some test cases related to using floating IP | -| | | addresses fail because of a known ODL bug. | -| | | | -+-----------+-----------+----------------------------------------------+ -| compass | odl_l2-\* | In some test cases, VM instance will failed | -| | | raising network interfaces. | | | | | +-----------+-----------+----------------------------------------------+ - Useful links ------------- +============ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning + - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser - Yardstick repo: https://git.opnfv.org/cgit/yardstick diff --git a/docs/release/results/euphrates_fraser_comparsion.rst b/docs/release/results/euphrates_fraser_comparsion.rst new file mode 100644 index 000000000..222dc8bb0 --- /dev/null +++ b/docs/release/results/euphrates_fraser_comparsion.rst @@ -0,0 +1,8 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + +======================================================= +Test results analysis for Euphrates and Fraser releases +======================================================= + diff --git a/docs/release/results/images/tc002_pod.png b/docs/release/results/images/tc002_pod.png Binary files differnew file mode 100644 index 000000000..7f92c471d --- /dev/null +++ b/docs/release/results/images/tc002_pod.png diff --git a/docs/release/results/images/tc002_scenario.png b/docs/release/results/images/tc002_scenario.png Binary files differnew file mode 100644 index 000000000..0ea1ecec6 --- /dev/null +++ b/docs/release/results/images/tc002_scenario.png diff --git a/docs/release/results/images/tc010_pod.png b/docs/release/results/images/tc010_pod.png Binary files differnew file mode 100644 index 000000000..c7c623481 --- /dev/null +++ b/docs/release/results/images/tc010_pod.png diff --git a/docs/release/results/images/tc010_scenario.png b/docs/release/results/images/tc010_scenario.png Binary files differnew file mode 100644 index 000000000..7c53a5fab --- /dev/null +++ b/docs/release/results/images/tc010_scenario.png diff --git a/docs/release/results/images/tc011_pod.png b/docs/release/results/images/tc011_pod.png Binary files differnew file mode 100644 index 000000000..8fec72f5a --- /dev/null +++ b/docs/release/results/images/tc011_pod.png diff --git a/docs/release/results/images/tc011_scenario.png b/docs/release/results/images/tc011_scenario.png Binary files differnew file mode 100644 index 000000000..2d78ea372 --- /dev/null +++ b/docs/release/results/images/tc011_scenario.png diff --git a/docs/release/results/images/tc012_pod.png b/docs/release/results/images/tc012_pod.png Binary files differnew file mode 100644 index 000000000..0f2a00910 --- /dev/null +++ b/docs/release/results/images/tc012_pod.png diff --git a/docs/release/results/images/tc012_scenario.png b/docs/release/results/images/tc012_scenario.png Binary files differnew file mode 100644 index 000000000..16257988d --- /dev/null +++ b/docs/release/results/images/tc012_scenario.png diff --git a/docs/release/results/images/tc014_pod.png b/docs/release/results/images/tc014_pod.png Binary files differnew file mode 100644 index 000000000..63aead2e8 --- /dev/null +++ b/docs/release/results/images/tc014_pod.png diff --git a/docs/release/results/images/tc014_scenario.png b/docs/release/results/images/tc014_scenario.png Binary files differnew file mode 100644 index 000000000..98f23ba1b --- /dev/null +++ b/docs/release/results/images/tc014_scenario.png diff --git a/docs/release/results/images/tc069_pod.png b/docs/release/results/images/tc069_pod.png Binary files differnew file mode 100644 index 000000000..66b272cb4 --- /dev/null +++ b/docs/release/results/images/tc069_pod.png diff --git a/docs/release/results/images/tc069_scenario.png b/docs/release/results/images/tc069_scenario.png Binary files differnew file mode 100644 index 000000000..caf12f8d5 --- /dev/null +++ b/docs/release/results/images/tc069_scenario.png diff --git a/docs/release/results/images/tc082_pod.png b/docs/release/results/images/tc082_pod.png Binary files differnew file mode 100644 index 000000000..89e01666b --- /dev/null +++ b/docs/release/results/images/tc082_pod.png diff --git a/docs/release/results/images/tc082_scenario.png b/docs/release/results/images/tc082_scenario.png Binary files differnew file mode 100644 index 000000000..637a739c3 --- /dev/null +++ b/docs/release/results/images/tc082_scenario.png diff --git a/docs/release/results/images/tc083_pod.png b/docs/release/results/images/tc083_pod.png Binary files differnew file mode 100644 index 000000000..f874191e4 --- /dev/null +++ b/docs/release/results/images/tc083_pod.png diff --git a/docs/release/results/images/tc083_scenario.png b/docs/release/results/images/tc083_scenario.png Binary files differnew file mode 100644 index 000000000..afd80aa02 --- /dev/null +++ b/docs/release/results/images/tc083_scenario.png diff --git a/docs/release/results/index.rst b/docs/release/results/index.rst index 0560152e0..3ec9e1cff 100644 --- a/docs/release/results/index.rst +++ b/docs/release/results/index.rst @@ -14,3 +14,4 @@ Yardstick test results .. include:: ./overview.rst .. include:: ./results.rst +.. include:: ./euphrates_fraser_comparsion.rst diff --git a/docs/release/results/os-nosdn-kvm-ha.rst b/docs/release/results/os-nosdn-kvm-ha.rst deleted file mode 100644 index a8a56f80e..000000000 --- a/docs/release/results/os-nosdn-kvm-ha.rst +++ /dev/null @@ -1,270 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -================================ -Test Results for os-nosdn-kvm-ha -================================ - -.. toctree:: - :maxdepth: 2 - - -fuel -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test -runs, each run on the Ericsson POD2_ or LF POD2_ between August 24 and 30 in -2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.44 and 0.75 ms. -A few runs start with a 0.65 - 0.68 ms RTT spike (This could be because of -normal ARP handling). One test run has a greater RTT spike of 1.49 ms. -To be able to draw conclusions more runs should be made. SLA set to 10 ms. -The SLA value is used as a reference, it has not been defined by OPNFV. - -TC005 ------ -The IO read bandwidth looks similar between different dates, with an -average between approx. 92 and 204 MB/s. Within each test run the results -vary, with a minimum 2 MB/s and maximum 819 MB/s on the totality. Most runs -have a minimum BW of 3 MB/s (one run at 2 MB/s). The maximum BW varies more in -absolute numbers between the dates, between 238 and 819 MB/s. -SLA set to 400 MB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC010 ------ -The measurements for memory latency are similar between test dates and result -in approx. 2.07 ns. The variations within each test run are similar, between -1.41 and 3.53 ns. -SLA set to 30 ns. The SLA value is used as a reference, it has not been defined -by OPNFV. - -TC011 ------ -Packet delay variation between 2 VMs on different blades is measured using -Iperf3. The reported packet delay variation varies between 0.0051 and 0.0243 ms, -with an average delay variation between 0.0081 ms and 0.0195 ms. - -TC012 ------ -Between test dates, the average measurements for memory bandwidth result in -approx. 13.6 GB/s. Within each test run the results vary more, with a minimal -BW of 6.09 GB/s and maximum of 16.47 GB/s on the totality. -SLA set to 15 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC014 ------ -The Unixbench processor test run results vary between scores 2316 and 3619, -one result each date. -No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -CPU utilization statistics are collected during UDP flows sent between the VMs -using pktgen as packet generator tool. The average measurements for CPU -utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio -appears around 7%. - -TC069 ------ -Between test dates, the average measurements for memory bandwidth vary between -22.6 and 29.1 GB/s. Within each test run the results vary more, with a minimal -BW of 20.0 GB/s and maximum of 29.5 GB/s on the totality. -SLA set to 6 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Memory utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for memory -utilization vary between 225MB to 246MB. The peak of memory utilization appears -around 340MB. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Cache utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for cache -utilization vary between 205MB to 212MB. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. Total number of packets received per -second was average on 200 kpps and total number of packets transmitted per -second was average on 600 kpps. - -Detailed test results ---------------------- -The scenario was run on Ericsson POD2_ and LF POD2_ with: -Fuel 9.0 -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - -Conclusions and recommendations -------------------------------- -The pktgen test configuration has a relatively large base effect on RTT in -TC037 compared to TC002, where there is no background load at all. Approx. -15 ms compared to approx. 0.5 ms, which is more than a 3000 percentage -difference in RTT results. -Especially RTT and throughput come out with better results than for instance -the *fuel-os-nosdn-nofeature-ha* scenario does. The reason for this should -probably be further analyzed and understood. Also of interest could be -to make further analyzes to find patterns and reasons for lost traffic. -Also of interest could be to see if there are continuous variations where -some test cases stand out with better or worse results than the general test -case. - diff --git a/docs/release/results/os-nosdn-nofeature-ha.rst b/docs/release/results/os-nosdn-nofeature-ha.rst deleted file mode 100644 index 9e52731d5..000000000 --- a/docs/release/results/os-nosdn-nofeature-ha.rst +++ /dev/null @@ -1,492 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -====================================== -Test Results for os-nosdn-nofeature-ha -====================================== - -.. toctree:: - :maxdepth: 2 - - -apex -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD1: https://wiki.opnfv.org/pharos?&#community_test_labs - - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test -runs, each run on the LF POD1_ between August 25 and 28 in -2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.74 and 1.08 ms. -A few runs start with a 0.99 - 1.07 ms RTT spike (This could be because of -normal ARP handling). One test run has a greater RTT spike of 1.35 ms. -To be able to draw conclusions more runs should be made. SLA set to 10 ms. -The SLA value is used as a reference, it has not been defined by OPNFV. - -TC005 ------ -The IO read bandwidth looks similar between different dates, with an -average between approx. 128 and 136 MB/s. Within each test run the results -vary, with a minimum 5 MB/s and maximum 446 MB/s on the totality. Most runs -have a minimum BW of 5 MB/s (one run at 6 MB/s). The maximum BW varies more in -absolute numbers between the dates, between 416 and 446 MB/s. -SLA set to 400 MB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC010 ------ -The measurements for memory latency are similar between test dates and result -in approx. 1.09 ns. The variations within each test run are similar, between -1.0860 and 1.0880 ns. -SLA set to 30 ns. The SLA value is used as a reference, it has not been defined -by OPNFV. - -TC011 ------ -Packet delay variation between 2 VMs on different blades is measured using -Iperf3. The reported packet delay variation varies between 0.0025 and 0.0148 ms, -with an average delay variation between 0.0056 ms and 0.0157 ms. - -TC012 ------ -Between test dates, the average measurements for memory bandwidth result in -approx. 19.70 GB/s. Within each test run the results vary more, with a minimal -BW of 18.16 GB/s and maximum of 20.13 GB/s on the totality. -SLA set to 15 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC014 ------ -The Unixbench processor test run results vary between scores 3224.4 and 3842.8, -one result each date. The average score on the total is 3659.5. -No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -CPU utilization statistics are collected during UDP flows sent between the VMs -using pktgen as packet generator tool. The average measurements for CPU -utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio -appears around 7%. - -TC069 ------ -Between test dates, the average measurements for memory bandwidth vary between -22.6 and 29.1 GB/s. Within each test run the results vary more, with a minimal -BW of 20.0 GB/s and maximum of 29.5 GB/s on the totality. -SLA set to 6 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Memory utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for memory -utilization vary between 225MB to 246MB. The peak of memory utilization appears -around 340MB. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Cache utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for cache -utilization vary between 205MB to 212MB. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. Total number of packets received per -second was average on 200 kpps and total number of packets transmitted per -second was average on 600 kpps. - -Detailed test results ---------------------- -The scenario was run on LF POD1_ with: -Apex -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - - -Joid -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD5: https://wiki.opnfv.org/pharos?&#community_test_labs - - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Intel POD5_ between September 11 and 14 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 1.59 and 1.70 ms. -Two test runs have reached the same greater RTT spike of 3.06 ms, which are -1.66 and 1.70 ms average, but only one has the lower RTT of 1.35 ms. The other -two runs have no similar spike at all. To be able to draw conclusions more runs -should be made. SLA set to be 10 ms. The SLA value is used as a reference, it -has not been defined by OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput and the -greatest IO read bandwidth of the four runs is 173.3 MB/s. The IO read -bandwidth of the four runs looks similar on different four days, with an -average between 32.7 and 60.4 MB/s. One of the runs has a minimum BW of 429 -KM/s and other has a maximum BW of 173.3 MB/s. The SLA of read bandwidth sets -to be 400 MB/s, which is used as a reference, and it has not been defined by -OPNFV. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is 1.1 ns on average. The -variations within each test run are different, some vary from a large range and -others have a small change. For example, the largest change is on September 14, -the memory read latency of which is ranging from 1.12 ns to 1.22 ns. However, -the results on September 12 change very little, which range from 1.14 ns to -1.17 ns. The SLA sets to be 30 ns. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC011 ------ -Iperf3 is a tool for evaluating the pocket delay variation between 2 VMs on -different blades. The reported pocket delay variations of the four test runs -differ from each other. The results on September 13 within the date look -similar and the values are between 0.0087 and 0.0190 ms, which is 0.0126 ms on -average. However, on the fourth day, the pocket delay variation has a large -wide change within the date, which ranges from 0.0032 ms to 0.0121 ms and has -the minimum average value. The pocket delay variations of other two test runs -look relatively similar, which are 0.0076 ms and 0.0152 ms on average. The SLA -value sets to be 10 ms. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the four test runs, the memory -bandwidth within the second day almost keep stable, which is 11.58 GB/s on -average. And the memory bandwidth of the fourth day look similar as that of the -second day, both of which remain stable. The other two test runs relatively -change from a large wide range, in which the minimum memory bandwidth is 11.22 -GB/s and the maximum bandwidth is 16.65 GB/s with an average bandwidth of about -12.20 GB/s. Here SLA set to be 15 GB/s. The SLA value is used as a reference, -it has not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to measure processing speed, that is instructions per -second. It can be seen from the dashboard that the processing test results -vary from scores 3272 to 3444, and there is only one result one date. The -overall average score is 3371. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the four test runs is 119.85, 128.02, 121.40 and -126.08 kpps, of which the result of the second is the highest. The RTT results -of all the test runs keep flat at approx. 37 ms. It is obvious that the PPS -results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 184 kpps and the -minimum throughput is 49 K respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar average vaue, that is 38 ms, of which the worest RTT is -93 ms on Sep. 14th. - -CPU load of the four test runs have a large change, since the minimum value and -the peak of CPU load is 0 percent and 51 percent respectively. And the best -result is obtained on Sep. 14th. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 22.3 GB/s to 26.8 GB/s and then to 18.5 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is 8 kb and 16 kb, the memory write bandwidth -look similar with a minimal BW of 22.5 GB/s and peak value of 28.7 GB/s. SLA -sets to be 7 GB/s. The SLA value is used as a a reference, it has not been -defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT can reach -more than 80 ms and the average RTT is usually approx. 38 ms. On the whole, the -average RTTs of the four runs keep flat. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 268 MiB on Sep -14, which also has the largest minimum memory. Besides, the rest three test -runs have the similar used memory. On the other hand, the free memory of the -four runs have the same smallest minimum value, that is about 223 MiB, and the -maximum free memory of three runs have the similar result, that is 337 MiB, -except that on Sep. 14th, whose maximum free memory is 254 MiB. On the whole, -all the test runs have similar average free memory. - -Network throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -network throughput of the four test runs seem quite different, ranging from -119.85 kpps to 128.02 kpps. The average number of flows in these tests is -24000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding packet throughput -differ between 49.4k and 193.3k with an average packet throughput of approx. -125k. On the whole, the PPS results seem consistent. Within each test run of -the four runs, when number of flows becomes larger, the packet throughput seems -not larger in the meantime. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT can reach -more than 94 ms and the average RTT is usually approx. 35 ms. On the whole, the -average RTTs of the four runs keep flat. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool.The largest -cache size is 212 MiB in the four runs, and the smallest cache size is 75 MiB. -On the whole, the average cache size of the four runs is approx. 208 MiB. -Meanwhile, the tread of the buffer size looks similar with each other. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs seem quite different, ranging from 119.85 kpps to 128.02 -kpps. The average number of flows in these tests is 239.7k, and each run has a -minimum number of flows of 2 and a maximum number of flows of 1.001 Mil. At the -same time, the corresponding packet throughput differ between 49.4k and 193.3k -with an average packet throughput of approx. 125k. On the whole, the PPS results -seem consistent. Within each test run of the four runs, when number of flows -becomes larger, the packet throughput seems not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 32 ms. The PPS results are not as consistent as the RTT results. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second differs from each other, in which the smallest number of -packets transmitted per second is 6 pps on Sep. 12ed and the largest of that is -210.8 kpps. Meanwhile, the largest total number of packets received per second -differs from each other, in which the smallest number of packets received per -second is 2 pps on Sep. 13rd and the largest of that is 250.2 kpps. - -In some test runs when running with less than approx. 90000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 1000000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Intel POD5_ with: -Joid -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - - diff --git a/docs/release/results/os-nosdn-nofeature-noha.rst b/docs/release/results/os-nosdn-nofeature-noha.rst deleted file mode 100644 index 8b7c184bb..000000000 --- a/docs/release/results/os-nosdn-nofeature-noha.rst +++ /dev/null @@ -1,259 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -======================================== -Test Results for os-nosdn-nofeature-noha -======================================== - -.. toctree:: - :maxdepth: 2 - - -Joid -===== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD5: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Intel POD5_ between September 12 and 15 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 1.50 and 1.68 ms. -Only one test run has reached greatest RTT spike of 2.92 ms, which has -the smallest RTT of 1.06 ms. The other three runs have no similar spike at all, -the minimum and average RTTs of which are approx. 1.50 ms and 1.68 ms. SLA set to -be 10 ms. The SLA value is used as a reference, it has not been defined by -OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 177.5 -MB/s. The IO read bandwidth of the four runs looks similar on different four -days, with an average between 46.7 and 62.5 MB/s. One of the runs has a minimum -BW of 680 KM/s and other has a maximum BW of 177.5 MB/s. The SLA of read -bandwidth sets to be 400 MB/s, which is used as a reference, and it has not -been defined by OPNFV. - -The results of storage IOPS for the four runs look similar with each other. The -test runs all have an approx. 1.55 K/s for IO reading with an minimum value of -less than 60 times per second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is between 1.134 ns and 1.227 -ns on average. The variations within each test run are quite different, some -vary from a large range and others have a small change. For example, the -largest change is on September 15, the memory read latency of which is ranging -from 1.116 ns to 1.393 ns. However, the results on September 12 change very -little, which mainly keep flat and range from 1.124 ns to 1.55 ns. The SLA sets -to be 30 ns. The SLA value is used as a reference, it has not been defined by -OPNFV. - -TC011 ------ -Iperf3 is a tool for evaluating the pocket delay variation between 2 VMs on -different blades. The reported pocket delay variations of the four test runs -differ from each other. The results on September 13 within the date look -similar and the values are between 0.0213 and 0.0225 ms, which is 0.0217 ms on -average. However, on the third day, the packet delay variation has a large -wide change within the date, which ranges from 0.008 ms to 0.0225 ms and has -the minimum value. On Sep. 12, the packet delay is quite long, for the value is -between 0.0236 and 0.0287 ms and it also has the maximum packet delay of 0.0287 -ms. The packet delay of the last test run is 0.0151 ms on average. The SLA -value sets to be 10 ms. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the four test runs, the memory -bandwidth of three test runs almost keep stable within each run, which is -11.65, 11.57 and 11.64 GB/s on average. However, the memory read and write -bandwidth on Sep. 14 has a large range, for it ranges from 11.36 GB/s to 16.68 -GB/s. Here SLA set to be 15 GB/s. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3222 to 3585, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the four test runs is 124.8, 160.1, 113.8 and -137.3 kpps, of which the result of the second is the highest. The RTT results -of all the test runs keep flat at approx. 37 ms. It is obvious that the PPS -results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 243.1 kpps and the -minimum throughput is 37.6 kpps respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar average vaue, that is between 32 ms and 41 ms, of which -the worest RTT is 155 ms on Sep. 14th. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and 9 percent respectively. And the best result is obtained on Sep. -15th, with an CPU load of nine percent. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 22.4 GB/s to 26.5 GB/s and then to 18.6 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is 8 kb and 16 kb, the memory write bandwidth -look similar with a minimal BW of 22.5 GB/s and peak value of 28.7 GB/s. And -then with the block size becoming larger, the memory write bandwidth tends to -decrease. SLA sets to be 7 GB/s. The SLA value is used as a a reference, it has -not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of three test runs look -similar with each other, and Within these test runs, the maximum RTT can reach -95 ms and the average RTT is usually approx. 36 ms. The network latency tested -on Sep. 14 shows that it has a peak latency of 155 ms. But on the whole, the -average RTTs of the four runs keep flat. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 270 MiB on Sep -13, which also has the smallest minimum memory utilization. Besides, the rest -three test runs have the similar used memory with an average memory usage of -264 MiB. On the other hand, the free memory of the four runs have the same -smallest minimum value, that is about 223 MiB, and the maximum free memory of -three runs have the similar result, that is 226 MiB, except that on Sep. 13th, -whose maximum free memory is 273 MiB. On the whole, all the test runs have -similar average free memory. - -Network throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -network throughput of the four test runs seem quite different, ranging from -119.85 kpps to 128.02 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding packet throughput -differ between 38k and 243k with an average packet throughput of approx. 134k. -On the whole, the PPS results seem consistent. Within each test run of the four -runs, when number of flows becomes larger, the packet throughput seems not -larger in the meantime. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT can reach -79 ms and the average RTT is usually approx. 35 ms. On the whole, the average -RTTs of the four runs keep flat. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool.The largest -cache size is 214 MiB in the four runs, and the smallest cache size is 100 MiB. -On the whole, the average cache size of the four runs is approx. 210 MiB. -Meanwhile, the tread of the buffer size looks similar with each other. On the -other hand, the mean buffer size of the four runs keep flat, since they have a -minimum value of approx. 7 MiB and a maximum value of 8 MiB, with an average -value of about 8 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs seem quite different, ranging from 113.8 kpps to 124.8 kpps. -The average number of flows in these tests is 240k, and each run has a minimum -number of flows of 2 and a maximum number of flows of 1.001 Mil. At the same -time, the corresponding packet throughput differ between 47.6k and 243.1k with -an average packet throughput between 113.8k and 160.1k. Within each test run of -the four runs, when number of flows becomes larger, the packet throughput seems -not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs -between 0 ms and 79 ms with an average leatency of approx. 35 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the four runs differ from 113.8 kpps to 124.8 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar on the first three runs with a minimum -number of 10 pps and a maximum number of 97 kpps, except the one on Sep. 15th, -in which the number of packets transmitted per second is 10 pps. Meanwhile, the -largest total number of packets received per second differs from each other, -in which the smallest number of packets received per second is 1 pps and the -largest of that is 276 kpps. - -In some test runs when running with less than approx. 90000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 1000000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Intel POD5_ with: -Joid -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. diff --git a/docs/release/results/os-odl_l2-bgpvpn-ha.rst b/docs/release/results/os-odl_l2-bgpvpn-ha.rst deleted file mode 100644 index 2bd6dc35d..000000000 --- a/docs/release/results/os-odl_l2-bgpvpn-ha.rst +++ /dev/null @@ -1,53 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -==================================== -Test Results for os-odl_l2-bgpvpn-ha -==================================== - -.. toctree:: - :maxdepth: 2 - - -fuel -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Ericsson POD2_ between September 7 and 11 in 2016. - -TC043 ------ -The round-trip-time (RTT) between 2 nodes is measured using -ping. Most test run measurements result on average between 0.21 and 0.28 ms. -A few runs start with a 0.32 - 0.35 ms RTT spike (This could be because of -normal ARP handling). To be able to draw conclusions more runs should be made. -SLA set to 10 ms. The SLA value is used as a reference, it has not been defined -by OPNFV. - -Detailed test results ---------------------- -The scenario was run on Ericsson POD2_ with: -Fuel 9.0 -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - diff --git a/docs/release/results/os-odl_l2-nofeature-ha.rst b/docs/release/results/os-odl_l2-nofeature-ha.rst deleted file mode 100644 index ac0c5bb59..000000000 --- a/docs/release/results/os-odl_l2-nofeature-ha.rst +++ /dev/null @@ -1,743 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -======================================= -Test Results for os-odl_l2-nofeature-ha -======================================= - -.. toctree:: - :maxdepth: 2 - - -apex -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD1: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the LF POD1_ between September 14 and 17 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.49 ms and 0.60 ms. -Only one test run has reached greatest RTT spike of 0.93 ms. Meanwhile, the -smallest network latency is 0.33 ms, which is obtained on Sep. 14th. -SLA set to be 10 ms. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 416 -MB/s. The IO read bandwidth of all four runs looks similar, with an average -between 128 and 131 MB/s. One of the runs has a minimum BW of 497 KB/s. The SLA -of read bandwidth sets to be 400 MB/s, which is used as a reference, and it has -not been defined by OPNFV. - -The results of storage IOPS for the four runs look similar with each other. The -IO read times per second of the four test runs have an average value at 1k per -second, and meanwhile, the minimum result is only 45 times per second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is between 1.0859 ns and -1.0869 ns on average. The variations within each test run are quite different, -some vary from a large range and others have a small change. For example, the -largest change is on September 14th, the memory read latency of which is ranging -from 1.091 ns to 1.086 ns. However. -The SLA sets to be 30 ns. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC011 ------ -Packet delay variation between 2 VMs on different blades is measured using -Iperf3. On the first two test runs the reported packet delay variation varies between -0.0037 and 0.0740 ms, with an average delay variation between 0.0096 ms and 0.0321. -On the second date the delay variation varies between 0.0063 and 0.0096 ms, with -an average delay variation of 0.0124 - 0.0141 ms. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the four test runs, the trend of -three memory bandwidth almost look similar, which all have a narrow range, and -the average result is 19.88 GB/s. Here SLA set to be 15 GB/s. The SLA value is -used as a reference, it has not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3754k to 3831k, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the four test runs is between 307.3 kpps and -447.1 kpps, of which the result of the third run is the highest. The RTT -results of all the test runs keep flat at approx. 15 ms. It is obvious that the -PPS results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 418.1 kpps and the -minimum throughput is 326.5 kpps respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar average vaue, that is approx. 15 ms. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and nine percent respectively. And the best result is obtained on Sep. -1, with an CPU load of nine percent. But on the whole, the CPU load is very -poor, since the average value is quite small. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 28.2 GB/s to 29.5 GB/s and then to 29.2 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is 2 kb or 16 kb, the memory write bandwidth -look similar with a minimal BW of 25.8 GB/s and peak value of 28.3 GB/s. And -then with the block size becoming larger, the memory write bandwidth tends to -decrease. SLA sets to be 7 GB/s. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other, and within these test runs, the maximum RTT can -reach 39 ms and the average RTT is usually approx. 15 ms. The network latency -tested on Sep. 1 and Sep. 8 have a peak latency of 39 ms. But on the whole, -the average RTTs of the five runs keep flat and the network latency is -relatively short. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 267 MiB for the -four runs. In general, the four test runs have very large memory utilization, -which can reach 257 MiB on average. On the other hand, for the mean free memory, -the four test runs have the similar trend with that of the mean used memory. -In general, the mean free memory change from 233 MiB to 241 MiB. - -Packet throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -packet throughput of the four test runs seem quite different, ranging from -305.3 kpps to 447.1 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding average packet -throughput is between 354.4 kpps and 381.8 kpps. In summary, the PPS results -seem consistent. Within each test run of the four runs, when number of flows -becomes larger, the packet throughput seems not larger at the same time. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT is only 42 -ms and the average RTT is usually approx. 15 ms. On the whole, the average -RTTs of the four runs keep stable and the network latency is relatively small. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool. The largest -cache size is 212 MiB, which is same for the four runs, and the smallest cache -size is 75 MiB. On the whole, the average cache size of the four runs look the -same and is between 197 MiB and 211 MiB. Meanwhile, the tread of the buffer -size keep flat, since they have a minimum value of 7 MiB and a maximum value of -8 MiB, with an average value of about 7.9 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs differ from 354.4 kpps to 381.8 kpps. The average number of -flows in these tests is 240k, and each run has a minimum number of flows of 2 -and a maximum number of flows of 1.001 Mil. At the same time, the corresponding -packet throughput differ between 305.3 kpps to 447.1 kpps. Within each test run -of the four runs, when number of flows becomes larger, the packet throughput -seems not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs -between 0 ms and 42 ms with an average leatency of less than 15 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the four runs differ from 354.4 kpps to 381.8 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar for three test runs, whose values change a -lot from 10 pps to 501 kpps. While results of the rest test run seem the same -and keep stable with the average number of packets transmitted per second of 10 -pps. However, the total number of packets received per second of the four runs -look similar, which have a large wide range of 2 pps to 815 kpps. - -In some test runs when running with less than approx. 251000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 251000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on LF POD1_ with: -Apex -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - - - -fuel -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Ericsson POD2_ or LF POD2_ between August 25 and 29 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.5 and 0.6 ms. -A few runs start with a 1 - 1.5 ms RTT spike (This could be because of normal ARP -handling). One test run has a greater RTT spike of 1.9 ms, which is the same -one with the 0.7 ms average. The other runs have no similar spike at all. -To be able to draw conclusions more runs should be made. -SLA set to 10 ms. The SLA value is used as a reference, it has not -been defined by OPNFV. - -TC005 ------ -The IO read bandwidth looks similar between different dates, with an -average between approx. 170 and 200 MB/s. Within each test run the results -vary, with a minimum 2 MB/s and maximum 838 MB/s on the totality. Most runs -have a minimum BW of 3 MB/s (two runs at 2 MB/s). The maximum BW varies more in -absolute numbers between the dates, between 617 and 838 MB/s. -SLA set to 400 MB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC010 ------ -The measurements for memory latency are similar between test dates and result -in approx. 1.2 ns. The variations within each test run are similar, between -1.215 and 1.219 ns. One exception is February 16, where the average is 1.222 -and varies between 1.22 and 1.28 ns. -SLA set to 30 ns. The SLA value is used as a reference, it has not been defined -by OPNFV. - -TC011 ------ -Packet delay variation between 2 VMs on different blades is measured using -Iperf3. On the first date the reported packet delay variation varies between -0.0025 and 0.011 ms, with an average delay variation of 0.0067 ms. -On the second date the delay variation varies between 0.002 and 0.006 ms, with -an average delay variation of 0.004 ms. - -TC012 ------ -Between test dates, the average measurements for memory bandwidth vary between -17.4 and 17.9 GB/s. Within each test run the results vary more, with a minimal -BW of 16.4 GB/s and maximum of 18.2 GB/s on the totality. -SLA set to 15 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC014 ------ -The Unixbench processor test run results vary between scores 3080 and 3240, -one result each date. The average score on the total is 3150. -No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -CPU utilization statistics are collected during UDP flows sent between the VMs -using pktgen as packet generator tool. The average measurements for CPU -utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio -appears around 7%. - -TC069 ------ -Between test dates, the average measurements for memory bandwidth vary between -15.5 and 25.4 GB/s. Within each test run the results vary more, with a minimal -BW of 9.7 GB/s and maximum of 29.5 GB/s on the totality. -SLA set to 6 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Memory utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for memory -utilization vary between 225MB to 246MB. The peak of memory utilization appears -around 340MB. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Cache utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for cache -utilization vary between 205MB to 212MB. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. Total number of packets received per -second was average on 200 kpps and total number of packets transmitted per -second was average on 600 kpps. - -Detailed test results ---------------------- -The scenario was run on Ericsson POD2_ and LF POD2_ with: -Fuel 9.0 -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - -Conclusions and recommendations -------------------------------- -The pktgen test configuration has a relatively large base effect on RTT in -TC037 compared to TC002, where there is no background load at all. Approx. -15 ms compared to approx. 0.5 ms, which is more than a 3000 percentage -difference in RTT results. -Especially RTT and throughput come out with better results than for instance -the *fuel-os-nosdn-nofeature-ha* scenario does. The reason for this should -probably be further analyzed and understood. Also of interest could be -to make further analyzes to find patterns and reasons for lost traffic. -Also of interest could be to see if there are continuous variations where -some test cases stand out with better or worse results than the general test -case. - - - -Joid -===== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Intel POD6_ between September 1 and 8 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 1.01 ms and 1.88 ms. -Only one test run has reached greatest RTT spike of 1.88 ms. Meanwhile, the -smallest network latency is 1.01 ms, which is obtained on Sep. 1st. In general, -the average of network latency of the four test runs are between 1.29 ms and -1.34 ms. SLA set to be 10 ms. The SLA value is used as a reference, it has not -been defined by OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 183.65 -MB/s. The IO read bandwidth of the three runs looks similar, with an average -between 62.9 and 64.3 MB/s, except one on Sep. 1, for its maximum storage -throughput is only 159.1 MB/s. One of the runs has a minimum BW of 685 KB/s and -other has a maximum BW of 183.6 MB/s. The SLA of read bandwidth sets to be -400 MB/s, which is used as a reference, and it has not been defined by OPNFV. - -The results of storage IOPS for the four runs look similar with each other. The -IO read times per second of the four test runs have an average value between -1.41k per second and 1.64k per second, and meanwhile, the minimum result is -only 55 times per second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is between 1.152 ns and 1.179 -ns on average. The variations within each test run are quite different, some -vary from a large range and others have a small change. For example, the -largest change is on September 8, the memory read latency of which is ranging -from 1.120 ns to 1.221 ns. However, the results on September 7 change very -little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC011 ------ -Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on -different blades. The reported packet delay variations of the four test runs -differ from each other. In general, the packet delay of the first two runs look -similar, for they both stay stable within each run. And the mean packet delay -of them are 0.0087 ms and 0.0127 ms respectively. Of the four runs, the fourth -has the worst result, because the packet delay reaches 0.0187 ms. The SLA value -sets to be 10 ms. The SLA value is used as a reference, it has not been defined -by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the four test runs, the trend of -three memory bandwidth almost look similar, which all have a narrow range, and -the average result is 11.78 GB/s. Here SLA set to be 15 GB/s. The SLA value is -used as a reference, it has not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3260k to 3328k, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the four test runs is between 307.3 kpps and -447.1 kpps, of which the result of the third run is the highest. The RTT -results of all the test runs keep flat at approx. 15 ms. It is obvious that the -PPS results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 418.1 kpps and the -minimum throughput is 326.5 kpps respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar average vaue, that is approx. 15 ms. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and nine percent respectively. And the best result is obtained on Sep. -1, with an CPU load of nine percent. But on the whole, the CPU load is very -poor, since the average value is quite small. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 21.9 GB/s to 25.9 GB/s and then to 17.8 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is 2 kb or 16 kb, the memory write bandwidth -look similar with a minimal BW of 24.8 GB/s and peak value of 27.8 GB/s. And -then with the block size becoming larger, the memory write bandwidth tends to -decrease. SLA sets to be 7 GB/s. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other, and within these test runs, the maximum RTT can -reach 39 ms and the average RTT is usually approx. 15 ms. The network latency -tested on Sep. 1 and Sep. 8 have a peak latency of 39 ms. But on the whole, -the average RTTs of the five runs keep flat and the network latency is -relatively short. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 267 MiB for the -four runs. In general, the four test runs have very large memory utilization, -which can reach 257 MiB on average. On the other hand, for the mean free memory, -the four test runs have the similar trend with that of the mean used memory. -In general, the mean free memory change from 233 MiB to 241 MiB. - -Packet throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -packet throughput of the four test runs seem quite different, ranging from -305.3 kpps to 447.1 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding average packet -throughput is between 354.4 kpps and 381.8 kpps. In summary, the PPS results -seem consistent. Within each test run of the four runs, when number of flows -becomes larger, the packet throughput seems not larger at the same time. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT is only 42 -ms and the average RTT is usually approx. 15 ms. On the whole, the average -RTTs of the four runs keep stable and the network latency is relatively small. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool. The largest -cache size is 212 MiB, which is same for the four runs, and the smallest cache -size is 75 MiB. On the whole, the average cache size of the four runs look the -same and is between 197 MiB and 211 MiB. Meanwhile, the tread of the buffer -size keep flat, since they have a minimum value of 7 MiB and a maximum value of -8 MiB, with an average value of about 7.9 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs differ from 354.4 kpps to 381.8 kpps. The average number of -flows in these tests is 240k, and each run has a minimum number of flows of 2 -and a maximum number of flows of 1.001 Mil. At the same time, the corresponding -packet throughput differ between 305.3 kpps to 447.1 kpps. Within each test run -of the four runs, when number of flows becomes larger, the packet throughput -seems not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs -between 0 ms and 42 ms with an average leatency of less than 15 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the four runs differ from 354.4 kpps to 381.8 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar for three test runs, whose values change a -lot from 10 pps to 501 kpps. While results of the rest test run seem the same -and keep stable with the average number of packets transmitted per second of 10 -pps. However, the total number of packets received per second of the four runs -look similar, which have a large wide range of 2 pps to 815 kpps. - -In some test runs when running with less than approx. 251000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 251000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Intel POD6_ with: -Joid -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - diff --git a/docs/release/results/os-odl_l2-sfc-ha.rst b/docs/release/results/os-odl_l2-sfc-ha.rst deleted file mode 100644 index e27562cae..000000000 --- a/docs/release/results/os-odl_l2-sfc-ha.rst +++ /dev/null @@ -1,231 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -================================== -Test Results for os-odl_l2-sfc-ha -================================== - -.. toctree:: - :maxdepth: 2 - - -Fuel -===== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the LF POD2_ or Ericsson POD2_ between September 16 and 20 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.32 ms and 1.42 ms. -Only one test run on Sep. 20 has reached greatest RTT spike of 4.66 ms. -Meanwhile, the smallest network latency is 0.16 ms, which is obtained on Sep. -17th. To sum up, the curve of network latency has very small wave, which is -less than 5 ms. SLA sets to be 10 ms. The SLA value is used as a reference, it -has not been defined by OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 734 -MB/s. The IO read bandwidth of the first three runs looks similar, with an -average of less than 100 KB/s, except one on Sep. 20, whose maximum storage -throughput can reach 734 MB/s. The SLA of read bandwidth sets to be 400 MB/s, -which is used as a reference, and it has not been defined by OPNFV. - -The results of storage IOPS for the four runs look similar with each other. The -IO read times per second of the four test runs have an average value between -1.8k per second and 3.27k per second, and meanwhile, the minimum result is -only 60 times per second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is between 1.085 ns and 1.218 -ns on average. The variations within each test run are quite small. For -Ericsson pod2, the average of memory latency is approx. 1.217 ms. While for LF -pod2, the average value is about 1.085 ms. It can be seen that the performance -of LF is better than Ericsson's. The SLA sets to be 30 ns. The SLA value is -used as a reference, it has not been defined by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. The four test runs all have a narrow range -of change with the average memory and write BW of 18.5 GB/s. Here SLA set to be -15 GB/s. The SLA value is used as a reference, it has not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3209k to 3843k, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the three test runs is between 439 kpps and -582 kpps, and the test run on Sep. 17th has the lowest average value of 371 -kpps. The RTT results of all the test runs keep flat at approx. 10 ms. It is -obvious that the PPS results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved, since the largest packet throughput is 680 kpps and the -minimum throughput is 319 kpps respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar trend of RTT with the average value of approx. 12 ms. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and ten percent respectively. And the best result is obtained on Sep. -17th, with an CPU load of ten percent. But on the whole, the CPU load is very -poor, since the average value is quite small. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the average memory write -bandwidth tends to become larger first and then smaller within every run test -for the two pods, which rangs from 25.1 GB/s to 29.4 GB/s and then to 19.2 GB/s -on average. Since the test id is one, it is that only the INT memory write -bandwidth is tested. On the whole, with the block size becoming larger, the -memory write bandwidth tends to decrease. SLA sets to be 7 GB/s. The SLA value -is used as a reference, it has not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other, and within these test runs, the maximum RTT can -reach 27 ms and the average RTT is usually approx. 12 ms. The network latency -tested on Sep. 27th has a peak latency of 27 ms. But on the whole, the average -RTTs of the four runs keep flat. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 269 MiB for the -four runs. In general, the four test runs have very large memory utilization, -which can reach 251 MiB on average. On the other hand, for the mean free memory, -the four test runs have the similar trend with that of the mean used memory. -In general, the mean free memory change from 231 MiB to 248 MiB. - -Packet throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -packet throughput of the four test runs seem quite different, ranging from -371 kpps to 582 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding average packet -throughput is between 319 kpps and 680 kpps. In summary, the PPS results -seem consistent. Within each test run of the four runs, when number of flows -becomes larger, the packet throughput seems not larger at the same time. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT is only 24 -ms and the average RTT is usually approx. 12 ms. On the whole, the average -RTTs of the four runs keep stable and the network latency is relatively small. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool. The largest -cache size is 213 MiB, and the smallest cache size is 99 MiB, which is same for -the four runs. On the whole, the average cache size of the four runs look the -same and is between 184 MiB and 205 MiB. Meanwhile, the tread of the buffer -size keep stable, since they have a minimum value of 7 MiB and a maximum value of -8 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs differ from 371 kpps to 582 kpps. The average number of -flows in these tests is 240k, and each run has a minimum number of flows of 2 -and a maximum number of flows of 1.001 Mil. At the same time, the corresponding -packet throughput differ between 319 kpps to 680 kpps. Within each test run -of the four runs, when number of flows becomes larger, the packet throughput -seems not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs -between 0 ms and 24 ms with an average leatency of less than 13 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the four runs differ from 370 kpps to 582 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar for the four test runs, whose values change a -lot from 10 pps to 697 kpps. However, the total number of packets received per -second of three runs look similar, which have a large wide range of 2 pps to -1.497 Mpps, while the results on Sep. 18th and 20th have very small maximum -number of packets received per second of 817 kpps. - -In some test runs when running with less than approx. 251000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 251000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Ericsson POD2_ and LF POD2_ with: -Fuel 9.0 -OpenStack Mitaka -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. diff --git a/docs/release/results/os-onos-nofeature-ha.rst b/docs/release/results/os-onos-nofeature-ha.rst deleted file mode 100644 index d8b3ace5f..000000000 --- a/docs/release/results/os-onos-nofeature-ha.rst +++ /dev/null @@ -1,257 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -====================================== -Test Results for os-onos-nofeature-ha -====================================== - -.. toctree:: - :maxdepth: 2 - - -Joid -===== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 5 scenario test runs, each run -on the Intel POD6_ between September 13 and 16 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 1.50 and 1.68 ms. -Only one test run has reached greatest RTT spike of 2.62 ms, which has -the smallest RTT of 1.00 ms. The other four runs have no similar spike at all, -the minimum and average RTTs of which are approx. 1.06 ms and 1.32 ms. SLA set -to be 10 ms. The SLA value is used as a reference, it has not been defined by -OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 175.4 -MB/s. The IO read bandwidth of the four runs looks similar on different four -days, with an average between 58.1 and 62.0 MB/s, except one on Sep. 14, for -its maximum storage throughput is only 133.0 MB/s. One of the runs has a -minimum BW of 497 KM/s and other has a maximum BW of 177.4 MB/s. The SLA of read -bandwidth sets to be 400 MB/s, which is used as a reference, and it has not -been defined by OPNFV. - -The results of storage IOPS for the five runs look similar with each other. The -IO read times per second of the five test runs have an average value between -1.20 K/s and 1.61 K/s, and meanwhile, the minimum result is only 41 times per -second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the five runs is between 1.146 ns and 1.172 -ns on average. The variations within each test run are quite different, some -vary from a large range and others have a small change. For example, the -largest change is on September 13, the memory read latency of which is ranging -from 1.152 ns to 1.221 ns. However, the results on September 14 change very -little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC011 ------ -Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on -different blades. The reported packet delay variations of the five test runs -differ from each other. In general, the packet delay of the first two runs look -similar, for they both stay stable within each run. And the mean packet delay of -of them are 0.07714 ms and 0.07982 ms respectively. Of the five runs, the third -has the worst result, because the packet delay reaches 0.08384 ms. The trend of -therest two runs look the same, for the average packet delay are 0.07808 ms and -0.07727 ms respectively. The SLA value sets to be 10 ms. The SLA value is used -as a reference, it has not been defined by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the five test runs, the memory -bandwidth of last three test runs almost keep stable within each run, which is -11.64, 11.71 and 11.61 GB/s on average. However, the memory read and write -bandwidth on Sep. 13 has a large range, for it ranges from 6.68 GB/s to 11.73 -GB/s. Here SLA set to be 15 GB/s. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3208 to 3314, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the five test runs is between 259.6 kpps and -318.4 kpps, of which the result of the second run is the highest. The RTT -results of all the test runs keep flat at approx. 20 ms. It is obvious that the -PPS results are not as consistent as the RTT results. - -The No. flows of the five test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 398.9 kpps and the -minimum throughput is 250.6 kpps respectively. - -There are no errors of packets received in the five runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the five -runs have the similar average vaue, that is between 17 ms and 22 ms, of which -the worest RTT is 53 ms on Sep. 14th. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and 10 percent respectively. And the best result is obtained on Sep. -13rd, with an CPU load of 10 percent. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 21.6 GB/s to 26.8 GB/s and then to 18.4 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is 8 kb and 16 kb, the memory write bandwidth -look similar with a minimal BW of 23.0 GB/s and peak value of 28.6 GB/s. And -then with the block size becoming larger, the memory write bandwidth tends to -decrease. SLA sets to be 7 GB/s. The SLA value is used as a a reference, it has -not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the five test runs -look similar with each other, and within these test runs, the maximum RTT can -reach 53 ms and the average RTT is usually approx. 18 ms. The network latency -tested on Sep. 14 shows that it has a peak latency of 53 ms. But on the whole, -the average RTTs of the five runs keep flat and the network latency is -relatively short. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 272 MiB on Sep -14. In general, the mean used memory of the five test runs have the similar -trend and the minimum memory used size is approx. 150 MiB, and the average -used memory size is about 250 MiB. On the other hand, for the mean free memory, -the five test runs have the similar trend, whose mean free memory change from -218 MiB to 342 MiB, with an average value of approx. 38 MiB. - -Packet throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -packet throughput of the five test runs seem quite different, ranging from -285.29 kpps to 297.76 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding packet throughput -differ between 250.6k and 398.9k with an average packet throughput between -277.2 K and 318.4 K. In summary, the PPS results seem consistent. Within each -test run of the five runs, when number of flows becomes larger, the packet -throughput seems not larger at the same time. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the five test runs -look similar with each other. Within each test run, the maximum RTT is only 49 -ms and the average RTT is usually approx. 20 ms. On the whole, the average -RTTs of the five runs keep stable and the network latency is relatively short. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool.The largest -cache size is 215 MiB in the four runs, and the smallest cache size is 95 MiB. -On the whole, the average cache size of the five runs change a little and is -about 200 MiB, except the one on Sep. 14th, the mean cache size is very small, -which keeps 102 MiB. Meanwhile, the tread of the buffer size keep flat, since -they have a minimum value of 7 MiB and a maximum value of 8 MiB, with an -average value of about 7.8 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs seem quite different, ranging from 285.29 kpps to 297.76 -kpps. The average number of flows in these tests is 239.7k, and each run has a -minimum number of flows of 2 and a maximum number of flows of 1.001 Mil. At the -same time, the corresponding packet throughput differ between 227.3k and 398.9k -with an average packet throughput between 277.2k and 318.4k. Within each test -run of the five runs, when number of flows becomes larger, the packet -throughput seems not larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs - between 0 ms and 49 ms with an average leatency of less than 22 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the five runs differ from 250.6 kpps to 398.9 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar for four test runs, whose values change a -lot from 10 pps to 399 kpps, except the one on Sep. 14th, whose total number -of transmitted per second keep stable, that is 10 pps. Similarly, the total -number of packets received per second look the same for four runs, except the -one on Sep. 14th, whose value is only 10 pps. - -In some test runs when running with less than approx. 90000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 250000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Intel POD6_ with: -Joid -OpenStack Mitaka -Onos Goldeneye -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. diff --git a/docs/release/results/os-onos-sfc-ha.rst b/docs/release/results/os-onos-sfc-ha.rst deleted file mode 100644 index e52ae3d55..000000000 --- a/docs/release/results/os-onos-sfc-ha.rst +++ /dev/null @@ -1,517 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International -.. License. -.. http://creativecommons.org/licenses/by/4.0 - - -=============================== -Test Results for os-onos-sfc-ha -=============================== - -.. toctree:: - :maxdepth: 2 - - -fuel -==== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Ericsson POD2_ or LF POD2_ between September 5 and 10 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 0.5 and 0.6 ms. -A few runs start with a 1 - 1.5 ms RTT spike (This could be because of normal ARP -handling). One test run has a greater RTT spike of 1.9 ms, which is the same -one with the 0.7 ms average. The other runs have no similar spike at all. -To be able to draw conclusions more runs should be made. -SLA set to 10 ms. The SLA value is used as a reference, it has not -been defined by OPNFV. - -TC005 ------ -The IO read bandwidth looks similar between different dates, with an -average between approx. 170 and 200 MB/s. Within each test run the results -vary, with a minimum 2 MB/s and maximum 838 MB/s on the totality. Most runs -have a minimum BW of 3 MB/s (two runs at 2 MB/s). The maximum BW varies more in -absolute numbers between the dates, between 617 and 838 MB/s. -SLA set to 400 MB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC010 ------ -The measurements for memory latency are similar between test dates and result -in approx. 1.2 ns. The variations within each test run are similar, between -1.215 and 1.219 ns. One exception is February 16, where the average is 1.222 -and varies between 1.22 and 1.28 ns. -SLA set to 30 ns. The SLA value is used as a reference, it has not been defined -by OPNFV. - -TC011 ------ -Packet delay variation between 2 VMs on different blades is measured using -Iperf3. On the first date the reported packet delay variation varies between -0.0025 and 0.011 ms, with an average delay variation of 0.0067 ms. -On the second date the delay variation varies between 0.002 and 0.006 ms, with -an average delay variation of 0.004 ms. - -TC012 ------ -Between test dates, the average measurements for memory bandwidth vary between -17.4 and 17.9 GB/s. Within each test run the results vary more, with a minimal -BW of 16.4 GB/s and maximum of 18.2 GB/s on the totality. -SLA set to 15 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC014 ------ -The Unixbench processor test run results vary between scores 3080 and 3240, -one result each date. The average score on the total is 3150. -No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -CPU utilization statistics are collected during UDP flows sent between the VMs -using pktgen as packet generator tool. The average measurements for CPU -utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio -appears around 7%. - -TC069 ------ -Between test dates, the average measurements for memory bandwidth vary between -15.5 and 25.4 GB/s. Within each test run the results vary more, with a minimal -BW of 9.7 GB/s and maximum of 29.5 GB/s on the totality. -SLA set to 6 GB/s. The SLA value is used as a reference, it has not been -defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Memory utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for memory -utilization vary between 225MB to 246MB. The peak of memory utilization appears -around 340MB. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Cache utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The average measurements for cache -utilization vary between 205MB to 212MB. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs at -approx. 15 ms. Some test runs show an increase with many flows, in the range -towards 16 to 17 ms. One exception standing out is Feb. 15 where the average -RTT is stable at approx. 13 ms. The PPS results are not as consistent as the -RTT results. -In some test runs when running with less than approx. 10000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. Around 20 percent decrease in the worst -case. For the other test runs there is however no significant change to the PPS -throughput when the number of flows are increased. In some test runs the PPS -is also greater with 1000000 flows compared to other test runs where the PPS -result is less with only 2 flows. - -The average PPS throughput in the different runs varies between 414000 and -452000 PPS. The total amount of packets in each test run is approx. 7500000 to -8200000 packets. One test run Feb. 15 sticks out with a PPS average of -558000 and approx. 1100000 packets in total (same as the on mentioned earlier -for RTT results). - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally range between 100 and 1000 per test run, -but there are spikes in the range of 10000 lost packets as well, and even -more in a rare cases. - -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. Total number of packets received per -second was average on 200 kpps and total number of packets transmitted per -second was average on 600 kpps. - -Detailed test results ---------------------- -The scenario was run on Ericsson POD2_ and LF POD2_ with: -Fuel 9.0 -OpenStack Mitaka -Onos Goldeneye -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - -Conclusions and recommendations -------------------------------- -The pktgen test configuration has a relatively large base effect on RTT in -TC037 compared to TC002, where there is no background load at all. Approx. -15 ms compared to approx. 0.5 ms, which is more than a 3000 percentage -difference in RTT results. -Especially RTT and throughput come out with better results than for instance -the *fuel-os-nosdn-nofeature-ha* scenario does. The reason for this should -probably be further analyzed and understood. Also of interest could be -to make further analyzes to find patterns and reasons for lost traffic. -Also of interest could be to see if there are continuous variations where -some test cases stand out with better or worse results than the general test -case. - - -Joid -===== - -.. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main -.. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs - -Overview of test results ------------------------- - -See Grafana_ for viewing test result metrics for each respective test case. It -is possible to chose which specific scenarios to look at, and then to zoom in -on the details of each run test scenario as well. - -All of the test case results below are based on 4 scenario test runs, each run -on the Intel POD6_ between September 8 and 11 in 2016. - -TC002 ------ -The round-trip-time (RTT) between 2 VMs on different blades is measured using -ping. Most test run measurements result on average between 1.35 ms and 1.57 ms. -Only one test run has reached greatest RTT spike of 2.58 ms. Meanwhile, the -smallest network latency is 1.11 ms, which is obtained on Sep. 11st. In -general, the average of network latency of the four test runs are between 1.35 -ms and 1.57 ms. SLA set to be 10 ms. The SLA value is used as a reference, it -has not been defined by OPNFV. - -TC005 ------ -The IO read bandwidth actually refers to the storage throughput, which is -measured by fio and the greatest IO read bandwidth of the four runs is 175.4 -MB/s. The IO read bandwidth of the three runs looks similar, with an average -between 43.7 and 56.3 MB/s, except one on Sep. 8, for its maximum storage -throughput is only 107.9 MB/s. One of the runs has a minimum BW of 478 KM/s and -other has a maximum BW of 168.6 MB/s. The SLA of read bandwidth sets to be -400 MB/s, which is used as a reference, and it has not been defined by OPNFV. - -The results of storage IOPS for the four runs look similar with each other. The -IO read times per second of the four test runs have an average value between -978 per second and 1.20 K/s, and meanwhile, the minimum result is only 36 times -per second. - -TC010 ------ -The tool we use to measure memory read latency is lmbench, which is a series of -micro benchmarks intended to measure basic operating system and hardware system -metrics. The memory read latency of the four runs is between 1.164 ns and 1.244 -ns on average. The variations within each test run are quite different, some -vary from a large range and others have a small change. For example, the -largest change is on September 10, the memory read latency of which is ranging -from 1.128 ns to 1.381 ns. However, the results on September 11 change very -little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has -not been defined by OPNFV. - -TC011 ------ -Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on -different blades. The reported packet delay variations of the four test runs -differ from each other. In general, the packet delay of two runs look similar, -for they both stay stable within each run. And the mean packet delay of them -are 0.0772 ms and 0.0788 ms respectively. Of the four runs, the fourth has the -worst result, because the packet delay reaches 0.0838 ms. The rest one has a -large wide range from 0.0666 ms to 0.0798 ms. The SLA value sets to be 10 ms. -The SLA value is used as a reference, it has not been defined by OPNFV. - -TC012 ------ -Lmbench is also used to measure the memory read and write bandwidth, in which -we use bw_mem to obtain the results. Among the four test runs, the trend of the -memory bandwidth almost look similar, which all have a large wide range, and -the minimum and maximum results are 9.02 GB/s and 18.14 GB/s. Here SLA set to -be 15 GB/s. The SLA value is used as a reference, it has not been defined by -OPNFV. - -TC014 ------ -The Unixbench is used to evaluate the IaaS processing speed with regards to -score of single cpu running and parallel running. It can be seen from the -dashboard that the processing test results vary from scores 3395 to 3475, and -there is only one result one date. No SLA set. - -TC037 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The mean packet throughput of the four test runs is between 362.1 kpps and -363.5 kpps, of which the result of the third run is the highest. The RTT -results of all the test runs keep flat at approx. 17 ms. It is obvious that the -PPS results are not as consistent as the RTT results. - -The No. flows of the four test runs are 240 k on average and the PPS results -look a little waved since the largest packet throughput is 418.1 kpps and the -minimum throughput is 326.5 kpps respectively. - -There are no errors of packets received in the four runs, but there are still -lost packets in all the test runs. The RTT values obtained by ping of the four -runs have the similar average vaue, that is approx. 17 ms, of which the worst -RTT is 39 ms on Sep. 11st. - -CPU load is measured by mpstat, and CPU load of the four test runs seem a -little similar, since the minimum value and the peak of CPU load is between 0 -percent and nine percent respectively. And the best result is obtained on Sep. -10, with an CPU load of nine percent. - -TC069 ------ -With the block size changing from 1 kb to 512 kb, the memory write bandwidth -tends to become larger first and then smaller within every run test, which -rangs from 25.9 GB/s to 26.6 GB/s and then to 18.1 GB/s on average. Since the -test id is one, it is that only the INT memory write bandwidth is tested. On -the whole, when the block size is from 2 kb to 16 kb, the memory write -bandwidth look similar with a minimal BW of 22.1 GB/s and peak value of 28.6 -GB/s. And then with the block size becoming larger, the memory write bandwidth -tends to decrease. SLA sets to be 7 GB/s. The SLA value is used as a reference, -it has not been defined by OPNFV. - -TC070 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other, and within these test runs, the maximum RTT can -reach 39 ms and the average RTT is usually approx. 17 ms. The network latency -tested on Sep. 11 shows that it has a peak latency of 39 ms. But on the whole, -the average RTTs of the five runs keep flat and the network latency is -relatively short. - -Memory utilization is measured by free, which can display amount of free and -used memory in the system. The largest amount of used memory is 270 MiB on the -first two runs. In general, the mean used memory of two test runs have very -large memory utilization, which can reach 264 MiB on average. And the other two -runs have a large wide range of memory usage with the minimum value of 150 MiB -and the maximum value of 270 MiB. On the other hand, for the mean free memory, -the four test runs have the similar trend with that of the mean used memory. -In general, the mean free memory change from 220 MiB to 342 MiB. - -Packet throughput and packet loss can be measured by pktgen, which is a tool -in the network for generating traffic loads for network experiments. The mean -packet throughput of the four test runs seem quite different, ranging from -326.5 kpps to 418.1 kpps. The average number of flows in these tests is -240000, and each run has a minimum number of flows of 2 and a maximum number -of flows of 1.001 Mil. At the same time, the corresponding packet throughput -differ between 326.5 kpps and 418.1 kpps with an average packet throughput between -361.7 kpps and 363.5 kpps. In summary, the PPS results seem consistent. Within each -test run of the four runs, when number of flows becomes larger, the packet -throughput seems not larger at the same time. - -TC071 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The network latency is measured by ping, and the results of the four test runs -look similar with each other. Within each test run, the maximum RTT is only 47 -ms and the average RTT is usually approx. 15 ms. On the whole, the average -RTTs of the four runs keep stable and the network latency is relatively small. - -Cache utilization is measured by cachestat, which can display size of cache and -buffer in the system. Cache utilization statistics are collected during UDP -flows sent between the VMs using pktgen as packet generator tool. The largest -cache size is 214 MiB, which is same for the four runs, and the smallest cache -size is 94 MiB. On the whole, the average cache size of the four runs look the -same and is between 198 MiB and 207 MiB. Meanwhile, the tread of the buffer -size keep flat, since they have a minimum value of 7 MiB and a maximum value of -8 MiB, with an average value of about 7.9 MiB. - -Packet throughput can be measured by pktgen, which is a tool in the network for -generating traffic loads for network experiments. The mean packet throughput of -the four test runs seem quite the same, which is approx. 363 kpps. The average -number of flows in these tests is 240k, and each run has a minimum number of -flows of 2 and a maximum number of flows of 1.001 Mil. At the same time, the -corresponding packet throughput differ between 327 kpps and 418 kpps with an -average packet throughput of about 363 kpps. Within each test run of the four -runs, when number of flows becomes larger, the packet throughput seems not -larger in the meantime. - -TC072 ------ -The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs -on different blades are measured when increasing the amount of UDP flows sent -between the VMs using pktgen as packet generator tool. - -Round trip times and packet throughput between VMs can typically be affected by -the amount of flows set up and result in higher RTT and less PPS throughput. - -The RTT results are similar throughout the different test dates and runs -between 0 ms and 47 ms with an average leatency of less than 16 ms. The PPS -results are not as consistent as the RTT results, for the mean packet -throughput of the four runs differ from 361.7 kpps to 365.0 kpps. - -Network utilization is measured by sar, that is system activity reporter, which -can display the average statistics for the time since the system was started. -Network utilization statistics are collected during UDP flows sent between the -VMs using pktgen as packet generator tool. The largest total number of packets -transmitted per second look similar for two test runs, whose values change a -lot from 10 pps to 432 kpps. While results of the other test runs seem the same -and keep stable with the average number of packets transmitted per second of 10 -pps. However, the total number of packets received per second of the four runs -look similar, which have a large wide range of 2 pps to 657 kpps. - -In some test runs when running with less than approx. 250000 flows the PPS -throughput is normally flatter compared to when running with more flows, after -which the PPS throughput decreases. For the other test runs there is however no -significant change to the PPS throughput when the number of flows are -increased. In some test runs the PPS is also greater with 250000 flows -compared to other test runs where the PPS result is less with only 2 flows. - -There are lost packets reported in most of the test runs. There is no observed -correlation between the amount of flows and the amount of lost packets. -The lost amount of packets normally differs a lot per test run. - -Detailed test results ---------------------- -The scenario was run on Intel POD6_ with: -Joid -OpenStack Mitaka -Onos Goldeneye -OpenVirtualSwitch 2.5.90 -OpenDayLight Beryllium - -Rationale for decisions ------------------------ -Pass - -Conclusions and recommendations -------------------------------- -Tests were successfully executed and metrics collected. -No SLA was verified. To be decided on in next release of OPNFV. - diff --git a/docs/release/results/overview.rst b/docs/release/results/overview.rst index b4a050545..9fd74797c 100644 --- a/docs/release/results/overview.rst +++ b/docs/release/results/overview.rst @@ -42,55 +42,31 @@ environment, features or test framework. The list of scenarios supported by each installer can be described as follows: -+-------------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=========================+=========+=========+=========+=========+ -| os-nosdn-nofeature-noha | | | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-nofeature-ha | X | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-ha | X | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-noha| | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-ha | X | X | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-noha| | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-onos-sfc-ha | X | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-onos-sfc-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-ha | X | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-sfc-ha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-sfc-noha | X | X | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-bgpvpn-ha | X | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-bgpvpn-noha | | X | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-ha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-noha | | X | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-ovs-ha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-ovs-noha | X | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-ocl-nofeature-ha | | | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-lxd-ha | | | | X | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-lxd-noha | | | | X | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-fdio-noha | X | | | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-moon-ha | | X | | | -+-------------------------+---------+---------+---------+---------+ ++-------------------------+------+---------+----------+------+------+-------+ +| Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy | ++=========================+======+=========+==========+======+======+=======+ +| os-nosdn-nofeature-noha | X | | | | X | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-nofeature-ha | X | | X | X | X | X | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-bar-noha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-bar-ha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-bgpvpn-ha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-calipso-noha | X | | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-kvm-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl_l3-nofeature-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-sfc-ha | | X | | | | | ++-------------------------+------+---------+----------+------+------+-------+ +| os-odl-nofeature-ha | | | | X | | X | ++-------------------------+------+---------+----------+------+------+-------+ +| os-nosdn-ovs-ha | | | | X | | | ++-------------------------+------+---------+----------+------+------+-------+ To qualify for release, the scenarios must have deployed and been successfully tested in four consecutive installations to establish stability of deployment @@ -103,4 +79,4 @@ References * IEEE Std 829-2008. "Standard for Software and System Test Documentation". -* OPNFV Colorado release note for Yardstick. +* OPNFV Fraser release note for Yardstick. diff --git a/docs/release/results/results.rst b/docs/release/results/results.rst index 04c6b9f87..c75f5ae94 100644 --- a/docs/release/results/results.rst +++ b/docs/release/results/results.rst @@ -2,13 +2,16 @@ .. License. .. http://creativecommons.org/licenses/by/4.0 -Results listed by scenario +Results listed by test cases ========================== -The following sections describe the yardstick results as evaluated for the -Colorado release scenario validation runs. Each section describes the -determined state of the specific scenario as deployed in the Colorado -release process. +.. _TOM: https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results + + +The following sections describe the yardstick test case results as evaluated +for the OPNFV Fraser release scenario validation runs. Each section describes +the determined state of the specific test case as executed in the Fraser release +process. All test date are analyzed using TOM_ tool. Scenario Results ================ @@ -16,21 +19,22 @@ Scenario Results .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main .. _Jenkins: https://build.opnfv.org/ci/view/yardstick/ + The following documents contain results of Yardstick test cases executed on -OPNFV labs, triggered by OPNFV CI pipeline, documented per scenario. +OPNFV labs, triggered by OPNFV CI pipeline, documented per test case. .. toctree:: :maxdepth: 1 - os-nosdn-nofeature-ha.rst - os-nosdn-nofeature-noha.rst - os-odl_l2-nofeature-ha.rst - os-odl_l2-bgpvpn-ha.rst - os-odl_l2-sfc-ha.rst - os-nosdn-kvm-ha.rst - os-onos-nofeature-ha.rst - os-onos-sfc-ha.rst + tc002-network-latency.rst + tc010-memory-read-latency.rst + tc011-packet-delay-variation.rst + tc012-memory-read-write-bandwidth.rst + tc014-cpu-processing-speed.rst + tc069-memory-write-bandwidth.rst + tc082-context-switches-under-load.rst + tc083-network-throughput-between-vm.rst Test results of executed tests are avilable in Dashboard_ and logs in Jenkins_. diff --git a/docs/release/results/tc002-network-latency.rst b/docs/release/results/tc002-network-latency.rst new file mode 100644 index 000000000..722423473 --- /dev/null +++ b/docs/release/results/tc002-network-latency.rst @@ -0,0 +1,317 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +====================================== +Test results for TC002 network latency +====================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC002 verifies that network latency is within acceptable boundaries when packets travel between hosts located on same or different compute blades. +Ping packets (ICMP protocol's mandatory ECHO_REQUEST datagram) are sent from host VM to target VM(s) to elicit ICMP ECHO_RESPONSE. + +Metric: RTT (Round Trip Time) +Unit: ms + + +Euphrates release +----------------- + +Test results per scenario and pod (lower is better): + +{ + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [0.214], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [0.309], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [0.3145], + + "os-nosdn-ovs-ha:lf-pod2:fuel": [0.3585], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [0.3765], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [0.403], + + "os-odl-sfc-ha:huawei-pod2:compass": [0.413], + + "os-nosdn-ovs-ha:ericsson-pod1:fuel": [0.494], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [0.5715], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [0.5785], + + "os-odl-sfc-ha:lf-pod1:apex": [0.617], + + "os-odl-nofeature-ha:lf-pod1:apex": [0.62], + + "os-nosdn-bar-noha:lf-pod1:apex": [0.632], + + "os-odl-nofeature-noha:lf-pod1:apex": [0.635], + + "os-odl-bgpvpn-ha:lf-pod1:apex": [0.658], + + "os-odl-sfc-noha:lf-pod1:apex": [0.663], + + "os-nosdn-bar-ha:lf-pod1:apex": [0.668], + + "os-ovn-nofeature-noha:lf-pod1:apex": [0.668], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [0.6815], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [0.7005], + + "os-nosdn-bar-ha:huawei-pod2:compass": [0.778], + + "os-nosdn-ovs-noha:ericsson-virtual4:fuel": [0.7825], + + "os-nosdn-ovs-noha:ericsson-virtual2:fuel": [0.7885], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [0.795], + + "os-nosdn-ovs-noha:ericsson-virtual1:fuel": [0.8045], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [0.8335], + + "os-nosdn-ovs-noha:ericsson-virtual3:fuel": [0.8755], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [0.8855], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [0.8895], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [0.901], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [0.956], + + "os-nosdn-lxd-noha:intel-pod5:joid": [1.131], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [1.173], + + "os-odl-sfc-ha:huawei-virtual8:compass": [1.2015], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [1.204], + + "os-nosdn-lxd-ha:intel-pod5:joid": [1.2245], + + "os-odl-nofeature-ha:lf-pod2:fuel": [1.2285], + + "os-odl-sfc-noha:huawei-virtual4:compass": [1.3055], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [1.309], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [1.313], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [1.319], + + "os-odl-nofeature-ha:ericsson-pod1:fuel": [1.3425], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [1.3475], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [1.348], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [1.432], + + "os-odl_l3-nofeature-noha:huawei-virtual9:compass": [1.442], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [1.4505], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [1.497], + + "os-odl-sfc-noha:huawei-virtual3:compass": [1.504], + + "os-odl-nofeature-ha:arm-pod5:fuel": [1.519], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [1.5415], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [1.5785], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [1.604], + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [1.61], + + "os-nosdn-nofeature-noha:intel-pod18:joid": [1.633], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [1.6485], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [1.7085], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [1.71], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [1.7955], + + "os-odl-nofeature-ha:arm-pod6:fuel": [1.838], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [1.88], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [1.8975], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [1.923], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [1.944], + + "os-odl-sfc-ha:huawei-virtual3:compass": [1.968], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [1.986], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [2.0415], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [2.071], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [2.0855], + + "os-odl-sfc-ha:huawei-virtual4:compass": [2.1085], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [2.1135], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [2.234], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [2.294], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [2.304], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [2.378], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [2.397], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [2.472], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [2.603], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [2.635], + + "os-odl-nofeature-noha:ericsson-virtual3:fuel": [2.9055], + + "os-odl-nofeature-noha:ericsson-virtual2:fuel": [3.1295], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [3.337], + + "os-odl-nofeature-noha:ericsson-virtual4:fuel": [3.634], + + "os-nosdn-nofeature-noha:ericsson-virtual1:fuel": [3.875], + + "os-odl-nofeature-noha:ericsson-virtual1:fuel": [3.9655], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [3.9795] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc002_scenario.png + :width: 800px + :alt: TC002 influence of scenario + +{ + + "os-odl_l2-moon-ha": [0.3415], + + "os-nosdn-ovs-ha": [0.3625], + + "os-nosdn-ovs_dpdk-noha": [0.378], + + "os-nosdn-ovs_dpdk-ha": [0.5265], + + "os-nosdn-bar-noha": [0.632], + + "os-odl-bgpvpn-ha": [0.658], + + "os-ovn-nofeature-noha": [0.668], + + "os-odl_l3-nofeature-ha": [0.8545], + + "os-nosdn-ovs-noha": [0.8575], + + "os-nosdn-bar-ha": [0.903], + + "os-odl-sfc-ha": [1.127], + + "os-nosdn-lxd-noha": [1.131], + + "os-nosdn-nofeature-ha": [1.152], + + "os-odl_l2-moon-noha": [1.1825], + + "os-nosdn-lxd-ha": [1.2245], + + "os-odl_l3-nofeature-noha": [1.337], + + "os-odl-nofeature-ha": [1.352], + + "os-odl-sfc-noha": [1.4255], + + "os-nosdn-kvm-noha": [1.5045], + + "os-nosdn-openbaton-ha": [1.5665], + + "os-nosdn-nofeature-noha": [1.729], + + "os-nosdn-kvm-ha": [1.7745], + + "os-odl-nofeature-noha": [3.106] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc002_pod.png + :width: 800px + :alt: TC002 influence of the POD + +{ + + "huawei-pod2": [0.3925], + + "lf-pod2": [0.5315], + + "lf-pod1": [0.62], + + "flex-pod2": [0.795], + + "huawei-pod12": [0.87], + + "intel-pod5": [1.25], + + "ericsson-virtual3": [1.2655], + + "ericsson-pod1": [1.372], + + "arm-pod5": [1.518], + + "huawei-virtual4": [1.5355], + + "huawei-virtual3": [1.606], + + "intel-pod18": [1.6575], + + "huawei-virtual8": [1.709], + + "huawei-virtual2": [1.872], + + "arm-pod6": [1.895], + + "huawei-virtual9": [2.0745], + + "huawei-virtual1": [2.495], + + "ericsson-virtual2": [2.7895], + + "ericsson-virtual4": [3.768], + + "ericsson-virtual1": [3.8035] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc010-memory-read-latency.rst b/docs/release/results/tc010-memory-read-latency.rst new file mode 100644 index 000000000..9a296b7a0 --- /dev/null +++ b/docs/release/results/tc010-memory-read-latency.rst @@ -0,0 +1,299 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +========================================== +Test results for TC010 memory read latency +========================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC010 measures the memory read latency for varying memory sizes and strides. +The test results shown below are for memory size of 16MB. + +Metric: Memory read latency +Unit: ns + + +Euphrates release +----------------- + +Test results per scenario and pod (lower is better): + +{ + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [5.3165], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [5.908], + + "os-nosdn-ovs-noha:ericsson-virtual1:fuel": [6.412], + + "os-nosdn-nofeature-noha:intel-pod18:joid": [6.545], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [6.592], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [6.5975], + + "os-nosdn-ovs-ha:ericsson-pod1:fuel": [6.7675], + + "os-odl-nofeature-ha:ericsson-pod1:fuel": [6.7675], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [6.7945], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [6.839], + + "os-nosdn-ovs-noha:ericsson-virtual4:fuel": [6.9695], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [7.123], + + "os-odl-nofeature-noha:ericsson-virtual4:fuel": [7.289], + + "os-nosdn-ovs-noha:ericsson-virtual2:fuel": [7.4315], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [7.9], + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [8.178], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [8.616], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [8.646], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [8.8615], + + "os-odl-sfc-ha:huawei-pod2:compass": [8.87], + + "os-nosdn-bar-ha:huawei-pod2:compass": [8.877], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [8.892], + + "os-nosdn-ovs-noha:ericsson-virtual3:fuel": [8.898], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [8.952], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [8.9745], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [9.0375], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [9.083], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [9.09], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [9.094], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [9.293], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [9.3525], + + "os-odl-sfc-noha:huawei-virtual4:compass": [9.477], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [9.5445], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [9.5575], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [9.6435], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [9.68], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [9.728], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [9.751], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [9.8645], + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [9.969], + + "os-odl-sfc-noha:huawei-virtual3:compass": [10.029], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [10.088], + + "os-odl-nofeature-noha:ericsson-virtual2:fuel": [10.2985], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [10.318], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [10.3215], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [10.617], + + "os-odl-nofeature-noha:ericsson-virtual3:fuel": [10.762], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [10.7715], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [10.866], + + "os-odl-sfc-ha:huawei-virtual3:compass": [10.871], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [11.1605], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [11.227], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [11.348], + + "os-odl-sfc-ha:huawei-virtual4:compass": [11.453], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [11.571], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [11.5925], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [11.689], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [11.8695], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [12.199], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [12.433], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [12.713], + + "os-nosdn-ovs-ha:lf-pod2:fuel": [15.328], + + "os-odl-nofeature-ha:lf-pod1:apex": [15.4265], + + "os-odl-nofeature-noha:lf-pod1:apex": [15.428], + + "os-ovn-nofeature-noha:lf-pod1:apex": [15.545], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [15.55], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [15.6395], + + "os-odl-sfc-noha:lf-pod1:apex": [15.696], + + "os-odl-sfc-ha:lf-pod1:apex": [15.774], + + "os-nosdn-bar-ha:lf-pod1:apex": [16.6455], + + "os-nosdn-bar-noha:lf-pod1:apex": [16.861], + + "os-odl-nofeature-ha:arm-pod5:fuel": [18.071], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [18.116], + + "os-odl-nofeature-ha:lf-pod2:fuel": [18.8365], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [18.927], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [29.557], + + "os-odl-sfc-ha:huawei-virtual8:compass": [32.492], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [37.623], + + "os-odl-nofeature-ha:arm-pod6:fuel": [41.345], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [42.3795], + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc010_scenario.png + :width: 800px + :alt: TC010 influence of scenario + +{ + + "os-nosdn-ovs-noha": [7.9], + + "os-nosdn-ovs_dpdk-noha": [8.641], + + "os-nosdn-ovs_dpdk-ha": [8.6815], + + "os-nosdn-openbaton-ha": [8.882], + + "os-odl_l2-moon-ha": [8.948], + + "os-odl_l3-nofeature-ha": [8.992], + + "os-nosdn-nofeature-ha": [9.118], + + "os-nosdn-nofeature-noha": [9.174], + + "os-odl_l2-moon-noha": [9.312], + + "os-odl_l3-nofeature-noha": [9.5535], + + "os-odl-nofeature-noha": [9.673], + + "os-odl-sfc-noha": [9.8385], + + "os-odl-sfc-ha": [9.98], + + "os-nosdn-kvm-noha": [10.088], + + "os-nosdn-kvm-ha": [11.1705], + + "os-nosdn-bar-ha": [12.1395], + + "os-nosdn-ovs-ha": [15.3195], + + "os-ovn-nofeature-noha": [15.545], + + "os-odl-nofeature-ha": [16.301], + + "os-nosdn-bar-noha": [16.861] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc010_pod.png + :width: 800px + :alt: TC010 influence of the POD + +{ + + "ericsson-pod1": [5.7785], + + "flex-pod2": [5.908], + + "ericsson-virtual1": [6.412], + + "intel-pod18": [6.5905], + + "intel-pod5": [6.6975], + + "ericsson-virtual4": [7.183], + + "ericsson-virtual2": [8.4985], + + "huawei-pod2": [8.877], + + "huawei-pod12": [9.091], + + "ericsson-virtual3": [9.719], + + "huawei-virtual4": [10.1195], + + "huawei-virtual3": [10.19], + + "huawei-virtual1": [10.3045], + + "huawei-virtual9": [10.318], + + "huawei-virtual2": [11.274], + + "lf-pod1": [15.7025], + + "lf-pod2": [15.8495], + + "arm-pod5": [18.092], + + "huawei-virtual8": [33.999], + + "arm-pod6": [41.5605] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc011-packet-delay-variation.rst b/docs/release/results/tc011-packet-delay-variation.rst new file mode 100644 index 000000000..b07ea8980 --- /dev/null +++ b/docs/release/results/tc011-packet-delay-variation.rst @@ -0,0 +1,262 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +============================================= +Test results for TC011 packet delay variation +============================================= + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC011 measures the packet delay variation sending the packets from one VM to the other. + +Metric: packet delay variation (jitter) +Unit: ms + + +Euphrates release +----------------- + +Test results per scenario and pod (lower is better): + +{ + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [2996], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [2996], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [2996], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [2996], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [2997], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [2997], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [2997], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [2997], + + "os-odl-sfc-ha:huawei-virtual4:compass": [2997], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [2997.5], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [2998], + + "os-odl-sfc-ha:huawei-virtual8:compass": [2998], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [2999], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [2999.5], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [3000], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [3001], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [3002], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [3002], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [3002], + + "os-odl-sfc-ha:huawei-virtual3:compass": [3002], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [3003], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [3003.5], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [3004], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [3004], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [3004.5], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [3005], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [3006], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [3006.5], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [3009], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [3010], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [3010], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [3012], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [3017], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [3017], + + "os-odl-sfc-noha:huawei-virtual4:compass": [3017], + + "os-nosdn-nofeature-noha:intel-pod18:joid": [3018], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [3020], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [3021], + + "os-nosdn-bar-ha:huawei-pod2:compass": [3022], + + "os-nosdn-bar-ha:lf-pod1:apex": [3022], + + "os-nosdn-bar-noha:lf-pod1:apex": [3022], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [3022], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [3022], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [3022], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [3022], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [3022], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [3022], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [3022], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [3022], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [3022], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [3022], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [3022], + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [3022], + + "os-odl-nofeature-ha:arm-pod5:fuel": [3022], + + "os-odl-sfc-ha:huawei-pod2:compass": [3022], + + "os-odl-sfc-ha:lf-pod1:apex": [3022], + + "os-odl-sfc-noha:huawei-virtual3:compass": [3022], + + "os-odl-sfc-noha:lf-pod1:apex": [3022], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [3022], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [3022], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [3022], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [3022], + + "os-ovn-nofeature-noha:lf-pod1:apex": [3022], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [3023], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [3023], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [3024] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc011_scenario.png + :width: 800px + :alt: TC011 influence of scenario + +{ + + "os-nosdn-ovs_dpdk-noha": [2996], + + "os-odl_l3-nofeature-noha": [2997], + + "os-nosdn-kvm-noha": [2999], + + "os-nosdn-ovs_dpdk-ha": [3002], + + "os-nosdn-kvm-ha": [3014.5], + + "os-odl-sfc-noha": [3018], + + "os-nosdn-nofeature-noha": [3020], + + "os-nosdn-openbaton-ha": [3020], + + "os-nosdn-bar-ha": [3022], + + "os-nosdn-bar-noha": [3022], + + "os-nosdn-nofeature-ha": [3022], + + "os-odl-nofeature-ha": [3022], + + "os-odl-sfc-ha": [3022], + + "os-odl_l2-moon-ha": [3022], + + "os-odl_l2-moon-noha": [3022], + + "os-odl_l3-nofeature-ha": [3022], + + "os-ovn-nofeature-noha": [3022] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc011_pod.png + :width: 800px + :alt: TC011 influence of the POD + +{ + + "huawei-virtual2": [2997], + + "flex-pod2": [2997.5], + + "huawei-virtual3": [2998], + + "huawei-virtual9": [3000], + + "huawei-virtual8": [3001], + + "huawei-virtual4": [3002], + + "ericsson-virtual3": [3006], + + "huawei-virtual1": [3007], + + "ericsson-virtual2": [3009], + + "intel-pod18": [3010], + + "ericsson-virtual4": [3017], + + "lf-pod2": [3021], + + "arm-pod5": [3022], + + "arm-pod6": [3022], + + "ericsson-pod1": [3022], + + "huawei-pod12": [3022], + + "huawei-pod2": [3022], + + "intel-pod5": [3022], + + "lf-pod1": [3022] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc012-memory-read-write-bandwidth.rst b/docs/release/results/tc012-memory-read-write-bandwidth.rst new file mode 100644 index 000000000..c28eb1f3c --- /dev/null +++ b/docs/release/results/tc012-memory-read-write-bandwidth.rst @@ -0,0 +1,299 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +================================================== +Test results for TC012 memory read/write bandwidth +================================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC012 measures the rate at which data can be read from and written to the memory (this includes all levels of memory). +In this test case, the bandwidth to read data from memory and then write data to the same memory location are measured. + +Metric: memory bandwidth +Unit: MBps + + +Euphrates release +----------------- + +Test results per scenario and pod (higher is better): + +{ + + "os-nosdn-nofeature-ha:lf-pod1:apex": [23126.325], + + "os-odl-nofeature-noha:lf-pod1:apex": [23123.975], + + "os-odl-nofeature-ha:lf-pod1:apex": [23068.965], + + "os-odl-nofeature-ha:lf-pod2:fuel": [22972.46], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [22912.015], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [22911.35], + + "os-ovn-nofeature-noha:lf-pod1:apex": [22900.93], + + "os-nosdn-bar-ha:lf-pod1:apex": [22767.56], + + "os-nosdn-bar-noha:lf-pod1:apex": [22721.83], + + "os-odl-sfc-noha:lf-pod1:apex": [22511.565], + + "os-nosdn-ovs-ha:lf-pod2:fuel": [22071.235], + + "os-odl-sfc-ha:lf-pod1:apex": [21646.415], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [20229.99], + + "os-nosdn-ovs-noha:ericsson-virtual4:fuel": [17491.18], + + "os-nosdn-ovs-noha:ericsson-virtual1:fuel": [17474.965], + + "os-nosdn-ovs-ha:ericsson-pod1:fuel": [17141.375], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [17134.99], + + "os-odl-nofeature-ha:ericsson-pod1:fuel": [17124.27], + + "os-nosdn-ovs-noha:ericsson-virtual2:fuel": [16599.325], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [16309.13], + + "os-odl-nofeature-noha:ericsson-virtual4:fuel": [16137.48], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [15960.76], + + "os-nosdn-ovs-noha:ericsson-virtual3:fuel": [15685.505], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [15536.65], + + "os-odl-nofeature-noha:ericsson-virtual3:fuel": [15431.795], + + "os-odl-nofeature-noha:ericsson-virtual2:fuel": [15129.27], + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [15125.51], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [15030.65], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [15019.89], + + "os-odl-sfc-ha:huawei-pod2:compass": [15005.11], + + "os-nosdn-bar-ha:huawei-pod2:compass": [14975.645], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [14968.97], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [14968.97], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [14741.425], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [14714.28], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [14674.38], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [14664.12], + + "os-odl-sfc-noha:huawei-virtual4:compass": [14587.62], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [14539.94], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [14534.54], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [14511.925], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [14496.875], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [14378.87], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [14366.69], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [14356.695], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [14341.605], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [14327.78], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [14313.81], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [14284.365], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [14157.99], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [14144.86], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [14138.9], + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [14117.7], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [14097.255], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [14085.675], + + "os-odl-sfc-noha:huawei-virtual3:compass": [14071.605], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [14059.51], + + "os-odl-sfc-ha:huawei-virtual4:compass": [14057.155], + + "os-odl-sfc-ha:huawei-virtual3:compass": [14051.945], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [14020.74], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [14017.915], + + "os-nosdn-nofeature-noha:intel-pod18:joid": [13954.27], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [13915.87], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [13874.59], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [13812.215], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [13777.59], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [13765.36], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [13559.905], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [13477.52], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [13255.17], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [13189.64], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [12718.545], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [12559.445], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [12409.66], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [8832.515], + + "os-odl-sfc-ha:huawei-virtual8:compass": [8823.955], + + "os-odl-nofeature-ha:arm-pod5:fuel": [4398.08], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [4375.75], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [4260.77], + + "os-odl-nofeature-ha:arm-pod6:fuel": [4259.62] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc012_scenario.png + :width: 800px + :alt: TC012 influence of scenario + +{ + + "os-ovn-nofeature-noha": [22900.93], + + "os-nosdn-bar-noha": [22721.83], + + "os-nosdn-ovs-ha": [22063.67], + + "os-odl-nofeature-ha": [17146.05], + + "os-odl-nofeature-noha": [16017.41], + + "os-nosdn-ovs-noha": [16005.74], + + "os-nosdn-nofeature-noha": [15290.94], + + "os-nosdn-nofeature-ha": [15038.74], + + "os-nosdn-bar-ha": [14972.975], + + "os-odl_l2-moon-ha": [14956.955], + + "os-odl_l3-nofeature-ha": [14839.21], + + "os-odl-sfc-ha": [14823.48], + + "os-nosdn-ovs_dpdk-ha": [14822.17], + + "os-nosdn-ovs_dpdk-noha": [14725.9], + + "os-odl_l2-moon-noha": [14665.4], + + "os-odl_l3-nofeature-noha": [14483.09], + + "os-odl-sfc-noha": [14373.21], + + "os-nosdn-openbaton-ha": [14135.325], + + "os-nosdn-kvm-noha": [14020.26], + + "os-nosdn-kvm-ha": [13996.02] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc012_pod.png + :width: 800px + :alt: TC012 influence of the POD + +{ + + "lf-pod1": [22912.39], + + "lf-pod2": [22637.67], + + "flex-pod2": [20229.99], + + "ericsson-virtual1": [17474.965], + + "ericsson-pod1": [17127.38], + + "ericsson-virtual4": [16219.97], + + "ericsson-virtual2": [15652.28], + + "ericsson-virtual3": [15551.26], + + "huawei-pod2": [15017.2], + + "huawei-virtual4": [14266.34], + + "huawei-virtual1": [14233.035], + + "huawei-virtual3": [14227.63], + + "huawei-pod12": [14147.245], + + "intel-pod18": [14058.33], + + "huawei-virtual2": [13862.85], + + "intel-pod5": [13280.32], + + "huawei-virtual9": [12559.445], + + "huawei-virtual8": [8998.02], + + "arm-pod5": [4388.875], + + "arm-pod6": [4260.2] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc014-cpu-processing-speed.rst b/docs/release/results/tc014-cpu-processing-speed.rst new file mode 100644 index 000000000..34d4ad0f9 --- /dev/null +++ b/docs/release/results/tc014-cpu-processing-speed.rst @@ -0,0 +1,298 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +=========================================== +Test results for TC014 cpu processing speed +=========================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC014 measures score of single cpu running using UnixBench. + +Metric: score of single CPU running +Unit: N/A + + +Euphrates release +----------------- + +Test results per scenario and pod (higher is better): + +{ + + "os-odl-sfc-noha:lf-pod1:apex": [3735.2], + + "os-nosdn-ovs-ha:lf-pod2:fuel": [3725.5], + + "os-odl-nofeature-ha:lf-pod2:fuel": [3711], + + "os-odl-nofeature-ha:lf-pod1:apex": [3708.4], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [3705.7], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [3704], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [3703.2], + + "os-odl-nofeature-noha:lf-pod1:apex": [3702.8], + + "os-odl-sfc-ha:lf-pod1:apex": [3698.7], + + "os-ovn-nofeature-noha:lf-pod1:apex": [3654.8], + + "os-nosdn-bar-ha:lf-pod1:apex": [3635.55], + + "os-nosdn-bar-noha:lf-pod1:apex": [3633.2], + + "os-nosdn-nofeature-noha:intel-pod18:joid": [3450.3], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [3406.4], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [3360.4], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [3340.65], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [3208.6], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [3134.8], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [3056.2], + + "os-nosdn-ovs-noha:ericsson-virtual1:fuel": [2988.9], + + "os-nosdn-ovs-ha:ericsson-pod1:fuel": [2773.7], + + "os-nosdn-ovs-noha:ericsson-virtual4:fuel": [2645.85], + + "os-nosdn-ovs-noha:ericsson-virtual2:fuel": [2625.3], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [2601.3], + + "os-odl-nofeature-noha:ericsson-virtual4:fuel": [2590.4], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [2570.2], + + "os-nosdn-ovs-noha:ericsson-virtual3:fuel": [2558.8], + + "os-odl-nofeature-ha:ericsson-pod1:fuel": [2556.5], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [2554.6], + + "os-odl-nofeature-noha:ericsson-virtual3:fuel": [2536.75], + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [2533.55], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [2531.85], + + "os-odl-sfc-ha:huawei-pod2:compass": [2531.7], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [2531.2], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [2531], + + "os-nosdn-bar-ha:huawei-pod2:compass": [2529.6], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [2520.5], + + "os-odl-nofeature-noha:ericsson-virtual2:fuel": [2481.15], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [2474], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [2472.6], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [2471], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [2470.6], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [2464.15], + + "os-odl-sfc-noha:huawei-virtual4:compass": [2455.9], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [2455.3], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [2446.85], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [2444.75], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [2430.9], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [2421.3], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [2415.7], + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [2399.4], + + "os-odl-sfc-ha:huawei-virtual3:compass": [2391.85], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [2391.45], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [2380.7], + + "os-odl-sfc-ha:huawei-virtual4:compass": [2379.6], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [2371.9], + + "os-odl-sfc-noha:huawei-virtual3:compass": [2364.6], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [2363.4], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [2362], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [2358.5], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [2358.45], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [2336], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [2326.6], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [2324.95], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [2320.2], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [2318.5], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [2312.8], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [2311.7], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [2301.15], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [2297.7], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [2232.8], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [2232.1], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [2230], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [2154], + + "os-odl-sfc-ha:huawei-virtual8:compass": [2150.1], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [2004.3], + + "os-odl-nofeature-ha:arm-pod5:fuel": [1754.5], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [1754.15], + + "os-odl-nofeature-ha:arm-pod6:fuel": [716.15], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [716.05] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc014_scenario.png + :width: 800px + :alt: TC014 influence of scenario + +{ + + "os-nosdn-ovs-ha": [3725.5], + + "os-ovn-nofeature-noha": [3654.8], + + "os-nosdn-bar-noha": [3633.2], + + "os-odl-nofeature-ha": [3407.8], + + "os-nosdn-ovs-noha": [2583.2], + + "os-odl-nofeature-noha": [2578.9], + + "os-nosdn-nofeature-noha": [2553.2], + + "os-nosdn-nofeature-ha": [2532.8], + + "os-odl_l2-moon-ha": [2530.5], + + "os-nosdn-bar-ha": [2527], + + "os-odl_l3-nofeature-ha": [2501.5], + + "os-nosdn-ovs_dpdk-noha": [2473.65], + + "os-odl-sfc-ha": [2472.9], + + "os-odl_l2-moon-noha": [2470.8], + + "os-nosdn-ovs_dpdk-ha": [2461.9], + + "os-odl_l3-nofeature-noha": [2442.8], + + "os-nosdn-kvm-noha": [2392.9], + + "os-odl-sfc-noha": [2370.5], + + "os-nosdn-kvm-ha": [2358.5], + + "os-nosdn-openbaton-ha": [2231.8] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc014_pod.png + :width: 800px + :alt: TC014 influence of the POD + +{ + + "lf-pod2": [3723.95], + + "lf-pod1": [3669], + + "intel-pod5": [3388.6], + + "intel-pod18": [3298.4], + + "flex-pod2": [3208.6], + + "ericsson-virtual1": [2988.9], + + "ericsson-pod1": [2669.1], + + "ericsson-virtual4": [2598.5], + + "ericsson-virtual3": [2553.15], + + "huawei-pod2": [2531.2], + + "ericsson-virtual2": [2526.9], + + "huawei-virtual4": [2407.4], + + "huawei-virtual3": [2374.6], + + "huawei-virtual2": [2326.4], + + "huawei-virtual9": [2324.95], + + "huawei-virtual1": [2302.6], + + "huawei-pod12": [2232.2], + + "huawei-virtual8": [2085.3], + + "arm-pod5": [1754.4], + + "arm-pod6": [716.15] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc069-memory-write-bandwidth.rst b/docs/release/results/tc069-memory-write-bandwidth.rst new file mode 100644 index 000000000..06e2ec922 --- /dev/null +++ b/docs/release/results/tc069-memory-write-bandwidth.rst @@ -0,0 +1,300 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +============================================= +Test results for TC069 memory write bandwidth +============================================= + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC069 measures the maximum possible cache and memory performance while reading and writing certain +blocks of data (starting from 1Kb and further in power of 2) continuously through ALU and FPU +respectively. Measure different aspects of memory performance via synthetic simulations. +Each simulation consists of four performances (Copy, Scale, Add, Triad). +The test results shown below are for writing 32MB integer block size. + +Metric: memory write bandwidth +Unit: MBps + + +Euphrates release +----------------- + +Test results per scenario and pod (higher is better): + +{ + + "os-nosdn-nofeature-noha:intel-pod18:joid": [20113.395], + + "os-nosdn-openbaton-ha:intel-pod18:joid": [19183.58], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [17851.35], + + "os-nosdn-nofeature-noha:intel-pod5:joid": [16312.37], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [15633.245], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [13332.065], + + "os-odl-nofeature-ha:arm-pod6:fuel": [13327.02], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [9462.74], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [9384.585], + + "os-odl-nofeature-ha:ericsson-pod1:fuel": [9235.98], + + "os-nosdn-nofeature-noha:huawei-pod12:joid": [9213.6], + + "os-nosdn-openbaton-ha:huawei-pod12:joid": [9152.18], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [9079.45], + + "os-odl_l2-moon-ha:huawei-pod2:compass": [9071.13], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [9068.06], + + "os-odl-sfc-ha:huawei-pod2:compass": [9031.24], + + "os-odl_l3-nofeature-ha:huawei-pod2:compass": [9019.53], + + "os-nosdn-bar-ha:huawei-pod2:compass": [8977.3], + + "os-nosdn-ovs_dpdk-ha:huawei-pod2:compass": [8960.635], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [8825.805], + + "os-nosdn-kvm-ha:huawei-pod2:compass": [8282.75], + + "os-odl_l2-moon-noha:huawei-virtual4:compass": [8116.33], + + "os-nosdn-ovs-noha:ericsson-virtual4:fuel": [8083.97], + + "os-odl_l2-moon-noha:huawei-virtual3:compass": [8083.52], + + "os-nosdn-nofeature-noha:huawei-virtual3:compass": [7799.145], + + "os-odl_l3-nofeature-noha:huawei-virtual3:compass": [7776.12], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual3:compass": [7680.37], + + "os-nosdn-ovs-noha:ericsson-virtual1:fuel": [7615.97], + + "os-nosdn-nofeature-noha:ericsson-virtual4:fuel": [7612.62], + + "os-nosdn-nofeature-noha:ericsson-virtual2:fuel": [7518.62], + + "os-nosdn-nofeature-noha:huawei-virtual2:compass": [7489.67], + + "os-nosdn-ovs-noha:ericsson-virtual2:fuel": [7478.57], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual4:compass": [7465.82], + + "os-nosdn-kvm-noha:huawei-virtual3:compass": [7443.16], + + "os-odl-nofeature-noha:ericsson-virtual4:fuel": [7442.855], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [7440.65], + + "os-odl-sfc-noha:huawei-virtual4:compass": [7401.16], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [7389.505], + + "os-odl-nofeature-ha:arm-pod5:fuel": [7385.76], + + "os-nosdn-nofeature-noha:huawei-virtual1:compass": [7382.345], + + "os-odl_l2-moon-ha:huawei-virtual3:compass": [7286.385], + + "os-odl_l3-nofeature-noha:huawei-virtual4:compass": [7272.06], + + "os-odl_l3-nofeature-ha:huawei-virtual4:compass": [7261.73], + + "os-nosdn-nofeature-noha:ericsson-virtual3:fuel": [7253.64], + + "os-odl-sfc-noha:huawei-virtual3:compass": [7247.89], + + "os-odl_l3-nofeature-ha:huawei-virtual2:compass": [7214.01], + + "os-nosdn-ovs_dpdk-ha:huawei-virtual3:compass": [7207.39], + + "os-nosdn-ovs_dpdk-noha:huawei-virtual4:compass": [7205.565], + + "os-nosdn-ovs-noha:ericsson-virtual3:fuel": [7201.005], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [7132.835], + + "os-odl-nofeature-noha:ericsson-virtual3:fuel": [7117.05], + + "os-odl_l3-nofeature-ha:huawei-virtual3:compass": [7064.18], + + "os-odl_l2-moon-ha:huawei-virtual4:compass": [6997.295], + + "os-odl-nofeature-ha:lf-pod1:apex": [6992.21], + + "os-odl-sfc-ha:huawei-virtual4:compass": [6975.63], + + "os-odl-nofeature-noha:lf-pod1:apex": [6972.63], + + "os-nosdn-nofeature-noha:lf-pod1:apex": [6955], + + "os-ovn-nofeature-noha:lf-pod1:apex": [6954.5], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [6953.35], + + "os-odl-sfc-noha:lf-pod1:apex": [6951.89], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [6932.29], + + "os-nosdn-nofeature-noha:huawei-virtual4:compass": [6929.54], + + "os-nosdn-kvm-noha:huawei-virtual4:compass": [6921.6], + + "os-nosdn-ovs-ha:lf-pod2:fuel": [6913.355], + + "os-odl-nofeature-ha:lf-pod2:fuel": [6848.58], + + "os-odl-sfc-ha:lf-pod1:apex": [6818.74], + + "os-nosdn-bar-noha:lf-pod1:apex": [6812.16], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [6808.18], + + "os-odl-nofeature-noha:ericsson-virtual2:fuel": [6807.565], + + "os-nosdn-bar-ha:lf-pod1:apex": [6774.76], + + "os-nosdn-bar-ha:huawei-virtual4:compass": [6759.4], + + "os-nosdn-nofeature-noha:huawei-virtual8:compass": [6756.9], + + "os-nosdn-bar-ha:huawei-virtual3:compass": [6543.46], + + "os-nosdn-kvm-ha:huawei-virtual3:compass": [6504.34], + + "os-odl-sfc-ha:huawei-virtual3:compass": [6481.005], + + "os-nosdn-kvm-ha:huawei-virtual4:compass": [6461.5], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [6152.375], + + "os-odl-sfc-ha:huawei-virtual8:compass": [5941.7], + + "os-nosdn-kvm-noha:huawei-virtual8:compass": [4564.515] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc069_scenario.png + :width: 800px + :alt: TC069 influence of scenario + +{ + + "os-nosdn-openbaton-ha": [9187.16], + + "os-odl_l2-moon-ha": [9010.57], + + "os-nosdn-nofeature-ha": [8886.75], + + "os-odl_l3-nofeature-ha": [8779.67], + + "os-odl_l2-moon-noha": [8114.995], + + "os-nosdn-ovs_dpdk-ha": [7864.07], + + "os-odl_l3-nofeature-noha": [7632.11], + + "os-odl-sfc-ha": [7624.67], + + "os-nosdn-nofeature-noha": [7470.66], + + "os-odl-nofeature-ha": [7372.23], + + "os-nosdn-ovs_dpdk-noha": [7311.54], + + "os-odl-sfc-noha": [7300.56], + + "os-nosdn-ovs-noha": [7280.005], + + "os-odl-nofeature-noha": [7162.67], + + "os-nosdn-kvm-ha": [7130.775], + + "os-nosdn-kvm-noha": [7041.13], + + "os-ovn-nofeature-noha": [6954.5], + + "os-nosdn-ovs-ha": [6913.355], + + "os-nosdn-bar-ha": [6829.17], + + "os-nosdn-bar-noha": [6812.16] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc069_pod.png + :width: 800px + :alt: TC069 influence of the POD + +{ + + "intel-pod18": [18871.79], + + "intel-pod5": [16055.79], + + "arm-pod6": [13327.02], + + "flex-pod2": [9384.585], + + "ericsson-pod1": [9331.535], + + "huawei-pod12": [9164.88], + + "huawei-pod2": [9026.52], + + "huawei-virtual9": [8825.805], + + "ericsson-virtual1": [7615.97], + + "ericsson-virtual4": [7539.23], + + "arm-pod5": [7403.38], + + "huawei-virtual3": [7247.89], + + "huawei-virtual2": [7205.35], + + "huawei-virtual1": [7196.405], + + "ericsson-virtual3": [7173.72], + + "huawei-virtual4": [7131.47], + + "ericsson-virtual2": [7129.08], + + "lf-pod1": [6928.18], + + "lf-pod2": [6875.88], + + "huawei-virtual8": [5729.705] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc082-context-switches-under-load.rst b/docs/release/results/tc082-context-switches-under-load.rst new file mode 100644 index 000000000..d8a9f5493 --- /dev/null +++ b/docs/release/results/tc082-context-switches-under-load.rst @@ -0,0 +1,129 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +================================================== +Test results for TC082 context switches under load +================================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC082 measures various software performance events using perf. +The test results shown below are for context-switches. + +Metric: context switches +Unit: N/A + + +Euphrates release +----------------- + +Test results per scenario and pod (lower is better): + +{ + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [316], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [340], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [357.5], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [384], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [394.5], + + "os-nosdn-nofeature-ha:lf-pod1:apex": [435], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [476], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [518], + + "os-odl-nofeature-ha:arm-pod5:fuel": [863], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [871], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [1002], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [1174], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [1239], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [1430], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [1489], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [1883.5] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc082_scenario.png + :width: 800px + :alt: TC082 influence of scenario + +the influence of the scenario + +{ + + "os-nosdn-nofeature-ha": [505], + + "os-odl-nofeature-ha": [863] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc082_pod.png + :width: 800px + :alt: TC082 influence of the POD + +the influence of the POD + +{ + + "huawei-pod12": [316], + + "intel-pod18": [340], + + "intel-pod5": [357.5], + + "ericsson-pod1": [384], + + "lf-pod2": [394.5], + + "lf-pod1": [435], + + "flex-pod2": [476], + + "huawei-pod2": [518], + + "arm-pod5": [869.5], + + "huawei-virtual9": [1002], + + "huawei-virtual4": [1174], + + "huawei-virtual3": [1239], + + "huawei-virtual2": [1430], + + "huawei-virtual1": [1489], + + "arm-pod6": [1883.5] + +} + + +Fraser release +-------------- diff --git a/docs/release/results/tc083-network-throughput-between-vm.rst b/docs/release/results/tc083-network-throughput-between-vm.rst new file mode 100644 index 000000000..f846571a5 --- /dev/null +++ b/docs/release/results/tc083-network-throughput-between-vm.rst @@ -0,0 +1,129 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 + + +===================================================== +Test results for TC083 network throughput between VMs +===================================================== + +.. toctree:: + :maxdepth: 2 + + +Overview of test case +===================== + +TC083 measures network latency and throughput between VMs using netperf. +The test results shown below are for UDP throughout. + +Metric: UDP stream throughput +Unit: 10^6bits/s + + +Euphrates release +----------------- + +Test results per scenario and pod (higher is better): + +{ + + "os-nosdn-nofeature-ha:lf-pod1:apex": [2204.42], + + "os-nosdn-nofeature-ha:intel-pod18:joid": [1835.55], + + "os-nosdn-nofeature-ha:lf-pod2:fuel": [1676.705], + + "os-nosdn-nofeature-ha:intel-pod5:joid": [1612.555], + + "os-nosdn-nofeature-ha:flex-pod2:apex": [1370.23], + + "os-nosdn-nofeature-ha:huawei-pod12:joid": [1300.12], + + "os-nosdn-nofeature-ha:huawei-pod2:compass": [1070.455], + + "os-nosdn-nofeature-ha:ericsson-pod1:fuel": [1004.32], + + "os-nosdn-nofeature-ha:huawei-virtual9:compass": [753.46], + + "os-nosdn-nofeature-ha:huawei-virtual4:compass": [735.07], + + "os-odl-nofeature-ha:arm-pod5:fuel": [531.63], + + "os-nosdn-nofeature-ha:huawei-virtual3:compass": [493.985], + + "os-nosdn-nofeature-ha:arm-pod5:fuel": [448.82], + + "os-nosdn-nofeature-ha:arm-pod6:fuel": [193.43], + + "os-nosdn-nofeature-ha:huawei-virtual1:compass": [189.99], + + "os-nosdn-nofeature-ha:huawei-virtual2:compass": [80.15] + +} + + +The influence of the scenario +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc083_scenario.png + :width: 800px + :alt: TC083 influence of scenario + +the influence of the scenario + +{ + + "os-nosdn-nofeature-ha": [1109.12], + + "os-odl-nofeature-ha": [531.63] + +} + + +The influence of the POD +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: images/tc083_pod.png + :width: 800px + :alt: TC083 influence of the POD + +the influence of the POD + +{ + + "lf-pod1": [2204.42], + + "intel-pod18": [1835.55], + + "lf-pod2": [1676.705], + + "intel-pod5": [1612.555], + + "flex-pod2": [1370.23], + + "huawei-pod12": [1300.12], + + "huawei-pod2": [1070.455], + + "ericsson-pod1": [1004.32], + + "huawei-virtual9": [753.46], + + "huawei-virtual4": [735.07], + + "huawei-virtual3": [493.985], + + "arm-pod5": [451.38], + + "arm-pod6": [193.43], + + "huawei-virtual1": [189.99], + + "huawei-virtual2": [80.15] + +} + + +Fraser release +-------------- diff --git a/docs/testing/developer/devguide/devguide.rst b/docs/testing/developer/devguide/devguide.rst index dade49b75..04d5350be 100755 --- a/docs/testing/developer/devguide/devguide.rst +++ b/docs/testing/developer/devguide/devguide.rst @@ -432,7 +432,8 @@ Yardstick committers and contributors to review your codes. :width: 800px :alt: Gerrit for code review -You can find Yardstick people info `here <https://wiki.opnfv.org/display/yardstick/People>`_. +You can find a list Yardstick people `here <https://wiki.opnfv.org/display/yardstick/People>`_, +or use the ``yardstick-reviewers`` and ``yardstick-committers`` groups in gerrit. Modify the code under review in Gerrit ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/testing/developer/devguide/devguide_nsb_prox.rst b/docs/testing/developer/devguide/devguide_nsb_prox.rst index fc533b2cf..22628413b 100755 --- a/docs/testing/developer/devguide/devguide_nsb_prox.rst +++ b/docs/testing/developer/devguide/devguide_nsb_prox.rst @@ -251,9 +251,11 @@ Now let's examine the components of the file in detail In this case it is ``handle_l2fwd-2.cfg`` A number of additional parameters can be added. This example - is taken from VPE:: + is for VPE:: options: + interface_speed_gbps: 10 + vnf__0: prox_path: /opt/nsb_bin/prox prox_config: ``configs/handle_vpe-4.cfg`` @@ -267,6 +269,12 @@ Now let's examine the components of the file in detail ``configs/vpe_rules.lua`` : ```` prox_generate_parameter: True + ``interface_speed_gbps`` - this specifies the speed of the interface + in Gigabits Per Second. This is used to calculate pps(packets per second). + If the interfaces are of different speeds, then this specifies the speed + of the slowest interface. This parameter is optional. If omitted the + interface speed defaults to 10Gbps. + ``prox_files`` - this specified that a number of addition files need to be provided for the test to run correctly. This files could provide routing information,hashing information or a diff --git a/docs/testing/user/userguide/01-introduction.rst b/docs/testing/user/userguide/01-introduction.rst index c1d5def98..d846e759c 100755 --- a/docs/testing/user/userguide/01-introduction.rst +++ b/docs/testing/user/userguide/01-introduction.rst @@ -42,43 +42,47 @@ This document consists of the following chapters: * Chapter :doc:`02-methodology` describes the methodology implemented by the *Yardstick* Project for :term:`NFVI` verification. -* Chapter :doc:`03-architecture` provides information on the software architecture - of *Yardstick*. +* Chapter :doc:`03-architecture` provides information on the software + architecture of *Yardstick*. * Chapter :doc:`04-installation` provides instructions to install *Yardstick*. -* Chapter :doc:`05-yardstick_plugin` provides information on how to integrate +* Chapter :doc:`05-operation` provides information on how to use *Yardstick* + to run and create testcases. + +* Chapter :doc:`06-yardstick-plugin` provides information on how to integrate other OPNFV testing projects into *Yardstick*. -* Chapter :doc:`06-result-store-InfluxDB` provides inforamtion on how to run +* Chapter :doc:`07-result-store-InfluxDB` provides inforamtion on how to run plug-in test cases and store test results into community's InfluxDB. -* Chapter :doc:`07-grafana` provides inforamtion on *Yardstick* grafana dashboard - and how to add a dashboard into *Yardstick* grafana dashboard. +* Chapter :doc:`08-grafana` provides inforamtion on *Yardstick* grafana + dashboard and how to add a dashboard into *Yardstick* grafana dashboard. -* Chapter :doc:`08-api` provides inforamtion on *Yardstick* ReST API and how to +* Chapter :doc:`09-api` provides inforamtion on *Yardstick* ReST API and how to use *Yardstick* API. -* Chapter :doc:`09-yardstick_user_interface` provides inforamtion on how to use +* Chapter :doc:`10-yardstick-user-interface` provides inforamtion on how to use yardstick report CLI to view the test result in table format and also values pinned on to a graph -* Chapter :doc:`10-vtc-overview` provides information on the :term:`VTC`. +* Chapter :doc:`11-vtc-overview` provides information on the :term:`VTC`. -* Chapter :doc:`13-nsb-overview` describes the methodology implemented by the +* Chapter :doc:`12-nsb-overview` describes the methodology implemented by the Yardstick - Network service benchmarking to test real world usecase for a given VNF. -* Chapter :doc:`14-nsb_installation` provides instructions to install - *Yardstick - Network service benchmarking testing*. +* Chapter :doc:`13-nsb_installation` provides instructions to install + *Yardstick - Network Service Benchmarking (NSB) testing*. + +* Chapter :doc:`14-nsb-operation` provides information on running *NSB* * Chapter :doc:`15-list-of-tcs` includes a list of available *Yardstick* test cases. - Contact Yardstick ================= Feedback? `Contact us`_ -.. _Contact us: opnfv-users@lists.opnfv.org +.. _Contact us: mailto:opnfv-users@lists.opnfv.org&subject="[yardstick]" diff --git a/docs/testing/user/userguide/03-architecture.rst b/docs/testing/user/userguide/03-architecture.rst index 8336b609d..622002ee4 100755 --- a/docs/testing/user/userguide/03-architecture.rst +++ b/docs/testing/user/userguide/03-architecture.rst @@ -9,8 +9,9 @@ Architecture Abstract ======== -This chapter describes the yardstick framework software architecture. we will introduce it from Use-Case View, -Logical View, Process View and Deployment View. More technical details will be introduced in this chapter. +This chapter describes the yardstick framework software architecture. We will +introduce it from Use-Case View, Logical View, Process View and Deployment +View. More technical details will be introduced in this chapter. Overview ======== @@ -23,8 +24,8 @@ files. Yardstick is inspired by Rally. Yardstick is intended to run on a computer with access and credentials to a cloud. The test case is described in a configuration file given as an argument. -How it works: the benchmark task configuration file is parsed and converted into -an internal model. The context part of the model is converted into a Heat +How it works: the benchmark task configuration file is parsed and converted +into an internal model. The context part of the model is converted into a Heat template and deployed into a stack. Each scenario is run using a runner, either serially or in parallel. Each runner runs in its own subprocess executing commands in a VM using SSH. The output of each scenario is written as json @@ -43,13 +44,15 @@ names, image names, affinity rules and network configurations. A context is converted into a simplified Heat template, which is used to deploy onto the Openstack environment. -**Data** - Output produced by running a benchmark, written to a file in json format +**Data** - Output produced by running a benchmark, written to a file in json +format **Runner** - Logic that determines how a test scenario is run and reported, for example the number of test iterations, input value stepping and test duration. Predefined runner types exist for re-usage, see `Runner types`_. -**Scenario** - Type/class of measurement for example Ping, Pktgen, (Iperf, LmBench, ...) +**Scenario** - Type/class of measurement for example Ping, Pktgen, (Iperf, +LmBench, ...) **SLA** - Relates to what result boundary a test case must meet to pass. For example a latency limit, amount or ratio of lost packets and so on. Action @@ -128,8 +131,8 @@ Snippet of an Iteration runner configuration: Use-Case View ============= Yardstick Use-Case View shows two kinds of users. One is the Tester who will -do testing in cloud, the other is the User who is more concerned with test result -and result analyses. +do testing in cloud, the other is the User who is more concerned with test +result and result analyses. For testers, they will run a single test case or test case suite to verify infrastructure compliance or bencnmark their own infrastructure performance. @@ -254,7 +257,8 @@ Yardstick Directory structure *tools/* - Currently contains tools to build image for VMs which are deployed by Heat. Currently contains how to build the yardstick-trusty-server - image with the different tools that are needed from within the image. + image with the different tools that are needed from within the + image. *plugin/* - Plug-in configuration files are stored here. diff --git a/docs/testing/user/userguide/04-installation.rst b/docs/testing/user/userguide/04-installation.rst index cac814667..a4846230e 100644 --- a/docs/testing/user/userguide/04-installation.rst +++ b/docs/testing/user/userguide/04-installation.rst @@ -39,18 +39,18 @@ Several prerequisites are needed for Yardstick: 4. Connectivity from the Jumphost to the SUT public/external network .. note:: *Jumphost* refers to any server which meets the previous -requirements. Normally it is the same server from where the OPNFV -deployment has been triggered. + requirements. Normally it is the same server from where the OPNFV + deployment has been triggered. .. warning:: Connectivity from Jumphost is essential and it is of paramount -importance to make sure it is working before even considering to install -and run Yardstick. Make also sure you understand how your networking is -designed to work. + importance to make sure it is working before even considering to install + and run Yardstick. Make also sure you understand how your networking is + designed to work. .. note:: If your Jumphost is operating behind a company http proxy and/or -Firewall, please first consult `Proxy Support`_ section which is towards the -end of this document. That section details some tips/tricks which *may* be of -help in a proxified environment. + Firewall, please first consult `Proxy Support`_ section which is towards + the end of this document. That section details some tips/tricks which *may* + be of help in a proxified environment. Install Yardstick using Docker (first option) (**recommended**) @@ -85,27 +85,30 @@ Run the Docker image to get a Yardstick container:: docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock \ -p 8888:5000 --name yardstick opnfv/yardstick:stable -.. table:: Description of the parameters used with ``docker run`` command - - ======================= ==================================================== - Parameters Detail - ======================= ==================================================== - -itd -i: interactive, Keep STDIN open even if not - attached - -t: allocate a pseudo-TTY detached mode, in the - background - ======================= ==================================================== - --privileged If you want to build ``yardstick-image`` in - Yardstick container, this parameter is needed - ======================= ==================================================== - -p 8888:5000 Redirect the a host port (8888) to a container port - (5000) - ======================= ==================================================== - -v /var/run/docker.sock If you want to use yardstick env grafana/influxdb to - :/var/run/docker.sock create a grafana/influxdb container out of Yardstick - container - ======================= ==================================================== - --name yardstick The name for this container +Description of the parameters used with ``docker run`` command + + +------------------------+--------------------------------------------------+ + | Parameters | Detail | + +========================+==================================================+ + | -itd | -i: interactive, Keep STDIN open even if not | + | | attached | + | +--------------------------------------------------+ + | | -t: allocate a pseudo-TTY detached mode, in the | + | | background | + +------------------------+--------------------------------------------------+ + | --privileged | If you want to build ``yardstick-image`` in | + | | Yardstick container, this parameter is needed | + +------------------------+--------------------------------------------------+ + | -p 8888:5000 | Redirect the a host port (8888) to a container | + | | port (5000) | + +------------------------+--------------------------------------------------+ + | -v /var/run/docker.sock| If you want to use yardstick env | + | :/var/run/docker.sock | grafana/influxdb to create a grafana/influxdb | + | | container out of Yardstick container | + +------------------------+--------------------------------------------------+ + | --name yardstick | The name for this container | + +------------------------+--------------------------------------------------+ + If the host is restarted ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -135,18 +138,18 @@ automatically:: yardstick env prepare .. note:: Since Euphrates release, the above command will not be able to -automatically configure the ``/etc/yardstick/openstack.creds`` file. So before -running the above command, it is necessary to create the -``/etc/yardstick/openstack.creds`` file and save OpenStack environment -variables into it manually. If you have the openstack credential file saved -outside the Yardstick Docker container, you can do this easily by mapping the -credential file into Yardstick container using:: + automatically configure the ``/etc/yardstick/openstack.creds`` file. So before + running the above command, it is necessary to create the + ``/etc/yardstick/openstack.creds`` file and save OpenStack environment + variables into it manually. If you have the openstack credential file saved + outside the Yardstick Docker container, you can do this easily by mapping the + credential file into Yardstick container using:: - '-v /path/to/credential_file:/etc/yardstick/openstack.creds' + '-v /path/to/credential_file:/etc/yardstick/openstack.creds' -when running the Yardstick container. For details of the required OpenStack -environment variables please refer to section `Export OpenStack environment -variables`_. + when running the Yardstick container. For details of the required OpenStack + environment variables please refer to section `Export OpenStack environment + variables`_. The ``env prepare`` command may take up to 6-8 minutes to finish building yardstick-image and other environment preparation. Meanwhile if you wish to @@ -222,8 +225,8 @@ Yardstick is installed:: sudo -EH tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh .. warning:: Before building the guest image inside the Yardstick container, -make sure the container is granted with privilege. The script will create files -by default in ``/tmp/workspace/yardstick`` and the files will be owned by root. + make sure the container is granted with privilege. The script will create files + by default in ``/tmp/workspace/yardstick`` and the files will be owned by root. The created image can be added to OpenStack using the OpenStack client or via the OpenStack Dashboard:: @@ -270,7 +273,7 @@ For usage of Yardstick GUI, please watch our demo video at `Yardstick GUI demo`_. .. note:: The Yardstick GUI is still in development, the GUI layout and -features may change. + features may change. Delete the Yardstick container ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -433,7 +436,7 @@ of Yardstick ``help`` command and ``ping.py`` test sample:: yardstick task start samples/ping.yaml .. note:: The above commands could be run in both the Yardstick container and -the Ubuntu directly. + the Ubuntu directly. Each testing tool supported by Yardstick has a sample configuration file. These configuration files can be found in the ``samples`` directory. @@ -468,10 +471,10 @@ Then you can run a test case and visit http://host_ip:1948 (``admin``/``admin``) to see the results. .. note:: Executing ``yardstick env`` command to deploy InfluxDB and Grafana -requires Jumphost's docker API version => 1.24. Run the following command to -check the docker API version on the Jumphost:: + requires Jumphost's docker API version => 1.24. Run the following command to + check the docker API version on the Jumphost:: - docker version + docker version Manual deployment of InfluxDB and Grafana containers @@ -537,200 +540,6 @@ Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**) --------------------------------------------------------- -Yardstick common CLI --------------------- - -List test cases -^^^^^^^^^^^^^^^ - -``yardstick testcase list``: This command line would list all test cases in -Yardstick. It would show like below:: - - +--------------------------------------------------------------------------------------- - | Testcase Name | Description - +--------------------------------------------------------------------------------------- - | opnfv_yardstick_tc001 | Measure network throughput using pktgen - | opnfv_yardstick_tc002 | measure network latency using ping - | opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio. - ... - +--------------------------------------------------------------------------------------- - - -Show a test case config file -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Take opnfv_yardstick_tc002 for an example. This test case measure network -latency. You just need to type in ``yardstick testcase show -opnfv_yardstick_tc002``, and the console would show the config yaml of this -test case:: - - --- - - schema: "yardstick:task:0.1" - description: > - Yardstick TC002 config file; - measure network latency using ping; - - {% set image = image or "cirros-0.3.5" %} - - {% set provider = provider or none %} - {% set physical_network = physical_network or 'physnet1' %} - {% set segmentation_id = segmentation_id or none %} - {% set packetsize = packetsize or 100 %} - - scenarios: - {% for i in range(2) %} - - - type: Ping - options: - packetsize: {{packetsize}} - host: athena.demo - target: ares.demo - - runner: - type: Duration - duration: 60 - interval: 10 - - sla: - max_rtt: 10 - action: monitor - {% endfor %} - - context: - name: demo - image: {{image}} - flavor: yardstick-flavor - user: cirros - - placement_groups: - pgrp1: - policy: "availability" - - servers: - athena: - floating_ip: true - placement: "pgrp1" - ares: - placement: "pgrp1" - - networks: - test: - cidr: '10.0.1.0/24' - {% if provider == "vlan" %} - provider: {{provider}} - physical_network: {{physical_network}}å - {% if segmentation_id %} - segmentation_id: {{segmentation_id}} - {% endif %} - {% endif %} - - -Start a task to run yardstick test case -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want run a test case, then you need to use ``yardstick task start -<test_case_path>`` this command support some parameters as below:: - - +---------------------+--------------------------------------------------+ - | Parameters | Detail | - +=====================+==================================================+ - | -d | show debug log of yardstick running | - | | | - +---------------------+--------------------------------------------------+ - | --task-args | If you want to customize test case parameters, | - | | use "--task-args" to pass the value. The format | - | | is a json string with parameter key-value pair. | - | | | - +---------------------+--------------------------------------------------+ - | --task-args-file | If you want to use yardstick | - | | env prepare command(or | - | | related API) to load the | - +---------------------+--------------------------------------------------+ - | --parse-only | | - | | | - | | | - +---------------------+--------------------------------------------------+ - | --output-file \ | Specify where to output the log. if not pass, | - | OUTPUT_FILE_PATH | the default value is | - | | "/tmp/yardstick/yardstick.log" | - | | | - +---------------------+--------------------------------------------------+ - | --suite \ | run a test suite, TEST_SUITE_PATH specify where | - | TEST_SUITE_PATH | the test suite locates | - | | | - +---------------------+--------------------------------------------------+ - - -Run Yardstick in a local environment ------------------------------------- - -We also have a guide about how to run Yardstick in a local environment. -This work is contributed by Tapio Tallgren. -You can find this guide at `How to run Yardstick in a local environment`_. - - -Create a test suite for Yardstick ------------------------------------- - -A test suite in yardstick is a yaml file which include one or more test cases. -Yardstick is able to support running test suite task, so you can customize your -own test suite and run it in one task. - -``tests/opnfv/test_suites`` is the folder where Yardstick puts CI test suite. -A typical test suite is like below (the ``fuel_test_suite.yaml`` example):: - - --- - # Fuel integration test task suite - - schema: "yardstick:suite:0.1" - - name: "fuel_test_suite" - test_cases_dir: "samples/" - test_cases: - - - file_name: ping.yaml - - - file_name: iperf3.yaml - -As you can see, there are two test cases in the ``fuel_test_suite.yaml``. The -``schema`` and the ``name`` must be specified. The test cases should be listed -via the tag ``test_cases`` and their relative path is also marked via the tag -``test_cases_dir``. - -Yardstick test suite also supports constraints and task args for each test -case. Here is another sample (the ``os-nosdn-nofeature-ha.yaml`` example) to -show this, which is digested from one big test suite:: - - --- - - schema: "yardstick:suite:0.1" - - name: "os-nosdn-nofeature-ha" - test_cases_dir: "tests/opnfv/test_cases/" - test_cases: - - - file_name: opnfv_yardstick_tc002.yaml - - - file_name: opnfv_yardstick_tc005.yaml - - - file_name: opnfv_yardstick_tc043.yaml - constraint: - installer: compass - pod: huawei-pod1 - task_args: - huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml", - "host": "node4.LF","target": "node5.LF"}' - -As you can see in test case ``opnfv_yardstick_tc043.yaml``, there are two -tags, ``constraint`` and ``task_args``. ``constraint`` is to specify which -installer or pod it can be run in the CI environment. ``task_args`` is to -specify the task arguments for each pod. - -All in all, to create a test suite in Yardstick, you just need to create a -yaml file and add test cases, constraint or task arguments if necessary. - - Proxy Support ------------- @@ -790,7 +599,7 @@ stop and delete the container:: sudo docker rm yardstick .. warning:: Be careful, the above ``rm`` command will delete the container -completely. Everything on this container will be lost. + completely. Everything on this container will be lost. Then follow the previous instructions `Prepare the Yardstick container`_ to rebuild the Yardstick container. @@ -804,4 +613,3 @@ References .. _`Cirros 0.3.5`: http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img .. _`Ubuntu 16.04`: https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img .. _`Yardstick GUI demo`: https://www.youtube.com/watch?v=M3qbJDp6QBk -.. _`How to run Yardstick in a local environment`: https://wiki.opnfv.org/display/yardstick/How+to+run+Yardstick+in+a+local+environment diff --git a/docs/testing/user/userguide/05-operation.rst b/docs/testing/user/userguide/05-operation.rst new file mode 100644 index 000000000..f390d1643 --- /dev/null +++ b/docs/testing/user/userguide/05-operation.rst @@ -0,0 +1,296 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intel, Ericsson AB, Huawei Technologies Co. Ltd and others. + +.. + Convention for heading levels in Yardstick: + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ^^^^^^^ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + Avoid deeper levels because they do not render well. + +=============== +Yardstick Usage +=============== + +Once you have yardstick installed, you can start using it to run testcases +immediately, through the CLI. You can also define and run new testcases and +test suites. This chapter details basic usage (running testcases), as well as +more advanced usage (creating your own testcases). + +Yardstick common CLI +-------------------- + +List test cases +^^^^^^^^^^^^^^^ + +``yardstick testcase list``: This command line would list all test cases in +Yardstick. It would show like below:: + + +--------------------------------------------------------------------------------------- + | Testcase Name | Description + +--------------------------------------------------------------------------------------- + | opnfv_yardstick_tc001 | Measure network throughput using pktgen + | opnfv_yardstick_tc002 | measure network latency using ping + | opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio. + ... + +--------------------------------------------------------------------------------------- + + +Show a test case config file +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Take opnfv_yardstick_tc002 for an example. This test case measure network +latency. You just need to type in ``yardstick testcase show +opnfv_yardstick_tc002``, and the console would show the config yaml of this +test case: + +.. literalinclude:: + ../../../../tests/opnfv/test_cases/opnfv_yardstick_tc002.yaml + :lines: 9- + +Run a Yardstick test case +^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you want run a test case, then you need to use ``yardstick task start +<test_case_path>`` this command support some parameters as below: + + +---------------------+--------------------------------------------------+ + | Parameters | Detail | + +=====================+==================================================+ + | -d | show debug log of yardstick running | + | | | + +---------------------+--------------------------------------------------+ + | --task-args | If you want to customize test case parameters, | + | | use "--task-args" to pass the value. The format | + | | is a json string with parameter key-value pair. | + | | | + +---------------------+--------------------------------------------------+ + | --task-args-file | If you want to use yardstick | + | | env prepare command(or | + | | related API) to load the | + +---------------------+--------------------------------------------------+ + | --parse-only | | + | | | + | | | + +---------------------+--------------------------------------------------+ + | --output-file \ | Specify where to output the log. if not pass, | + | OUTPUT_FILE_PATH | the default value is | + | | "/tmp/yardstick/yardstick.log" | + | | | + +---------------------+--------------------------------------------------+ + | --suite \ | run a test suite, TEST_SUITE_PATH specify where | + | TEST_SUITE_PATH | the test suite locates | + | | | + +---------------------+--------------------------------------------------+ + + +Run Yardstick in a local environment +------------------------------------ + +We also have a guide about `How to run Yardstick in a local environment`_. +This work is contributed by Tapio Tallgren. + +Create a new testcase for Yardstick +----------------------------------- + +As a user, you may want to define a new testcase in addition to the ones +already available in Yardstick. This section will show you how to do this. + +Each testcase consists of two sections: + +* ``scenarios`` describes what will be done by the test +* ``context`` describes the environment in which the test will be run. + +Defining the testcase scenarios +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +TODO + +Defining the testcase context(s) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each testcase consists of one or more contexts, which describe the environment +in which the testcase will be run. +Current available contexts are: + +* ``Dummy``: this is a no-op context, and is used when there is no environment + to set up e.g. when testing whether OpenStack services are available +* ``Node``: this context is used to perform operations on baremetal servers +* ``Heat``: uses OpenStack to provision the required hosts, networks, etc. +* ``Kubernetes``: uses Kubernetes to provision the resources required for the + test. + +Regardless of the context type, the ``context`` section of the testcase will +consist of the following:: + + context: + name: demo + type: Dummy|Node|Heat|Kubernetes + +The content of the ``context`` section will vary based on the context type. + +Dummy Context ++++++++++++++ + +No additional information is required for the Dummy context:: + + context: + name: my_context + type: Dummy + +Node Context +++++++++++++ + +TODO + +Heat Context +++++++++++++ + +In addition to ``name`` and ``type``, a Heat context requires the following +arguments: + +* ``image``: the image to be used to boot VMs +* ``flavor``: the flavor to be used for VMs in the context +* ``user``: the username for connecting into the VMs +* ``networks``: The networks to be created, networks are identified by name + + * ``name``: network name (required) + * (TODO) Any optional attributes + +* ``servers``: The servers to be created + + * ``name``: server name + * (TODO) Any optional attributes + +In addition to the required arguments, the following optional arguments can be +passed to the Heat context: + +* ``placement_groups``: + + * ``name``: the name of the placement group to be created + * ``policy``: either ``affinity`` or ``availability`` +* ``server_groups``: + + * ``name``: the name of the server group + * ``policy``: either ``affinity`` or ``anti-affinity`` + +Combining these elements together, a sample Heat context config looks like: + +.. literalinclude:: + ../../../../yardstick/tests/integration/dummy-scenario-heat-context.yaml + :start-after: --- + :empahsise-lines: 14- + +Using exisiting HOT Templates +''''''''''''''''''''''''''''' + +TODO + +Kubernetes Context +++++++++++++++++++ + +TODO + +Using multiple contexts in a testcase ++++++++++++++++++++++++++++++++++++++ + +When using multiple contexts in a testcase, the ``context`` section is replaced +by a ``contexts`` section, and each context is separated with a ``-`` line:: + + contexts: + - + name: context1 + type: Heat + ... + - + name: context2 + type: Node + ... + + +Reusing a context ++++++++++++++++++ + +Typically, a context is torn down after a testcase is run, however, the user +may wish to keep an context intact after a testcase is complete. + +.. note:: + This feature has been implemented for the Heat context only + +To keep or reuse a context, the ``flags`` option must be specified: + +* ``no_setup``: skip the deploy stage, and fetch the details of a deployed + context/Heat stack. +* ``no_teardown``: skip the undeploy stage, thus keeping the stack intact for + the next test + +If either of these ``flags`` are ``True``, the context information must still +be given. By default, these flags are disabled:: + + context: + name: mycontext + type: Heat + flags: + no_setup: True + no_teardown: True + ... + +Create a test suite for Yardstick +--------------------------------- + +A test suite in Yardstick is a .yaml file which includes one or more test +cases. Yardstick is able to support running test suite task, so you can +customize your own test suite and run it in one task. + +``tests/opnfv/test_suites`` is the folder where Yardstick puts CI test suite. +A typical test suite is like below (the ``fuel_test_suite.yaml`` example): + +.. literalinclude:: + ../../../../tests/opnfv/test_suites/fuel_test_suite.yaml + :lines: 9- + +As you can see, there are two test cases in the ``fuel_test_suite.yaml``. The +``schema`` and the ``name`` must be specified. The test cases should be listed +via the tag ``test_cases`` and their relative path is also marked via the tag +``test_cases_dir``. + +Yardstick test suite also supports constraints and task args for each test +case. Here is another sample (the ``os-nosdn-nofeature-ha.yaml`` example) to +show this, which is digested from one big test suite:: + + --- + + schema: "yardstick:suite:0.1" + + name: "os-nosdn-nofeature-ha" + test_cases_dir: "tests/opnfv/test_cases/" + test_cases: + - + file_name: opnfv_yardstick_tc002.yaml + - + file_name: opnfv_yardstick_tc005.yaml + - + file_name: opnfv_yardstick_tc043.yaml + constraint: + installer: compass + pod: huawei-pod1 + task_args: + huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml", + "host": "node4.LF","target": "node5.LF"}' + +As you can see in test case ``opnfv_yardstick_tc043.yaml``, there are two +tags, ``constraint`` and ``task_args``. ``constraint`` is to specify which +installer or pod it can be run in the CI environment. ``task_args`` is to +specify the task arguments for each pod. + +All in all, to create a test suite in Yardstick, you just need to create a +yaml file and add test cases, constraint or task arguments if necessary. + +References +---------- + +.. _`How to run Yardstick in a local environment`: https://wiki.opnfv.org/display/yardstick/How+to+run+Yardstick+in+a+local+environment diff --git a/docs/testing/user/userguide/05-yardstick_plugin.rst b/docs/testing/user/userguide/06-yardstick-plugin.rst index 679ce7900..bc35e239d 100644 --- a/docs/testing/user/userguide/05-yardstick_plugin.rst +++ b/docs/testing/user/userguide/06-yardstick-plugin.rst @@ -31,7 +31,7 @@ In this introduction we will install Storperf on Jump Host. Step 0: Environment preparation ->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +------------------------------- Running Storperf on Jump Host Requirements: @@ -47,24 +47,26 @@ environment and other dependencies: 1. Make sure docker is installed. 2. Make sure Keystone, Nova, Neutron, Glance, Heat are installed correctly. 3. Make sure Jump Host have access to the OpenStack Controller API. -4. Make sure Jump Host must have internet connectivity for downloading docker image. -5. You need to know where to get basic openstack Keystone authorization info, such as - OS_PASSWORD, OS_PROJECT_NAME, OS_AUTH_URL, OS_USERNAME. -6. To run a Storperf container, you need to have OpenStack Controller environment - variables defined and passed to Storperf container. The best way to do this is to - put environment variables in a "storperf_admin-rc" file. The storperf_admin-rc - should include credential environment variables at least: - -* OS_AUTH_URL -* OS_USERNAME -* OS_PASSWORD -* OS_PROJECT_NAME -* OS_PROJECT_ID -* OS_USER_DOMAIN_ID - -*Yardstick* has a "prepare_storperf_admin-rc.sh" script which can be used to -generate the "storperf_admin-rc" file, this script is located at -test/ci/prepare_storperf_admin-rc.sh +4. Make sure Jump Host must have internet connectivity for downloading docker + image. +5. You need to know where to get basic openstack Keystone authorization info, + such as OS_PASSWORD, OS_PROJECT_NAME, OS_AUTH_URL, OS_USERNAME. +6. To run a Storperf container, you need to have OpenStack Controller + environment variables defined and passed to Storperf container. The best way + to do this is to put environment variables in a "storperf_admin-rc" file. + The storperf_admin-rc should include credential environment variables at + least: + + * OS_AUTH_URL + * OS_USERNAME + * OS_PASSWORD + * OS_PROJECT_NAME + * OS_PROJECT_ID + * OS_USER_DOMAIN_ID + +*Yardstick* has a ``prepare_storperf_admin-rc.sh`` script which can be used to +generate the ``storperf_admin-rc`` file, this script is located at +``test/ci/prepare_storperf_admin-rc.sh`` :: @@ -92,18 +94,18 @@ test/ci/prepare_storperf_admin-rc.sh echo "OS_USER_DOMAIN_ID="$USER_DOMAIN_ID >> ~/storperf_admin-rc -The generated "storperf_admin-rc" file will be stored in the root directory. If -you installed *Yardstick* using Docker, this file will be located in the +The generated ``storperf_admin-rc`` file will be stored in the root directory. +If you installed *Yardstick* using Docker, this file will be located in the container. You may need to copy it to the root directory of the Storperf deployed host. Step 1: Plug-in configuration file preparation ->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +---------------------------------------------- To install a plug-in, first you need to prepare a plug-in configuration file in -YAML format and store it in the "plugin" directory. The plugin configration file -work as the input of yardstick "plugin" command. Below is the Storperf plug-in -configuration file sample: +YAML format and store it in the "plugin" directory. The plugin configration +file work as the input of yardstick "plugin" command. Below is the Storperf +plug-in configuration file sample: :: --- @@ -123,28 +125,28 @@ Here the Storperf will be installed on IP 192.168.23.2 which is the Jump Host in my local environment. Step 2: Plug-in install/remove scripts preparation ->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +-------------------------------------------------- -In "yardstick/resource/scripts" directory, there are two folders: a "install" -folder and a "remove" folder. You need to store the plug-in install/remove -scripts in these two folders respectively. +In ``yardstick/resource/scripts`` directory, there are two folders: an +``install`` folder and a ``remove`` folder. You need to store the plug-in +install/remove scripts in these two folders respectively. The detailed installation or remove operation should de defined in these two scripts. The name of both install and remove scripts should match the plugin-in name that you specified in the plug-in configuration file. -For example, the install and remove scripts for Storperf are both named to -"storperf.bash". +For example, the install and remove scripts for Storperf are both named +``storperf.bash``. Step 3: Install and remove Storperf ->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +----------------------------------- To install Storperf, simply execute the following command:: # Install Storperf yardstick plugin install plugin/storperf.yaml -removing Storperf from yardstick +Removing Storperf from yardstick ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To remove Storperf, simply execute the following command:: diff --git a/docs/testing/user/userguide/06-result-store-InfluxDB.rst b/docs/testing/user/userguide/07-result-store-InfluxDB.rst index 747927889..cde931376 100644 --- a/docs/testing/user/userguide/06-result-store-InfluxDB.rst +++ b/docs/testing/user/userguide/07-result-store-InfluxDB.rst @@ -24,7 +24,7 @@ Store Storperf Test Results into Community's InfluxDB ===================================================== .. _Influxdb: https://git.opnfv.org/cgit/yardstick/tree/yardstick/dispatcher/influxdb.py -.. _Mingjiang: limingjiang@huawei.com +.. _Mingjiang: mailto:limingjiang@huawei.com .. _Visual: https://wiki.opnfv.org/download/attachments/6827660/tc074.PNG?version=1&modificationDate=1470298075000&api=v2 .. _Login: http://testresults.opnfv.org/grafana/login @@ -40,12 +40,13 @@ into community's InfluxDB: will be supported in the future. Our plan is to support rest-api in D release so that other testing projects can -call the rest-api to use yardstick dispatcher service to push data to yardstick's -influxdb database. +call the rest-api to use yardstick dispatcher service to push data to +Yardstick's InfluxDB database. -For now, influxdb only support line protocol, and the json protocol is deprecated. +For now, InfluxDB only supports line protocol, and the json protocol is +deprecated. -Take ping test case for example, the raw_result is json format like this: +Take ping test case for example, the ``raw_result`` is json format like this: :: "benchmark": { @@ -61,23 +62,24 @@ Take ping test case for example, the raw_result is json format like this: "runner_id": 2625 } -With the help of "influxdb_line_protocol", the json is transform to like below as a line string: -:: +With the help of "influxdb_line_protocol", the json is transform to like below +as a line string:: 'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown,pod_name=unknown, runner_id=2625,scenarios=Ping,target=ares.demo,task_id=77755f38-1f6a-4667-a7f3- 301c99963656,version=unknown rtt.ares=1.125 1470315409868094976' -So, for data output of json format, you just need to transform json into line format and call -influxdb api to post the data into the database. All this function has been implemented in Influxdb_. -If you need support on this, please contact Mingjiang_. +So, for data output of json format, you just need to transform json into line +format and call influxdb api to post the data into the database. All this +function has been implemented in Influxdb_. If you need support on this, please +contact Mingjiang_. :: curl -i -XPOST 'http://104.197.68.199:8086/write?db=yardstick' -- data-binary 'ping,deploy_scenario=unknown,host=athena.demo,installer=unknown, ...' -Grafana will be used for visualizing the collected test data, which is shown in Visual_. Grafana -can be accessed by Login_. +Grafana will be used for visualizing the collected test data, which is shown in +Visual_. Grafana can be accessed by Login_. .. image:: images/results_visualization.png diff --git a/docs/testing/user/userguide/07-grafana.rst b/docs/testing/user/userguide/08-grafana.rst index 416857b71..29bc23a08 100644 --- a/docs/testing/user/userguide/07-grafana.rst +++ b/docs/testing/user/userguide/08-grafana.rst @@ -108,8 +108,10 @@ There are 6 steps to go. 5. When finished with all Grafana configuration changes in this temporary dashboard then chose "export" of the updated dashboard copy into a JSON file - and put it up for review in Gerrit, in file /yardstick/dashboard/Yardstick-TCxxx-yyyyyyyyyyyyy. - For instance a typical default name of the file would be "Yardstick-TC001 Copy-1234567891234". + and put it up for review in Gerrit, in file + ``/yardstick/dashboard/Yardstick-TCxxx-yyyyyyyyyyyyy``. + For instance a typical default name of the file would be + ``Yardstick-TC001 Copy-1234567891234``. 6. Once you finish your dashboard, the next step is exporting the configuration file and propose a patch into Yardstick. Yardstick team will review and diff --git a/docs/testing/user/userguide/08-api.rst b/docs/testing/user/userguide/09-api.rst index 92fa408c8..f0ae3980b 100644 --- a/docs/testing/user/userguide/08-api.rst +++ b/docs/testing/user/userguide/09-api.rst @@ -3,25 +3,29 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, Huawei Technologies Co.,Ltd and others. +===================== Yardstick Restful API -====================== +===================== Abstract --------- +======== Yardstick support restful API since Danube. Available API -------------- +============= /yardstick/env/action -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +--------------------- -Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports: +Description: This API is used to prepare Yardstick test environment. +For Euphrates, it supports: -1. Prepare yardstick test environment, including set external network environment variable, load Yardstick VM images and create flavors; +1. Prepare yardstick test environment, including setting the + ``EXTERNAL_NETWORK`` environment variable, load Yardstick VM images and + create flavors; 2. Start an InfluxDB Docker container and config Yardstick output to InfluxDB; 3. Start a Grafana Docker container and config it with the InfluxDB. @@ -35,34 +39,37 @@ Prepare Yardstick test environment Example:: { - 'action': 'prepareYardstickEnv' + 'action': 'prepare_env' } -This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result. +This is an asynchronous API. You need to call ``/yardstick/asynctask`` API to +get the task result. Start and config an InfluxDB docker container Example:: { - 'action': 'createInfluxDBContainer' + 'action': 'create_influxdb' } -This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result. +This is an asynchronous API. You need to call ``/yardstick/asynctask`` API to +get the task result. Start and config a Grafana docker container Example:: { - 'action': 'createGrafanaContainer' + 'action': 'create_grafana' } -This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result. +This is an asynchronous API. You need to call ``/yardstick/asynctask`` API to +get the task result. /yardstick/asynctask -^^^^^^^^^^^^^^^^^^^^ +-------------------- Description: This API is used to get the status of asynchronous tasks @@ -73,13 +80,18 @@ Method: GET Get the status of asynchronous tasks Example:: - http://localhost:8888/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c The returned status will be 0(running), 1(finished) and 2(failed). +NOTE:: + + <SERVER IP>: The ip of the host where you start your yardstick container + <PORT>: The outside port of port mapping which set when you start start yardstick container + /yardstick/testcases -^^^^^^^^^^^^^^^^^^^^ +-------------------- Description: This API is used to list all released Yardstick test cases. @@ -90,11 +102,11 @@ Method: GET Get a list of released test cases Example:: - http://localhost:8888/yardstick/testcases + http://<SERVER IP>:<PORT>/yardstick/testcases /yardstick/testcases/release/action -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------- Description: This API is used to run a Yardstick released test case. @@ -106,18 +118,19 @@ Run a released test case Example:: { - 'action': 'runTestCase', + 'action': 'run_test_case', 'args': { 'opts': {}, - 'testcase': 'tc002' + 'testcase': 'opnfv_yardstick_tc002' } } -This is an asynchronous API. You need to call /yardstick/results to get the result. +This is an asynchronous API. You need to call ``/yardstick/results`` to get the +result. /yardstick/testcases/samples/action -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------- Description: This API is used to run a Yardstick sample test case. @@ -129,20 +142,22 @@ Run a sample test case Example:: { - 'action': 'runTestCase', + 'action': 'run_test_case', 'args': { 'opts': {}, 'testcase': 'ping' } } -This is an asynchronous API. You need to call /yardstick/results to get the result. +This is an asynchronous API. You need to call ``/yardstick/results`` to get +the result. /yardstick/testcases/<testcase_name>/docs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------------- -Description: This API is used to the documentation of a certain released test case. +Description: This API is used to the documentation of a certain released test +case. Method: GET @@ -151,11 +166,11 @@ Method: GET Get the documentation of a certain test case Example:: - http://localhost:8888/yardstick/taskcases/opnfv_yardstick_tc002/docs + http://<SERVER IP>:<PORT>/yardstick/taskcases/opnfv_yardstick_tc002/docs /yardstick/testsuites/action -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------- Description: This API is used to run a Yardstick test suite. @@ -167,17 +182,19 @@ Run a test suite Example:: { - 'action': 'runTestSuite', + 'action': 'run_test_suite', 'args': { 'opts': {}, - 'testcase': 'smoke' + 'testsuite': 'opnfv_smoke' } } -This is an asynchronous API. You need to call /yardstick/results to get the result. +This is an asynchronous API. You need to call /yardstick/results to get the +result. /yardstick/tasks/<task_id>/log +------------------------------ Description: This API is used to get the real time log of test case execution. @@ -188,13 +205,15 @@ Method: GET Get real time of test case execution Example:: - http://localhost:8888/yardstick/tasks/14795be8-f144-4f54-81ce-43f4e3eab33f/log?index=0 + http://<SERVER IP>:<PORT>/yardstick/tasks/14795be8-f144-4f54-81ce-43f4e3eab33f/log?index=0 /yardstick/results -^^^^^^^^^^^^^^^^^^ +------------------ -Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by using this API. +Description: This API is used to get the test results of tasks. If you call +/yardstick/testcases/samples/action API, it will return a task id. You can use +the returned task id to get the results by using this API. Method: GET @@ -203,17 +222,19 @@ Method: GET Get test results of one task Example:: - http://localhost:8888/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c This API will return a list of test case result -/api/v2/yardstick/openrcs/action +/api/v2/yardstick/openrcs +------------------------- -Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: +Description: This API provides functionality of handling OpenStack credential +file (openrc). For Euphrates, it supports: 1. Upload an openrc file for an OpenStack environment; -2. Update an openrc file; +2. Update an openrc; 3. Get openrc file information; 4. Delete an openrc file. @@ -260,12 +281,21 @@ Example:: } +/api/v2/yardstick/openrcs/<openrc_id> +------------------------------------- + +Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: + +1. Get openrc file information; +2. Delete an openrc file. + + METHOD: GET Get openrc file information Example:: - http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c METHOD: DELETE @@ -274,16 +304,16 @@ METHOD: DELETE Delete openrc file Example:: - http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c -/api/v2/yardstick/pods/action +/api/v2/yardstick/pods +---------------------- -Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: +Description: This API provides functionality of handling Yardstick pod file +(pod.yaml). For Euphrates, it supports: 1. Upload a pod file; -2. Get pod file information; -3. Delete an openrc file. Which API to call will depend on the parameters. @@ -303,12 +333,20 @@ Example:: } +/api/v2/yardstick/pods/<pod_id> +------------------------------- + +Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: + +1. Get pod file information; +2. Delete an openrc file. + METHOD: GET Get pod file information Example:: - http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c METHOD: DELETE @@ -316,16 +354,16 @@ METHOD: DELETE Delete openrc file Example:: - http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c -/api/v2/yardstick/images/action +/api/v2/yardstick/images +------------------------ -Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: +Description: This API is used to do some work related to Yardstick VM images. +For Euphrates, it supports: 1. Load Yardstick VM images; -2. Get image's information; -3. Delete images. Which API to call will depend on the parameters. @@ -337,16 +375,27 @@ Load VM images Example:: { - 'action': 'load_images' + 'action': 'load_image', + 'args': { + 'name': 'yardstick-image' + } } +/api/v2/yardstick/images/<image_id> +----------------------------------- + +Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: + +1. Get image's information; +2. Delete images + METHOD: GET Get image information Example:: - http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c METHOD: DELETE @@ -354,19 +403,16 @@ METHOD: DELETE Delete images Example:: - http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c -/api/v2/yardstick/tasks/action +/api/v2/yardstick/tasks +----------------------- -Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: +Description: This API is used to do some work related to yardstick tasks. For +Euphrates, it supports: 1. Create a Yardstick task; -2. run a Yardstick task; -3. Add a test case to a task; -4. Add a test suite to a task; -5. Get a tasks' information; -6. Delete a task. Which API to call will depend on the parameters. @@ -386,20 +432,35 @@ Example:: } +/api/v2/yardstick/tasks/<task_id> +-------------------------------- + +Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: + +1. Add a environment to a task +2. Add a test case to a task; +3. Add a test suite to a task; +4. run a Yardstick task; +5. Get a tasks' information; +6. Delete a task. + + METHOD: PUT +Add a environment to a task -Run a task Example:: { - 'action': 'run' + 'action': 'add_environment', + 'args': { + 'environment_id': 'e3cadbbb-0419-4fed-96f1-a232daa0422a' + } } METHOD: PUT - Add a test case to a task Example:: @@ -412,8 +473,8 @@ Example:: } -METHOD: PUT +METHOD: PUT Add a test suite to a task Example:: @@ -427,29 +488,43 @@ Example:: } +METHOD: PUT + +Run a task + +Example:: + + { + 'action': 'run' + } + + + METHOD: GET Get a task's information Example:: - http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c METHOD: DELETE Delete a task + Example:: - http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c + http://<SERVER IP>:<PORT>/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c -/api/v2/yardstick/testcases/action -Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: +/api/v2/yardstick/testcases +--------------------------- + +Description: This API is used to do some work related to Yardstick testcases. +For Euphrates, it supports: 1. Upload a test case; 2. Get all released test cases' information; -3. Get a certain released test case's information; -4. Delete a test case. Which API to call will depend on the parameters. @@ -474,16 +549,24 @@ METHOD: GET Get all released test cases' information Example:: - http://localhost:8888/api/v2/yardstick/testcases + http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases + +/api/v2/yardstick/testcases/<case_name> +--------------------------------------- + +Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: + +1. Get certain released test case's information; +2. Delete a test case. METHOD: GET -Get a certain released test case's information +Get certain released test case's information Example:: - http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002 + http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases/opnfv_yardstick_tc002 METHOD: DELETE @@ -491,17 +574,18 @@ METHOD: DELETE Delete a certain test case Example:: - http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002 + http://<SERVER IP>:<PORT>/api/v2/yardstick/testcases/opnfv_yardstick_tc002 -/api/v2/yardstick/testsuites/action -Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: +/api/v2/yardstick/testsuites +---------------------------- + +Description: This API is used to do some work related to yardstick test suites. +For Euphrates, it supports: 1. Create a test suite; -2. Get a certain test suite's information; -3. Get all test suites; -4. Delete a test case. +2. Get all test suites; Which API to call will depend on the parameters. @@ -513,7 +597,7 @@ Create a test suite Example:: { - 'action': 'create_sutie', + 'action': 'create_suite', 'args': { 'name': <suite_name>, 'testcases': [ @@ -526,19 +610,27 @@ Example:: METHOD: GET -Get a certain test suite's information +Get all test suite Example:: - http://localhost:8888/api/v2/yardstick/testsuites/<suite_name> + http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites + + +/api/v2/yardstick/testsuites +---------------------------- + +Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: +1. Get certain test suite's information; +2. Delete a test case. METHOD: GET -Get all test suite +Get certain test suite's information Example:: - http://localhost:8888/api/v2/yardstick/testsuites + http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites/<suite_name> METHOD: DELETE @@ -547,17 +639,17 @@ METHOD: DELETE Delete a certain test suite Example:: - http://localhost:8888/api/v2/yardstick/testsuites/<suite_name> + http://<SERVER IP>:<PORT>/api/v2/yardstick/testsuites/<suite_name> -/api/v2/yardstick/projects/action +/api/v2/yardstick/projects +-------------------------- -Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: +Description: This API is used to do some work related to Yardstick test +projects. For Euphrates, it supports: 1. Create a Yardstick project; -2. Get a certain project's information; -3. Get all projects; -4. Delete a project. +2. Get all projects; Which API to call will depend on the parameters. @@ -579,19 +671,27 @@ Example:: METHOD: GET -Get a certain project's information +Get all projects' information Example:: - http://localhost:8888/api/v2/yardstick/projects/<project_id> + http://<SERVER IP>:<PORT>/api/v2/yardstick/projects + +/api/v2/yardstick/projects +-------------------------- + +Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: + +1. Get certain project's information; +2. Delete a project. METHOD: GET -Get all projects' information +Get certain project's information Example:: - http://localhost:8888/api/v2/yardstick/projects + http://<SERVER IP>:<PORT>/api/v2/yardstick/projects/<project_id> METHOD: DELETE @@ -600,17 +700,17 @@ METHOD: DELETE Delete a certain project Example:: - http://localhost:8888/api/v2/yardstick/projects/<project_id> + http://<SERVER IP>:<PORT>/api/v2/yardstick/projects/<project_id> -/api/v2/yardstick/containers/action +/api/v2/yardstick/containers +---------------------------- -Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: +Description: This API is used to do some work related to Docker containers. +For Euphrates, it supports: 1. Create a Grafana Docker container; 2. Create an InfluxDB Docker container; -3. Get a certain container's information; -4. Delete a container. Which API to call will depend on the parameters. @@ -643,13 +743,21 @@ Example:: } +/api/v2/yardstick/containers/<container_id> +------------------------------------------- + +Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: + +1. Get certain container's information; +2. Delete a container. + METHOD: GET -Get a certain container's information +Get certain container's information Example:: - http://localhost:8888/api/v2/yardstick/containers/<container_id> + http://<SERVER IP>:<PORT>/api/v2/yardstick/containers/<container_id> METHOD: DELETE @@ -658,4 +766,4 @@ METHOD: DELETE Delete a certain container Example:: - http://localhost:8888/api/v2/yardstick/containers/<container_id> + http://<SERVER IP>:<PORT>/api/v2/yardstick/containers/<container_id> diff --git a/docs/testing/user/userguide/09-yardstick_user_interface.rst b/docs/testing/user/userguide/10-yardstick-user-interface.rst index 9058dd46d..cadec78ef 100644 --- a/docs/testing/user/userguide/09-yardstick_user_interface.rst +++ b/docs/testing/user/userguide/10-yardstick-user-interface.rst @@ -1,3 +1,4 @@ +======================== Yardstick User Interface ======================== @@ -6,14 +7,14 @@ in table format and also values pinned on to a graph. Command -------- +======= :: yardstick report generate <task-ID> <testcase-filename> Description ------------ +=========== 1. When the command is triggered using the task-id and the testcase name provided the respective values are retrieved from the diff --git a/docs/testing/user/userguide/10-vtc-overview.rst b/docs/testing/user/userguide/11-vtc-overview.rst index 8ed17873d..47582358c 100644 --- a/docs/testing/user/userguide/10-vtc-overview.rst +++ b/docs/testing/user/userguide/11-vtc-overview.rst @@ -29,10 +29,10 @@ to run the :term:`VNF`. The exploitation of Deep Packet Inspection assumptions: * third parties unaffiliated with either source or recipient are able to -inspect each IP packet’s payload + inspect each IP packet's payload -* the classifier knows the relevant syntax of each application’s packet -payloads (protocol signatures, data patterns, etc.). +* the classifier knows the relevant syntax of each application's packet + payloads (protocol signatures, data patterns, etc.). The proposed :term:`DPI` based approach will only use an indicative, small number of the initial packets from each flow in order to identify the content @@ -47,14 +47,14 @@ Concepts ======== * *Traffic Inspection*: The process of packet analysis and application -identification of network traffic that passes through the :term:`VTC`. + identification of network traffic that passes through the :term:`VTC`. * *Traffic Forwarding*: The process of packet forwarding from an incoming -network interface to a pre-defined outgoing network interface. + network interface to a pre-defined outgoing network interface. * *Traffic Rule Application*: The process of packet tagging, based on a -predefined set of rules. Packet tagging may include e.g. Type of Service -(:term:`ToS`) field modification. + predefined set of rules. Packet tagging may include e.g. Type of Service + (:term:`ToS`) field modification. Architecture ============ diff --git a/docs/testing/user/userguide/11-nsb-overview.rst b/docs/testing/user/userguide/12-nsb-overview.rst index 8ce90f65d..71a5c1130 100644 --- a/docs/testing/user/userguide/11-nsb-overview.rst +++ b/docs/testing/user/userguide/12-nsb-overview.rst @@ -3,11 +3,12 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2017 Intel Corporation. +=================================== Network Services Benchmarking (NSB) =================================== Abstract --------- +======== .. _Yardstick: https://wiki.opnfv.org/yardstick @@ -15,10 +16,10 @@ This chapter provides an overview of the NSB, a contribution to OPNFV Yardstick_ from Intel. Overview --------- +======== -The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi Characterization and -benchmarking with repeatable and deterministic methods. +The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi +Characterization and benchmarking with repeatable and deterministic methods. The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization and benchmarking in three different execution @@ -70,17 +71,17 @@ NSB extension includes: - VNF KPIs, e.g., packet_in, packet_drop, packet_fwd etc Architecture ------------- +============ The Network Service (NS) defines a set of Virtual Network Functions (VNF) connected together using NFV infrastructure. The Yardstick NSB extension can support multiple VNFs created by different vendors including traffic generators. Every VNF being tested has its -own data model. The Network service defines a VNF modelling on base of performed -network functionality. The part of the data model is a set of the configuration -parameters, number of connection points used and flavor including core and -memory amount. +own data model. The Network service defines a VNF modelling on base of +performed network functionality. The part of the data model is a set of the +configuration parameters, number of connection points used and flavor including +core and memory amount. The ETSI defines a Network Service as a set of configurable VNFs working in some NFV Infrastructure connecting each other using Virtual Links available @@ -112,7 +113,7 @@ Network Service framework performs the necessary test steps. It may involve - Read the KPI's provided by particular VNF Components of Network Service -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------- * *Models for Network Service benchmarking*: The Network Service benchmarking requires the proper modelling approach. The NSB provides models using Python @@ -132,9 +133,9 @@ Components of Network Service same way as other VNFs being a part of benchmarked network service. Same as other VNFs the traffic generator are instantiated and terminated. - Every traffic generator has own configuration defined as a traffic profile and - a set of KPIs supported. The python models for TG is extended by specific calls - to listen and generate traffic. + Every traffic generator has own configuration defined as a traffic profile + and a set of KPIs supported. The python models for TG is extended by + specific calls to listen and generate traffic. * *The stateless TREX traffic generator*: The main traffic generator used as Network Service stimulus is open source TREX tool. @@ -165,7 +166,7 @@ Components of Network Service - RFC2544 throughput for various loss rate defined (1% is a default) Graphical Overview ------------------- +================== NSB Testing with yardstick framework facilitate performance testing of various VNFs provided. @@ -192,12 +193,14 @@ VNFs provided. Figure 1: Network Service - 2 server configuration VNFs supported for chracterization: -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +----------------------------------- 1. CGNAPT - Carrier Grade Network Address and port Translation 2. vFW - Virtual Firewall 3. vACL - Access Control List -5. Prox - Packet pROcessing eXecution engine: - - VNF can act as Drop, Basic Forwarding (no touch), L2 Forwarding (change MAC), GRE encap/decap, Load balance based on packet fields, Symmetric load balancing, +4. Prox - Packet pROcessing eXecution engine: + - VNF can act as Drop, Basic Forwarding (no touch), + L2 Forwarding (change MAC), GRE encap/decap, Load balance based on + packet fields, Symmetric load balancing - QinQ encap/decap IPv4/IPv6, ARP, QoS, Routing, Unmpls, Policing, ACL -6. UDP_Replay +5. UDP_Replay diff --git a/docs/testing/user/userguide/12-nsb_installation.rst b/docs/testing/user/userguide/13-nsb-installation.rst index a584ca231..3e0ed0bfb 100644 --- a/docs/testing/user/userguide/12-nsb_installation.rst +++ b/docs/testing/user/userguide/13-nsb-installation.rst @@ -3,11 +3,12 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) OPNFV, 2016-2017 Intel Corporation. +===================================== Yardstick - NSB Testing -Installation ===================================== Abstract --------- +======== The Network Service Benchmarking (NSB) extends the yardstick framework to do VNF characterization and benchmarking in three different execution @@ -26,79 +27,65 @@ The steps needed to run Yardstick with NSB testing are: Prerequisites -------------- +============= Refer chapter Yardstick Installation for more information on yardstick prerequisites -Several prerequisites are needed for Yardstick(VNF testing): - - - Python Modules: pyzmq, pika. - - - flex - - - bison - - - build-essential - - - automake - - - libtool +Several prerequisites are needed for Yardstick (VNF testing): - - librabbitmq-dev - - - rabbitmq-server - - - collectd - - - intel-cmt-cat + * Python Modules: pyzmq, pika. + * flex + * bison + * build-essential + * automake + * libtool + * librabbitmq-dev + * rabbitmq-server + * collectd + * intel-cmt-cat Hardware & Software Ingredients -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------- SUT requirements: - +-----------+--------------------+ - | Item | Description | - +-----------+--------------------+ - | Memory | Min 20GB | - +-----------+--------------------+ - | NICs | 2 x 10G | - +-----------+--------------------+ - | OS | Ubuntu 16.04.3 LTS | - +-----------+--------------------+ - | kernel | 4.4.0-34-generic | - +-----------+--------------------+ - | DPDK | 17.02 | - +-----------+--------------------+ + ======= =================== + Item Description + ======= =================== + Memory Min 20GB + NICs 2 x 10G + OS Ubuntu 16.04.3 LTS + kernel 4.4.0-34-generic + DPDK 17.02 + ======= =================== Boot and BIOS settings: - +------------------+---------------------------------------------------+ - | Boot settings | default_hugepagesz=1G hugepagesz=1G hugepages=16 | - | | hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 | - | | nohz_full=1-11,22-33 rcu_nocbs=1-11,22-33 | - | | iommu=on iommu=pt intel_iommu=on | - | | Note: nohz_full and rcu_nocbs is to disable Linux | - | | kernel interrupts | - +------------------+---------------------------------------------------+ - |BIOS | CPU Power and Performance Policy <Performance> | - | | CPU C-state Disabled | - | | CPU P-state Disabled | - | | Enhanced Intel® Speedstep® Tech Disabled | - | | Hyper-Threading Technology (If supported) Enabled | - | | Virtualization Techology Enabled | - | | Intel(R) VT for Direct I/O Enabled | - | | Coherency Enabled | - | | Turbo Boost Disabled | - +------------------+---------------------------------------------------+ + ============= ================================================= + Boot settings default_hugepagesz=1G hugepagesz=1G hugepages=16 + hugepagesz=2M hugepages=2048 isolcpus=1-11,22-33 + nohz_full=1-11,22-33 rcu_nocbs=1-11,22-33 + iommu=on iommu=pt intel_iommu=on + Note: nohz_full and rcu_nocbs is to disable Linux + kernel interrupts + BIOS CPU Power and Performance Policy <Performance> + CPU C-state Disabled + CPU P-state Disabled + Enhanced Intel® Speedstep® Tech Disabl + Hyper-Threading Technology (If supported) Enabled + Virtualization Techology Enabled + Intel(R) VT for Direct I/O Enabled + Coherency Enabled + Turbo Boost Disabled + ============= ================================================= Install Yardstick (NSB Testing) -------------------------------- +=============================== Download the source code and install Yardstick from it @@ -116,11 +103,13 @@ Configure the network proxy, either using the environment variables or setting the global environment file: .. code-block:: ini + cat /etc/environment http_proxy='http://proxy.company.com:port' https_proxy='http://proxy.company.com:port' .. code-block:: console + export http_proxy='http://proxy.company.com:port' export https_proxy='http://proxy.company.com:port' @@ -128,6 +117,7 @@ The last step is to modify the Yardstick installation inventory, used by Ansible: .. code-block:: ini + cat ./ansible/yardstick-install-inventory.ini [jumphost] localhost ansible_connection=local @@ -145,6 +135,15 @@ Ansible: ansible_user=root ansible_pass=root +.. note:: + + SSH access without password needs to be configured for all your nodes defined in + ``yardstick-install-inventory.ini`` file. + If you want to use password authentication you need to install sshpass + + .. code-block:: console + + sudo -EH apt-get install sshpass To execute an installation for a Bare-Metal or a Standalone context: @@ -165,11 +164,12 @@ Above command setup docker with latest yardstick code. To execute docker exec -it yardstick bash -It will also automatically download all the packages needed for NSB Testing setup. -Refer chapter :doc:`04-installation` for more on docker **Install Yardstick using Docker (recommended)** +It will also automatically download all the packages needed for NSB Testing +setup. Refer chapter :doc:`04-installation` for more on docker +**Install Yardstick using Docker (recommended)** System Topology: ----------------- +================ .. code-block:: console @@ -184,13 +184,15 @@ System Topology: Environment parameters and credentials --------------------------------------- +====================================== Config yardstick conf -^^^^^^^^^^^^^^^^^^^^^ +--------------------- -If user did not run 'yardstick env influxdb' inside the container, which will generate -correct yardstick.conf, then create the config file manually (run inside the container): +If user did not run 'yardstick env influxdb' inside the container, which will +generate correct ``yardstick.conf``, then create the config file manually (run +inside the container): +:: cp ./etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf vi /etc/yardstick/yardstick.conf @@ -216,11 +218,11 @@ Add trex_path, trex_client_lib and bin_path in 'nsb' section. trex_client_lib=/opt/nsb_bin/trex_client/stl Run Yardstick - Network Service Testcases ------------------------------------------ +========================================= NS testing - using yardstick CLI -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------- See :doc:`04-installation` @@ -233,13 +235,13 @@ NS testing - using yardstick CLI yardstick --debug task start yardstick/samples/vnf_samples/nsut/<vnf>/<test case> Network Service Benchmarking - Bare-Metal ------------------------------------------ +========================================= Bare-Metal Config pod.yaml describing Topology -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------------------------------------- -Bare-Metal 2-Node setup: -######################## +Bare-Metal 2-Node setup +^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console +----------+ +----------+ @@ -251,8 +253,8 @@ Bare-Metal 2-Node setup: +----------+ +----------+ trafficgen_1 vnf -Bare-Metal 3-Node setup - Correlated Traffic: -############################################# +Bare-Metal 3-Node setup - Correlated Traffic +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console +----------+ +----------+ +------------+ @@ -267,7 +269,7 @@ Bare-Metal 3-Node setup - Correlated Traffic: Bare-Metal Config pod.yaml -^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------- Before executing Yardstick test cases, make sure that pod.yaml reflects the topology and update all the required fields.:: @@ -342,13 +344,13 @@ topology and update all the required fields.:: Network Service Benchmarking - Standalone Virtualization --------------------------------------------------------- +======================================================== -SR-IOV: -^^^^^^^ +SR-IOV +------ SR-IOV Pre-requisites -##################### +^^^^^^^^^^^^^^^^^^^^^ On Host: a) Create a bridge for VM to connect to external network @@ -384,10 +386,10 @@ On Host: SR-IOV Config pod.yaml describing Topology -########################################## +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ SR-IOV 2-Node setup: -#################### +^^^^^^^^^^^^^^^^^^^^ .. code-block:: console +--------------------+ @@ -415,7 +417,7 @@ SR-IOV 2-Node setup: SR-IOV 3-Node setup - Correlated Traffic -######################################## +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console +--------------------+ @@ -451,7 +453,7 @@ topology and update all the required fields. .. note:: Update all the required fields like ip, user, password, pcis, etc... SR-IOV Config pod_trex.yaml -########################### +^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: YAML @@ -480,7 +482,7 @@ SR-IOV Config pod_trex.yaml local_mac: "00:00.00:00:00:02" SR-IOV Config host_sriov.yaml -############################# +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: YAML @@ -492,7 +494,8 @@ SR-IOV Config host_sriov.yaml user: "" password: "" -SR-IOV testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` +SR-IOV testcase update: +``<yardstick>/samples/vnf_samples/nsut/vfw/tc_sriov_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` Update "contexts" section """"""""""""""""""""""""" @@ -539,11 +542,11 @@ Update "contexts" section -OVS-DPDK: -^^^^^^^^^ +OVS-DPDK +-------- OVS-DPDK Pre-requisites -####################### +^^^^^^^^^^^^^^^^^^^^^^^ On Host: a) Create a bridge for VM to connect to external network @@ -582,10 +585,10 @@ On Host: OVS-DPDK Config pod.yaml describing Topology -############################################ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -OVS-DPDK 2-Node setup: -###################### +OVS-DPDK 2-Node setup +^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -616,7 +619,7 @@ OVS-DPDK 2-Node setup: OVS-DPDK 3-Node setup - Correlated Traffic -########################################## +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: console @@ -656,7 +659,7 @@ topology and update all the required fields. .. note:: Update all the required fields like ip, user, password, pcis, etc... OVS-DPDK Config pod_trex.yaml -############################# +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: YAML @@ -684,7 +687,7 @@ OVS-DPDK Config pod_trex.yaml local_mac: "00:00.00:00:00:02" OVS-DPDK Config host_ovs.yaml -############################# +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: YAML @@ -696,7 +699,8 @@ OVS-DPDK Config host_ovs.yaml user: "" password: "" -ovs_dpdk testcase update: ``<yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` +ovs_dpdk testcase update: +``<yardstick>/samples/vnf_samples/nsut/vfw/tc_ovs_rfc2544_ipv4_1rule_1flow_64B_trex.yaml`` Update "contexts" section """"""""""""""""""""""""" @@ -753,62 +757,309 @@ Update "contexts" section gateway_ip: '152.16.100.20' +Network Service Benchmarking - OpenStack with SR-IOV support +============================================================ + +This section describes how to run a Sample VNF test case, using Heat context, +with SR-IOV. It also covers how to install OpenStack in Ubuntu 16.04, using +DevStack, with SR-IOV support. + + +Single node OpenStack setup with external TG +-------------------------------------------- + +.. code-block:: console + + +----------------------------+ + |OpenStack(DevStack) | + | | + | +--------------------+ | + | |sample-VNF VM | | + | | | | + | | DUT | | + | | (VNF) | | + | | | | + | +--------+ +--------+ | + | | VF NIC | | VF NIC | | + | +-----+--+--+----+---+ | + | ^ ^ | + | | | | + +----------+ +---------+----------+-------+ + | | | VF0 VF1 | + | | | ^ ^ | + | | | | SUT | | + | TG | (PF0)<----->(PF0) +---------+ | | + | | | | | + | | (PF1)<----->(PF1) +--------------------+ | + | | | | + +----------+ +----------------------------+ + trafficgen_1 host + + +Host pre-configuration +^^^^^^^^^^^^^^^^^^^^^^ + +.. warning:: The following configuration requires sudo access to the system. Make + sure that your user have the access. + +Enable the Intel VT-d or AMD-Vi extension in the BIOS. Some system manufacturers +disable this extension by default. + +Activate the Intel VT-d or AMD-Vi extension in the kernel by modifying the GRUB +config file ``/etc/default/grub``. + +For the Intel platform: + +.. code:: bash + + ... + GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on" + ... + +For the AMD platform: + +.. code:: bash + + ... + GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on" + ... + +Update the grub configuration file and restart the system: + +.. warning:: The following command will reboot the system. + +.. code:: bash + + sudo update-grub + sudo reboot + +Make sure the extension has been enabled: + +.. code:: bash + + sudo journalctl -b 0 | grep -e IOMMU -e DMAR + + Feb 06 14:50:14 hostname kernel: ACPI: DMAR 0x000000006C406000 0001E0 (v01 INTEL S2600WF 00000001 INTL 20091013) + Feb 06 14:50:14 hostname kernel: DMAR: IOMMU enabled + Feb 06 14:50:14 hostname kernel: DMAR: Host address width 46 + Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000d37fc000 flags: 0x0 + Feb 06 14:50:14 hostname kernel: DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020de + Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0 + Feb 06 14:50:14 hostname kernel: DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020de + Feb 06 14:50:14 hostname kernel: DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 + +Setup system proxy (if needed). Add the following configuration into the +``/etc/environment`` file: + +.. note:: The proxy server name/port and IPs should be changed according to + actuall/current proxy configuration in the lab. + +.. code:: bash + + export http_proxy=http://proxy.company.com:port + export https_proxy=http://proxy.company.com:port + export ftp_proxy=http://proxy.company.com:port + export no_proxy=localhost,127.0.0.1,company.com,<IP-OF-HOST1>,<IP-OF-HOST2>,... + export NO_PROXY=localhost,127.0.0.1,company.com,<IP-OF-HOST1>,<IP-OF-HOST2>,... + +Upgrade the system: + +.. code:: bash + + sudo -EH apt-get update + sudo -EH apt-get upgrade + sudo -EH apt-get dist-upgrade + +Install dependencies needed for the DevStack + +.. code:: bash + + sudo -EH apt-get install python + sudo -EH apt-get install python-dev + sudo -EH apt-get install python-pip + +Setup SR-IOV ports on the host: + +.. note:: The ``enp24s0f0``, ``enp24s0f0`` are physical function (PF) interfaces + on a host and ``enp24s0f3`` is a public interface used in OpenStack, so the + interface names should be changed according to the HW environment used for + testing. + +.. code:: bash + + sudo ip link set dev enp24s0f0 up + sudo ip link set dev enp24s0f1 up + sudo ip link set dev enp24s0f3 up + + # Create VFs on PF + echo 2 | sudo tee /sys/class/net/enp24s0f0/device/sriov_numvfs + echo 2 | sudo tee /sys/class/net/enp24s0f1/device/sriov_numvfs + + +DevStack installation +^^^^^^^^^^^^^^^^^^^^^ + +Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ +documentation to install OpenStack on a host. Please note, that stable +``pike`` branch of devstack repo should be used during the installation. +The required `local.conf`` configuration file are described below. + +DevStack configuration file: + +.. note:: Update the devstack configuration file by replacing angluar brackets + with a short description inside. + +.. note:: Use ``lspci | grep Ether`` & ``lspci -n | grep <PCI ADDRESS>`` + commands to get device and vendor id of the virtual function (VF). + +.. literalinclude:: code/single-devstack-local.conf + :language: console + +Start the devstack installation on a host. + + +TG host configuration +^^^^^^^^^^^^^^^^^^^^^ + +Yardstick automatically install and configure Trex traffic generator on TG +host based on provided POD file (see below). Anyway, it's recommended to check +the compatibility of the installed NIC on the TG server with software Trex using +the manual at https://trex-tgn.cisco.com/trex/doc/trex_manual.html. + + +Run the Sample VNF test case +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There is an example of Sample VNF test case ready to be executed in an +OpenStack environment with SR-IOV support: ``samples/vnf_samples/nsut/vfw/ +tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``. + +Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +context. + +Create pod file for TG in the yardstick repo folder located in the yardstick +container: + +.. note:: The ``ip``, ``user``, ``password`` and ``vpci`` fields show be changed + according to HW environment used for the testing. Use ``lshw -c network -businfo`` + command to get the PF PCI address for ``vpci`` field. + +.. literalinclude:: code/single-yardstick-pod.conf + :language: console + +Run the Sample vFW RFC2544 SR-IOV TC (``samples/vnf_samples/nsut/vfw/ +tc_heat_sriov_external_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) in the heat +context using steps described in `NS testing - using yardstick CLI`_ section. + + +Multi node OpenStack TG and VNF setup (two nodes) +------------------------------------------------- + +.. code-block:: console + + +----------------------------+ +----------------------------+ + |OpenStack(DevStack) | |OpenStack(DevStack) | + | | | | + | +--------------------+ | | +--------------------+ | + | |sample-VNF VM | | | |sample-VNF VM | | + | | | | | | | | + | | TG | | | | DUT | | + | | trafficgen_1 | | | | (VNF) | | + | | | | | | | | + | +--------+ +--------+ | | +--------+ +--------+ | + | | VF NIC | | VF NIC | | | | VF NIC | | VF NIC | | + | +----+---+--+----+---+ | | +-----+--+--+----+---+ | + | ^ ^ | | ^ ^ | + | | | | | | | | + +--------+-----------+-------+ +---------+----------+-------+ + | VF0 VF1 | | VF0 VF1 | + | ^ ^ | | ^ ^ | + | | SUT2 | | | | SUT1 | | + | | +-------+ (PF0)<----->(PF0) +---------+ | | + | | | | | | + | +-------------------+ (PF1)<----->(PF1) +--------------------+ | + | | | | + +----------------------------+ +----------------------------+ + host2 (compute) host1 (controller) + + +Controller/Compute pre-configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Pre-configuration of the controller and compute hosts are the same as +described in `Host pre-configuration`_ section. Follow the steps in the section. + + +DevStack configuration +^^^^^^^^^^^^^^^^^^^^^^ + +Use official `Devstack <https://docs.openstack.org/devstack/pike/>`_ +documentation to install OpenStack on a host. Please note, that stable +``pike`` branch of devstack repo should be used during the installation. +The required `local.conf`` configuration file are described below. + +.. note:: Update the devstack configuration files by replacing angluar brackets + with a short description inside. + +.. note:: Use ``lspci | grep Ether`` & ``lspci -n | grep <PCI ADDRESS>`` + commands to get device and vendor id of the virtual function (VF). + +DevStack configuration file for controller host: + +.. literalinclude:: code/multi-devstack-controller-local.conf + :language: console + +DevStack configuration file for compute host: + +.. literalinclude:: code/multi-devstack-compute-local.conf + :language: console + +Start the devstack installation on the controller and compute hosts. + + +Run the sample vFW TC +^^^^^^^^^^^^^^^^^^^^^ + +Install yardstick using `Install Yardstick (NSB Testing)`_ steps for OpenStack +context. + +Run sample vFW RFC2544 SR-IOV TC (``samples/vnf_samples/nsut/vfw/ +tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml``) in the heat +context using steps described in `NS testing - using yardstick CLI`_ section +and the following yardtick command line arguments: + +.. code:: bash + + yardstick -d task start --task-args='{"provider": "sriov"}' \ + samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml + + Enabling other Traffic generator --------------------------------- +================================ -IxLoad: -^^^^^^^ +IxLoad +^^^^^^ -1. Software needed: IxLoadAPI ``<IxLoadTclApi verson>Linux64.bin.tgz and <IxOS version>Linux64.bin.tar.gz`` (Download from ixia support site) - Install - ``<IxLoadTclApi verson>Linux64.bin.tgz & <IxOS version>Linux64.bin.tar.gz`` - If the installation was not done inside the container, after installing the IXIA client, - check /opt/ixia/ixload/<ver>/bin/ixloadpython and make sure you can run this cmd - inside the yardstick container. Usually user is required to copy or link /opt/ixia/python/<ver>/bin/ixiapython - to /usr/bin/ixiapython<ver> inside the container. +1. Software needed: IxLoadAPI ``<IxLoadTclApi verson>Linux64.bin.tgz`` and + ``<IxOS version>Linux64.bin.tar.gz`` (Download from ixia support site) + Install - ``<IxLoadTclApi verson>Linux64.bin.tgz`` and + ``<IxOS version>Linux64.bin.tar.gz`` + If the installation was not done inside the container, after installing + the IXIA client, check ``/opt/ixia/ixload/<ver>/bin/ixloadpython`` and make + sure you can run this cmd inside the yardstick container. Usually user is + required to copy or link ``/opt/ixia/python/<ver>/bin/ixiapython`` to + ``/usr/bin/ixiapython<ver>`` inside the container. -2. Update pod_ixia.yaml file with ixia details. +2. Update ``pod_ixia.yaml`` file with ixia details. .. code-block:: console cp <repo>/etc/yardstick/nodes/pod.yaml.nsb.sample.ixia etc/yardstick/nodes/pod_ixia.yaml - Config pod_ixia.yaml + Config ``pod_ixia.yaml`` - .. code-block:: yaml - - - nodes: - - - name: trafficgen_1 - role: IxNet - ip: 1.2.1.1 #ixia machine ip - user: user - password: r00t - key_filename: /root/.ssh/id_rsa - tg_config: - ixchassis: "1.2.1.7" #ixia chassis ip - tcl_port: "8009" # tcl server port - lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0" - root_dir: "/opt/ixia/ixos-api/8.01.0.2/" - py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/" - py_lib_path: "/opt/ixia/ixnetwork/8.01.1029.14/lib/PythonApi" - dut_result_dir: "/mnt/ixia" - version: 8.1 - interfaces: - xe0: # logical name from topology.yaml and vnfd.yaml - vpci: "2:5" # Card:port - driver: "none" - dpdk_port_num: 0 - local_ip: "152.16.100.20" - netmask: "255.255.0.0" - local_mac: "00:98:10:64:14:00" - xe1: # logical name from topology.yaml and vnfd.yaml - vpci: "2:6" # [(Card, port)] - driver: "none" - dpdk_port_num: 1 - local_ip: "152.40.40.20" - netmask: "255.255.0.0" - local_mac: "00:98:28:28:14:00" + .. literalinclude:: code/pod_ixia.yaml + :language: console for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration @@ -816,23 +1067,24 @@ IxLoad: You will also need to configure the IxLoad machine to start the IXIA IxosTclServer. This can be started like so: - - Connect to the IxLoad machine using RDP - - Go to: - ``Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1`` + * Connect to the IxLoad machine using RDP + * Go to: + ``Start->Programs->Ixia->IxOS->IxOS 8.01-GA-Patch1->Ixia Tcl Server IxOS 8.01-GA-Patch1`` or - ``"C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"`` + ``"C:\Program Files (x86)\Ixia\IxOS\8.01-GA-Patch1\ixTclServer.exe"`` + +4. Create a folder ``Results`` in c:\ and share the folder on the network. -4. Create a folder "Results" in c:\ and share the folder on the network. +5. Execute testcase in samplevnf folder e.g. + ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml`` -5. execute testcase in samplevnf folder. - eg ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_http_ixload_1b_Requests-65000_Concurrency.yaml`` +IxNetwork +--------- -IxNetwork: -^^^^^^^^^^ +IxNetwork testcases use IxNetwork API Python Bindings module, which is +installed as part of the requirements of the project. -1. Software needed: ``IxNetworkAPI<ixnetwork verson>Linux64.bin.tgz`` (Download from ixia support site) - Install - ``IxNetworkAPI<ixnetwork verson>Linux64.bin.tgz`` -2. Update pod_ixia.yaml file with ixia details. +1. Update ``pod_ixia.yaml`` file with ixia details. .. code-block:: console @@ -840,50 +1092,19 @@ IxNetwork: Config pod_ixia.yaml - .. code-block:: yaml - - nodes: - - - name: trafficgen_1 - role: IxNet - ip: 1.2.1.1 #ixia machine ip - user: user - password: r00t - key_filename: /root/.ssh/id_rsa - tg_config: - ixchassis: "1.2.1.7" #ixia chassis ip - tcl_port: "8009" # tcl server port - lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0" - root_dir: "/opt/ixia/ixos-api/8.01.0.2/" - py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/" - py_lib_path: "/opt/ixia/ixnetwork/8.01.1029.14/lib/PythonApi" - dut_result_dir: "/mnt/ixia" - version: 8.1 - interfaces: - xe0: # logical name from topology.yaml and vnfd.yaml - vpci: "2:5" # Card:port - driver: "none" - dpdk_port_num: 0 - local_ip: "152.16.100.20" - netmask: "255.255.0.0" - local_mac: "00:98:10:64:14:00" - xe1: # logical name from topology.yaml and vnfd.yaml - vpci: "2:6" # [(Card, port)] - driver: "none" - dpdk_port_num: 1 - local_ip: "152.40.40.20" - netmask: "255.255.0.0" - local_mac: "00:98:28:28:14:00" + .. literalinclude:: code/pod_ixia.yaml + :language: console for sriov/ovs_dpdk pod files, please refer to above Standalone Virtualization for ovs-dpdk/sriov configuration -3. Start IxNetwork TCL Server +2. Start IxNetwork TCL Server You will also need to configure the IxNetwork machine to start the IXIA IxNetworkTclServer. This can be started like so: - - Connect to the IxNetwork machine using RDP - - Go to: ``Start->Programs->Ixia->IxNetwork->IxNetwork 7.21.893.14 GA->IxNetworkTclServer`` (or ``IxNetworkApiServer``) - -4. execute testcase in samplevnf folder. - eg ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml`` + * Connect to the IxNetwork machine using RDP + * Go to: + ``Start->Programs->Ixia->IxNetwork->IxNetwork 7.21.893.14 GA->IxNetworkTclServer`` + (or ``IxNetworkApiServer``) +3. Execute testcase in samplevnf folder e.g. + ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_ixia.yaml`` diff --git a/docs/testing/user/userguide/13-nsb_operation.rst b/docs/testing/user/userguide/14-nsb-operation.rst index 8c477fa3f..2e741822e 100644 --- a/docs/testing/user/userguide/13-nsb_operation.rst +++ b/docs/testing/user/userguide/14-nsb-operation.rst @@ -23,13 +23,14 @@ provider/external networks. Provider networks ^^^^^^^^^^^^^^^^^ -The VNFs require a clear L2 connect to the external network in order to generate -realistic traffic from multiple address ranges and port +The VNFs require a clear L2 connect to the external network in order to +generate realistic traffic from multiple address ranges and ports. -In order to prevent Neutron from filtering traffic we have to disable Neutron Port Security. -We also disable DHCP on the data ports because we are binding the ports to DPDK and do not need -DHCP addresses. We also disable gateways because multiple default gateways can prevent SSH access -to the VNF from the floating IP. We only want a gateway on the mgmt network +In order to prevent Neutron from filtering traffic we have to disable Neutron +Port Security. We also disable DHCP on the data ports because we are binding +the ports to DPDK and do not need DHCP addresses. We also disable gateways +because multiple default gateways can prevent SSH access to the VNF from the +floating IP. We only want a gateway on the mgmt network .. code-block:: yaml @@ -42,8 +43,9 @@ to the VNF from the floating IP. We only want a gateway on the mgmt network Heat Topologies ^^^^^^^^^^^^^^^ -By default Heat will attach every node to every Neutron network that is created. -For scale-out tests we do not want to attach every node to every network. +By default Heat will attach every node to every Neutron network that is +created. For scale-out tests we do not want to attach every node to every +network. For each node you can specify which ports are on which network using the network_ports dictionary. @@ -85,11 +87,11 @@ In this example we have ``TRex xe0 <-> xe0 VNF xe1 <-> xe0 UDP_Replay`` Collectd KPIs ------------- -NSB can collect KPIs from collected. We have support for various plugins enabled by the -Barometer project. +NSB can collect KPIs from collected. We have support for various plugins +enabled by the Barometer project. -The default yardstick-samplevnf has collectd installed. This allows for collecting KPIs -from the VNF. +The default yardstick-samplevnf has collectd installed. This allows for +collecting KPIs from the VNF. Collecting KPIs from the NFVi is more complicated and requires manual setup. We assume that collectd is not installed on the compute nodes. @@ -126,40 +128,80 @@ To collectd KPIs from the NFVi compute nodes: Scale-Up ------------------- +-------- VNFs performance data with scale-up - * Helps to figure out optimal number of cores specification in the Virtual Machine template creation or VNF + * Helps to figure out optimal number of cores specification in the Virtual + Machine template creation or VNF * Helps in comparison between different VNF vendor offerings - * Better the scale-up index, indicates the performance scalability of a particular solution + * Better the scale-up index, indicates the performance scalability of a + particular solution Heat ^^^^ - -For VNF scale-up tests we increase the number for VNF worker threads. In the case of VNFs -we also need to increase the number of VCPUs and memory allocated to the VNF. +For VNF scale-up tests we increase the number for VNF worker threads. In the +case of VNFs we also need to increase the number of VCPUs and memory allocated +to the VNF. An example scale-up Heat testcase is: +.. literalinclude:: /submodules/yardstick/samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml + :language: yaml + +This testcase template requires specifying the number of VCPUs, Memory and Ports. +We set the VCPUs and memory using the ``--task-args`` options + .. code-block:: console - <repo>/samples/vnf_samples/nsut/acl/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml + yardstick task start --task-args='{"mem": 10480, "vcpus": 4, "ports": 2}' \ + samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml -This testcase template requires specifying the number of VCPUs and Memory. -We set the VCPUs and memory using the --task-args options +In order to support ports scale-up, traffic and topology templates need to be used in testcase. -.. code-block:: console +A example topology template is: + +.. literalinclude:: /submodules/yardstick/samples/vnf_samples/nsut/vfw/vfw-tg-topology-scale-up.yaml + :language: yaml + +This template has ``vports`` as an argument. To pass this argument it needs to +be configured in ``extra_args`` scenario definition. Please note that more +argument can be defined in that section. All of them will be passed to topology +and traffic profile templates + +For example: + +.. code-block:: yaml - yardstick --debug task start --task-args='{"mem": 20480, "vcpus": 10}' samples/vnf_samples/nsut/acl/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml + schema: yardstick:task:0.1 + scenarios: + - type: NSPerf + traffic_profile: ../../traffic_profiles/ipv4_throughput-scale-up.yaml + extra_args: + vports: {{ vports }} + topology: vfw-tg-topology-scale-up.yaml + +A example traffic profile template is: + +.. literalinclude:: /submodules/yardstick/samples/vnf_samples/traffic_profiles/ipv4_throughput-scale-up.yaml + :language: yaml + +There is an option to provide predefined config for SampleVNFs. Path to config +file may by specified in ``vnf_config`` scenario section. + +.. code-block:: yaml + + vnf__0: + rules: acl_1rule.yaml + vnf_config: {lb_config: 'SW', file: vfw_vnf_pipeline_cores_4_ports_2_lb_1_sw.conf } Baremetal ^^^^^^^^^ 1. Follow above traffic generator section to setup. - 2. edit num of threads in ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml`` - - e.g, 6 Threads for given VNF + 2. Edit num of threads in + ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_1rule_1flow_64B_trex_scale_up.yaml`` + e.g, 6 Threads for given VNF .. code-block:: yaml @@ -202,11 +244,12 @@ Baremetal Scale-Out -------------------- -VNFs performance data with scale-out +VNFs performance data with scale-out helps - * Helps in capacity planning to meet the given network node requirements - * Helps in comparison between different VNF vendor offerings - * Better the scale-out index, provides the flexibility in meeting future capacity requirements + * in capacity planning to meet the given network node requirements + * in comparison between different VNF vendor offerings + * better the scale-out index, provides the flexibility in meeting future + capacity requirements Standalone @@ -236,7 +279,8 @@ Scale-out not supported on Baremetal. Heat ^^^^ -There are sample scale-out all-VM Heat tests. These tests only use VMs and don't use external traffic. +There are sample scale-out all-VM Heat tests. These tests only use VMs and +don't use external traffic. The tests use UDP_Replay and correlated traffic. @@ -250,11 +294,14 @@ To run the test you need to increase OpenStack CPU, Memory and Port quotas. Traffic Generator tuning ------------------------ -The TRex traffic generator can be setup to use multiple threads per core, this is for multiqueue testing. +The TRex traffic generator can be setup to use multiple threads per core, this +is for multiqueue testing. -TRex does not automatically enable multiple threads because we currently cannot detect the number of queues on a device. +TRex does not automatically enable multiple threads because we currently cannot +detect the number of queues on a device. -To enable multiple queue set the queues_per_port value in the TG VNF options section. +To enable multiple queue set the ``queues_per_port`` value in the TG VNF +options section. .. code-block:: yaml @@ -266,5 +313,3 @@ To enable multiple queue set the queues_per_port value in the TG VNF options sec options: tg_0: queues_per_port: 2 - - diff --git a/docs/testing/user/userguide/15-list-of-tcs.rst b/docs/testing/user/userguide/15-list-of-tcs.rst index 47526cdda..37ce819f1 100644 --- a/docs/testing/user/userguide/15-list-of-tcs.rst +++ b/docs/testing/user/userguide/15-list-of-tcs.rst @@ -14,11 +14,11 @@ This chapter lists available Yardstick test cases. Yardstick test cases are divided in two main categories: * *Generic NFVI Test Cases* - Test Cases developed to realize the methodology -described in :doc:`02-methodology` + described in :doc:`02-methodology` * *OPNFV Feature Test Cases* - Test Cases developed to verify one or more -aspect of a feature delivered by an OPNFV Project, including the test cases -developed for the :term:`VTC`. + aspect of a feature delivered by an OPNFV Project, including the test cases + developed for the :term:`VTC`. Generic NFVI Test Case Descriptions =================================== @@ -83,6 +83,9 @@ H A opnfv_yardstick_tc056.rst opnfv_yardstick_tc057.rst opnfv_yardstick_tc058.rst + opnfv_yardstick_tc087.rst + opnfv_yardstick_tc092.rst + opnfv_yardstick_tc093.rst IPv6 ---- @@ -108,8 +111,8 @@ Parser opnfv_yardstick_tc040.rst - StorPerf ------------ +StorPerf +-------- .. toctree:: :maxdepth: 1 diff --git a/docs/testing/user/userguide/code/multi-devstack-compute-local.conf b/docs/testing/user/userguide/code/multi-devstack-compute-local.conf new file mode 100644 index 000000000..b0b3cc5d4 --- /dev/null +++ b/docs/testing/user/userguide/code/multi-devstack-compute-local.conf @@ -0,0 +1,53 @@ +[[local|localrc]] +HOST_IP=<HOST_IP_ADDRESS> +MYSQL_PASSWORD=password +DATABASE_PASSWORD=password +RABBIT_PASSWORD=password +ADMIN_PASSWORD=password +SERVICE_PASSWORD=password +HORIZON_PASSWORD=password +# Controller node +SERVICE_HOST=<CONTROLLER_IP_ADDRESS> +MYSQL_HOST=$SERVICE_HOST +RABBIT_HOST=$SERVICE_HOST +GLANCE_HOSTPORT=$SERVICE_HOST:9292 + +# Internet access. +RECLONE=False +PIP_UPGRADE=True +IP_VERSION=4 + +# Neutron +enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike + +# Services +ENABLED_SERVICES=n-cpu,rabbit,q-agt,placement-api,q-sriov-agt + +# Neutron Options +PUBLIC_INTERFACE=<PUBLIC INTERFACE> + +# ML2 Configuration +Q_PLUGIN=ml2 +Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch +Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve + +# Open vSwitch provider networking configuration +PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME> + + +[[post-config|$NOVA_CONF]] +[DEFAULT] +scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter +# Whitelist PCI devices +pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" } +pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" } + +[libvirt] +cpu_mode = host-model + + +# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function +[[post-config|/$Q_PLUGIN_CONF_FILE]] +[ml2_sriov] +agent_required = True +supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID> diff --git a/docs/testing/user/userguide/code/multi-devstack-controller-local.conf b/docs/testing/user/userguide/code/multi-devstack-controller-local.conf new file mode 100644 index 000000000..fb61cdcbd --- /dev/null +++ b/docs/testing/user/userguide/code/multi-devstack-controller-local.conf @@ -0,0 +1,64 @@ +[[local|localrc]] +HOST_IP=<HOST_IP_ADDRESS> +ADMIN_PASSWORD=password +MYSQL_PASSWORD=$ADMIN_PASSWORD +DATABASE_PASSWORD=$ADMIN_PASSWORD +RABBIT_PASSWORD=$ADMIN_PASSWORD +SERVICE_PASSWORD=$ADMIN_PASSWORD +HORIZON_PASSWORD=$ADMIN_PASSWORD +# Controller node +SERVICE_HOST=$HOST_IP +MYSQL_HOST=$SERVICE_HOST +RABBIT_HOST=$SERVICE_HOST +GLANCE_HOSTPORT=$SERVICE_HOST:9292 + +# Internet access. +RECLONE=False +PIP_UPGRADE=True +IP_VERSION=4 + +# Services +disable_service n-net +ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt,q-sriov-agt + +# Heat +enable_plugin heat https://git.openstack.org/openstack/heat stable/pike + +# Neutron +enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike + +# Neutron Options +FLOATING_RANGE=<RANGE_IN_THE_PUBLIC_INTERFACE_NETWORK> +Q_FLOATING_ALLOCATION_POOL=start=<START_IP_ADDRESS>,end=<END_IP_ADDRESS> +PUBLIC_NETWORK_GATEWAY=<PUBLIC_NETWORK_GATEWAY> +PUBLIC_INTERFACE=<PUBLIC INTERFACE> + +# ML2 Configuration +Q_PLUGIN=ml2 +Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch +Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve + +# Open vSwitch provider networking configuration +Q_USE_PROVIDERNET_FOR_PUBLIC=True +OVS_PHYSICAL_BRIDGE=br-ex +OVS_BRIDGE_MAPPINGS=public:br-ex +PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME> +PHYSICAL_NETWORK=physnet1,physnet2 + + +[[post-config|$NOVA_CONF]] +[DEFAULT] +scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter +# Whitelist PCI devices +pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" } +pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" } + +[libvirt] +cpu_mode = host-model + + +# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function +[[post-config|/$Q_PLUGIN_CONF_FILE]] +[ml2_sriov] +agent_required = True +supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID> diff --git a/docs/testing/user/userguide/code/pod_ixia.yaml b/docs/testing/user/userguide/code/pod_ixia.yaml new file mode 100644 index 000000000..4ab56fe4e --- /dev/null +++ b/docs/testing/user/userguide/code/pod_ixia.yaml @@ -0,0 +1,31 @@ +nodes: +- + name: trafficgen_1 + role: IxNet + ip: 1.2.1.1 #ixia machine ip + user: user + password: r00t + key_filename: /root/.ssh/id_rsa + tg_config: + ixchassis: "1.2.1.7" #ixia chassis ip + tcl_port: "8009" # tcl server port + lib_path: "/opt/ixia/ixos-api/8.01.0.2/lib/ixTcl1.0" + root_dir: "/opt/ixia/ixos-api/8.01.0.2/" + py_bin_path: "/opt/ixia/ixload/8.01.106.3/bin/" + dut_result_dir: "/mnt/ixia" + version: 8.1 + interfaces: + xe0: # logical name from topology.yaml and vnfd.yaml + vpci: "2:5" # Card:port + driver: "none" + dpdk_port_num: 0 + local_ip: "152.16.100.20" + netmask: "255.255.0.0" + local_mac: "00:98:10:64:14:00" + xe1: # logical name from topology.yaml and vnfd.yaml + vpci: "2:6" # [(Card, port)] + driver: "none" + dpdk_port_num: 1 + local_ip: "152.40.40.20" + netmask: "255.255.0.0" + local_mac: "00:98:28:28:14:00" diff --git a/docs/testing/user/userguide/code/single-devstack-local.conf b/docs/testing/user/userguide/code/single-devstack-local.conf new file mode 100644 index 000000000..4c44f729d --- /dev/null +++ b/docs/testing/user/userguide/code/single-devstack-local.conf @@ -0,0 +1,62 @@ +[[local|localrc]] +HOST_IP=<HOST_IP_ADDRESS> +ADMIN_PASSWORD=password +MYSQL_PASSWORD=$ADMIN_PASSWORD +DATABASE_PASSWORD=$ADMIN_PASSWORD +RABBIT_PASSWORD=$ADMIN_PASSWORD +SERVICE_PASSWORD=$ADMIN_PASSWORD +HORIZON_PASSWORD=$ADMIN_PASSWORD + +# Internet access. +RECLONE=False +PIP_UPGRADE=True +IP_VERSION=4 + +# Services +disable_service n-net +ENABLED_SERVICES+=,q-svc,q-dhcp,q-meta,q-agt,q-sriov-agt + +# Heat +enable_plugin heat https://git.openstack.org/openstack/heat stable/pike + +# Neutron +enable_plugin neutron https://git.openstack.org/openstack/neutron.git stable/pike + +# Neutron Options +FLOATING_RANGE=<RANGE_IN_THE_PUBLIC_INTERFACE_NETWORK> +Q_FLOATING_ALLOCATION_POOL=start=<START_IP_ADDRESS>,end=<END_IP_ADDRESS> +PUBLIC_NETWORK_GATEWAY=<PUBLIC_NETWORK_GATEWAY> +PUBLIC_INTERFACE=<PUBLIC INTERFACE> + +# ML2 Configuration +Q_PLUGIN=ml2 +Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,sriovnicswitch +Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,flat,local,vxlan,gre,geneve + +# Open vSwitch provider networking configuration +Q_USE_PROVIDERNET_FOR_PUBLIC=True +OVS_PHYSICAL_BRIDGE=br-ex +OVS_BRIDGE_MAPPINGS=public:br-ex +PHYSICAL_DEVICE_MAPPINGS=physnet1:<PF0_IFNAME>,physnet2:<PF1_IFNAME> +PHYSICAL_NETWORK=physnet1,physnet2 + + +[[post-config|$NOVA_CONF]] +[DEFAULT] +scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter +# Whitelist PCI devices +pci_passthrough_whitelist = {\\"devname\\": \\"<PF0_IFNAME>\\", \\"physical_network\\": \\"physnet1\\" } +pci_passthrough_whitelist = {\\"devname\\": \\"<PF1_IFNAME>\\", \\"physical_network\\": \\"physnet2\\" } + +[filter_scheduler] +enabled_filters = RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter + +[libvirt] +cpu_mode = host-model + + +# ML2 plugin bits for SR-IOV enablement of Intel Corporation XL710/X710 Virtual Function +[[post-config|/$Q_PLUGIN_CONF_FILE]] +[ml2_sriov] +agent_required = True +supported_pci_vendor_devs = <VF_DEV_ID:VF_VEN_ID> diff --git a/docs/testing/user/userguide/code/single-yardstick-pod.conf b/docs/testing/user/userguide/code/single-yardstick-pod.conf new file mode 100644 index 000000000..421246d60 --- /dev/null +++ b/docs/testing/user/userguide/code/single-yardstick-pod.conf @@ -0,0 +1,22 @@ +nodes: +- + name: trafficgen_1 + role: tg__0 + ip: <TG-HOST-IP> + user: <TG-USER> + password: <TG-PASS> + interfaces: + xe0: # logical name from topology.yaml and vnfd.yaml + vpci: "0000:18:00.0" + driver: i40e # default kernel driver + dpdk_port_num: 0 + local_ip: "10.1.1.150" + netmask: "255.255.255.0" + local_mac: "00:00:00:00:00:01" + xe1: # logical name from topology.yaml and vnfd.yaml + vpci: "0000:18:00.1" + driver: i40e # default kernel driver + dpdk_port_num: 1 + local_ip: "10.1.1.151" + netmask: "255.255.255.0" + local_mac: "00:00:00:00:00:02" diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 61e157e52..b936e723d 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -17,15 +17,16 @@ Yardstick User Guide 02-methodology 03-architecture 04-installation - 05-yardstick_plugin - 06-result-store-InfluxDB - 07-grafana - 08-api - 09-yardstick_user_interface - 10-vtc-overview - 11-nsb-overview - 12-nsb_installation - 13-nsb_operation + 05-operation + 06-yardstick-plugin + 07-result-store-InfluxDB + 08-grafana + 09-api + 10-yardstick-user-interface + 11-vtc-overview + 12-nsb-overview + 13-nsb-installation + 14-nsb-operation 15-list-of-tcs nsb/nsb-list-of-tcs glossary diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc019.rst b/docs/testing/user/userguide/opnfv_yardstick_tc019.rst index 57e8ddf79..8d79e011a 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc019.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc019.rst @@ -125,7 +125,12 @@ Yardstick Test Case Description TC019 +--------------+--------------------------------------------------------------+ |post-action | It is the action when the test cases exist. It will check | | | the status of the specified process on the host, and restart | -| | the process if it is not running for next test cases | +| | the process if it is not running for next test cases. | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc045.rst b/docs/testing/user/userguide/opnfv_yardstick_tc045.rst index 0b0993c34..378176090 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc045.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc045.rst @@ -128,9 +128,14 @@ Yardstick Test Case Description TC045 | | Result: The test case is passed or not. | | | | +--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the specified process on the host, and restart | +| | the process if it is not running for next test cases. | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc046.rst b/docs/testing/user/userguide/opnfv_yardstick_tc046.rst index cce6c6884..5308c8e7b 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc046.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc046.rst @@ -127,9 +127,14 @@ Yardstick Test Case Description TC046 | | Result: The test case is passed or not. | | | | +--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the specified process on the host, and restart | +| | the process if it is not running for next test cases. | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc047.rst b/docs/testing/user/userguide/opnfv_yardstick_tc047.rst index 95158cfd6..bb8ffc6ab 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc047.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc047.rst @@ -128,9 +128,14 @@ Yardstick Test Case Description TC047 | | Result: The test case is passed or not. | | | | +--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the specified process on the host, and restart | +| | the process if it is not running for next test cases. | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc048.rst b/docs/testing/user/userguide/opnfv_yardstick_tc048.rst index 21c00d1fe..1bf627282 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc048.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc048.rst @@ -128,9 +128,14 @@ Yardstick Test Case Description TC048 | | Result: The test case is passed or not. | | | | +--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the specified process on the host, and restart | +| | the process if it is not running for next test case | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc049.rst b/docs/testing/user/userguide/opnfv_yardstick_tc049.rst index f58bb9989..12ed94b7d 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc049.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc049.rst @@ -128,9 +128,14 @@ Yardstick Test Case Description TC049 | | Result: The test case is passed or not. | | | | +--------------+--------------------------------------------------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the specified process on the host, and restart the | -| | process if it is not running for next test cases | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the specified process on the host, and restart | +| | the process if it is not running for next test cases. | +| | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | | | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc050.rst b/docs/testing/user/userguide/opnfv_yardstick_tc050.rst index 8890c9d53..82a491b72 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc050.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc050.rst @@ -34,23 +34,20 @@ Yardstick Test Case Description TC050 | | 2) host: which is the name of a control node being attacked. | | | 3) interface: the network interface to be turned off. | | | | -| | There are four instance of the "close-interface" monitor: | -| | attacker1(for public netork): | -| | -fault_type: "close-interface" | -| | -host: node1 | -| | -interface: "br-ex" | -| | attacker2(for management netork): | -| | -fault_type: "close-interface" | -| | -host: node1 | -| | -interface: "br-mgmt" | -| | attacker3(for storage netork): | -| | -fault_type: "close-interface" | -| | -host: node1 | -| | -interface: "br-storage" | -| | attacker4(for private netork): | -| | -fault_type: "close-interface" | -| | -host: node1 | -| | -interface: "br-mesh" | +| | The interface to be closed by the attacker can be set by the | +| | variable of "{{ interface_name }}" | +| | | +| | attackers: | +| | - | +| | fault_type: "general-attacker" | +| | host: {{ attack_host }} | +| | key: "close-br-public" | +| | attack_key: "close-interface" | +| | action_parameter: | +| | interface: {{ interface_name }} | +| | rollback_parameter: | +| | interface: {{ interface_name }} | +| | | +--------------+--------------------------------------------------------------+ |monitors | In this test case, the monitor named "openstack-cmd" is | | | needed. The monitor needs needs two parameters: | @@ -61,17 +58,17 @@ Yardstick Test Case Description TC050 | | | | | There are four instance of the "openstack-cmd" monitor: | | | monitor1: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "nova image-list" | +| | - monitor_type: "openstack-cmd" | +| | - command_name: "nova image-list" | | | monitor2: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "neutron router-list" | +| | - monitor_type: "openstack-cmd" | +| | - command_name: "neutron router-list" | | | monitor3: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "heat stack-list" | +| | - monitor_type: "openstack-cmd" | +| | - command_name: "heat stack-list" | | | monitor4: | -| | -monitor_type: "openstack-cmd" | -| | -command_name: "cinder list" | +| | - monitor_type: "openstack-cmd" | +| | - command_name: "cinder list" | +--------------+--------------------------------------------------------------+ |metrics | In this test case, there is one metric: | | | 1)service_outage_time: which indicates the maximum outage | @@ -109,9 +106,9 @@ Yardstick Test Case Description TC050 +--------------+--------------------------------------------------------------+ |step 2 | do attacker: connect the host through SSH, and then execute | | | the turnoff network interface script with param value | -| | specified by "interface". | +| | specified by "{{ interface_name }}". | | | | -| | Result: Network interfaces will be turned down. | +| | Result: The specified network interface will be down. | | | | +--------------+--------------------------------------------------------------+ |step 3 | stop monitors after a period of time specified by | @@ -133,3 +130,4 @@ Yardstick Test Case Description TC050 | | execution problem. | | | | +--------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc053.rst b/docs/testing/user/userguide/opnfv_yardstick_tc053.rst index 3c6bbc628..7308babb8 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc053.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc053.rst @@ -135,6 +135,11 @@ Yardstick Test Case Description TC053 | | the status of the specified process on the host, and restart | | | the process if it is not running for next test cases. | | | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | +| | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc056.rst b/docs/testing/user/userguide/opnfv_yardstick_tc056.rst index e6e06df57..cd8cc2f20 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc056.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc056.rst @@ -10,7 +10,7 @@ Yardstick Test Case Description TC056 +-----------------------------------------------------------------------------+ |OpenStack Controller Messaging Queue Service High Availability | -+==============+==============================================================+ ++--------------+--------------------------------------------------------------+ |test case id | OPNFV_YARDSTICK_TC056:OpenStack Controller Messaging Queue | | | Service High Availability | +--------------+--------------------------------------------------------------+ @@ -142,6 +142,11 @@ Yardstick Test Case Description TC056 | | the status of the specified process on the host, and restart | | | the process if it is not running for next test cases. | | | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | +| | | +--------------+--------------------------------------------------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc057.rst b/docs/testing/user/userguide/opnfv_yardstick_tc057.rst index 2a4ce40c0..1bb43c9e7 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc057.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc057.rst @@ -10,8 +10,11 @@ Yardstick Test Case Description TC057 +-----------------------------------------------------------------------------+ |OpenStack Controller Cluster Management Service High Availability | -+==============+==============================================================+ -|test case id | | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC057_HA: OpenStack Controller Cluster | +| | Management Service High Availability | +| | | +--------------+--------------------------------------------------------------+ |test purpose | This test case will verify the quorum configuration of the | | | cluster manager(pacemaker) on controller nodes. When a | @@ -53,10 +56,11 @@ Yardstick Test Case Description TC057 | | "openstack-cmd" for this monitor. | | | 2) command_name: which is the command name used for request | | | | -| | In this case, the command_name of monitor1 should be services| -| | that are managed by the cluster manager. (Since rabbitmq and | -| | haproxy are managed by pacemaker, most Openstack Services | -| | can be used to check high availability in this case) | +| | In this case, the command_name of monitor1 should be | +| | services that are managed by the cluster manager. | +| | (Since rabbitmq and haproxy are managed by pacemaker, | +| | most Openstack Services can be used to check high | +| | availability in this case) | | | | | | (e.g.) | | | monitor1: | @@ -155,11 +159,17 @@ Yardstick Test Case Description TC057 | | Result: The test case is passed or not. | | | | +--------------+------+----------------------------------+--------------------+ -|post-action | It is the action when the test cases exist. It will check the| -| | status of the cluster messaging process(corosync) on the | +|post-action | It is the action when the test cases exist. It will check | +| | the status of the cluster messaging process(corosync) on the | | | host, and restart the process if it is not running for next | -| | test cases | +| | test cases. | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | +| | | +--------------+------+----------------------------------+--------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc058.rst b/docs/testing/user/userguide/opnfv_yardstick_tc058.rst index fb9a4c2d1..9e8427b50 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc058.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc058.rst @@ -10,8 +10,9 @@ Yardstick Test Case Description TC058 +-----------------------------------------------------------------------------+ |OpenStack Controller Virtual Router Service High Availability | -+==============+==============================================================+ -|test case id | OPNFV_YARDSTICK_TC058:OpenStack Controller Virtual Router | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC058: OpenStack Controller Virtual Router | | | Service High Availability | +--------------+--------------------------------------------------------------+ |test purpose | This test case will verify the high availability of virtual | @@ -108,8 +109,9 @@ Yardstick Test Case Description TC058 |conditions | with cachestat included in the image. | | | | +--------------+--------------------------------------------------------------+ -|step 1 | Two host VMs are booted, these two hosts are in two different| -| | networks, the networks are connected by a virtual router | +|step 1 | Two host VMs are booted, these two hosts are in two | +| | different networks, the networks are connected by a virtual | +| | router. | | | | +--------------+--------------------------------------------------------------+ |step 1 | start monitors: | @@ -142,7 +144,13 @@ Yardstick Test Case Description TC058 | | Virtual machines and network created in the test case will | | | be destoryed. | | | | +| | Notice: This post-action uses 'lsb_release' command to check | +| | the host linux distribution and determine the OpenStack | +| | service name to restart the process. Lack of 'lsb_release' | +| | on the host may cause failure to restart the process. | +| | | +--------------+------+----------------------------------+--------------------+ |test verdict | Fails only if SLA is not passed, or if there is a test case | | | execution problem. | +| | | +--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc087.rst b/docs/testing/user/userguide/opnfv_yardstick_tc087.rst new file mode 100644 index 000000000..99bfeebfc --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc087.rst @@ -0,0 +1,182 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson and others. + +************************************* +Yardstick Test Case Description TC087 +************************************* + ++-----------------------------------------------------------------------------+ +|SDN Controller resilience in non-HA configuration | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC087: SDN controller resilience in | +| | non-HA configuration | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test validates that network data plane services are | +| | highly available in the event of an SDN Controller failure, | +| | even if the SDN controller is deployed in a non-HA | +| | configuration. Specifically, the test verifies that | +| | existing data plane connectivity is not impacted, i.e. all | +| | configured network services such as DHCP, ARP, L2, | +| | L3 Security Groups should continue to operate | +| | between the existing VMs while the SDN controller is | +| | offline or rebooting. | +| | | +| | The test also validates that new network service operations | +| | (creating a new VM in the existing L2/L3 network or in a new | +| | network, etc.) are operational after the SDN controller | +| | has recovered from a failure. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case fails the SDN controller service running | +| | on the OpenStack controller node, then checks if already | +| | configured DHCP/ARP/L2/L3/SNAT connectivity is not | +| | impacted between VMs and the system is able to execute | +| | new virtual network operations once the SDN controller | +| | is restarted and has fully recovered | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called “kill-process” is | +| | needed. This attacker includes three parameters: | +| | 1. fault_type: which is used for finding the attacker's | +| | scripts. It should be set to 'kill-process' in this test | +| | | +| | 2. process_name: should be set to the name of the SDN | +| | controller process | +| | | +| | 3. host: which is the name of a control node where the | +| | SDN controller process is running | +| | | +| | e.g. -fault_type: "kill-process" | +| | -process_name: "opendaylight" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | This test case utilizes two monitors of type "ip-status" | +| | and one monitor of type "process" to track the following | +| | conditions: | +| | 1. "ping_same_network_l2": monitor ICMP traffic between | +| | VMs in the same Neutron network | +| | | +| | 2. "ping_external_snat": monitor ICMP traffic from VMs to | +| | an external host on the Internet to verify SNAT | +| | functionality. | +| | | +| | 3. "SDN controller process monitor": a monitor checking the | +| | state of a specified SDN controller process. It measures | +| | the recovery time of the given process. | +| | | +| | Monitors of type "ip-status" use the "ping" utility to | +| | verify reachability of a given target IP. | +| | | ++--------------+--------------------------------------------------------------+ +|operations | In this test case, the following operations are needed: | +| | 1. "nova-create-instance-in_network": create a VM instance | +| | in one of the existing Neutron network. | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1. process_recover_time: which indicates the maximun | +| | time (seconds) from the process being killed to | +| | recovered | +| | | +| | 2. packet_drop: measure the packets that have been dropped | +| | by the monitors using pktgen. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | none | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1. test case file: opnfv_yardstick_tc087.yaml | +| | - Attackers: see above “attackers” discription | +| | - waiting_time: which is the time (seconds) from the | +| | process being killed to stoping monitors the monitors | +| | - Monitors: see above “monitors” discription | +| | - SLA: see above “metrics” discription | +| | | +| | 2. POD file: pod.yaml The POD configuration should record | +| | on pod.yaml first. the “host” item in this test case | +| | will use the node name in the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|pre-action | 1. The OpenStack cluster is set up with a single SDN | +| | controller in a non-HA configuration. | +| | | +| | 2. One or more Neutron networks are created with two or | +| | more VMs attached to each of the Neutron networks. | +| | | +| | 3. The Neutron networks are attached to a Neutron router | +| | which is attached to an external network towards the | +| | DCGW. | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Start IP connectivity monitors: | +| | 1. Check the L2 connectivity between the VMs in the same | +| | Neutron network. | +| | | +| | 2. Check connectivity from one VM to an external host on | +| | the Internet to verify SNAT functionality. +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Start attacker: | +| | SSH connect to the VIM node and kill the SDN controller | +| | process | +| | | +| | Result: the SDN controller service will be shutdown | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Verify the results of the IP connectivity monitors. | +| | | +| | Result: The outage_time metric reported by the monitors | +| | is zero. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Restart the SDN controller. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | Create a new VM in the existing Neutron network | +| | | ++--------------+--------------------------------------------------------------+ +|step 6 | Verify connectivity between VMs as follows: | +| | 1. Check the L2 connectivity between the previously | +| | existing VM and the newly created VM on the same | +| | Neutron network by sending ICMP messages | +| | | ++--------------+--------------------------------------------------------------+ +|step 7 | Stop IP connectivity monitors after a period of time | +| | specified by “waiting_time” | +| | | +| | Result: The monitor info will be aggregated | +| | | ++--------------+--------------------------------------------------------------+ +|step 8 | Verify the IP connectivity monitor results | +| | | +| | Result: IP connectivity monitor should not have any packet | +| | drop failures reported | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | This test fails if the SLAs are not met or if there is a | +| | test case execution problem. The SLAs are define as follows | +| | for this test: | +| | * SDN Controller recovery | +| | * process_recover_time <= 30 sec | +| | | +| | * no impact on data plane connectivity during SDN | +| | controller failure and recovery. | +| | * packet_drop == 0 | +| | | ++--------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc088.rst b/docs/testing/user/userguide/opnfv_yardstick_tc088.rst new file mode 100644 index 000000000..2423a6b31 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc088.rst @@ -0,0 +1,129 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Yin Kanglin and others. +.. 14_ykl@tongji.edu.cn + +************************************* +Yardstick Test Case Description TC088 +************************************* + ++-----------------------------------------------------------------------------+ +|Control Node Openstack Service High Availability - Nova Scheduler | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC088: Control node Openstack service down - | +| | nova scheduler | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of the | +| | compute scheduler service provided by OpenStack (nova- | +| | scheduler) on control node. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of nova-scheduler service | +| | on a selected control node, then checks whether the request | +| | of the related OpenStack command is OK and the killed | +| | processes are recovered. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the specified | +| | OpenStack service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | In this case. This parameter should always set to "nova- | +| | scheduler". | +| | 3) host: which is the name of a control node being attacked. | +| | | +| | e.g. | +| | -fault_type: "kill-process" | +| | -process_name: "nova-scheduler" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, one kind of monitor is needed: | +| | 1. the "process" monitor check whether a process is running | +| | on a specific node, which needs three parameters: | +| | 1) monitor_type: which used for finding the monitor class and| +| | related scripts. It should be always set to "process" | +| | for this monitor. | +| | 2) process_name: which is the process name for monitor | +| | 3) host: which is the name of the node running the process | +| | | +| | e.g. | +| | monitor: | +| | -monitor_type: "process" | +| | -process_name: "nova-scheduler" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|operations | In this test case, the following operations are needed: | +| | 1. "nova-create-instance": create a VM instance to check | +| | whether the nova-scheduler works normally. | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are one metric: | +| | 1)process_recover_time: which indicates the maximum time | +| | (seconds) from the process being killed to recovered | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc088.yaml | +| | -Attackers: see above "attackers" description | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to stopping monitors the monitors | +| | -Monitors: see above "monitors" description | +| | -SLA: see above "metrics" description | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | create a new instance to check whether the nova scheduler | +| | works normally. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | stop the monitor after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the specified process on the host, and restart the | +| | process if it is not running for next test cases | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc089.rst b/docs/testing/user/userguide/opnfv_yardstick_tc089.rst new file mode 100644 index 000000000..0a8b2570b --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc089.rst @@ -0,0 +1,129 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Yin Kanglin and others. +.. 14_ykl@tongji.edu.cn + +************************************* +Yardstick Test Case Description TC089 +************************************* + ++-----------------------------------------------------------------------------+ +|Control Node Openstack Service High Availability - Nova Conductor | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC089: Control node Openstack service down - | +| | nova conductor | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of the | +| | compute database proxy service provided by OpenStack (nova- | +| | conductor) on control node. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of nova-conductor service | +| | on a selected control node, then checks whether the request | +| | of the related OpenStack command is OK and the killed | +| | processes are recovered. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the specified | +| | OpenStack service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | In this case. This parameter should always set to "nova- | +| | conductor". | +| | 3) host: which is the name of a control node being attacked. | +| | | +| | e.g. | +| | -fault_type: "kill-process" | +| | -process_name: "nova-conductor" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, one kind of monitor is needed: | +| | 1. the "process" monitor check whether a process is running | +| | on a specific node, which needs three parameters: | +| | 1) monitor_type: which used for finding the monitor class and| +| | related scripts. It should be always set to "process" | +| | for this monitor. | +| | 2) process_name: which is the process name for monitor | +| | 3) host: which is the name of the node running the process | +| | | +| | e.g. | +| | monitor: | +| | -monitor_type: "process" | +| | -process_name: "nova-conductor" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|operations | In this test case, the following operations are needed: | +| | 1. "nova-create-instance": create a VM instance to check | +| | whether the nova-conductor works normally. | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are one metric: | +| | 1)process_recover_time: which indicates the maximum time | +| | (seconds) from the process being killed to recovered | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc089.yaml | +| | -Attackers: see above "attackers" description | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to stopping monitors the monitors | +| | -Monitors: see above "monitors" description | +| | -SLA: see above "metrics" description | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | create a new instance to check whether the nova conductor | +| | works normally. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | stop the monitor after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the specified process on the host, and restart the | +| | process if it is not running for next test cases | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc090.rst b/docs/testing/user/userguide/opnfv_yardstick_tc090.rst new file mode 100644 index 000000000..1f8747b2b --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc090.rst @@ -0,0 +1,151 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Yin Kanglin and others. +.. 14_ykl@tongji.edu.cn + +************************************* +Yardstick Test Case Description TC090 +************************************* + ++-----------------------------------------------------------------------------+ +|Control Node OpenStack Service High Availability - Database Instances | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC090: Control node OpenStack service down - | +| | database instances | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of the | +| | data base instances used by OpenStack (mysql) on control | +| | node. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of database service on a | +| | selected control node, then checks whether the request of | +| | the related OpenStack command is OK and the killed processes | +| | are recovered. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the specified | +| | OpenStack service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | In this case. This parameter should always set to the name | +| | of the database service of OpenStack. | +| | 3) host: which is the name of a control node being attacked. | +| | | +| | e.g. | +| | -fault_type: "kill-process" | +| | -process_name: "mysql" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, two kinds of monitor are needed: | +| | 1. the "openstack-cmd" monitor constantly request a specific | +| | Openstack command, which needs two parameters: | +| | 1) monitor_type: which is used for finding the monitor class | +| | and related scritps. It should be always set to | +| | "openstack-cmd" for this monitor. | +| | 2) command_name: which is the command name used for request. | +| | In this case, the command name should be neutron related | +| | commands. | +| | | +| | 2. the "process" monitor check whether a process is running | +| | on a specific node, which needs three parameters: | +| | 1) monitor_type: which used for finding the monitor class and| +| | related scripts. It should be always set to "process" | +| | for this monitor. | +| | 2) process_name: which is the process name for monitor | +| | 3) host: which is the name of the node running the process | +| | | +| | The examples of monitors show as follows, there are four | +| | instance of the "openstack-cmd" monitor, in order to check | +| | the database connection of different OpenStack components. | +| | | +| | monitor1: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "openstack image list" | +| | monitor2: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "openstack router list" | +| | monitor3: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "openstack stack list" | +| | monitor4: | +| | -monitor_type: "openstack-cmd" | +| | -api_name: "openstack volume list" | +| | monitor5: | +| | -monitor_type: "process" | +| | -process_name: "mysql" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1)service_outage_time: which indicates the maximum outage | +| | time (seconds) of the specified OpenStack command request. | +| | 2)process_recover_time: which indicates the maximum time | +| | (seconds) from the process being killed to recovered | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc090.yaml | +| | -Attackers: see above "attackers" description | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to stopping monitors the monitors | +| | -Monitors: see above "monitors" description | +| | -SLA: see above "metrics" description | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | stop monitors after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | verify the SLA | +| | | +| | Result: The test case is passed or not. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the specified process on the host, and restart the | +| | process if it is not running for next test cases | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc091.rst b/docs/testing/user/userguide/opnfv_yardstick_tc091.rst new file mode 100644 index 000000000..8e89b6425 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc091.rst @@ -0,0 +1,138 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Yin Kanglin and others. +.. 14_ykl@tongji.edu.cn + +************************************* +Yardstick Test Case Description TC091 +************************************* + ++-----------------------------------------------------------------------------+ +|Control Node Openstack Service High Availability - Heat Api | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC091: Control node OpenStack service down - | +| | heat api | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the high availability of the | +| | orchestration service provided by OpenStack (heat-api) on | +| | control node. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of heat-api service on a | +| | selected control node, then checks whether the request of | +| | the related OpenStack command is OK and the killed processes | +| | are recovered. | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the specified | +| | OpenStack service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | In this case. This parameter should always set to "heat-api".| +| | 3) host: which is the name of a control node being attacked. | +| | | +| | e.g. | +| | -fault_type: "kill-process" | +| | -process_name: "heat-api" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, two kinds of monitor are needed: | +| | 1. the "openstack-cmd" monitor constantly request a specific | +| | OpenStack command, which needs two parameters: | +| | 1) monitor_type: which is used for finding the monitor class | +| | and related scripts. It should be always set to | +| | "openstack-cmd" for this monitor. | +| | 2) command_name: which is the command name used for request. | +| | In this case, the command name should be neutron related | +| | commands. | +| | | +| | 2. the "process" monitor check whether a process is running | +| | on a specific node, which needs three parameters: | +| | 1) monitor_type: which used for finding the monitor class and| +| | related scripts. It should be always set to "process" | +| | for this monitor. | +| | 2) process_name: which is the process name for monitor | +| | 3) host: which is the name of the node running the process | +| | | +| | e.g. | +| | monitor1: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "heat stack list" | +| | monitor2: | +| | -monitor_type: "process" | +| | -process_name: "heat-api" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1)service_outage_time: which indicates the maximum outage | +| | time (seconds) of the specified OpenStack command request. | +| | 2)process_recover_time: which indicates the maximum time | +| | (seconds) from the process being killed to recovered | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc091.yaml | +| | -Attackers: see above "attackers" description | +| | -waiting_time: which is the time (seconds) from the process | +| | being killed to the monitor stopped | +| | -Monitors: see above "monitors" description | +| | -SLA: see above "metrics" description | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | stop monitors after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | verify the SLA | +| | | +| | Result: The test case is passed or not. | +| | | ++--------------+--------------------------------------------------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the specified process on the host, and restart the | +| | process if it is not running for next test cases | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc092.rst b/docs/testing/user/userguide/opnfv_yardstick_tc092.rst new file mode 100644 index 000000000..895074a85 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc092.rst @@ -0,0 +1,196 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Ericsson and others. + +************************************* +Yardstick Test Case Description TC092 +************************************* + ++-----------------------------------------------------------------------------+ +|SDN Controller resilience in HA configuration | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC092: SDN controller resilience and high | +| | availability HA configuration | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test validates SDN controller node high availability by | +| | verifying there is no impact on the data plane connectivity | +| | when one SDN controller fails in a HA configuration, | +| | i.e. all existing configured network services DHCP, ARP, L2, | +| | L3VPN, Security Groups should continue to operate | +| | between the existing VMs while one SDN controller instance | +| | is offline and rebooting. | +| | | +| | The test also validates that network service operations such | +| | as creating a new VM in an existing or new L2 network | +| | network remain operational while one instance of the | +| | SDN controller is offline and recovers from the failure. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This test case: | +| | 1. fails one instance of a SDN controller cluster running | +| | in a HA configuration on the OpenStack controller node | +| | | +| | 2. checks if already configured L2 connectivity between | +| | existing VMs is not impacted | +| | | +| | 3. verifies that the system never loses the ability to | +| | execute virtual network operations, even when the | +| | failed SDN Controller is still recovering | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called “kill-process” is | +| | needed. This attacker includes three parameters: | +| | 1. ``fault_type``: which is used for finding the attacker's | +| | scripts. It should be set to 'kill-process' in this test | +| | | +| | 2. ``process_name``: should be set to sdn controller | +| | process | +| | | +| | 3. ``host``: which is the name of a control node where | +| | opendaylight process is running | +| | | +| | example: | +| | - ``fault_type``: “kill-process” | +| | - ``process_name``: “opendaylight-karaf” (TBD) | +| | - ``host``: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, the following monitors are needed | +| | 1. ``ping_same_network_l2``: monitor pinging traffic | +| | between the VMs in same neutron network | +| | | +| | 2. ``ping_external_snat``: monitor ping traffic from VMs to | +| | external destinations (e.g. google.com) | +| | | +| | 3. ``SDN controller process monitor``: a monitor checking | +| | the state of a specified SDN controller process. It | +| | measures the recovery time of the given process. | +| | | ++--------------+--------------------------------------------------------------+ +|operations | In this test case, the following operations are needed: | +| | 1. "nova-create-instance-in_network": create a VM instance | +| | in one of the existing neutron network. | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1. process_recover_time: which indicates the maximun | +| | time (seconds) from the process being killed to | +| | recovered | +| | | +| | 2. packet_drop: measure the packets that have been dropped | +| | by the monitors using pktgen. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | TBD | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1. test case file: opnfv_yardstick_tc092.yaml | +| | - Attackers: see above “attackers” discription | +| | - Monitors: see above “monitors” discription | +| | - waiting_time: which is the time (seconds) from the | +| | process being killed to stoping monitors the | +| | monitors | +| | - SLA: see above “metrics” discription | +| | | +| | 2. POD file: pod.yaml The POD configuration should record | +| | on pod.yaml first. the “host” item in this test case | +| | will use the node name in the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|pre-action | 1. The OpenStack cluster is set up with an SDN controller | +| | running in a three node cluster configuration. | +| | | +| | 2. One or more neutron networks are created with two or | +| | more VMs attached to each of the neutron networks. | +| | | +| | 3. The neutron networks are attached to a neutron router | +| | which is attached to an external network the towards | +| | DCGW. | +| | | +| | 4. The master node of SDN controller cluster is known. | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Start ip connectivity monitors: | +| | 1. Check the L2 connectivity between the VMs in the same | +| | neutron network. | +| | | +| | 2. Check the external connectivity of the VMs. | +| | | +| | Each monitor runs in an independent process. | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Start attacker: | +| | SSH to the VIM node and kill the SDN controller process | +| | determined in step 2. | +| | | +| | Result: One SDN controller service will be shut down | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Restart the SDN controller. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Create a new VM in the existing Neutron network while the | +| | SDN controller is offline or still recovering. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | Stop IP connectivity monitors after a period of time | +| | specified by “waiting_time” | +| | | +| | Result: The monitor info will be aggregated | +| | | ++--------------+--------------------------------------------------------------+ +|step 6 | Verify the IP connectivity monitor result | +| | | +| | Result: IP connectivity monitor should not have any packet | +| | drop failures reported | +| | | ++--------------+--------------------------------------------------------------+ +|step 7 | Verify process_recover_time, which indicates the maximun | +| | time (seconds) from the process being killed to recovered, | +| | is within the SLA. This step blocks until either the | +| | process has recovered or a timeout occurred. | +| | | +| | Result: process_recover_time is within SLA limits, if not, | +| | test case failed and stopped. | +| | | ++--------------+--------------------------------------------------------------+ +|step 8 | Start IP connectivity monitors for the new VM: | +| | 1. Check the L2 connectivity from the existing VMs to the | +| | new VM in the Neutron network. | +| | | +| | 2. Check connectivity from one VM to an external host on | +| | the Internet to verify SNAT functionality. | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 9 | Stop IP connectivity monitors after a period of time | +| | specified by “waiting_time” | +| | | +| | Result: The monitor info will be aggregated | +| | | ++--------------+--------------------------------------------------------------+ +|step 10 | Verify the IP connectivity monitor result | +| | | +| | Result: IP connectivity monitor should not have any packet | +| | drop failures reported | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | +| | | ++--------------+--------------------------------------------------------------+ + diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc093.rst b/docs/testing/user/userguide/opnfv_yardstick_tc093.rst new file mode 100644 index 000000000..31fa5d3d3 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc093.rst @@ -0,0 +1,184 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Intracom Telecom and others. +.. mardim@intracom-telecom.com + +************************************* +Yardstick Test Case Description TC093 +************************************* + ++-----------------------------------------------------------------------------+ +|SDN Vswitch resilience in non-HA or HA configuration | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC093: SDN Vswitch resilience in | +| | non-HA or HA configuration | ++--------------+--------------------------------------------------------------+ +|test purpose | This test validates that network data plane services are | +| | resilient in the event of Virtual Switch failure | +| | in compute nodes. Specifically, the test verifies that | +| | existing data plane connectivity is not permanently impacted | +| | i.e. all configured network services such as DHCP, ARP, L2, | +| | L3 Security Groups continue to operate between the existing | +| | VMs eventually after the Virtual Switches have finished | +| | rebooting. | +| | | +| | The test also validates that new network service operations | +| | (creating a new VM in the existing L2/L3 network or in a new | +| | network, etc.) are operational after the Virtual Switches | +| | have recovered from a failure. | +| | | ++--------------+--------------------------------------------------------------+ +|test method | This testcase first checks if the already configured | +| | DHCP/ARP/L2/L3/SNAT connectivity is proper. After | +| | it fails and restarts again the VSwitch services which are | +| | running on both OpenStack compute nodes, and then checks if | +| | already configured DHCP/ARP/L2/L3/SNAT connectivity is not | +| | permanently impacted (even if there are some packet | +| | loss events) between VMs and the system is able to execute | +| | new virtual network operations once the Vswitch services | +| | are restarted and have been fully recovered | +| | | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, two attackers called “kill-process” are | +| | needed. These attackers include three parameters: | +| | 1. fault_type: which is used for finding the attacker's | +| | scripts. It should be set to 'kill-process' in this test | +| | | +| | 2. process_name: should be set to the name of the Vswitch | +| | process | +| | | +| | 3. host: which is the name of the compute node where the | +| | Vswitch process is running | +| | | +| | e.g. -fault_type: "kill-process" | +| | -process_name: "openvswitch" | +| | -host: node1 | +| | | ++--------------+--------------------------------------------------------------+ +|monitors | This test case utilizes two monitors of type "ip-status" | +| | and one monitor of type "process" to track the following | +| | conditions: | +| | 1. "ping_same_network_l2": monitor ICMP traffic between | +| | VMs in the same Neutron network | +| | | +| | 2. "ping_external_snat": monitor ICMP traffic from VMs to | +| | an external host on the Internet to verify SNAT | +| | functionality. | +| | | +| | 3. "Vswitch process monitor": a monitor checking the | +| | state of the specified Vswitch process. It measures | +| | the recovery time of the given process. | +| | | +| | Monitors of type "ip-status" use the "ping" utility to | +| | verify reachability of a given target IP. | +| | | ++--------------+--------------------------------------------------------------+ +|operations | In this test case, the following operations are needed: | +| | 1. "nova-create-instance-in_network": create a VM instance | +| | in one of the existing Neutron network. | +| | | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1. process_recover_time: which indicates the maximun | +| | time (seconds) from the process being killed to | +| | recovered | +| | | +| | 2. outage_time: measures the total time in which | +| | monitors were failing in their tasks (e.g. total time of | +| | Ping failure) | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | Developed by the project. Please see folder: | +| | "yardstick/benchmark/scenarios/availability/ha_tools" | +| | | ++--------------+--------------------------------------------------------------+ +|references | none | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1. test case file: opnfv_yardstick_tc093.yaml | +| | - Attackers: see above “attackers” description | +| | - monitor_time: which is the time (seconds) from | +| | starting to stoping the monitors | +| | - Monitors: see above “monitors” discription | +| | - SLA: see above “metrics” description | +| | | +| | 2. POD file: pod.yaml The POD configuration should record | +| | on pod.yaml first. the “host” item in this test case | +| | will use the node name in the pod.yaml. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | Description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|pre-action | 1. The Vswitches are set up in both compute nodes. | +| | | +| | 2. One or more Neutron networks are created with two or | +| | more VMs attached to each of the Neutron networks. | +| | | +| | 3. The Neutron networks are attached to a Neutron router | +| | which is attached to an external network towards the | +| | DCGW. | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Start IP connectivity monitors: | +| | 1. Check the L2 connectivity between the VMs in the same | +| | Neutron network. | +| | | +| | 2. Check connectivity from one VM to an external host on | +| | the Internet to verify SNAT functionality. | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Start attackers: | +| | SSH connect to the VIM compute nodes and kill the Vswitch | +| | processes | +| | | +| | Result: the SDN Vswitch services will be shutdown | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Verify the results of the IP connectivity monitors. | +| | | +| | Result: The outage_time metric reported by the monitors | +| | is not greater than the max_outage_time. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Restart the SDN Vswitch services. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | Create a new VM in the existing Neutron network | +| | | ++--------------+--------------------------------------------------------------+ +|step 6 | Verify connectivity between VMs as follows: | +| | 1. Check the L2 connectivity between the previously | +| | existing VM and the newly created VM on the same | +| | Neutron network by sending ICMP messages | +| | | ++--------------+--------------------------------------------------------------+ +|step 7 | Stop IP connectivity monitors after a period of time | +| | specified by “monitor_time” | +| | | +| | Result: The monitor info will be aggregated | +| | | ++--------------+--------------------------------------------------------------+ +|step 8 | Verify the IP connectivity monitor results | +| | | +| | Result: IP connectivity monitor should not have any packet | +| | drop failures reported | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | This test fails if the SLAs are not met or if there is a | +| | test case execution problem. The SLAs are define as follows | +| | for this test: | +| | * SDN Vswitch recovery | +| | * process_recover_time <= 30 sec | +| | | +| | * no impact on data plane connectivity during SDN | +| | Vswitch failure and recovery. | +| | * packet_drop == 0 | +| | | ++--------------+--------------------------------------------------------------+ + |
