diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/release/release-notes/release-notes.rst | 875 | ||||
| -rw-r--r-- | docs/testing/user/userguide/08-api.rst | 521 | ||||
| -rw-r--r-- | docs/testing/user/userguide/15-list-of-tcs.rst | 6 | ||||
| -rw-r--r-- | docs/testing/user/userguide/opnfv_yardstick_tc002.rst | 4 | ||||
| -rw-r--r-- | docs/testing/user/userguide/opnfv_yardstick_tc078.rst | 3 | ||||
| -rw-r--r-- | docs/testing/user/userguide/opnfv_yardstick_tc080.rst | 117 | ||||
| -rw-r--r-- | docs/testing/user/userguide/opnfv_yardstick_tc081.rst | 122 |
7 files changed, 1065 insertions, 583 deletions
diff --git a/docs/release/release-notes/release-notes.rst b/docs/release/release-notes/release-notes.rst index bbd7e8417..c998f0fbf 100644 --- a/docs/release/release-notes/release-notes.rst +++ b/docs/release/release-notes/release-notes.rst @@ -1,18 +1,16 @@ -======= License ======= -OPNFV Danube release note for Yardstick Docs +OPNFV Euphrates release note for Yardstick Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>. -The *Yardstick framework*, the *Yardstick test cases* are opensource software, +The *Yardstick framework*, the *Yardstick test cases* are open-source software, licensed under the terms of the Apache License, Version 2.0. -========================================= -OPNFV Danube Release Note for Yardstick -========================================= +OPNFV Euphrates Release Note for Yardstick +========================================== .. toctree:: :maxdepth: 2 @@ -25,35 +23,24 @@ OPNFV Danube Release Note for Yardstick Abstract -======== +-------- This document describes the release note of Yardstick project. Version History -=============== - -+----------------+--------------------+---------------------------------+ -| *Date* | *Version* | *Comment* | -| | | | -+----------------+--------------------+---------------------------------+ -| | 3.2 | Yardstick for Danube release | -| | | | -| | | Note: The 3.2 tag is due to a | -| | | code issue during Danube 3.1 | -| | | release | -| | | | -+----------------+--------------------+---------------------------------+ -| May 4th, 2017 | 2.0 | Yardstick for Danube release | -| | | | -+----------------+--------------------+---------------------------------+ -| Mar 31st, 2017 | 1.0 | Yardstick for Danube release | -| | | | -+----------------+--------------------+---------------------------------+ +--------------- ++------------------+-----------+---------------------------------+ +| *Date* | *Version* | *Comment* | +| | | | ++------------------+-----------+---------------------------------+ +| October 20, 2017 | 5.0.0 | Yardstick for Euphrates release | +| | | | ++------------------+-----------+---------------------------------+ Important Notes -=============== +--------------- The software delivered in the OPNFV Yardstick_ Project, comprising the *Yardstick framework*, the *Yardstick test cases* and the experimental @@ -63,11 +50,10 @@ NFV-TST001_. The *Yardstick* framework is *installer*, *infrastructure* and *application* independent. +OPNFV Euphrates Release +----------------------- -OPNFV Danube Release -====================== - -This Danube release provides *Yardstick* as a framework for NFVI testing +This Euphrates release provides *Yardstick* as a framework for NFVI testing and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Documentation generated with Sphinx @@ -93,9 +79,9 @@ and OPNFV feature testing, automated in the OPNFV CI pipeline, including: * Yardstick test cases yaml files -* Yardstick plug-in configration yaml files, plug-in install/remove scripts +* Yardstick plug-in configuration yaml files, plug-in install/remove scripts -For Danube release, the *Yardstick framework* is used for the following +For Euphrates release, the *Yardstick framework* is used for the following testing: * OPNFV platform testing - generic test cases to measure the categories: @@ -127,8 +113,7 @@ testing: * virtual Traffic Classifier The *Yardstick framework* is developed in the OPNFV community, by the -Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick -Project. +Yardstick_ team. .. note:: The test case description template used for the Yardstick test cases is based on the document ETSI-ISG NFV-TST001_; the results report template @@ -136,336 +121,201 @@ Project. Release Data -============ - -+--------------------------------------+--------------------------------------+ -| **Project** | Yardstick | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | yardstick/Danube.3.2 | -| | | -+--------------------------------------+--------------------------------------+ -| **Yardstick Docker image tag** | Danube.3.2 | -| | | -+--------------------------------------+--------------------------------------+ -| **Release designation** | Danube | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | August 15th, 2017 | -| | | -+--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | OPNFV Danube release 3.0 | -| | | -+--------------------------------------+--------------------------------------+ +------------ + ++--------------------------------+-----------------------+ +| **Project** | Yardstick | +| | | ++--------------------------------+-----------------------+ +| **Repo/tag** | yardstick/opnfv-5.0.0 | +| | | ++--------------------------------+-----------------------+ +| **Yardstick Docker image tag** | opnfv-5.0.0 | +| | | ++--------------------------------+-----------------------+ +| **Release designation** | Euphrates | +| | | ++--------------------------------+-----------------------+ +| **Release date** | October 20, 2017 | +| | | ++--------------------------------+-----------------------+ +| **Purpose of the delivery** | OPNFV Euphrates 5.0.0 | +| | | ++--------------------------------+-----------------------+ Deliverables -============ +------------ Documents ---------- +^^^^^^^^^ - - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html + - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html - - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html + - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html Software Deliverables ---------------------- - - - - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.3.2) - - -**Contexts** - -+---------------------+-------------------------------------------------------+ -| **Context** | **Description** | -| | | -+---------------------+-------------------------------------------------------+ -| *Heat* | Models orchestration using OpenStack Heat | -| | | -+---------------------+-------------------------------------------------------+ -| *Node* | Models Baremetal, Controller, Compute | -| | | -+---------------------+-------------------------------------------------------+ -| *Standalone* | Models VM running on Non-Managed NFVi | -| | | -+---------------------+-------------------------------------------------------+ - - -**Runners** - -+---------------------+-------------------------------------------------------+ -| **Runner** | **Description** | -| | | -+---------------------+-------------------------------------------------------+ -| *Arithmetic* | Steps every run arithmetically according to specified | -| | input value | -| | | -+---------------------+-------------------------------------------------------+ -| *Duration* | Runs for a specified period of time | -| | | -+---------------------+-------------------------------------------------------+ -| *Iteration* | Runs for a specified number of iterations | -| | | -+---------------------+-------------------------------------------------------+ -| *Sequence* | Selects input value to a scenario from an input file | -| | and runs all entries sequentially | -| | | -+---------------------+-------------------------------------------------------+ - - -**Scenarios** - -+---------------------+-------------------------------------------------------+ -| **Category** | **Delivered** | -| | | -+---------------------+-------------------------------------------------------+ -| *Availability* | Attacker: | -| | | -| | * baremetal, process | -| | | -| | HA tools: | -| | | -| | * check host, openstack, process, service | -| | * kill process | -| | * start/stop service | -| | | -| | Monitor: | -| | | -| | * command, process | -| | | -+---------------------+-------------------------------------------------------+ -| *Compute* | * cpuload | -| | | -| | * cyclictest | -| | | -| | * lmbench | -| | | -| | * lmbench_cache | -| | | -| | * perf | -| | | -| | * unixbench | -| | | -| | * ramspeed | -| | | -| | * cachestat | -| | | -| | * memeoryload | -| | | -| | * computecapacity | -| | | -+---------------------+-------------------------------------------------------+ -| *Networking* | * iperf3 | -| | | -| | * netperf | -| | | -| | * netperf_node | -| | | -| | * ping | -| | | -| | * ping6 | -| | | -| | * pktgen | -| | | -| | * sfc | -| | | -| | * sfc with tacker | -| | | -| | * vtc instantion validation | -| | | -| | * vtc instantion validation with noisy neighbors | -| | | -| | * vtc throughput | -| | | -| | * vtc throughput in the presence of noisy neighbors | -| | | -| | * networkcapacity | -| | | -| | * netutilization | -| | | -| | * nstat | -| | | -+---------------------+-------------------------------------------------------+ -| *Parser* | Tosca2Heat | -| | | -+---------------------+-------------------------------------------------------+ -| *Storage* | fio | -| | | -| | storagecapacity | -| | | -+---------------------+-------------------------------------------------------+ -| *StorPerf* | storperf | -| | | -+---------------------+-------------------------------------------------------+ -| *NSB* | vPE thoughput test case | -| | | -+---------------------+-------------------------------------------------------+ - - -**Test Results Output** - -+-----------------------------+-----------------------------------------------+ -| **Dispatcher** | **Description** | -| | | -+-----------------------------+-----------------------------------------------+ -| file | Log to a file. | -| | | -+-----------------------------+-----------------------------------------------+ -| http | Post data to html. | -| | | -+-----------------------------+-----------------------------------------------+ -| influxdb | Post data to influxDB. | -| | | -+-----------------------------+-----------------------------------------------+ - - -Delivered Test cases --------------------- - -* Generic NFVI test cases - - * OPNFV_YARDSTICK_TCOO1 - NW Performance - - * OPNFV_YARDSTICK_TCOO2 - NW Latency - - * OPNFV_YARDSTICK_TCOO4 - Cache Utilization - - * OPNFV_YARDSTICK_TCOO5 - Storage Performance - - * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test - - * OPNFV_YARDSTICK_TCOO9 - Packet Loss - - * OPNFV_YARDSTICK_TCO10 - Memory Latency - - * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs - - * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth - - * OPNFV_YARDSTICK_TCO14 - Processing Speed - - * OPNFV_YARDSTICK_TCO24 - CPU Load - - * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss - - * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended - Test - - * OPNFV_YARDSTICK_TCO42 - Network Performance - - * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes - - * OPNFV_YARDSTICK_TCO44 - Memory Utilization - - * OPNFV_YARDSTICK_TCO55 - Compute Capacity - - * OPNFV_YARDSTICK_TCO61 - Network Utilization - - * OPNFV_YARDSTICK_TCO63 - Storage Capacity - - * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth - - * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet - Loss - - * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss - - * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet - Loss - - * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes - - * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale - - * OPNFV_YARDSTICK_TCO76 - Network frame error rate - -* Test Cases for OPNFV HA Project: - - * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down - - * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down - - * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron - server +^^^^^^^^^^^^^^^^^^^^^ + + + - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.0.0) + + +New Contexts +############ + ++--------------+-------------------------------------------+ +| **Context** | **Description** | +| | | ++--------------+-------------------------------------------+ +| *Heat* | Models orchestration using OpenStack Heat | +| | | ++--------------+-------------------------------------------+ +| *Node* | Models Baremetal, Controller, Compute | +| | | ++--------------+-------------------------------------------+ +| *Standalone* | Models VM running on Non-Managed NFVi | +| | | ++--------------+-------------------------------------------+ +| *Kubernetes* | Models VM running on Non-Managed NFVi | +| | | ++--------------+-------------------------------------------+ + + +New Runners +########### + ++--------------+-------------------------------------------------------+ +| **Runner** | **Description** | +| | | ++--------------+-------------------------------------------------------+ +| *Arithmetic* | Steps every run arithmetically according to specified | +| | input value | +| | | ++--------------+-------------------------------------------------------+ +| *Duration* | Runs for a specified period of time | +| | | ++--------------+-------------------------------------------------------+ +| *Iteration* | Runs for a specified number of iterations | +| | | ++--------------+-------------------------------------------------------+ +| *Sequence* | Selects input value to a scenario from an input file | +| | and runs all entries sequentially | +| | | ++--------------+-------------------------------------------------------+ + + +New Scenarios +############# + ++----------------+-----------------------------------------------------+ +| **Category** | **Delivered** | +| | | ++----------------+-----------------------------------------------------+ +| *Availability* | Attacker: | +| | | +| | * baremetal, process | +| | | +| | HA tools: | +| | | +| | * check host, openstack, process, service | +| | * kill process | +| | * start/stop service | +| | | +| | Monitor: | +| | | +| | * command, process | +| | | ++----------------+-----------------------------------------------------+ +| *Compute* | * cpuload | +| | | +| | * cyclictest | +| | | +| | * lmbench | +| | | +| | * lmbench_cache | +| | | +| | * perf | +| | | +| | * unixbench | +| | | +| | * ramspeed | +| | | +| | * cachestat | +| | | +| | * memeoryload | +| | | +| | * computecapacity | +| | | ++----------------+-----------------------------------------------------+ +| *Networking* | * iperf3 | +| | | +| | * netperf | +| | | +| | * netperf_node | +| | | +| | * ping | +| | | +| | * ping6 | +| | | +| | * pktgen | +| | | +| | * sfc | +| | | +| | * sfc with tacker | +| | | +| | * vtc instantion validation | +| | | +| | * vtc instantion validation with noisy neighbors | +| | | +| | * vtc throughput | +| | | +| | * vtc throughput in the presence of noisy neighbors | +| | | +| | * networkcapacity | +| | | +| | * netutilization | +| | | +| | * nstat | +| | | ++----------------+-----------------------------------------------------+ +| *Parser* | Tosca2Heat | +| | | ++----------------+-----------------------------------------------------+ +| *Storage* | fio | +| | | +| | storagecapacity | +| | | ++----------------+-----------------------------------------------------+ +| *StorPerf* | storperf | +| | | ++----------------+-----------------------------------------------------+ +| *NSB* | vPE thoughput test case | +| | | ++----------------+-----------------------------------------------------+ + + + +New Test cases +^^^^^^^^^^^^^^ - * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone - - * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance - api - - * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder - api - - * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift - proxy - - * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High - Availability - - * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High - Availability - - * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High - Availability - - * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High - Availability - - * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability - -* Test Case for OPNFV IPv6 Project: - - * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity - -* Test Case for OPNFV KVM Project: - - * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements - -* Test Case for OPNFV Parser Project: - - * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca - -* Test Case for OPNFV StorPerf Project: - - * OPNFV_YARDSTICK_TCO74 - Storperf - -* Test Cases for Virtual Traffic Classifier: - - * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput - -* Benchmarking Test: - - * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput - -* Benchmarking in presence of noisy neighbors Test: - - * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test - - * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in - presence of noisy neighbors Test Version Change -============== +-------------- Module Version Changes ----------------------- +^^^^^^^^^^^^^^^^^^^^^^ This is the fourth tracked release of Yardstick. It is based on following upstream versions: -- ONOS Ibis - -- OpenStack Newton - -- OpenDaylight Boron +- OpenStack Ocata Document Version Changes ------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^ This is the fourth tracked version of the Yardstick framework in OPNFV. It includes the following documentation updates: @@ -480,7 +330,7 @@ It includes the following documentation updates: Feature additions ------------------ +^^^^^^^^^^^^^^^^^ - Yardstick RESTful API support @@ -494,7 +344,7 @@ Feature additions - yardstick report CLI - - Node context support openstack configuration via Ansible + - Node context support OpenStack configuration via Ansible - Https support @@ -502,78 +352,78 @@ Feature additions Scenario Matrix -=============== - -For Danube 3.0, Yardstick was tested on the following scenarios: - -+-------------------------+---------+---------+---------+---------+ -| Scenario | Apex | Compass | Fuel | Joid | -+=========================+=========+=========+=========+=========+ -| os-nosdn-nofeature-noha | | | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-nofeature-ha | X | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-ha | | X | X | X | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-nofeature-noha| | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-ha | X | X | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l3-nofeature-noha| | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-onos-sfc-ha | | | | | -+-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-ha | | X | | X | -+-------------------------+---------+---------+---------+---------+ -| os-onos-nofeature-noha | | | | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-sfc-ha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-sfc-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-bgpvpn-ha | X | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-bgpvpn-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-ha | X | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-ovs-ha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-ovs-noha | | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-ocl-nofeature-ha | | X | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-lxd-ha | | | | X | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-lxd-noha | | | | X | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-fdio-ha | X | | | | -+-------------------------+---------+---------+---------+---------+ -| os-odl_l2-fdio-noha | X | | | | -+-------------------------+---------+---------+---------+---------+ -| os-odl-gluon-noha | X | | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-openo-ha | | X | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| -noha | | | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm_ovs_dpdk-ha| | | X | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-ha | | | | | -+-------------------------+---------+---------+---------+---------+ -| os-nosdn-kvm_ovs_dpdk | | | X | | -| _bar-noha | | | | | -+-------------------------+---------+---------+---------+---------+ -| opnfv_os-ovn-nofeature- | X | | | | -| noha_daily | | | | | -+-------------------------+---------+---------+---------+---------+ +--------------- + +For Euphrates 5.0.0, Yardstick was tested on the following scenarios: + ++--------------------------+------+---------+------+------+ +| Scenario | Apex | Compass | Fuel | Joid | ++==========================+======+=========+======+======+ +| os-nosdn-nofeature-noha | | | X | X | ++--------------------------+------+---------+------+------+ +| os-nosdn-nofeature-ha | X | X | X | X | ++--------------------------+------+---------+------+------+ +| os-odl_l2-nofeature-ha | | X | X | X | ++--------------------------+------+---------+------+------+ +| os-odl_l2-nofeature-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-odl_l3-nofeature-ha | X | X | X | | ++--------------------------+------+---------+------+------+ +| os-odl_l3-nofeature-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-onos-sfc-ha | | | | | ++--------------------------+------+---------+------+------+ +| os-onos-nofeature-ha | | X | | X | ++--------------------------+------+---------+------+------+ +| os-onos-nofeature-noha | | | | | ++--------------------------+------+---------+------+------+ +| os-odl_l2-sfc-ha | | | X | | ++--------------------------+------+---------+------+------+ +| os-odl_l2-sfc-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-odl_l2-bgpvpn-ha | X | | X | | ++--------------------------+------+---------+------+------+ +| os-odl_l2-bgpvpn-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm-ha | X | | X | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-nosdn-ovs-ha | | | X | | ++--------------------------+------+---------+------+------+ +| os-nosdn-ovs-noha | | | X | | ++--------------------------+------+---------+------+------+ +| os-ocl-nofeature-ha | | X | | | ++--------------------------+------+---------+------+------+ +| os-nosdn-lxd-ha | | | | X | ++--------------------------+------+---------+------+------+ +| os-nosdn-lxd-noha | | | | X | ++--------------------------+------+---------+------+------+ +| os-nosdn-fdio-ha | X | | | | ++--------------------------+------+---------+------+------+ +| os-odl_l2-fdio-noha | X | | | | ++--------------------------+------+---------+------+------+ +| os-odl-gluon-noha | X | | | | ++--------------------------+------+---------+------+------+ +| os-nosdn-openo-ha | | X | | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| -noha | | | | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm_ovs_dpdk-ha | | | X | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| _bar-ha | | | | | ++--------------------------+------+---------+------+------+ +| os-nosdn-kvm_ovs_dpdk | | | X | | +| _bar-noha | | | | | ++--------------------------+------+---------+------+------+ +| opnfv_os-ovn-nofeature- | X | | | | +| noha_daily | | | | | ++--------------------------+------+---------+------+------+ Test results -============ +------------ Test results are available in: @@ -581,180 +431,81 @@ Test results are available in: The reporting pages can be found at: - * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html - * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html - * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html - * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html - ++---------------+-------------------------------------------------------------------------------------+ +| apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html | ++---------------+-------------------------------------------------------------------------------------+ +| compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html | ++---------------+-------------------------------------------------------------------------------------+ +| fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html | ++---------------+-------------------------------------------------------------------------------------+ +| fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html | ++---------------+-------------------------------------------------------------------------------------+ +| joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html | ++---------------+-------------------------------------------------------------------------------------+ Known Issues/Faults ------------- +^^^^^^^^^^^^^^^^^^^ - - Floating IP not supported in bgpvpn scenario - - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario +Corrected Faults +^^^^^^^^^^^^^^^^ -.. note:: The faults not related to *Yardstick* framework, addressing scenarios - which were not fully verified, are listed in the OPNFV installer's release - notes. +Euphrates 5.0.0: ++---------------------+--------------------------------------------+ +| **JIRA REFERENCE** | **DESCRIPTION** | +| | | ++---------------------+--------------------------------------------+ +| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using | +| | 'openstack -h' | ++---------------------+--------------------------------------------+ +| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they | +| | are dynamic | ++---------------------+--------------------------------------------+ -Corrected Faults ----------------- - -Danube.3.2: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-776 | Bugfix: cannot run task if without | -| | yardstick.conf in danube | -+----------------------------+------------------------------------------------+ - - -Danube.3.1: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-714 | Add yardstick env influxdb/grafana command for | -| | CentOS | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-655 | Monitor command in tc019 may not show the | -| | real nova-api service status | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-397 | HA testing framework improvement | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-660 | Improve monitor_process pass criteria | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-657 | HA monitor_multi bug, | -| | KeyError: 'max_outage_time' | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-647 | TC025 fault_type value is wrong when using | -| | baremetal pod scripts | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-659 | Terminate openstack service process using kill | -| | command in HA test cases | -+----------------------------+------------------------------------------------+ -| JIRA: ARMBAND-275 | Yardstick TC005 fails with | -| | "Cannot map zero-fill pages" error | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-561 | Bugfix: AttributeError: 'dict' object has no | -| | attribute 'split' if run sample/ping-hot.yaml | -+----------------------------+------------------------------------------------+ -| JIRA: ARMBAND-268 | ERROR No JSON object could be decoded from | -| | LMBENCH in TC010 | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-680 | storperf test case tc074 do not get results | -| | | -+----------------------------+------------------------------------------------+ - -Danube.2.0: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-608 | Set work directory in Yardstick container | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure | -| | situation | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-620 | Yardstick virtualenv support | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-622 | Bugfix: take test case modification into | -| | effect in load_images.sh | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-623 | change openrc file path to | -| | /etc/yardstick/openstack.creds | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test | -| | suite | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept | -| | --task-args options | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not | -| | support other installer | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA | -| | Test Cases | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test | -| | frameworks | -+----------------------------+------------------------------------------------+ - -Danube.1.0: - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using | -| | 'openstack -h' | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they | -| | are dynamic | -+----------------------------+------------------------------------------------+ - - -Danube 3.2 known restrictions/issues -==================================== + +Euphratess 5.0.0 known restrictions/issues +------------------------------------------ +-----------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | +| Installer | Scenario | Issue | +===========+===========+==============================================+ -| any | *-bgpvpn | Floating ips not supported. Some Test cases | +| any | \*-bgpvpn | Floating ips not supported. Some Test cases | | | | related to floating ips are excluded. | +-----------+-----------+----------------------------------------------+ -| any | odl_l3-* | Some test cases related to using floating IP | +| any | odl_l3-\* | Some test cases related to using floating IP | | | | addresses fail because of a known ODL bug. | | | | | +-----------+-----------+----------------------------------------------+ -| compass | odl_l2-* | In some test cases, VM instance will failed | +| compass | odl_l2-\* | In some test cases, VM instance will failed | | | | raising network interfaces. | | | | | +-----------+-----------+----------------------------------------------+ Open JIRA tickets -================= - -+----------------------------+------------------------------------------------+ -| **JIRA REFERENCE** | **DESCRIPTION** | -| | | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 | -| | image | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-603 | Timeout waiting for floating ip | -| | (which actually pingable) | -+----------------------------+------------------------------------------------+ -| JIRA: YARDSTICK-412 | IPv6 test case should add support for newton | -| | | -+----------------------------+------------------------------------------------+ +----------------- + ++---------------------+----------------------------------------------+ +| **JIRA REFERENCE** | **DESCRIPTION** | +| | | ++---------------------+----------------------------------------------+ +| JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 | +| | image | ++---------------------+----------------------------------------------+ +| JIRA: YARDSTICK-603 | Timeout waiting for floating ip | +| | (which actually pingable) | ++---------------------+----------------------------------------------+ +| JIRA: YARDSTICK-412 | IPv6 test case should add support for newton | +| | | ++---------------------+----------------------------------------------+ Useful links -============ +------------ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick - - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning + - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning - Yardstick repo: https://git.opnfv.org/cgit/yardstick diff --git a/docs/testing/user/userguide/08-api.rst b/docs/testing/user/userguide/08-api.rst index 1d9ea6d64..ff6e62228 100644 --- a/docs/testing/user/userguide/08-api.rst +++ b/docs/testing/user/userguide/08-api.rst @@ -10,7 +10,7 @@ Yardstick Restful API Abstract -------- -Yardstick support restful API in danube. +Yardstick support restful API since Danube. Available API @@ -19,19 +19,19 @@ Available API /yardstick/env/action ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to do some work related to environment. For now, we support: +Description: This API is used to prepare Yardstick test environment. For Euphrates, it supports: -1. Prepare yardstick environment(Including fetch openrc file, get external network and load images) -2. Start a InfluxDB docker container and config yardstick output to InfluxDB. -3. Start a Grafana docker container and config with the InfluxDB. +1. Prepare yardstick test environment, including set external network environment variable, load Yardstick VM images and create flavors; +2. Start an InfluxDB Docker container and config Yardstick output to InfluxDB; +3. Start a Grafana Docker container and config it with the InfluxDB. -Which API to call will depend on the Parameters. +Which API to call will depend on the parameters. Method: POST -Prepare Yardstick Environment +Prepare Yardstick test environment Example:: { @@ -41,7 +41,7 @@ Example:: This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result. -Start and Config InfluxDB docker container +Start and config an InfluxDB docker container Example:: { @@ -51,7 +51,7 @@ Example:: This is an asynchronous API. You need to call /yardstick/asynctask API to get the task result. -Start and Config Grafana docker container +Start and config a Grafana docker container Example:: { @@ -64,13 +64,13 @@ This is an asynchronous API. You need to call /yardstick/asynctask API to get th /yardstick/asynctask ^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to get the status of asynchronous task +Description: This API is used to get the status of asynchronous tasks Method: GET -Get the status of asynchronous task +Get the status of asynchronous tasks Example:: http://localhost:8888/yardstick/asynctask?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c @@ -81,13 +81,13 @@ The returned status will be 0(running), 1(finished) and 2(failed). /yardstick/testcases ^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to list all release test cases now in yardstick. +Description: This API is used to list all released Yardstick test cases. Method: GET -Get a list of release test cases +Get a list of released test cases Example:: http://localhost:8888/yardstick/testcases @@ -96,13 +96,13 @@ Example:: /yardstick/testcases/release/action ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to run a yardstick release test case. +Description: This API is used to run a Yardstick released test case. Method: POST -Run a release test case +Run a released test case Example:: { @@ -119,7 +119,7 @@ This is an asynchronous API. You need to call /yardstick/results to get the resu /yardstick/testcases/samples/action ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to run a yardstick sample test case. +Description: This API is used to run a Yardstick sample test case. Method: POST @@ -139,10 +139,25 @@ Example:: This is an asynchronous API. You need to call /yardstick/results to get the result. +/yardstick/testcases/<testcase_name>/docs +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Description: This API is used to the documentation of a certain released test case. + + +Method: GET + + +Get the documentation of a certain test case +Example:: + + http://localhost:8888/yardstick/taskcases/opnfv_yardstick_tc002/docs + + /yardstick/testsuites/action ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Description: This API is used to run a yardstick test suite. +Description: This API is used to run a Yardstick test suite. Method: POST @@ -162,11 +177,27 @@ Example:: This is an asynchronous API. You need to call /yardstick/results to get the result. +/yardstick/tasks/<task_id>/log + +Description: This API is used to get the real time log of test case execution. + + +Method: GET + + +Get real time of test case execution +Example:: + + http://localhost:8888/yardstick/tasks/14795be8-f144-4f54-81ce-43f4e3eab33f/log?index=0 + + /yardstick/results ^^^^^^^^^^^^^^^^^^ +Description: This API is used to get the test results of tasks. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by using this API. + -Description: This API is used to get the test results of certain task. If you call /yardstick/testcases/samples/action API, it will return a task id. You can use the returned task id to get the results by using this API. +Method: GET Get test results of one task @@ -175,3 +206,457 @@ Example:: http://localhost:8888/yardstick/results?task_id=3f3f5e03-972a-4847-a5f8-154f1b31db8c This API will return a list of test case result + + +/api/v2/yardstick/openrcs/action + +Description: This API provides functionality of handling OpenStack credential file (openrc). For Euphrates, it supports: + +1. Upload an openrc file for an OpenStack environment; +2. Update an openrc file; +3. Get openrc file information; +4. Delete an openrc file. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Upload an openrc file for an OpenStack environment +Example:: + + { + 'action': 'upload_openrc', + 'args': { + 'file': file, + 'environment_id': environment_id + } + } + + +METHOD: POST + + +Update an openrc file +Example:: + + { + 'action': 'update_openrc', + 'args': { + 'openrc': { + "EXTERNAL_NETWORK": "ext-net", + "OS_AUTH_URL": "http://192.168.23.51:5000/v3", + "OS_IDENTITY_API_VERSION": "3", + "OS_IMAGE_API_VERSION": "2", + "OS_PASSWORD": "console", + "OS_PROJECT_DOMAIN_NAME": "default", + "OS_PROJECT_NAME": "admin", + "OS_TENANT_NAME": "admin", + "OS_USERNAME": "admin", + "OS_USER_DOMAIN_NAME": "default" + }, + 'environment_id': environment_id + } + } + + +METHOD: GET + +Get openrc file information +Example:: + + http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +METHOD: DELETE + + +Delete openrc file +Example:: + + http://localhost:8888/api/v2/yardstick/openrcs/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +/api/v2/yardstick/pods/action + +Description: This API provides functionality of handling Yardstick pod file (pod.yaml). For Euphrates, it supports: + +1. Upload a pod file; +2. Get pod file information; +3. Delete an openrc file. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Upload a pod.yaml file +Example:: + + { + 'action': 'upload_pod_file', + 'args': { + 'file': file, + 'environment_id': environment_id + } + } + + +METHOD: GET + +Get pod file information +Example:: + + http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +METHOD: DELETE + +Delete openrc file +Example:: + + http://localhost:8888/api/v2/yardstick/pods/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +/api/v2/yardstick/images/action + +Description: This API is used to do some work related to Yardstick VM images. For Euphrates, it supports: + +1. Load Yardstick VM images; +2. Get image's information; +3. Delete images. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Load VM images +Example:: + + { + 'action': 'load_images' + } + + +METHOD: GET + +Get image information +Example:: + + http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +METHOD: DELETE + +Delete images +Example:: + + http://localhost:8888/api/v2/yardstick/images/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +/api/v2/yardstick/tasks/action + +Description: This API is used to do some work related to yardstick tasks. For Euphrates, it supports: + +1. Create a Yardstick task; +2. run a Yardstick task; +3. Add a test case to a task; +4. Add a test suite to a task; +5. Get a tasks' information; +6. Delete a task. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Create a Yardstick task +Example:: + + { + 'action': 'create_task', + 'args': { + 'name': 'task1', + 'project_id': project_id + } + } + + +METHOD: PUT + + +Run a task +Example:: + + { + 'action': 'run' + } + + +METHOD: PUT + + +Add a test case to a task +Example:: + + { + 'action': 'add_case', + 'args': { + 'case_name': 'opnfv_yardstick_tc002', + 'case_content': case_content + } + } + + +METHOD: PUT + + +Add a test suite to a task +Example:: + + { + 'action': 'add_suite', + 'args': { + 'suite_name': 'opnfv_smoke', + 'suite_content': suite_content + } + } + + +METHOD: GET + +Get a task's information +Example:: + + http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +METHOD: DELETE + +Delete a task +Example:: + http://localhost:8888/api/v2/yardstick/tasks/5g6g3e02-155a-4847-a5f8-154f1b31db8c + + +/api/v2/yardstick/testcases/action + +Description: This API is used to do some work related to yardstick testcases. For Euphrates, it supports: + +1. Upload a test case; +2. Get all released test cases' information; +3. Get a certain released test case's information; +4. Delete a test case. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Upload a test case +Example:: + + { + 'action': 'upload_case', + 'args': { + 'file': file + } + } + + +METHOD: GET + + +Get all released test cases' information +Example:: + + http://localhost:8888/api/v2/yardstick/testcases + + +METHOD: GET + + +Get a certain released test case's information +Example:: + + http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002 + + +METHOD: DELETE + + +Delete a certain test case +Example:: + http://localhost:8888/api/v2/yardstick/testcases/opnfv_yardstick_tc002 + + +/api/v2/yardstick/testsuites/action + +Description: This API is used to do some work related to yardstick test suites. For Euphrates, it supports: + +1. Create a test suite; +2. Get a certain test suite's information; +3. Get all test suites; +4. Delete a test case. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Create a test suite +Example:: + + { + 'action': 'create_sutie', + 'args': { + 'name': <suite_name>, + 'testcases': [ + 'opnfv_yardstick_tc002' + ] + } + } + + +METHOD: GET + + +Get a certain test suite's information +Example:: + + http://localhost:8888/api/v2/yardstick/testsuites/<suite_name> + + +METHOD: GET + + +Get all test suite +Example:: + + http://localhost:8888/api/v2/yardstick/testsuites + + +METHOD: DELETE + + +Delete a certain test suite +Example:: + + http://localhost:8888/api/v2/yardstick/testsuites/<suite_name> + + +/api/v2/yardstick/projects/action + +Description: This API is used to do some work related to yardstick test projects. For Euphrates, it supports: + +1. Create a Yardstick project; +2. Get a certain project's information; +3. Get all projects; +4. Delete a project. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Create a Yardstick project +Example:: + + { + 'action': 'create_project', + 'args': { + 'name': 'project1' + } + } + + +METHOD: GET + + +Get a certain project's information +Example:: + + http://localhost:8888/api/v2/yardstick/projects/<project_id> + + +METHOD: GET + + +Get all projects' information +Example:: + + http://localhost:8888/api/v2/yardstick/projects + + +METHOD: DELETE + + +Delete a certain project +Example:: + + http://localhost:8888/api/v2/yardstick/projects/<project_id> + + +/api/v2/yardstick/containers/action + +Description: This API is used to do some work related to Docker containers. For Euphrates, it supports: + +1. Create a Grafana Docker container; +2. Create an InfluxDB Docker container; +3. Get a certain container's information; +4. Delete a container. + +Which API to call will depend on the parameters. + + +METHOD: POST + + +Create a Grafana Docker container +Example:: + + { + 'action': 'create_grafana', + 'args': { + 'environment_id': <environment_id> + } + } + + +METHOD: POST + + +Create an InfluxDB Docker container +Example:: + + { + 'action': 'create_influxdb', + 'args': { + 'environment_id': <environment_id> + } + } + + +METHOD: GET + + +Get a certain container's information +Example:: + + http://localhost:8888/api/v2/yardstick/containers/<container_id> + + +METHOD: DELETE + + +Delete a certain container +Example:: + + http://localhost:8888/api/v2/yardstick/containers/<container_id> diff --git a/docs/testing/user/userguide/15-list-of-tcs.rst b/docs/testing/user/userguide/15-list-of-tcs.rst index 1b5806cd9..b62bf6390 100644 --- a/docs/testing/user/userguide/15-list-of-tcs.rst +++ b/docs/testing/user/userguide/15-list-of-tcs.rst @@ -50,8 +50,14 @@ Generic NFVI Test Case Descriptions opnfv_yardstick_tc071.rst opnfv_yardstick_tc072.rst opnfv_yardstick_tc073.rst + opnfv_yardstick_tc074.rst opnfv_yardstick_tc075.rst opnfv_yardstick_tc076.rst + opnfv_yardstick_tc078.rst + opnfv_yardstick_tc079.rst + opnfv_yardstick_tc080.rst + opnfv_yardstick_tc081.rst + opnfv_yardstick_tc083.rst OPNFV Feature Test Cases ======================== diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc002.rst b/docs/testing/user/userguide/opnfv_yardstick_tc002.rst index c98780fd5..bf2466fb2 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc002.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc002.rst @@ -106,8 +106,8 @@ Yardstick Test Case Description TC002 | | | +--------------+--------------------------------------------------------------+ |step 2 | Yardstick is connected with the server VM by using ssh. | -| | 'ping_benchmark' bash script is copyied from Jump Host to | -| | the server VM via the ssh tunnel. | +| | 'ping_benchmark' bash script is copied from Jump Host to the | +| | server VM via the ssh tunnel. | | | | +--------------+--------------------------------------------------------------+ |step 3 | Ping is invoked. Ping packets are sent from server VM to | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc078.rst b/docs/testing/user/userguide/opnfv_yardstick_tc078.rst index 560d55e07..b5a6545ba 100644 --- a/docs/testing/user/userguide/opnfv_yardstick_tc078.rst +++ b/docs/testing/user/userguide/opnfv_yardstick_tc078.rst @@ -114,7 +114,8 @@ Yardstick Test Case Description TC078 |step 3 | Yardstick is connected with the target server by using ssh. | | | If custom runspec config file is used, this file is copyied | | | from yardstick to the target server via the ssh tunnel. | ---------------+---------------------------------------------------------------+ +| | | ++--------------+--------------------------------------------------------------+ |step 4 | SPEC CPU2006 benchmark is invoked and SPEC CPU 2006 metrics | | | are generated. | | | | diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc080.rst b/docs/testing/user/userguide/opnfv_yardstick_tc080.rst new file mode 100644 index 000000000..05eca502e --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc080.rst @@ -0,0 +1,117 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC080 +************************************* + +.. _Ping: https://linux.die.net/man/8/ping + ++-----------------------------------------------------------------------------+ +|Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC080_NETWORK_LATENCY_BETWEEN_CONTAINER | +| | | ++--------------+--------------------------------------------------------------+ +|metric | RTT (Round Trip Time) | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The purpose of TC080 is to do a basic verification that | +| | network latency is within acceptable boundaries when packets | +| | travel between containers located in two different | +| | Kubernetes pods. | +| | | +| | The purpose is also to be able to spot the trends. | +| | Test results, graphs and similar shall be stored for | +| | comparison reasons and product evolution understanding | +| | between different OPNFV versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | ping | +| | | +| | Ping is a computer network administration software utility | +| | used to test the reachability of a host on an Internet | +| | Protocol (IP) network. It measures the round-trip time for | +| | packet sent from the originating host to a destination | +| | computer that are echoed back to the source. | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Docker image. | +| | | ++--------------+--------------------------------------------------------------+ +|test topology | Ping packets (ICMP protocol's mandatory ECHO_REQUEST | +| | datagram) are sent from host container to target container | +| | to elicit ICMP ECHO_RESPONSE. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc080.yaml | +| | | +| | Packet size 200 bytes. Test duration 60 seconds. | +| | SLA RTT is set to maximum 10 ms. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | This test case can be configured with different: | +| | | +| | * packet sizes; | +| | * burst sizes; | +| | * ping intervals; | +| | * test durations; | +| | * test iterations. | +| | | +| | Default values exist. | +| | | +| | SLA is optional. The SLA in this test case serves as an | +| | example. Considerably lower RTT is expected, and also normal | +| | to achieve in balanced L2 environments. However, to cover | +| | most configurations, both bare metal and fully virtualized | +| | ones, this value should be possible to achieve and | +| | acceptable for black box testing. Many real time | +| | applications start to suffer badly if the RTT time is higher | +| | than this. Some may suffer bad also close to this RTT, while | +| | others may not suffer at all. It is a compromise that may | +| | have to be tuned for different configuration purposes. | +| | | ++--------------+--------------------------------------------------------------+ +|usability | This test case should be run in Kunernetes environment. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case Docker image (openretriever/yardstick) needs | +|conditions | to be pulled into Kubernetes environment. | +| | | +| | No further requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | Two containers are booted, as server and client. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Yardstick is connected with the server container by using | +| | ssh. 'ping_benchmark' bash script is copied from Jump Host | +| | to the server container via the ssh tunnel. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Ping is invoked. Ping packets are sent from server container | +| | to client container. RTT results are calculated and checked | +| | against the SLA. Logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | Two containers are deleted. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any RTT is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc081.rst b/docs/testing/user/userguide/opnfv_yardstick_tc081.rst new file mode 100644 index 000000000..90af8a382 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc081.rst @@ -0,0 +1,122 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +************************************* +Yardstick Test Case Description TC080 +************************************* + +.. _cirros-image: https://download.cirros-cloud.net +.. _Ping: https://linux.die.net/man/8/ping + ++-----------------------------------------------------------------------------+ +|Network Latency | +| | ++--------------+--------------------------------------------------------------+ +|test case id | OPNFV_YARDSTICK_TC081_NETWORK_LATENCY_BETWEEN_CONTAINER_AND_ | +| | VM | +| | | ++--------------+--------------------------------------------------------------+ +|metric | RTT (Round Trip Time) | +| | | ++--------------+--------------------------------------------------------------+ +|test purpose | The purpose of TC080 is to do a basic verification that | +| | network latency is within acceptable boundaries when packets | +| | travel between a containers and a VM. | +| | | +| | The purpose is also to be able to spot the trends. | +| | Test results, graphs and similar shall be stored for | +| | comparison reasons and product evolution understanding | +| | between different OPNFV versions and/or configurations. | +| | | ++--------------+--------------------------------------------------------------+ +|test tool | ping | +| | | +| | Ping is a computer network administration software utility | +| | used to test the reachability of a host on an Internet | +| | Protocol (IP) network. It measures the round-trip time for | +| | packet sent from the originating host to a destination | +| | computer that are echoed back to the source. | +| | | +| | Ping is normally part of any Linux distribution, hence it | +| | doesn't need to be installed. It is also part of the | +| | Yardstick Docker image. (For example also a Cirros image can | +| | be downloaded from cirros-image_, it includes ping) | +| | | ++--------------+--------------------------------------------------------------+ +|test topology | Ping packets (ICMP protocol's mandatory ECHO_REQUEST | +| | datagram) are sent from host container to target vm to | +| | elicit ICMP ECHO_RESPONSE. | +| | | ++--------------+--------------------------------------------------------------+ +|configuration | file: opnfv_yardstick_tc081.yaml | +| | | +| | Packet size 200 bytes. Test duration 60 seconds. | +| | SLA RTT is set to maximum 10 ms. | +| | | ++--------------+--------------------------------------------------------------+ +|applicability | This test case can be configured with different: | +| | | +| | * packet sizes; | +| | * burst sizes; | +| | * ping intervals; | +| | * test durations; | +| | * test iterations. | +| | | +| | Default values exist. | +| | | +| | SLA is optional. The SLA in this test case serves as an | +| | example. Considerably lower RTT is expected, and also normal | +| | to achieve in balanced L2 environments. However, to cover | +| | most configurations, both bare metal and fully virtualized | +| | ones, this value should be possible to achieve and | +| | acceptable for black box testing. Many real time | +| | applications start to suffer badly if the RTT time is higher | +| | than this. Some may suffer bad also close to this RTT, while | +| | others may not suffer at all. It is a compromise that may | +| | have to be tuned for different configuration purposes. | +| | | ++--------------+--------------------------------------------------------------+ +|usability | This test case should be run in Kunernetes environment. | +| | | ++--------------+--------------------------------------------------------------+ +|references | Ping_ | +| | | +| | ETSI-NFV-TST001 | +| | | ++--------------+--------------------------------------------------------------+ +|pre-test | The test case Docker image (openretriever/yardstick) needs | +|conditions | to be pulled into Kubernetes environment. | +| | The VM image (cirros-image) needs to be installed into | +| | Glance with ping included in it. | +| | | +| | No further requirements have been identified. | +| | | ++--------------+--------------------------------------------------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | A containers is booted, as server and a VM is booted as | +| | client. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | Yardstick is connected with the server container by using | +| | ssh. 'ping_benchmark' bash script is copied from Jump Host | +| | to the server container via the ssh tunnel. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | Ping is invoked. Ping packets are sent from server container | +| | to client VM. RTT results are calculated and checked against | +| | the SLA. Logs are produced and stored. | +| | | +| | Result: Logs are stored. | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | The container and VM are deleted. | +| | | ++--------------+--------------------------------------------------------------+ +|test verdict | Test should not PASS if any RTT is above the optional SLA | +| | value, or if there is a test case execution problem. | +| | | ++--------------+--------------------------------------------------------------+ |