setup.py


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

from __future__ import absolute_import
from setuptools import setup, find_packages


setup(
    name="yardstick",
    version="0.1.dev0",
    packages=find_packages(),
    include_package_data=True,
    package_data={
        'yardstick': [
            'benchmark/scenarios/availability/*.yaml',
            'benchmark/scenarios/availability/attacker/*.yaml',
            'benchmark/scenarios/availability/ha_tools/*.bash',
            'benchmark/scenarios/availability/ha_tools/*/*.bash',
            'benchmark/scenarios/availability/attacker/scripts/*.bash',
            'benchmark/scenarios/availability/monitor/*.yaml',
            'benchmark/scenarios/availability/monitor/script_tools/*.bash',
            'benchmark/scenarios/compute/*.bash',
            'benchmark/scenarios/networking/*.bash',
            'benchmark/scenarios/networking/*.txt',
            'benchmark/scenarios/parser/*.sh',
            'benchmark/scenarios/storage/*.bash',
            'resources/files/*',
            'resources/scripts/install/*.bash',
            'resources/scripts/remove/*.bash'
        ],
        'etc': [
            'yardstick/nodes/*/*.yaml',
            'yardstick/*.sample'
        ],
        'tests': [
            'opnfv/*/*.yaml',
            'ci/*.sh'
        ]
    },
    url="https://www.opnfv.org",
    extras_require={
        'plot': ["matplotlib>=1.4.2"]
    },
    entry_points={
        'console_scripts': [
            'yardstick=yardstick.main:main',
            'yardstick-plot=yardstick.plot.plotter:main [plot]'
        ],
    },
    scripts=[
        'tools/yardstick-img-modify',
        'tools/yardstick-img-lxd-modify',
        'tools/yardstick-img-dpdk-modify'
    ]
)
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0

The *Yardstick framework*, the *Yardstick test cases* are open-source software,
 licensed under the terms of the Apache License, Version 2.0.

=======================
Yardstick Release Notes
=======================

.. toctree::
   :maxdepth: 2

.. _Yardstick: https://wiki.opnfv.org/display/yardstick

.. _Dashboard: http://testresults.opnfv.org/grafana/

.. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf


Abstract
========

This document compiles the release notes for the Gambia release of OPNFV Yardstick.


Version History
===============
+-------------------+-----------+---------------------------------+
| *Date*            | *Version* | *Comment*                       |
|                   |           |                                 |
+-------------------+-----------+---------------------------------+
| November 9, 2018  | 7.0.0     | Yardstick for Gambia release    |
|                   |           |                                 |
+-------------------+-----------+---------------------------------+
| December 14, 2018 | 7.1.0     | Yardstick for Gambia release    |
|                   |           |                                 |
+-------------------+-----------+---------------------------------+
| January 25, 2019  | 7.2.0     | Yardstick for Gambia release    |
|                   |           |                                 |
+-------------------+-----------+---------------------------------+


Important Notes
===============

The software delivered in the OPNFV Yardstick_ Project, comprising the
*Yardstick framework*, and the *Yardstick test cases* is a realization of
the methodology in ETSI-ISG NFV-TST001_.

The *Yardstick* framework is *installer*, *infrastructure* and *application*
independent.

OPNFV Gambia Release
====================

This Gambia release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:

* Documentation generated with Sphinx

  * User Guide
  * Developer Guide
  * Release notes (this document)
  * Results

* Automated Yardstick test suite (daily, weekly)

  * Jenkins Jobs for OPNFV community labs

* Automated Yardstick test results visualization

  * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
    backend

* Yardstick framework source code

* Yardstick test cases yaml files

* Yardstick plug-in configuration yaml files, plug-in install/remove scripts

For Gambia release, the *Yardstick framework* is used for the following
testing:

* OPNFV platform testing - generic test cases to measure the categories:

  * Compute
  * Network
  * Storage

* OPNFV platform network service benchmarking (NSB)

  * NSB

* Test cases for the following OPNFV Projects:

  * Container4NFV
  * High Availability
  * IPv6
  * KVM
  * Parser
  * StorPerf
  * VSperf

The *Yardstick framework* is developed in the OPNFV community, by the
Yardstick_ team.

.. note:: The test case description template used for the Yardstick test cases
  is based on the document ETSI-ISG NFV-TST001_; the results report template
  used for the Yardstick results is based on the IEEE Std 829-2008.


Release Data
============

+--------------------------------+-----------------------+
| **Project**                    | Yardstick             |
|                                |                       |
+--------------------------------+-----------------------+
| **Repo/tag**                   | yardstick/opnfv-7.2.0 |
|                                |                       |
+--------------------------------+-----------------------+
| **Yardstick Docker image tag** | opnfv-7.2.0           |
|                                |                       |
+--------------------------------+-----------------------+
| **Release designation**        | Gambia 7.2            |
|                                |                       |
+--------------------------------+-----------------------+
| **Release date**               | January 25, 2019     |
|                                |                       |
+--------------------------------+-----------------------+
| **Purpose of the delivery**    | OPNFV Gambia 7.2.0    |
|                                |                       |
+--------------------------------+-----------------------+


Deliverables
============

Documents
---------

 - User Guide: :ref:`<yardstick:userguide>`

 - Developer Guide: :ref:`<yardstick:devguide>`


Software Deliverables
---------------------

 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)

List of Contexts
^^^^^^^^^^^^^^^^

+--------------+-------------------------------------------+
| **Context**  | **Description**                           |
|              |                                           |
+--------------+-------------------------------------------+
| *Heat*       | Models orchestration using OpenStack Heat |
|              |                                           |
+--------------+-------------------------------------------+
| *Node*       | Models Baremetal, Controller, Compute     |
|              |                                           |
+--------------+-------------------------------------------+
| *Standalone* | Models VM running on Non-Managed NFVi     |
|              |                                           |
+--------------+-------------------------------------------+
| *Kubernetes* | Models VM running on Non-Managed NFVi     |
|              |                                           |
+--------------+-------------------------------------------+


List of Runners
^^^^^^^^^^^^^^^

.. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".

+----------------+-------------------------------------------------------+
| **Runner**     | **Description**                                       |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Arithmetic*   | Steps every run arithmetically according to specified |
|                | input value                                           |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Duration*     | Runs for a specified period of time                   |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Iteration*    | Runs for a specified number of iterations             |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *IterationIPC* | Runs a configurable number of times before it         |
|                | returns. Each iteration has a configurable timeout.   |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Sequence*     | Selects input value to a scenario from an input file  |
|                | and runs all entries sequentially                     |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Dynamictp*    | A runner that searches for the max throughput with    |
|                | binary search                                         |
|                |                                                       |
+----------------+-------------------------------------------------------+
| *Search*       | A runner that runs a specific time before it returns  |
|                |                                                       |
+----------------+-------------------------------------------------------+


List of Scenarios
^^^^^^^^^^^^^^^^^

+----------------+-----------------------------------------------------+
| **Category**   | **Delivered**                                       |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *Availability* | Attacker:                                           |
|                |                                                     |
|                | * baremetal, process                                |
|                |                                                     |
|                | HA tools:                                           |
|                |                                                     |
|                | * check host, openstack, process, service           |
|                | * kill process                                      |
|                | * start/stop service                                |
|                |                                                     |
|                | Monitor:                                            |
|                |                                                     |
|                | * command, process                                  |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *Compute*      | * cpuload                                           |
|                | * cyclictest                                        |
|                | * lmbench                                           |
|                | * lmbench_cache                                     |
|                | * perf                                              |
|                | * unixbench                                         |
|                | * ramspeed                                          |
|                | * cachestat                                         |
|                | * memeoryload                                       |
|                | * computecapacity                                   |
|                | * SpecCPU2006                                       |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *Networking*   | * iperf3                                            |
|                | * netperf                                           |
|                | * netperf_node                                      |
|                | * ping                                              |
|                | * ping6                                             |
|                | * pktgen                                            |
|                | * sfc                                               |
|                | * sfc with tacker                                   |
|                | * networkcapacity                                   |
|                | * netutilization                                    |
|                | * nstat                                             |
|                | * pktgenDPDK                                        |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *Parser*       | Tosca2Heat                                          |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *Storage*      | * fio                                               |
|                | * bonnie++                                          |
|                | * storagecapacity                                   |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *StorPerf*     | storperf                                            |
|                |                                                     |
+----------------+-----------------------------------------------------+
| *NSB*          | vFW thoughput test case                             |
|                |                                                     |
+----------------+-----------------------------------------------------+


New Test cases
--------------

.. note:: Yardstick Gambia 7.2.0 adds no new test cases.

* Generic NFVI test cases

 * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM

* HA Test cases

 * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration


Version Change
==============

Module Version Changes
----------------------

This is the seventh tracked release of Yardstick. It is based on following
upstream versions:

- OpenStack Queens
- OpenDayLight Oxygen


Document Version Changes
------------------------

This is the seventh tracked version of the Yardstick framework in OPNFV.
It includes the following documentation updates:

- Yardstick User Guide:

  - Remove vTC chapter;

- Yardstick Developer Guide
- Yardstick Release Notes for Yardstick: this document


Feature additions
-----------------

- Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
- Spirent support.
- vEPC testcases.
- Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
- PROX enhancements and the addition of Standalone test case using SRIOV and
  OVS-DPDK.
- Ixia enhancements for vBNG and PPPoE traffic.
- Improvements of unit tests and gating.
- Add DPDK pktgen traffic generator.
- Kubernetes context enhancements.
- Kubernetes sample test cases of fio and lmbench added.


Scenario Matrix
===============

For Gambia 7.2.0, Yardstick was tested on the following scenarios:

+-------------------------+------+---------+----------+------+
|        Scenario         | Apex | Compass | Fuel-arm | Fuel |
+=========================+======+=========+==========+======+
| os-nosdn-nofeature-noha |   X  |         |          |      |
+-------------------------+------+---------+----------+------+
| os-nosdn-nofeature-ha   |   X  |         |          |      |
+-------------------------+------+---------+----------+------+
| os-odl-bgpvpn-noha      |   X  |         |          |      |
+-------------------------+------+---------+----------+------+
| os-nosdn-calipso-noha   |   X  |         |          |      |
+-------------------------+------+---------+----------+------+
| os-nosdn-kvm-ha         |      |    X    |          |      |
+-------------------------+------+---------+----------+------+
| os-odl-nofeature-ha     |      |         |     X    |   X  |
+-------------------------+------+---------+----------+------+
| os-odl-sfc-noha         |   X  |         |          |      |
+-------------------------+------+---------+----------+------+
| os-nosdn-ovs-ha         |      |         |          |   X  |
+-------------------------+------+---------+----------+------+
| k8-nosdn-nofeature-ha   |      |    X    |          |      |
+-------------------------+------+---------+----------+------+
| k8-nosdn-stor4nfv-noha  |      |    X    |          |      |
+-------------------------+------+---------+----------+------+
| k8-nosdn-stor4nfv-ha    |      |    X    |          |      |
+-------------------------+------+---------+----------+------+


Test results
============

Test results are available in:

 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/


Known Issues/Faults
-------------------


Corrected Faults
----------------

Gambia 7.2.0:

+--------------------+--------------------------------------------------------------------------+
| **JIRA REFERENCE** |                             **DESCRIPTION**                              |
+====================+==========================================================================+
| YARDSTICK-1512     | [dovetail] split the sla check results into process recovery and service |
|                    | recovery for HA test cases.                                              |
+--------------------+--------------------------------------------------------------------------+

Gambia 7.2.0 known restrictions/issues
======================================


Useful links
============

 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick

 - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia

 - Yardstick repo: https://git.opnfv.org/yardstick

 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick

 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/

 - Yardstick IRC channel: #opnfv-yardstick