summaryrefslogtreecommitdiffstats
path: root/docs/testing/ecosystem
diff options
context:
space:
mode:
authorStuart Mackie <wsmackie@juniper.net>2017-09-14 23:26:31 -0700
committerStuart Mackie <wsmackie@juniper.net>2017-09-14 23:26:31 -0700
commitfd876b7dbc7d517a706b22e52bf6f0e8f79a0b4b (patch)
tree996858dd4abe0221f8f9d54a2aeeb4ffb9971b8e /docs/testing/ecosystem
parentfce102283bab73ed08c292fce03e39c52f4a1fe2 (diff)
Docs
Change-Id: Iea3001f8414267f1535353f28d30d45daf9a3e66 Signed-off-by: Stuart Mackie <wsmackie@juniper.net>
Diffstat (limited to 'docs/testing/ecosystem')
-rw-r--r--docs/testing/ecosystem/index.rst14
-rw-r--r--docs/testing/ecosystem/overview.rst274
2 files changed, 0 insertions, 288 deletions
diff --git a/docs/testing/ecosystem/index.rst b/docs/testing/ecosystem/index.rst
deleted file mode 100644
index f51fa19..0000000
--- a/docs/testing/ecosystem/index.rst
+++ /dev/null
@@ -1,14 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) Christopher Price (Ericsson AB)
-
-========================
-Test Framework Overview
-========================
-
-.. toctree::
- :maxdepth: 2
-
- ./abstract
- ./overview
-
diff --git a/docs/testing/ecosystem/overview.rst b/docs/testing/ecosystem/overview.rst
deleted file mode 100644
index ed1657c..0000000
--- a/docs/testing/ecosystem/overview.rst
+++ /dev/null
@@ -1,274 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. SPDX-License-Identifier: CC-BY-4.0
-
-=============
-OPNFV testing
-=============
-
-Introduction
-============
-
-Testing is one of the key activities in OPNFV and includes unit, feature, component, system
-level testing for development, automated deployment, performance characterization or stress
-testing.
-
-Test projects are dedicated to provide frameworks, tooling and test-cases categorized as
-functional, performance or compliance testing. Test projects fulfill different roles such as
-verifying VIM functionality, benchmarking components and platforms or analysis of measured
-KPIs for the scenarios released in OPNFV.
-
-Feature projects also provide their own test suites that either run independently or within a
-test project.
-
-This document details the OPNFV testing ecosystem, describes common test components used
-by individual OPNFV projects and provides links to project specific documentation.
-
-
-OPNFV testing ecosystem
-=======================
-
-The testing projects
---------------------
-
-The OPNFV testing projects may be summarized as follows:
-
-.. figure:: ../../images/OPNFV_testing_working_group.png
- :align: center
- :alt: Overview of OPNFV Testing projects
-
-The major testing projects are described in the table below:
-
-+----------------+---------------------------------------------------------+
-| Project | Description |
-+================+=========================================================+
-| Bottlenecks | This project aims to find system bottlenecks by testing |
-| | and verifying OPNFV infrastructure in a staging |
-| | environment before committing it to a production |
-| | environment. Instead of debugging a deployment in |
-| | production environment, an automatic method for |
-| | executing benchmarks which plans to validate the |
-| | deployment during staging is adopted. This project |
-| | forms a staging framework to find bottlenecks and to do |
-| | analysis of the OPNFV infrastructure. |
-+----------------+---------------------------------------------------------+
-| CPerf | SDN Controller benchmarks and performance testing, |
-| | applicable to controllers in general. Collaboration of |
-| | upstream controller testing experts, external test tool |
-| | developers and the standards community. Primarily |
-| | contribute to upstream/external tooling, then add jobs |
-| | to run those tools on OPNFV's infrastructure. |
-+----------------+---------------------------------------------------------+
-| Dovetail | This project intends to define and provide a set of |
-| | OPNFV related validation criteria that will provide |
-| | input for the evaluation of the use of OPNFV trademarks.|
-| | The dovetail project is executed with the guidance and |
-| | oversight of the Compliance and Certification committee |
-| | and work to secure the goals of the C&C committee for |
-| | each release. The project intends to incrementally |
-| | define qualification criteria that establish the |
-| | foundations of how we are able to measure the ability to|
-| | utilize the OPNFV platform, how the platform itself |
-| | should behave, and how applications may be deployed on |
-| | the platform. |
-+----------------+---------------------------------------------------------+
-| Functest | This project deals with the functional testing of the |
-| | VIM and NFVI. It leverages several upstream test suites |
-| | (OpenStack, ODL, ONOS, etc.) and can be used by feature |
-| | project to launch feature test suites in CI/CD. |
-| | The project is used for scenario validation. |
-+----------------+---------------------------------------------------------+
-| Qtip | QTIP as the project for "Platform Performance |
-| | Benchmarking" in OPNFV aims to provide user a simple |
-| | indicator for performance, supported by comprehensive |
-| | testing data and transparent calculation formula. |
-| | It provides a platform with common services for |
-| | performance benchmarking which helps users to build |
-| | indicators by themselves with ease. |
-+----------------+---------------------------------------------------------+
-| Storperf | The purpose of this project is to provide a tool to |
-| | measure block and object storage performance in an NFVI.|
-| | When complemented with a characterization of typical VF |
-| | storage performance requirements, it can provide |
-| | pass/fail thresholds for test, staging, and production |
-| | NFVI environments. |
-+----------------+---------------------------------------------------------+
-| VSperf | This project provides a framework for automation of NFV |
-| | data-plane performance testing and benchmarking. The |
-| | NFVI fast-path includes switch technology and network |
-| | with physical and virtual interfaces. VSperf can be |
-| | used to evaluate the suitability of different Switch |
-| | implementations and features, quantify data-path |
-| | performance and optimize platform configurations. |
-+----------------+---------------------------------------------------------+
-| Yardstick | The goal of the Project is to verify the infrastructure |
-| | compliance when running VNF applications. NFV Use Cases |
-| | described in ETSI GS NFV 001 show a large variety of |
-| | applications, each defining specific requirements and |
-| | complex configuration on the underlying infrastructure |
-| | and test tools.The Yardstick concept decomposes typical |
-| | VNF work-load performance metrics into a number of |
-| | characteristics/performance vectors, which each of them |
-| | can be represented by distinct test-cases. |
-+----------------+---------------------------------------------------------+
-
-
-===================================
-The testing working group resources
-===================================
-
-The assets
-==========
-
-Overall Architecture
---------------------
-The Test result management can be summarized as follows::
-
- +-------------+ +-------------+ +-------------+
- | | | | | |
- | Test | | Test | | Test |
- | Project #1 | | Project #2 | | Project #N |
- | | | | | |
- +-------------+ +-------------+ +-------------+
- | | |
- V V V
- +---------------------------------------------+
- | |
- | Test Rest API front end |
- | http://testresults.opnfv.org/test |
- | |
- +---------------------------------------------+
- ^ | ^
- | V |
- | +-------------------------+ |
- | | | |
- | | Test Results DB | |
- | | Mongo DB | |
- | | | |
- | +-------------------------+ |
- | |
- | |
- +----------------------+ +----------------------+
- | | | |
- | Testing Dashboards | | Landing page |
- | | | |
- +----------------------+ +----------------------+
-
-
-The testing databases
----------------------
-A Mongo DB Database has been introduced for the Brahmaputra release.
-The following collections are declared in this database:
- * pods: the list of pods used for production CI
- * projects: the list of projects providing test cases
- * testcases: the test cases related to a given project
- * results: the results of the test cases
- * scenarios: the OPNFV scenarios tested in CI
-
-This database can be used by any project through the testapi.
-Please note that projects may also use additional databases. This database is
-mainly use to colelct CI results and scenario trust indicators.
-
-This database is also cloned for OPNFV Plugfest.
-
-
-The test API
-------------
-The Test API is used to declare pods, projects, test cases and test results.
-Pods correspond to the cluster of machines (3 controller and 2 compute nodes in
-HA mode) used to run the tests and defined in Pharos project.
-The results pushed in the database are related to pods, projects and cases.
-If you try to push results of test done on non referenced pod, the API will
-return an error message.
-
-An additional method dashboard has been added to post-process the raw results in
-the Brahmaputra release (deprecated in Colorado release).
-
-The data model is very basic, 5 objects are available:
- * Pods
- * Projects
- * Testcases
- * Results
- * Scenarios
-
-For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html
-
-
-The reporting
--------------
-The reporting page for the test projects is http://testresults.opnfv.org/reporting/
-
-.. figure:: ../../images/reporting_page.png
- :align: center
- :alt: Testing group reporting page
-
-This page provides a reporting per OPNFV release and per testing project.
-
-.. figure:: ../../images/reporting_danube_page.png
- :align: center
- :alt: Testing group Danube reporting page
-
-An evolution of this page is planned.
-It was decided to unify the reporting by creating a landing page that should give
-the scenario status in one glance (it was previously consolidated manually
-on a wiki page).
-
-The landing page (planned for Danube 2.0) will be displayed per scenario:
- * the status of the deployment
- * the score of the test projectS
- * a trust indicator
-
-Additional filters (version, installer, test collection time window,... ) are
-included.
-
-The test case catalog
----------------------
-Until the Colorado release, each testing project was managing the list of its
-test cases. It was very hard to have a global view of the available test cases
-among the different test projects. A common view was possible through the API
-but it was not very user friendly.
-In fact you may know all the cases per project calling:
-
- http://testresults.opnfv.org/test/api/v1/projects/<project_name>/cases
-
-with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick
-
-It was decided to build a web site providing a consistent view of the test cases
-per project and allow any scenario owner to build his/her custom list of tests
-(Danube 2.0).
-
-Other resources
-===============
-
-wiki: https://wiki.opnfv.org/testing
-
-mailing list: test-wg@lists.opnfv.org
-
-IRC chan: #opnfv-testperf
-
-weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf):
- * Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST
- * APAC time: 2nd Wednesday of the month 8:00-9:00 UTC
-
-=======================
-Reference documentation
-=======================
-
-+----------------+---------------------------------------------------------+
-| Project | Documentation links |
-+================+=========================================================+
-| Bottlenecks | https://wiki.opnfv.org/display/bottlenecks/Bottlenecks |
-+----------------+---------------------------------------------------------+
-| CPerf | https://wiki.opnfv.org/display/cperf |
-+----------------+---------------------------------------------------------+
-| Dovetail | https://wiki.opnfv.org/display/dovetail |
-+----------------+---------------------------------------------------------+
-| Functest | https://wiki.opnfv.org/display/functest/ |
-+----------------+---------------------------------------------------------+
-| Qtip | https://wiki.opnfv.org/display/qtip |
-+----------------+---------------------------------------------------------+
-| Storperf | https://wiki.opnfv.org/display/storperf/Storperf |
-+----------------+---------------------------------------------------------+
-| VSperf | https://wiki.opnfv.org/display/vsperf |
-+----------------+---------------------------------------------------------+
-| Yardstick | https://wiki.opnfv.org/display/yardstick/Yardstick |
-+----------------+---------------------------------------------------------+