diff options
Diffstat (limited to 'docs/testing/ecosystem')
| -rw-r--r-- | docs/testing/ecosystem/index.rst | 14 | ||||
| -rw-r--r-- | docs/testing/ecosystem/overview.rst | 274 |
2 files changed, 0 insertions, 288 deletions
diff --git a/docs/testing/ecosystem/index.rst b/docs/testing/ecosystem/index.rst deleted file mode 100644 index f51fa19..0000000 --- a/docs/testing/ecosystem/index.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) Christopher Price (Ericsson AB) - -======================== -Test Framework Overview -======================== - -.. toctree:: - :maxdepth: 2 - - ./abstract - ./overview - diff --git a/docs/testing/ecosystem/overview.rst b/docs/testing/ecosystem/overview.rst deleted file mode 100644 index ed1657c..0000000 --- a/docs/testing/ecosystem/overview.rst +++ /dev/null @@ -1,274 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. SPDX-License-Identifier: CC-BY-4.0 - -============= -OPNFV testing -============= - -Introduction -============ - -Testing is one of the key activities in OPNFV and includes unit, feature, component, system -level testing for development, automated deployment, performance characterization or stress -testing. - -Test projects are dedicated to provide frameworks, tooling and test-cases categorized as -functional, performance or compliance testing. Test projects fulfill different roles such as -verifying VIM functionality, benchmarking components and platforms or analysis of measured -KPIs for the scenarios released in OPNFV. - -Feature projects also provide their own test suites that either run independently or within a -test project. - -This document details the OPNFV testing ecosystem, describes common test components used -by individual OPNFV projects and provides links to project specific documentation. - - -OPNFV testing ecosystem -======================= - -The testing projects --------------------- - -The OPNFV testing projects may be summarized as follows: - -.. figure:: ../../images/OPNFV_testing_working_group.png - :align: center - :alt: Overview of OPNFV Testing projects - -The major testing projects are described in the table below: - -+----------------+---------------------------------------------------------+ -| Project | Description | -+================+=========================================================+ -| Bottlenecks | This project aims to find system bottlenecks by testing | -| | and verifying OPNFV infrastructure in a staging | -| | environment before committing it to a production | -| | environment. Instead of debugging a deployment in | -| | production environment, an automatic method for | -| | executing benchmarks which plans to validate the | -| | deployment during staging is adopted. This project | -| | forms a staging framework to find bottlenecks and to do | -| | analysis of the OPNFV infrastructure. | -+----------------+---------------------------------------------------------+ -| CPerf | SDN Controller benchmarks and performance testing, | -| | applicable to controllers in general. Collaboration of | -| | upstream controller testing experts, external test tool | -| | developers and the standards community. Primarily | -| | contribute to upstream/external tooling, then add jobs | -| | to run those tools on OPNFV's infrastructure. | -+----------------+---------------------------------------------------------+ -| Dovetail | This project intends to define and provide a set of | -| | OPNFV related validation criteria that will provide | -| | input for the evaluation of the use of OPNFV trademarks.| -| | The dovetail project is executed with the guidance and | -| | oversight of the Compliance and Certification committee | -| | and work to secure the goals of the C&C committee for | -| | each release. The project intends to incrementally | -| | define qualification criteria that establish the | -| | foundations of how we are able to measure the ability to| -| | utilize the OPNFV platform, how the platform itself | -| | should behave, and how applications may be deployed on | -| | the platform. | -+----------------+---------------------------------------------------------+ -| Functest | This project deals with the functional testing of the | -| | VIM and NFVI. It leverages several upstream test suites | -| | (OpenStack, ODL, ONOS, etc.) and can be used by feature | -| | project to launch feature test suites in CI/CD. | -| | The project is used for scenario validation. | -+----------------+---------------------------------------------------------+ -| Qtip | QTIP as the project for "Platform Performance | -| | Benchmarking" in OPNFV aims to provide user a simple | -| | indicator for performance, supported by comprehensive | -| | testing data and transparent calculation formula. | -| | It provides a platform with common services for | -| | performance benchmarking which helps users to build | -| | indicators by themselves with ease. | -+----------------+---------------------------------------------------------+ -| Storperf | The purpose of this project is to provide a tool to | -| | measure block and object storage performance in an NFVI.| -| | When complemented with a characterization of typical VF | -| | storage performance requirements, it can provide | -| | pass/fail thresholds for test, staging, and production | -| | NFVI environments. | -+----------------+---------------------------------------------------------+ -| VSperf | This project provides a framework for automation of NFV | -| | data-plane performance testing and benchmarking. The | -| | NFVI fast-path includes switch technology and network | -| | with physical and virtual interfaces. VSperf can be | -| | used to evaluate the suitability of different Switch | -| | implementations and features, quantify data-path | -| | performance and optimize platform configurations. | -+----------------+---------------------------------------------------------+ -| Yardstick | The goal of the Project is to verify the infrastructure | -| | compliance when running VNF applications. NFV Use Cases | -| | described in ETSI GS NFV 001 show a large variety of | -| | applications, each defining specific requirements and | -| | complex configuration on the underlying infrastructure | -| | and test tools.The Yardstick concept decomposes typical | -| | VNF work-load performance metrics into a number of | -| | characteristics/performance vectors, which each of them | -| | can be represented by distinct test-cases. | -+----------------+---------------------------------------------------------+ - - -=================================== -The testing working group resources -=================================== - -The assets -========== - -Overall Architecture --------------------- -The Test result management can be summarized as follows:: - - +-------------+ +-------------+ +-------------+ - | | | | | | - | Test | | Test | | Test | - | Project #1 | | Project #2 | | Project #N | - | | | | | | - +-------------+ +-------------+ +-------------+ - | | | - V V V - +---------------------------------------------+ - | | - | Test Rest API front end | - | http://testresults.opnfv.org/test | - | | - +---------------------------------------------+ - ^ | ^ - | V | - | +-------------------------+ | - | | | | - | | Test Results DB | | - | | Mongo DB | | - | | | | - | +-------------------------+ | - | | - | | - +----------------------+ +----------------------+ - | | | | - | Testing Dashboards | | Landing page | - | | | | - +----------------------+ +----------------------+ - - -The testing databases ---------------------- -A Mongo DB Database has been introduced for the Brahmaputra release. -The following collections are declared in this database: - * pods: the list of pods used for production CI - * projects: the list of projects providing test cases - * testcases: the test cases related to a given project - * results: the results of the test cases - * scenarios: the OPNFV scenarios tested in CI - -This database can be used by any project through the testapi. -Please note that projects may also use additional databases. This database is -mainly use to colelct CI results and scenario trust indicators. - -This database is also cloned for OPNFV Plugfest. - - -The test API ------------- -The Test API is used to declare pods, projects, test cases and test results. -Pods correspond to the cluster of machines (3 controller and 2 compute nodes in -HA mode) used to run the tests and defined in Pharos project. -The results pushed in the database are related to pods, projects and cases. -If you try to push results of test done on non referenced pod, the API will -return an error message. - -An additional method dashboard has been added to post-process the raw results in -the Brahmaputra release (deprecated in Colorado release). - -The data model is very basic, 5 objects are available: - * Pods - * Projects - * Testcases - * Results - * Scenarios - -For detailed information, please go to http://artifacts.opnfv.org/releng/docs/testapi.html - - -The reporting -------------- -The reporting page for the test projects is http://testresults.opnfv.org/reporting/ - -.. figure:: ../../images/reporting_page.png - :align: center - :alt: Testing group reporting page - -This page provides a reporting per OPNFV release and per testing project. - -.. figure:: ../../images/reporting_danube_page.png - :align: center - :alt: Testing group Danube reporting page - -An evolution of this page is planned. -It was decided to unify the reporting by creating a landing page that should give -the scenario status in one glance (it was previously consolidated manually -on a wiki page). - -The landing page (planned for Danube 2.0) will be displayed per scenario: - * the status of the deployment - * the score of the test projectS - * a trust indicator - -Additional filters (version, installer, test collection time window,... ) are -included. - -The test case catalog ---------------------- -Until the Colorado release, each testing project was managing the list of its -test cases. It was very hard to have a global view of the available test cases -among the different test projects. A common view was possible through the API -but it was not very user friendly. -In fact you may know all the cases per project calling: - - http://testresults.opnfv.org/test/api/v1/projects/<project_name>/cases - -with project_name: bottlenecks, functest, qtip, storperf, vsperf, yardstick - -It was decided to build a web site providing a consistent view of the test cases -per project and allow any scenario owner to build his/her custom list of tests -(Danube 2.0). - -Other resources -=============== - -wiki: https://wiki.opnfv.org/testing - -mailing list: test-wg@lists.opnfv.org - -IRC chan: #opnfv-testperf - -weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf): - * Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST - * APAC time: 2nd Wednesday of the month 8:00-9:00 UTC - -======================= -Reference documentation -======================= - -+----------------+---------------------------------------------------------+ -| Project | Documentation links | -+================+=========================================================+ -| Bottlenecks | https://wiki.opnfv.org/display/bottlenecks/Bottlenecks | -+----------------+---------------------------------------------------------+ -| CPerf | https://wiki.opnfv.org/display/cperf | -+----------------+---------------------------------------------------------+ -| Dovetail | https://wiki.opnfv.org/display/dovetail | -+----------------+---------------------------------------------------------+ -| Functest | https://wiki.opnfv.org/display/functest/ | -+----------------+---------------------------------------------------------+ -| Qtip | https://wiki.opnfv.org/display/qtip | -+----------------+---------------------------------------------------------+ -| Storperf | https://wiki.opnfv.org/display/storperf/Storperf | -+----------------+---------------------------------------------------------+ -| VSperf | https://wiki.opnfv.org/display/vsperf | -+----------------+---------------------------------------------------------+ -| Yardstick | https://wiki.opnfv.org/display/yardstick/Yardstick | -+----------------+---------------------------------------------------------+ |