diff options
Diffstat (limited to 'docs/testing/ecosystem')
| -rw-r--r-- | docs/testing/ecosystem/index.rst | 14 | ||||
| -rw-r--r-- | docs/testing/ecosystem/overview.rst | 338 |
2 files changed, 352 insertions, 0 deletions
diff --git a/docs/testing/ecosystem/index.rst b/docs/testing/ecosystem/index.rst new file mode 100644 index 000000000..3ffb13bba --- /dev/null +++ b/docs/testing/ecosystem/index.rst @@ -0,0 +1,14 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Christopher Price (Ericsson AB) + +***************************** +OPNFV Test Framework document +***************************** + +.. toctree:: + :maxdepth: 2 + + ./abstract + ./overview + diff --git a/docs/testing/ecosystem/overview.rst b/docs/testing/ecosystem/overview.rst new file mode 100644 index 000000000..940e720ab --- /dev/null +++ b/docs/testing/ecosystem/overview.rst @@ -0,0 +1,338 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +============= +OPNFV testing +============= + +Introduction +============ + +Testing is one of the key activities in OPNFV, including component level testing, +system testing, automated deployment validation and performance characteristics +or stress testing. + +Some projects are fully dedicated to testing, they provide frameworks or tooling. +The feature projects also include their own test suites. +he test projects fulfill different roles (such as functional checks, performance +measurement, platform and component benchmarks, and analysis) on the scenarios +released in OPNFV. + +This document will detail the OPNFV testing ecosystem, describe the commons used +by the projects and provide the links to project specific documentation. + + +OPNFV testing ecosystem +======================= + +The testing projects +-------------------- + +The OPNFV testing projects may be summarized as follows: + +.. figure:: https://wiki.opnfv.org/download/attachments/8688867/EcoSystem%20Copy.png + :align: center + :alt: Overview of OPNFV Testing projects + +The main projects of the testing projects are described in the table below: + ++----------------+---------------------------------------------------------+ +| Project | Description | ++================+=========================================================+ +| Bottlenecks | This project aims to find system bottlenecks by testing | +| | and verifying OPNFV infrastructure in a staging | +| | environment before committing it to a production | +| | environment. Instead of debugging a deployment in | +| | production environment, an automatic method for | +| | executing benchmarks which plans to validate the | +| | deployment during staging is adopted. This project | +| | provides a framework to find the bottlenecks of OPNFV | +| | infrastructure. | ++----------------+---------------------------------------------------------+ +| CPerf | SDN Controller benchmarks and performance testing, | +| | applicable to controllers in general. | ++----------------+---------------------------------------------------------+ +| Dovetail | This project intends to define and provide a set of | +| | OPNFV related validation criteria that will provide | +| | input for the evaluation of the use of OPNFV trademarks.| +| | The dovetail project is executed with the guidance and | +| | oversight of the Compliance and Certification committee | +| | and work to secure the goals of the C&C committee for | +| | each release. The project intends to incrementally | +| | define qualification criteria that establish the | +| | foundations of how we are able to measure the ability to| +| | utilize the OPNFV platform, how the platform itself | +| | should behave, and how applications may be deployed on | +| | the platform. | ++----------------+---------------------------------------------------------+ +| Functest | This project deals with the functional testing of the | +| | VIM and NFVI. It leverages several upstream test suites | +| | (OpenStack, ODL, ONOS,..) and can be used by feature | +| | project to launch feature test suites in CI/CD. | +| | The project is used for scenario validation. | ++----------------+---------------------------------------------------------+ +| Qtip | QTIP as the project for "Platform Performance | +| | Benchmarking" in OPNFV aims to provide user a simple | +| | indicator for performance, supported by comprehensive | +| | testing data and transparent calculation formula. | +| | It provides a platform with common services for | +| | performance benchmarking which helps users to build | +| | indicators by themselves with ease. | ++----------------+---------------------------------------------------------+ +| Storperf | The purpose of this project is to provide a tool to | +| | measure block and object storage performance in an NFVI.| +| | When complemented with a characterization of typical VF | +| | storage performance requirements, it can provide | +| | pass/fail thresholds for test, staging, and production | +| | NFVI environments. | ++----------------+---------------------------------------------------------+ +| VSperf | The Project “vSwitch Performance" develops a generic and| +| | architecture agnostic vSwitch testing framework and | +| | associated tests, that will serve as a basis for | +| | validating the suitability of different vSwitch | +| | implementations in a Telco NFV deployment environment. | +| | The output of this project will be utilized by the OPNFV| +| | Performance and Test group and its associated projects, | +| | as part of OPNFV Platform and VNF level testing and | +| | validation. | ++----------------+---------------------------------------------------------+ +| Yardstick | The goal of the Project is to verify the infrastructure | +| | compliance when running VNF applications. NFV Use Cases | +| | described in ETSI GS NFV 001 show a large variety of | +| | applications, each defining specific requirements and | +| | complex configuration on the underlying infrastructure | +| | and test tools.The Yardstick concept decomposes typical | +| | VNF work-load performance metrics into a number of | +| | characteristics/performance vectors, which each of them | +| | can be represented by distinct test-cases. | ++----------------+---------------------------------------------------------+ + + +=================================== +The testing working group resources +=================================== + +The assets +========== + +Overall Architecture +-------------------- +The Test result management can be summarized as follows:: + + +-------------+ +-------------+ +-------------+ + | | | | | | + | Test | | Test | | Test | + | Project #1 | | Project #2 | | Project #N | + | | | | | | + +-------------+ +-------------+ +-------------+ + | | | + V V V + +---------------------------------------------+ + | | + | Test Rest API front end | + | http://testresults.opnfv.org/test | + | | + +---------------------------------------------+ + ^ | ^ + | V | + | +-------------------------+ | + | | | | + | | Test Results DB | | + | | Mongo DB | | + | | | | + | +-------------------------+ | + | | + | | + +----------------------+ +----------------------+ + | | | | + | Testing Dashboards | | Landing page | + | | | | + +----------------------+ +----------------------+ + + +The testing databases +--------------------- +A Mongo DB Database has been introduced in Brahmaputra. +The following collections are declared in this database: + * pods: the list of pods used for production CI + * projects: the list of projects providing test cases + * testcases: the test cases related to a given project + * results: the results of the test cases + * scenarios: the OPNFV scenarios tested in CI + +This database can be used by any project through the testapi. +Please note that projects may also use additional databases. This database is +mainly use to colelct CI results and scenario trust indicators. + +This database is cloned for plugfest. + + +The test API +------------ +The Test API is used to declare pods, projects, test cases and test +results. Pods are the pods used to run the tests. +The results pushed in the database are related to pods, projects and +cases. If you try to push results of test done on non referenced pod, +the API will return an error message. + +An additional method dashboard has been added to post-process +the raw results in release Brahmaputra (deprecated in Colorado). + +The data model is very basic, 4 objects are created: + + * Pods + * Projects + * Testcases + * Results + +Pods:: + + { + "id": <ID>, + "details": <URL description of the POD>, + "creation_date": "YYYY-MM-DD HH:MM:SS", + "name": <The POD Name>, + "mode": <metal or virtual>, + "role": <ci-pod or community-pod or single-node> + }, + +Projects:: + + { + "id": <ID>, + "name": <Name of the Project>, + "creation_date": "YYYY-MM-DD HH:MM:SS", + "description": <Short description> + }, + +Testcases:: + + { + "id": <ID>, + "name":<Name of the test case>, + "project_name":<Name of belonged project>, + "creation_date": "YYYY-MM-DD HH:MM:SS", + "description": <short description>, + "url":<URL for longer description> + }, + +Results:: + + { + "_id": <ID>, + "case_name": <Reference to the test case>, + "project_name": <Reference to project>, + "pod_name": <Reference to POD where the test was executed>, + "installer": <Installer Apex or Compass or Fuel or Joid>, + "version": <master or Colorado or Brahmaputra>, + "start_date": "YYYY-MM-DD HH:MM:SS", + "stop_date": "YYYY-MM-DD HH:MM:SS", + "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">, + "scenario": <Scenario on which the test was executed>, + "criteria": <PASS or FAILED>, + "trust_indicator": { + "current": 0, + "histories": [] + } + } + + Scenarios:: + + { + "id": <ID>, + "name":<Name of the test case>, + "name" : "os-odl_l2-nofeature-ha", + "installers":[ + { + "installer" : <installer name>, + "versions": [ + { + "version": <version name>, + "owner": <scenario owner>, + "custom list": { "projects": [{ + "functest" : [ "vping_ssh", "vping_userdata", "tempest_smoke_serial", "rally_sanity", "odl", "doctor"], + "yardstick" : [ "tc002","tc005","tc010","tc011","tc012","tc014","tc037","tc055","tc063","tc069","tc070","tc071","tc072","tc075"]}]}, + "score": { "projects": [{ + "functest" : [{"date": YYY-MM-DD HH:MM, "score":<score>}, {"date": YYY-MM-DD HH:MM, "score":<score>}, ...], + "yardstick" : [{"date": YYY-MM-DD HH:MM, "score":<score>}, {"date": YYY-MM-DD HH:MM, "score":<score>}, ...]}]}, + "trust_indicator": { "projects": [{ + "functest" : [{"date": YYY-MM-DD HH:MM,"status":<status>}, {"date": YYY-MM-DD HH:MM,"status":<status>},...], + "yardstick" : [{"date": YYY-MM-DD HH:MM,"status":<status>}, {"date": YYY-MM-DD HH:MM,"status":<status>},...]}]}}, + { .... + }, + +For detailed information, please go to + + http://testresults.opnfv.org/test/swagger/spec.html + + Authentication: opnfv/api@opnfv + +Please notes that POST/DELETE/PUT operations for test or study purpose via +swagger website is not allowed, because it will change the real data in +the database. + + +The reporting +------------- +Until Colorado, each testing project was reporting a status on a dedicated page. +It was decided to unify the reporting by creating a landing page that should give +the scenario status in one glance (it was previously consolidated manually +on a wiki page). The landing page will be display per scenario: + * the status of the deployment + * the score of the test projectS + * a trust indicator + +Additional filters (version, installer, test collection time window,... ) + +This landing page has been dockerized. The back end relies on the testing DB. + + TODO: add picture + +The test case catalog +---------------------- +Until Colorado, each testing project was managing the list of its test cases. It +was very hard to have a global view of the available test cases among the +different test projects. A common view was possible through the API but it was +not very user friendly. +It was decided to build a web site providing a consistent view of the test cases +per project and allow any scenario owner to build his/her custom list of tests. +The test catalog can be described as below:: + + TODO: add picture + +Other resources +=============== + +wiki: https://wiki.opnfv.org/testing + +mailing list: test-wg@lists.opnfv.org + +IRC chan: #opnfv-testperf + +weekly meeting (https://wiki.opnfv.org/display/meetings/TestPerf): + * Usual time: Every Thursday 15:00-16:00 UTC / 7:00-8:00 PST + * APAC time: 2nd Wednesday of the month 8:00-9:00 UTC + +======================= +Reference documentation +======================= + ++----------------+---------------------------------------------------------+ +| Project | Documentation links | ++================+=========================================================+ +| Bottlenecks | https://wiki.opnfv.org/display/bottlenecks/Bottlenecks | ++----------------+---------------------------------------------------------+ +| CPerf | https://wiki.opnfv.org/display/meetings/CPerf | ++----------------+---------------------------------------------------------+ +| Dovetail | https://wiki.opnfv.org/display/dovetail/Dovetail | ++----------------+---------------------------------------------------------+ +| Functest | https://wiki.opnfv.org/display/functest/ | ++----------------+---------------------------------------------------------+ +| Qtip | https://wiki.opnfv.org/display/qtip | ++----------------+---------------------------------------------------------+ +| Storperf | https://wiki.opnfv.org/display/storperf/Storperf | ++----------------+---------------------------------------------------------+ +| VSperf | https://wiki.opnfv.org/display/vsperf/VSperf | ++----------------+---------------------------------------------------------+ +| Yardstick | https://wiki.opnfv.org/display/yardstick/Yardstick | ++----------------+---------------------------------------------------------+ |