diff options
| m--------- | docs/submodules/apex | 0 | ||||
| m--------- | docs/submodules/armband | 0 | ||||
| m--------- | docs/submodules/barometer | 0 | ||||
| m--------- | docs/submodules/bottlenecks | 0 | ||||
| m--------- | docs/submodules/compass4nfv | 0 | ||||
| m--------- | docs/submodules/daisy | 0 | ||||
| m--------- | docs/submodules/doctor | 0 | ||||
| m--------- | docs/submodules/dovetail | 0 | ||||
| m--------- | docs/submodules/fds | 0 | ||||
| m--------- | docs/submodules/fuel | 0 | ||||
| m--------- | docs/submodules/functest | 0 | ||||
| m--------- | docs/submodules/joid | 0 | ||||
| m--------- | docs/submodules/octopus | 0 | ||||
| m--------- | docs/submodules/onosfw | 0 | ||||
| m--------- | docs/submodules/openretriever | 0 | ||||
| m--------- | docs/submodules/parser | 0 | ||||
| m--------- | docs/submodules/pharos | 0 | ||||
| m--------- | docs/submodules/qtip | 0 | ||||
| m--------- | docs/submodules/releng | 0 | ||||
| m--------- | docs/submodules/sdnvpn | 0 | ||||
| m--------- | docs/submodules/sfc | 0 | ||||
| m--------- | docs/submodules/snaps | 0 | ||||
| m--------- | docs/submodules/storperf | 0 | ||||
| m--------- | docs/submodules/vswitchperf | 0 | ||||
| m--------- | docs/submodules/yardstick | 0 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 353 |
26 files changed, 353 insertions, 0 deletions
diff --git a/docs/submodules/apex b/docs/submodules/apex -Subproject 1f346dd0052a8a24309a55058c54299803d95e0 +Subproject 68bb8c986bfdca528a06e18e58507b4750fdf40 diff --git a/docs/submodules/armband b/docs/submodules/armband -Subproject 7dca6641608ea4085cedaf25107c9f702b4cdeb +Subproject de0b0f160f6ac408c2ba302507ede4d0a537b73 diff --git a/docs/submodules/barometer b/docs/submodules/barometer -Subproject 477bc4fd49e4eb2481809cc99297eac02d0d6db +Subproject f56d04d684f5ccae7815960e9282c394545e385 diff --git a/docs/submodules/bottlenecks b/docs/submodules/bottlenecks -Subproject e4a17446f0f02d6ea950278074d88db6c09c5bb +Subproject b39f17a60e91552ffe94c520a30e6b271077f29 diff --git a/docs/submodules/compass4nfv b/docs/submodules/compass4nfv -Subproject 8ed47ae6ab2e57a37cd90707adcd0b2ea53362a +Subproject a0edd1b2b4d4df6fc5c3e773eba66224930fdb4 diff --git a/docs/submodules/daisy b/docs/submodules/daisy -Subproject e0d6c8a4e5c84bedbcbe91ca51c8bbabf1d8fdf +Subproject 69a1af39f02bb5c6a471768648c15f8ae1e8577 diff --git a/docs/submodules/doctor b/docs/submodules/doctor -Subproject d182b8a69eeef01f37b123e7d331972b7c57911 +Subproject 7e0ab6e2149df22455e58eb363eb291a985514d diff --git a/docs/submodules/dovetail b/docs/submodules/dovetail -Subproject 2c170360711d8307ffab3fe8070a771650e4fd4 +Subproject cc034a7135cf70f137915f8b55c8ef00ab61e96 diff --git a/docs/submodules/fds b/docs/submodules/fds -Subproject 4c8ccbb8594b7091f498dcf83b9b8c28835fe71 +Subproject 5363f1bd4b2007bc66f4171069251f813ff2b41 diff --git a/docs/submodules/fuel b/docs/submodules/fuel -Subproject d433573b98d288feaff518dd3ac9a5b19e35f1e +Subproject d676c91a67684a7efb317d58429c49db7c56247 diff --git a/docs/submodules/functest b/docs/submodules/functest -Subproject a48bf0c8306e566f15877f3bd72882cf8b7663d +Subproject 0323218afb7500bac979e46aafc9c33d614c99d diff --git a/docs/submodules/joid b/docs/submodules/joid -Subproject 703cfe7ce2639d3074cf39bffd748e59b5ed7b4 +Subproject aa443647aa36a04cf34358647310ca3baa4ed21 diff --git a/docs/submodules/octopus b/docs/submodules/octopus -Subproject b41f1567f781aebc5fcd9c95c050978ddc95936 +Subproject 3b530194ef79082aebb5afc4cf61cedd83dfc03 diff --git a/docs/submodules/onosfw b/docs/submodules/onosfw -Subproject ee2ebb70cd79c10cb9322e04172b0a0903c6656 +Subproject 0570a96f65db547dcf2f88ac71dfad9b17a0764 diff --git a/docs/submodules/openretriever b/docs/submodules/openretriever -Subproject ddaccb38f85dba6bec1fcd79b0b4cce45d92192 +Subproject b4804ed0b8d88fc91324dfbbf9e285d0691bb0f diff --git a/docs/submodules/parser b/docs/submodules/parser -Subproject a1eebb743002a10194d8f7c5448c02a99f1f0fe +Subproject 5b3f19f30ff02faa80ad02762ae23ad020b0421 diff --git a/docs/submodules/pharos b/docs/submodules/pharos -Subproject d4d63f63f1ab4cd636499adee319f2a0b851302 +Subproject 7cbe4e7447280d39fdb5a56b3d9a0fea437b5b1 diff --git a/docs/submodules/qtip b/docs/submodules/qtip -Subproject 26c2a8f2d556a8777d443b02d931e7df7fec8f6 +Subproject d218f94274ac94c36d2c843f73b83290f761a8f diff --git a/docs/submodules/releng b/docs/submodules/releng -Subproject 26f47542741f17b0d534565fc1cc6a6e6948dd3 +Subproject 1bfef523c2abd5024e2ec95a2fec3262178d194 diff --git a/docs/submodules/sdnvpn b/docs/submodules/sdnvpn -Subproject de80ee56c458b127be3497f3882bc1c962f42df +Subproject 45df38fe0cd7a604827a5cde6ad6480971b0456 diff --git a/docs/submodules/sfc b/docs/submodules/sfc -Subproject a29f1fcf91f130f74dbd9d7c83bb7445faec613 +Subproject d0260c5d35d753e2bce23c74d877da648d47c48 diff --git a/docs/submodules/snaps b/docs/submodules/snaps -Subproject c8212122569c2dbf6290b43a0fbde0171c2ffdc +Subproject 2a6d3a0f73eb58762bdbe10567cfbb6edec54ab diff --git a/docs/submodules/storperf b/docs/submodules/storperf -Subproject c7806a0f08f6114d8b1f037a77af041a2b0364d +Subproject 7602a54309adbe5c5346ee6befecc2e59697650 diff --git a/docs/submodules/vswitchperf b/docs/submodules/vswitchperf -Subproject 80c9f96f5e60ee6a2a93494a0aae82014c2311b +Subproject d3b124a22bf3aa2c05a5cb030f37b97db3d27db diff --git a/docs/submodules/yardstick b/docs/submodules/yardstick -Subproject 0509cf3e141b550653e642083907a584450a7fd +Subproject 29484c3071028df6c897797706117082a4b8c3f diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst new file mode 100644 index 000000000..a23231b03 --- /dev/null +++ b/docs/testing/developer/devguide/index.rst @@ -0,0 +1,353 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. SPDX-License-Identifier: CC-BY-4.0 + +********************* +Test developer guide +********************* + +.. toctree:: + :numbered: + :maxdepth: 2 + + +============ +Introduction +============ + +The OPNFV testing ecosystem is wide. + +The goal of this guide consists in providing some guidelines for new developers +involved in test areas. + +For the description of the ecosystem, see `[1]`_. + + +================= +Developer journey +================= + +Be involved in the testing group +================================ + +Best practices +============== + +Unit tests +---------- + +Dockerization +------------- + +API +--- + +CLI +--- + +Traffic generators +------------------ + +Towards a pseudo micro services approach +---------------------------------------- + +====================================== +Testing group configuration parameters +====================================== + + +Testing categories +================== + +The testing group defined several categories also known as tiers. These +categories can be used to group test suites. + ++----------------+-------------------------------------------------------------+ +| healthcheck | Simple and quick healthcheck tests case | ++----------------+-------------------------------------------------------------+ +| Smoke | Set of smoke test cases/suites to validate the release | ++----------------+-------------------------------------------------------------+ +| Features | Test cases that validate a specific feature on top of OPNFV.| +| | Those come from Feature projects and need a bit of support | +| | for integration | ++----------------+-------------------------------------------------------------+ +| Components | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)| +| | It may extend smoke tests | ++----------------+-------------------------------------------------------------+ +| Performance | Performance qualification | ++----------------+-------------------------------------------------------------+ +| VNF | Test cases related to deploy an open source VNF including | +| | an orchestrator | ++----------------+-------------------------------------------------------------+ +| Stress | Stress and robustness tests | ++----------------+-------------------------------------------------------------+ +| In Service | In service testing | ++----------------+-------------------------------------------------------------+ + +Testing domains +=============== + +The domains deal with the technical scope of the tests. It shall correspond to +domains defined for the certification program: + + * compute + * network + * storage + * hypervisor + * container + * vim + * mano + * vnf + * ... + +Testing coverage +================= +One of the goals of the testing working group is to identify the poorly covered +areas and avoid testing overlap. +Ideally based on the declaration of the test cases, through the tags, domains +and tier fields, it shall be possible to create heuristic maps. + + +============================== +Testing group generic enablers +============================== + + +TestAPI framework +================= + +The OPNFV testing group created a test collection database to collect +the test results from CI: + + + http://testresults.opnfv.org/test/swagger/spec.html + +Any test project running on any lab integrated in CI can push the +results to this database. +This database can be used to see the evolution of the tests and compare +the results versus the installers, the scenarios or the labs. +It is used to produce a dashboard with the current test status of the project. + + +Overall Architecture +-------------------- +The Test result management can be summarized as follows:: + + +-------------+ +-------------+ +-------------+ + | | | | | | + | Test | | Test | | Test | + | Project #1 | | Project #2 | | Project #N | + | | | | | | + +-------------+ +-------------+ +-------------+ + | | | + V V V + +-----------------------------------------+ + | | + | Test Rest API front end | + | | + +-----------------------------------------+ + A | + | V + | +-------------------------+ + | | | + | | Test Results DB | + | | Mongo DB | + | | | + | +-------------------------+ + | + | + +----------------------+ + | | + | test Dashboard | + | | + +----------------------+ + +TestAPI description +------------------- +The TestAPI is used to declare pods, projects, test cases and test +results. Pods are the sets of bare metal or virtual servers and networking +equipments used to run the tests. + +The results pushed in the database are related to pods, projects and test cases. +If you try to push results of test done on non referenced pod, the API will +return an error message. + +An additional method dashboard has been added to post-process +the raw results in release Brahmaputra (deprecated in Colorado). + +The data model is very basic, 5 objects are created: + + * Pods + * Projects + * Testcases + * Results + * Scenarios + +The code of the API is hosted in the releng repository `[6]`_. +The static documentation of the API can be found at `[7]`_. +The TestAPI has been dockerized and may be installed locally in your +lab. See `[15]`_ for details. + +The deployment of the TestAPI has been automated. +A jenkins job manages: + + * the unit tests of the TestAPI + * the creation of a new docker file + * the deployment of the new TestAPI + * the archive of the old TestAPI + * the backup of the Mongo DB + +TestAPI Authorization +~~~~~~~~~~~~~~~~~~~~~ + +PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs +to be added in the request using a header 'X-Auth-Token' for access to the database. + +e.g:: + headers['X-Auth-Token'] + +The value of the header i.e the token can be accessed in the jenkins environment variable +*TestApiToken*. The token value is added as a masked password. + +.. code-block:: python + + headers['X-Auth-Token'] = os.environ.get('TestApiToken') + +The above example is in Python. Token based authentication has been added so that only ci pods +jenkins job can have access to the database. + +Please note that currently token authorization is implemented but is not yet enabled. + +Reporting +========= + +An automatic reporting page has been created in order to provide a +consistent view of the scenarios. + +In this page, each scenario is evaluated according to test criteria. +The code for the automatic reporting is available at `[8]`_. + +The results are collected from the centralized database every day and, +per scenario. A score is calculated based on the results from the last +10 days. + +Dashboard +========= + +Dashboard is used to provide a consistent view of the results collected in CI. +The results showed on the dashboard are post processed from the Database, +which only contains raw results. + +It can be used in addition of the reporting page (high level view) to allow +the creation of specific graphs according to what the test owner wants to show. + +In Brahmaputra, a basic home made dashboard was created in Functest. +In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex +graphs). +Since Danube, the testing community decided to adopt ELK framework and to rely +on bitergia. It was not implemented for Danube but it is planned for Euphrates. + +Bitergia already provides a dashboard for code and infrastructure. +A new Test tab will be added. The dataset will be built by consuming +the TestAPI. + +See `[3]`_ for details. + + +======= +How TOs +======= + +Where can I find information on the different test projects? +=========================================================== + + +How can I contribute to a test project? +======================================= + + +Where can I find hardware resources? +==================================== + + +How do I integrate my tests in CI? +================================== + + +How to declare my tests in the test Database? +============================================= + + +How to push your results into the Test Database? +================================================ + +The test database is used to collect test results. By default it is +enabled only for CI tests from Production CI pods. + +Please note that it is possible to create your own local database. + +A dedicated database is for instance created for each plugfest. + +The architecture and associated API is described in previous chapter. +If you want to push your results from CI, you just have to call the API +at the end of your script. + +You can also reuse a python function defined in functest_utils.py `[5]`_ + + +Where can I find the documentation on the test API? +=================================================== + +http://artifacts.opnfv.org/releng/docs/testapi.html + + + +I have tests, to which category should I declare them? +====================================================== + + + +The main ambiguity could be between features and VNF. +In fact sometimes you have to spawn VMs to demonstrate the capabilities of the +feature you introduced. +We recommend to declare your test in the feature category. + +VNF category is really dedicated to test including: + + * creation of resources + * deployement of an orchestrator/VNFM + * deployment of the VNF + * test of the VNFM + * free resources + +The goal is not to study a particular feature on the infrastructure but to have +a whole end to end test of a VNF automatically deployed in CI. +Moreover VNF are run in weekly jobs (one a week), feature tests are in daily +jobs and use to get a scenario score. + +Where are the logs of CI runs? +============================== + +Logs and configuration files can be pushed to artifact server from the CI under +http://artifacts.opnfv.org/<project name> + + +========== +References +========== + +_`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html + +_`[2]`: http://www.opnfv.org + +_`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing + +_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming + +_`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176 + +_`[6]`: https://git.opnfv.org/functest/tree/releng + +_`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html + + +IRC support chan: #opnfv-testperf |