.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 *********************** Testing developer guide *********************** .. toctree:: :numbered: :maxdepth: 2 ============ Introduction ============ The OPNFV testing ecosystem is wide. The goal of this guide consists in providing some guidelines for new developers involved in test areas. For the description of the ecosystem, see `[1]`_. ================= Developer journey ================= Be involved in the testing group ================================ Best practices ============== Unit tests ---------- Dockerization ------------- API --- CLI --- Traffic generators ------------------ Towards a pseudo micro services approach ---------------------------------------- ====================================== Testing group configuration parameters ====================================== Testing categories ================== The testing group defined several categories also known as tiers. These categories can be used to group test suites. +----------------+-------------------------------------------------------------+ | Healthcheck | Simple and quick healthcheck tests case | +----------------+-------------------------------------------------------------+ | Smoke | Set of smoke test cases/suites to validate the release | +----------------+-------------------------------------------------------------+ | Features | Test cases that validate a specific feature on top of OPNFV.| | | Those come from Feature projects and need a bit of support | | | for integration | +----------------+-------------------------------------------------------------+ | Components | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)| | | It may extend smoke tests | +----------------+-------------------------------------------------------------+ | Performance | Performance qualification | +----------------+-------------------------------------------------------------+ | VNF | Test cases related to deploy an open source VNF including | | | an orchestrator | +----------------+-------------------------------------------------------------+ | Stress | Stress and robustness tests | +----------------+-------------------------------------------------------------+ | In Service | In service testing | +----------------+-------------------------------------------------------------+ Testing domains =============== The domains deal with the technical scope of the tests. It shall correspond to domains defined for the certification program: * compute * network * storage * hypervisor * container * vim * mano * vnf * ... Testing coverage ================= One of the goals of the testing working group is to identify the poorly covered areas and avoid testing overlap. Ideally based on the declaration of the test cases, through the tags, domains and tier fields, it shall be possible to create heuristic maps. ============================== Testing group generic enablers ============================== TestAPI framework ================= The OPNFV testing group created a test collection database to collect the test results from CI: http://testresults.opnfv.org/test/swagger/spec.html Any test project running on any lab integrated in CI can push the results to this database. This database can be used to see the evolution of the tests and compare the results versus the installers, the scenarios or the labs. It is used to produce a dashboard with the current test status of the project. Overall Architecture -------------------- The Test result management can be summarized as follows:: +-------------+ +-------------+ +-------------+ | | | | | | | Test | | Test | | Test | | Project #1 | | Project #2 | | Project #N | | | | | | | +-------------+ +-------------+ +-------------+ | | | V V V +-----------------------------------------+ | | | Test Rest API front end | | | +-----------------------------------------+ A | | V | +-------------------------+ | | | | | Test Results DB | | | Mongo DB | | | | | +-------------------------+ | | +----------------------+ | | | test Dashboard | | | +----------------------+ TestAPI description ------------------- The TestAPI is used to declare pods, projects, test cases and test results. Pods are the sets of bare metal or virtual servers and networking equipments used to run the tests. The results pushed in the database are related to pods, projects and test ca