aboutsummaryrefslogtreecommitdiffstats
path: root/src
AgeCommit message (Expand)AuthorFilesLines
2016-10-03cmd_timeout: Add ovs command timeout configuration optionChristian Trautman1-2/+3
2016-09-29ci: Update default DPDK and OVS versionsMartin Klozik1-4/+2
2016-09-15paths: Support binary packagesMartin Klozik4-62/+17
2016-09-01multi VM: Multi VMs in serial or parallelMartin Klozik1-2/+2
2016-08-18cuse: Remove vHost Cuse supportMartin Klozik3-9/+1
2016-07-26dpdk: Support of DPDK16.07-rc5 and newerMartin Klozik1-6/+7
2016-07-12ovs/ofctl: Fix validation method for complex flows.Antonio Fischetti1-0/+15
2016-07-11Merge "Enable BURST_MODE for l2fwd"Maryam Tahhan1-1/+37
2016-07-07rstp-stp: Add basic functions for stp/rstp enable on ovsChristian Trautman1-0/+31
2016-06-16src: update dpdk, ovs and qemu versionsMaryam Tahhan1-3/+3
2016-06-08Enable BURST_MODE for l2fwdMesut Ali Ergin1-1/+37
2016-05-12dpdk: Support of DPDK v16.04Martin Klozik1-10/+40
2016-05-11Merge "ovs: update to OVS 2.5"Maryam Tahhan1-1/+3
2016-05-11ovs: update to OVS 2.5Maryam Tahhan1-1/+3
2016-05-11qemu: add python path to configureMaryam Tahhan1-1/+1
2016-05-06dpdk: Support new way of DPDK configuration in ovs-vswitchdMartin Klozik3-165/+7
2016-05-04Merge "makefile: Remove obsolete copy operations"Maryam Tahhan2-12/+0
2016-05-04bugfix: Graceful shutdown of VM - improvementMartin Klozik1-5/+12
2016-04-29makefile: Remove obsolete copy operationsMartin Klozik2-12/+0
2016-04-27integration: Support of PVP and PVVP integration TCsMartin Klozik2-11/+11
2016-04-14sriov: Support of SRIOV and Qemu PCI passthroughMartin Klozik1-34/+22
2016-03-21bugfix: Fix errors related to removal of kernel modulesMartin Klozik1-2/+3
2016-03-21integration: Support of integration testcasesMartin Klozik1-0/+24
2016-03-12bugfix: Eliminate error and warning messagesMartin Klozik1-7/+2
2016-03-08dpdk: enable vfio_pci supportMaryam Tahhan1-113/+56
2016-02-16src: update make install for DPDKMaryam Tahhan1-1/+1
2016-02-03Add OVS tunnel encapsulation performance testEnable PVVP deployment for DPDK Vhost User and Vhost CuseMartin Klozik2-23/+26
2015-10-07The 'make' creates all required variants of vSwitchRadek Zetik6-11/+109
2015-09-09Add DNAT/SNAT supportGene Snider1-10/+218
2015-09-01src/dpdk: Rebind DPDK ports to the original driverMaryam Tahhan1-3/+32
2015-08-25vnfs: Enable PVP using vhost-cuseDino Simeon Madarang1-0/+3
2015-08-18vnfs: Enable PVP using vhost-userDino Simeon Madarang2-2/+4
2015-08-12Merge "src: Add QEMU makefile"Maryam Tahhan3-0/+86
2015-08-06Vanilla OVS support implementationMichal Weglicki2-29/+29
heck | Simple and quick healthcheck tests case | +----------------+-------------------------------------------------------------+ | Smoke | Set of smoke test cases/suites to validate the release | +----------------+-------------------------------------------------------------+ | Features | Test cases that validate a specific feature on top of OPNFV.| | | Those come from Feature projects and need a bit of support | | | for integration | +----------------+-------------------------------------------------------------+ | Components | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)| | | It may extend smoke tests | +----------------+-------------------------------------------------------------+ | Performance | Performance qualification | +----------------+-------------------------------------------------------------+ | VNF | Test cases related to deploy an open source VNF including | | | an orchestrator | +----------------+-------------------------------------------------------------+ | Stress | Stress and robustness tests | +----------------+-------------------------------------------------------------+ | In Service | In service testing | +----------------+-------------------------------------------------------------+ Testing domains =============== The domains deal with the technical scope of the tests. It shall correspond to domains defined for the certification program: * compute * network * storage * hypervisor * container * vim * mano * vnf * ... Testing coverage ================= One of the goals of the testing working group is to identify the poorly covered areas and avoid testing overlap. Ideally based on the declaration of the test cases, through the tags, domains and tier fields, it shall be possible to create heuristic maps. ============================================= Reliability, Stress and Long Duration Testing ============================================= Resiliency of NFV refers to the ability of the NFV framework to limit disruption and return to normal or at a minimum acceptable service delivery level in the face of a fault, failure, or an event that disrupts the normal operation `[DEV5]`_. **Reliability** testing evaluates the ability of SUT to recover in face of fault, failure or disrupts in normal operation or simply the ability of SUT absorbing "disruptions". Reliability tests use different forms of faults as stimulus, and the test must measure the reaction in terms of the outage time or impairments to transmission. **Stress testing** involves producing excess load as stimulus, and the test must measure the reaction in terms of unexpected outages or (more likely) impairments to transmission. These kinds of "load" will cause "disruption" which could be easily found in system logs. It is the purpose to raise such "load" to evaluate the SUT if it could provide an acceptable level of service or level of confidence during such circumstances. In Danube and Euphrates, we only considered the stress test with excess load over OPNFV Platform. In Danube, Bottlenecks and Yardstick project jointly implemented 2 stress tests (concurrently create/destroy VM pairs and do ping, system throughput limit) while Bottlenecks acts as the load manager calling yardstick to execute each test iteration. These tests are designed to test for breaking points and provide level of confidence of the system to users. Summary of the test cases are listed in the following addresses: * https://wiki.opnfv.org/display/bottlenecks/Stress+Testing+over+OPNFV+Platform * https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%20Long%20Duration%20POD.pptx?version=2&modificationDate=1502943821000&api=v2 **Stress test cases** for OPNFV Euphrates (OS Ocata) release can be seen as extension/enhancement of those in D release. These tests are located in Bottlenecks/Yardstick repo (Bottlenecks as load manager while Yardstick execute each test iteration): * VNF scale out/up tests (also plan to measure storage usage simultaneously): https://wiki.opnfv.org/pages/viewpage.action?pageId=12390101 * Life-cycle event with throughputs (measure NFVI to support concurrent network usage from different VM pairs): https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Baseline+Stress+Test+Case+for+Bottlenecks+E+Release In OPNFV E release, we also plan to do **long duration testing** over OS Ocata. A separate CI pipe testing OPNFV XCI (OSA) is proposed to accomplish the job. We have applied specific pod for the testing. Proposals and details are listed below: * https://wiki.opnfv.org/display/testing/Euphrates+Testing+needs * https://wiki.opnfv.org/download/attachments/2926539/testing%20evolution%20v1_4.pptx?version=1&modificationDate=1503937629000&api=v2 * https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%20Long%20Duration%20POD.pptx?version=2&modificationDate=1502943821000&api=v2 The long duration testing is supposed to be started when OPNFV E release is published. A simple monitoring module for these tests is also planned to be added: https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Monitoring+Stress+Testing+for+Bottlenecks+E+Release ======= How TOs ======= Where can I find information on the different test projects? =========================================================== On http://docs.opnfv.org! A section is dedicated to the testing projects. You will find the overview of the ecosystem and the links to the project documents. Another source is the testing wiki on https://wiki.opnfv.org/display/testing You may also contact the testing group on the IRC channel #opnfv-testperf or by mail at test-wg AT lists.opnfv.org (testing group) or opnfv-tech-discuss AT lists.opnfv.org (generic technical discussions). How can I contribute to a test project? ======================================= As any project, the best solution is to contact the project. The project members with their email address can be found under https://git.opnfv.org/<project>/tree/INFO You may also send a mail to the testing mailing list or use the IRC channel #opnfv-testperf Where can I find hardware resources? ==================================== You should discuss this topic with the project you are working with. If you need access to an OPNFV community POD, it is possible to contact the infrastructure group. Depending on your needs (scenario/installer/tooling), it should be possible to find free time slots on one OPNFV community POD from the Pharos federation. Create a JIRA ticket to describe your needs on https://jira.opnfv.org/projects/INFRA. You must already be an OPNFV contributor. See https://wiki.opnfv.org/display/DEV/Developer+Getting+Started. Please note that lots of projects have their own "how to contribute" or "get started" page on the OPNFV wiki. How do I integrate my tests in CI? ================================== It shall be discussed directly with the project you are working with. It is done through jenkins jobs calling testing project files but the way to onboard cases differ from one project to another. How to declare my tests in the test Database? ============================================= If you have access to the test API swagger (access granted to contributors), you may use the swagger interface of the test API to declare your project. The URL is http://testresults.opnfv.org/test/swagger/spec.html. .. figure:: ../../../images/swaggerUI.png :align: center :alt: Testing Group Test API swagger Click on *Spec*, the list of available methods must be displayed. .. figure:: ../../../images/API-operations.png :align: center :alt: Testing Group Test API swagger For the declaration of a new project use the POST /api/v1/projects method. For the declaration of new test cases in an existing project, use the POST /api/v1/projects/{project_name}/cases method .. figure:: ../../../images/CreateCase.png :align: center :alt: Testing group declare new test case How to push your results into the Test Database? ================================================ The test database is used to collect test results. By default it is enabled only for CI tests from Production CI pods. Please note that it is possible to create your own local database. A dedicated database is for instance created for each plugfest. The architecture and associated API is described in previous chapter. If you want to push your results from CI, you just have to call the API at the end of your script. You can also reuse a python function defined in functest_utils.py `[DEV2]`_ Where can I find the documentation on the test API? =================================================== The Test API is now documented in this document (see sections above). You may also find autogenerated documentation in http://artifacts.opnfv.org/releng/docs/testapi.html A web protal is also under construction for certification at http://testresults.opnfv.org/test/#/ I have tests, to which category should I declare them? ====================================================== See table above. The main ambiguity could be between features and VNF. In fact sometimes you have to spawn VMs to demonstrate the capabilities of the feature you introduced. We recommend to declare your test in the feature category. VNF category is really dedicated to test including: * creation of resources * deployement of an orchestrator/VNFM * deployment of the VNF * test of the VNFM * free resources The goal is not to study a particular feature on the infrastructure but to have a whole end to end test of a VNF automatically deployed in CI. Moreover VNF are run in weekly jobs (one a week), feature tests are in daily jobs and use to get a scenario score. Where are the logs of CI runs? ============================== Logs and configuration files can be pushed to artifact server from the CI under http://artifacts.opnfv.org/<project name> ========== References ========== `[DEV1]`_: OPNFV Testing Ecosystem `[DEV2]`_: Python code sample to push results into the Database `[DEV3]`_: Testing group wiki page `[DEV4]`_: Conversation with the testing community, OPNFV Beijing Summit `[DEV5]`_: GS NFV 003 .. _`[DEV1]`: http://docs.opnfv.org/en/latest/testing/ecosystem/index.html .. _`[DEV2]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176 .. _`[DEV3]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting .. _`[DEV4]`: https://www.youtube.com/watch?v=f9VAUdEqHoA .. _`[DEV5]`: http://www.etsi.org/deliver/etsi_gs/NFV/001_099/003/01.01.01_60/gs_NFV003v010101p.pdf IRC support chan: #opnfv-testperf