diff options
Diffstat (limited to 'docs/testing/user/ovpaddendum/index.rst')
| -rw-r--r-- | docs/testing/user/ovpaddendum/index.rst | 456 |
1 files changed, 456 insertions, 0 deletions
diff --git a/docs/testing/user/ovpaddendum/index.rst b/docs/testing/user/ovpaddendum/index.rst new file mode 100644 index 00000000..811c2bcd --- /dev/null +++ b/docs/testing/user/ovpaddendum/index.rst @@ -0,0 +1,456 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Intel and others + +.. _dovetail-ovp-addendum: + +======================================= +Guidelines Addendum for 2019.12 release +======================================= + +.. toctree:: + :maxdepth: 2 + + +Introduction +============ + +This addendum provides a high-level description of the testing scope and +pass/fail criteria used in the OPNFV Verification Program (OVP) for the 2019.12 +release. This information is intended as an overview for OVP testers and for +the Dovetail Project to help guide test-tool and test-case development for the +OVP 2019.12 release. The Dovetail project is responsible for documenting +test-case specifications as well as implementing the OVP tool-chain through +collaboration with the OPNFV and ONAP testing communities. OVP testing focuses on +establishing the ability of the System Under Test (SUT) to perform NFVI and VIM +operations and support Service Provider oriented features that ensure +manageable, resilient and secure networks. + + +Meaning of Compliance +===================== + +OPNFV Compliance indicates adherence of an NFV platform and VNF to behaviors +defined through specific platform capabilities, allowing to prepare, instantiate, +operate and remove VNFs running on the NFVI. OVP 2019.12 compliance evaluates +the ability of a platform to support Service Provider network capabilities and +workloads that are supported in the OPNFV and ONAP platforms as of this release. +Test cases are designated as compulsory or optional based on the maturity +of capabilities as well as industry expectations. Compulsory test cases may for +example include NFVI management capabilities whereas tests for certain +high-availability features may be deemed as optional. + +Test coverage and pass/fail criteria are designed to ensure an acceptable level +of compliance but not be so restrictive as to disqualify variations in platform +implementations, capabilities and features. + + +SUT Assumptions +=============== + +Assumptions about the NFVI System Under Test (SUT) for the OVP Infrastructure +badge include ... + +- The minimal specification of physical infrastructure, including controller + nodes, compute nodes and networks, is defined for the NFVI by the + `Pharos specification`_. + +- The SUT is fully deployed and operational, i.e. SUT deployment tools are + out of scope of testing. + +Assumptions about the VNF System Under Test (SUT) for the OVP VNF +badge include ... + +- The VNF templates and disk image(s) file are available, and the disk image(s) + have been deployed to the ONAP Cloud Site. + +- The required value for the VNF pre-load files are available for the selected + ONAP Cloud Site. + +Scope of Testing +================ + +The `OVP Governance Guidelines`_, as approved by the Board of Directors, +outlines the key objectives of the OVP as follows: + +- Help build the market for + + - LFN based infrastructure + + - applications designed to run on that infrastructure + +- Reduce adoption risks for end-users + +- Decrease testing costs by verifying hardware and software platform + interfaces and components + +- Enhance interoperability + +The guidelines further directs the scope to be constrained to "features, +capabilities, components, and interfaces included in an OPNFV and ONAP releases +that are generally available in the industry (e.g., through adoption by an upstream +community)", and that compliance verification is evaluated using "functional +tests that focus on defined interfaces and/or behaviors without regard to the +implementation of the underlying system under test". + +OPNFV provides a broad range of capabilities, including the reference platform +itself as well as tools-chains and methodologies for building infrastructures, +and deploying and testing the platform. Not all these aspects are in scope for +OVP and not all functions and components are tested in the initial versions of +OVP. For example, the deployment tools for the SUT and CI/CD toolchain are +currently out of scope. Similarly, performance benchmarking related testing is +also out of scope or for further study. Newer functional areas such as MANO +(outside of APIs in the NFVI and VIM) are still developing and are for future +considerations. + +ONAP provides a comprehensive platform for real-time, policy-driven orchestration +and automation of physical and virtual network functions that will enable software, +network, IT and cloud providers and developers to rapidly automate new services and +support complete lifecycle management. By unifying member resources, ONAP is +accelerating the development of a vibrant ecosystem around a globally shared +architecture and implementation for network automation–with an open standards focus– +faster than any one product could on its own. + +General Approach +---------------- + +In order to meet the above objectives for OVP, we aim to follow a general +approach by first identifying the overall requirements for all stake-holders, +then analyzing what OPNFV and the upstream communities can effectively test and +verify presently to derive an initial working scope for OVP, and to recommend +what the community should strive to achieve in future releases. + +The overall requirements for OVP can be categorized by the basic cloud +capabilities representing common operations needed by basic VNFs, and +additional requirements for VNFs that go beyond the common cloud capabilities +including functional extensions, operational capabilities and additional +carrier grade requirements. + +For the basic NFV requirements, we will analyze the required test cases, +leverage or improve upon existing test cases in OPNFV projects and upstream +projects whenever we can, and bridge the gaps when we must, to meet these basic +requirements. + +We are not yet ready to include compliance requirements for capabilities such +as hardware portability, carrier grade performance, fault management and other +operational features, security, MANO and VNF verification. These areas are +being studied for consideration in future OVP releases. + +In some areas, we will start with a limited level of verification initially, +constrained by what community resources are able to support at this time, but +still serve a basic need that is not being fulfilled elsewhere. In these +areas, we bring significant value to the community we serve by starting a new +area of verification, breaking new ground and expanding it in the future. + +In other areas, the functions being verified have yet to reach wide adoption +but are seen as important requirements in NFV, or features are only needed for +specific NFV use cases but an industry consensus about the APIs and behaviors +is still deemed beneficial. In such cases, we plan to incorporate the test +areas as optional. An optional test area will not have to be run or passed in +order to achieve compliance. Optional tests provide an opportunity for vendors +to demonstrate compliance with specific OPNFV features beyond the mandatory +test scope. + + +Analysis of Scope +----------------- + +In order to define the scope of the 2019.12 release of the compliance and +verification program, this section analyzes NFV-focused platform capabilities +with respect to the high-level objectives and the general approach outlined in +the previous section. The analysis determines which capabilities are suitable +for inclusion in this release of the OVP and which capabilities are to be +addressed in future releases. + +1. Basic Cloud Capabilities + +The intent of these tests is to verify that the SUT has the required +capabilities that a basic VNF needs, and these capabilities are implemented in +a way that enables this basic VNF to run on any OPNFV compliant deployment. + +A basic VNF can be thought of as a single virtual machine that is networked and +can perform the simplest network functions, for example, a simple forwarding +gateway, or a set of such virtual machines connected only by simple virtual +network services. Running such basic VNF leads to a set of common requirements, +including: + +- image management (testing Glance API) +- identity management (testing Keystone Identity API) +- virtual compute (testing Nova Compute API) +- virtual storage (testing Cinder API) +- virtual networks (testing Neutron Network API) +- forwarding packets through virtual networks in data path +- filtering packets based on security rules and port security in data path +- dynamic network runtime operations through the life of a VNF (e.g. attach/detach, + enable/disable, read stats) +- correct behavior after common virtual machine life cycles events (e.g. + suspend/resume, reboot, migrate) +- simple virtual machine resource scheduling on multiple nodes + +OPNFV mainly supports OpenStack as the VIM up to the 2019.12 release. The VNFs +used in the OVP NFVI program, and features in scope for the program which are +considered to be basic to all VNFs, require commercial OpenStack distributions +to support a common basic level of cloud capabilities, and to be compliant to a +common specification for these capabilities. This requirement significantly +overlaps with OpenStack community's Interop working group's goals, but they are +not identical. The OVP runs the OpenStack Refstack-Compute test cases to verify +compliance to the basic common API requirements of cloud management functions +and VNF (as a VM) management for OPNFV. Additional NFV specific requirements +are added in network data path validation, packet filtering by security group +rules and port security, life cycle runtime events of virtual networks, +multiple networks in a topology, validation of VNF's functional state after +common life-cycle events including reboot, pause, suspense, stop/start and cold +migration. In addition, the basic requirement also verifies that the SUT can +allocate VNF resources based on simple anti-affinity rules. + +The combined test cases help to ensure that these basic operations are always +supported by a compliant platform and they adhere to a common standard to +enable portability across OPNFV compliant platforms. + +2. NFV specific functional requirements + +NFV has functional requirements beyond the basic common cloud capabilities, +esp. in the networking area. Examples like BGPVPN, IPv6, SFC may be considered +additional NFV requirements beyond general purpose cloud computing. These +feature requirements expand beyond common OpenStack (or other VIM) +requirements. OPNFV OVP will incorporate test cases to verify compliance in +these areas as they become mature. Because these extensions may impose new API +demands, maturity and industry adoption is a prerequisite for making them a +mandatory requirement for OPNFV compliance. At the time of the 2019.12 release, +we have promoted tests of the OpenStack IPv6 API from optional to mandatory +while keeping BGPVPN as optional test area. Passing optional tests will not be +required to pass OPNFV compliance verification. + +BGPVPNs are relevant due to the wide adoption of MPLS/BGP based VPNs in wide +area networks, which makes it necessary for data centers hosting VNFs to be +able to seamlessly interconnect with such networks. SFC is also an important +NFV requirement, however its implementation has not yet been accepted or +adopted in the upstream at the time of the 2019.12 release. + +3. High availability + +High availability is a common carrier grade requirement. Availability of a +platform involves many aspects of the SUT, for example hardware or lower layer +system failures or system overloads, and is also highly dependent on +configurations. The current OPNFV high availability verification focuses on +OpenStack control service failures and resource overloads, and verifies service +continuity when the system encounters such failures or resource overloads, and +also verifies the system heals after a failure episode within a reasonable time +window. These service HA capabilities are commonly adopted in the industry and +should be a mandatory requirement. + +The current test cases in HA cover the basic area of failure and resource +overload conditions for a cloud platform's service availability, including all +of the basic cloud capability services, and basic compute and storage loads, so +it is a meaningful first step for OVP. We expect additional high availability +scenarios be extended in future releases. + +4. Stress Testing + +Resiliency testing involves stressing the SUT and verifying its ability to +absorb stress conditions and still provide an acceptable level of service. +Resiliency is an important requirement for end-users. + +The 2019.12 release of OVP includes a load test which spins up a number of VMs +pairs in parallel to assert that the system under test can process the workload +spike in a stable and deterministic fashion. + +5. Security + +Security is among the top priorities as a carrier grade requirement by the +end-users. Some of the basic common functions, including virtual network +isolation, security groups, port security and role based access control are +already covered as part of the basic cloud capabilities that are verified in +OVP. These test cases however do not yet cover the basic required security +capabilities expected of an end-user deployment. It is an area that we should +address in the near future, to define a common set of requirements and develop +test cases for verifying those requirements. + +The 2019.12 release includes new test cases which verify that the role-based +access control (RBAC) functionality of the VIM is behaving as expected. + +Another common requirement is security vulnerability scanning. While the OPNFV +security project integrated tools for security vulnerability scanning, this has +not been fully analyzed or exercised in 2019.12 release. This area needs +further work to identify the required level of security for the purpose of +OPNFV in order to be integrated into the OVP. End-user inputs on specific +requirements in security is needed. + +6. Service assurance + +Service assurance (SA) is a broad area of concern for reliability of the +NFVI/VIM and VNFs, and depends upon multiple subsystems of an NFV platform for +essential information and control mechanisms. These subsystems include +telemetry, fault management (e.g. alarms), performance management, audits, and +control mechanisms such as security and configuration policies. + +The current 2019.12 release implements some enabling capabilities in NFVI/VIM +such as telemetry, policy, and fault management. However, the specification of +expected system components, behavior and the test cases to verify them have not +yet been adequately developed. We will therefore not be testing this area at +this time but defer to future study. + +7. Use case testing + +Use-case test cases exercise multiple functional capabilities of a platform in +order to realize a larger end-to-end scenario. Such end-to-end use cases do +not necessarily add new API requirements to the SUT per se, but exercise +aspects of the SUT's functional capabilities in more complex ways. For +instance, they allow for verifying the complex interactions among multiple VNFs +and between VNFs and the cloud platform in a more realistic fashion. End-users +consider use-case-level testing as a significant tool in verifying OPNFV +compliance because it validates design patterns and support for the types of +NFVI features that users care about. + +There are a lot of projects in OPNFV developing use cases and sample VNFs. The +2019.12 release of OVP features two such use-case tests, spawning and verifying +a vIMS and a vEPC, correspondingly. + +8. Additional NFVI capabilities + +In addition to the capabilities analyzed above, there are further system +aspects which are of importance for the OVP. These comprise operational and +management aspects such as platform in-place upgrades and platform operational +insights such as telemetry and logging. Further aspects include API backward +compatibility / micro-versioning, workload migration, multi-site federation and +interoperability with workload automation platforms, e.g. ONAP. Finally, +efficiency aspects such as the hardware and energy footprint of the platform +are worth considering in the OVP. + +OPNFV is addressing these items on different levels of details in different +projects. However, the contributions developed in these projects are not yet +considered widely available in commercial systems in order to include them in +the OVP. Hence, these aspects are left for inclusion in future releases of the +OVP. + +9. VNF Compliance + +VNF Compliance verifies the VNF template files conform to the requirements documented +in by ONAP VNFRQTS project. + +10. VNF Validation + +VNF Validation verifies the VNF is able to onbroad within ONAP and ONAP is able to +perform basic orchestration operations with the VNF, including instantiating the +VNF on the Cloud Site. + +Scope of the 2019.12 release of the OVP +--------------------------------------- + +Summarizing the results of the analysis above, the scope of the 2019.12 release +of OVP is as follows: + +- Mandatory NFVI test scope: + + - functest.vping.userdata + - functest.vping.ssh + - functest.tempest.osinterop\* + - functest.tempest.compute + - functest.tempest.identity_v3 + - functest.tempest.image + - functest.tempest.network_api + - functest.tempest.volume + - functest.tempest.neutron_trunk_ports + - functest.tempest.ipv6_api + - functest.security.patrole + - yardstick.ha.nova_api + - yardstick.ha.neutron_server + - yardstick.ha.keystone + - yardstick.ha.glance_api + - yardstick.ha.cinder_api + - yardstick.ha.cpu_load + - yardstick.ha.disk_load + - yardstick.ha.haproxy + - yardstick.ha.rabbitmq + - yardstick.ha.database + - bottlenecks.stress.ping + +- Optional NFVI test scope: + + - functest.tempest.ipv6_scenario + - functest.tempest.multi_node_scheduling + - functest.tempest.network_security + - functest.tempest.vm_lifecycle + - functest.tempest.network_scenario + - functest.tempest.bgpvpn + - yardstick.ha.neutron_l3_agent + - yardstick.ha.controller_restart + - functest.vnf.vims + - functest.vnf.vepc + +- Mandatory VNF test scope: + + - Refer to `ONAP VNF Test Case Descriptions <https://docs.onap.org/en/elalto/submodules/vnfrqts/testcases.git/docs/index.html>`_ + +\* The OPNFV OVP utilizes the same set of test cases as the OpenStack +interoperability program *OpenStack Powered Compute*. Passing the OPNFV OVP +does **not** imply that the SUT is certified according to the *OpenStack +Powered Compute* program. *OpenStack Powered Compute* is a trademark of the +OpenStack foundation and the corresponding certification label can only be +awarded by the OpenStack foundation. + +Note: The SUT is limited to NFVI and VIM functions. While testing MANO +component capabilities is out of scope, certain APIs exposed towards MANO are +used by the current OPNFV compliance testing suite. MANO and other operational +elements may be part of the test infrastructure; for example used for workload +deployment and provisioning. + + +Scope considerations for future OVP releases +-------------------------------------------- + +Based on the previous analysis, the following items are outside the scope of +the 2019.12 release of OVP but are being considered for inclusion in future +releases: + +- service assurance +- use case testing +- platform in-place upgrade +- API backward compatibility / micro-versioning +- workload migration +- multi-site federation +- service function chaining +- platform operational insights, e.g. telemetry, logging +- efficiency, e.g. hardware and energy footprint of the platform +- interoperability with workload automation platforms e.g. ONAP +- resilience +- security and vulnerability scanning +- performance measurements + + +Criteria for Awarding Compliance +================================ + +This section provides guidance on compliance criteria for each test area. The +criteria described here are high-level, detailed pass/fail metrics are +documented in Dovetail test specifications. + +1. All mandatory test cases must pass. + +Exceptions to this rule may be legitimate, e.g. due to imperfect test tools or +reasonable circumstances that we can not foresee. These exceptions must be +documented and accepted by the reviewers. + +2. Optional test cases are optional to run. Its test results, pass or fail, + do not impact compliance. + +Applicants who choose to run the optional test cases can include the results +of the optional test cases to highlight the additional compliance. + + +Exemption from strict API response validation +============================================= + +Vendors of commercial NFVI products may have extended the Nova API to support +proprietary add-on features. These additions can cause Nova Tempest API tests +to fail due to unexpected data in API responses. In order to resolve this +transparently in the context of OVP, a temporary exemption process has been +created. More information on the exemption can be found in section +:ref:`dovetail-exemption_process_api_response_validation`. + + +.. References +.. _`OVP Governance Guidelines`: https://www.opnfv.org/wp-content/uploads/sites/12/2018/01/OVP-Governance-Guidelines-1.0.1-012218.pdf +.. _`Pharos specification`: https://wiki.opnfv.org/display/pharos/Pharos+Specification + |