From 8f16dae9dae6d13bc6f5934bae5d26d420efd892 Mon Sep 17 00:00:00 2001 From: Georg Kunz Date: Fri, 22 Dec 2017 14:48:10 +0100 Subject: Changing CVP to OVP in the documentation Due to the name change from 'CVP' to 'OVP', the documentation has to be updated. Change-Id: Id934e8fc07f60802f7aa6b3d295d268c4c9b71f8 Signed-off-by: Georg Kunz --- docs/testing/user/ovpaddendum/index.rst | 420 ++++++++++++++++++++++++++++++++ 1 file changed, 420 insertions(+) create mode 100644 docs/testing/user/ovpaddendum/index.rst (limited to 'docs/testing/user/ovpaddendum/index.rst') diff --git a/docs/testing/user/ovpaddendum/index.rst b/docs/testing/user/ovpaddendum/index.rst new file mode 100644 index 00000000..2bb7e8c8 --- /dev/null +++ b/docs/testing/user/ovpaddendum/index.rst @@ -0,0 +1,420 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Intel and others + +==================================================================== +OPNFV Verified Program - Guidelines Addendum for Danube +==================================================================== + +.. toctree:: + :maxdepth: 2 + + +Introduction +============ + +This addendum provides a high-level description of the testing scope and +pass/fail criteria used in the OPNFV Verified Program (OVP) for the +OPNFV Danube release. This information is intended as an overview for OVP +testers and for the Dovetail Project to help guide test-tool and test-case +development for the OPNFV Danube release. The Dovetail project is responsible for documenting +test-case specifications as well as implementing the OVP tool-chain through collaboration +with the OPNFV testing community. OVP testing focuses on establishing the +ability of the System Under Test (SUT) to perform NFVI and VIM operations and support +Service Provider oriented features that ensure manageable, resilient and secure +networks. + + +Meaning of Compliance +===================== + +OPNFV Compliance indicates adherence of an NFV platform to behaviors defined +through specific platform capabilities, allowing to prepare, instantiate, +operate and remove VNFs running on the NFVI. Danube compliance evaluates the +ability of a platform to support Service Provider network capabilities and +workloads that are supported in the OPNFV platform as of this release. +Compliance test cases are designated as compulsory or optional based on the +maturity of OPNFV capabilities as well as industry expectations. Compulsory +test cases may for example include NFVI management capabilities whereas tests +for certain high-availability features may be deemed as optional. + +Test coverage and pass/fail criteria are +designed to ensure an acceptable level of compliance but not be so restrictive +as to disqualify variations in platform implementations, capabilities and features. + + +SUT Assumptions +=============== + +Assumptions about the System Under Test (SUT) include ... + +- The minimal specification of physical infrastructure, including controller + nodes, compute nodes and networks, is defined by the `Pharos specification`_. + +- The SUT is fully deployed and operational, i.e. SUT deployment tools are + out of scope of testing. + + +Scope of Testing +================ + +The `OPNFV OVP Guidelines`_, as approved by the Board of Directors, outlines +the key objectives of the OVP as follows: + +- Help build the market for + + - OPNFV based infrastructure + + - applications designed to run on that infrastructure + +- Reduce adoption risks for end-users + +- Decrease testing costs by verifying hardware and software platform + interfaces and components + +- Enhance interoperability + +The guidelines further directs the scope to be constrained to "features, +capabilities, components, and interfaces included in an OPNFV release that are +generally available in the industry (e.g., through adoption by an upstream +community)", and that compliance verification is evaluated using "functional tests +that focus on defined interfaces and/or behaviors without regard to the +the implementation of the underlying system under test". + +OPNFV provides a broad range of capabilities, including the reference platform itself +as well as tools-chains and methodologies for building infrastructures, and +deploying and testing the platform. +Not all these aspects are in scope for OVP and not all functions and +components are tested in the initial version of OVP. For example, the deployment tools +for the SUT and CI/CD toolchain are currently out of scope. +Similarly, performance benchmarking related testing is also out of scope or +for further study. Newer functional areas such as MANO (outside of APIs in the NFVI and +VIM) are still developing and are for future considerations. + + +General Approach +---------------- + +In order to meet the above objectives for OVP, we aim to follow a general approach +by first identifying the overall requirements for all stake-holders, +then analyzing what OPNFV and the upstream communities can effectively test and verify +presently to derive an initial working scope for OVP, and to recommend what the +community should strive to achieve in future releases. + +The overall requirements for OVP can be categorized by the basic cloud +capabilities representing common operations needed by basic VNFs, and additional +requirements for VNFs that go beyond the common cloud capabilities including +functional extensions, operational capabilities and additional carrier grade +requirements. + +For the basic NFV requirements, we will analyze the required test cases, +leverage or improve upon existing test cases in OPNFV projects +and upstream projects whenever we can, and bridge the gaps when we must, to meet +these basic requirements. + +We are not yet ready to include compliance requirements for capabilities such +as hardware portability, carrier grade performance, fault management and other +operational features, security, MANO and VNF verification. These areas are +being studied for consideration in future OVP releases. + +In some areas, we will start with a limited level of verification +initially, constrained by what community resources are able to support at this +time, but still serve a basic need that is not being fulfilled elsewhere. +In these areas, we bring significant value to the community we +serve by starting a new area of verification, breaking new ground and +expanding it in the future. + +In other areas, the functions being verified have yet to reach +wide adoption but are seen as important requirements in NFV, +or features are only needed for specific NFV use cases but +an industry consensus about the APIs and behaviors is still deemed beneficial. In such +cases, we plan to incorporate the test areas as optional. An optional test +area will not have to be run or passed in order to achieve compliance. +Optional tests provide an opportunity for vendors to demonstrate compliance with specific OPNFV +features beyond the mandatory test scope. + + +Analysis of Scope +----------------- + +In order to define the scope of the Danube-release of the compliance and +verification program, this section analyzes NFV-focused platform capabilities +with respect to the high-level objectives and the general approach outlined in +the previous section. The analysis determines which capabilities are suitable +for inclusion in this release of the OVP and which capabilities are to be +addressed in future releases. + +1. Basic Cloud Capabilities + +The intent of these tests is to verify that the SUT has the required +capabilities that a basic VNF needs, and these capabilities are implemented +in a way that enables this basic VNF to run on any OPNFV compliant +deployment. + +A basic VNF can be thought of as a single virtual machine that is networked +and can perform the simplest network functions, for example, a simple forwarding +gateway, or a set of such virtual machines connected only by simple virtual network +services. Running such basic VNF leads to a set of common requirements, including: + +- image management (Refstack testing Glance API) +- identity management (Refstack testing Keystone Identity API) +- virtual compute (Refstack testing Nova Compute API) +- virtual storage (Refstack testing Cinder API) +- virtual networks (Refstack testing Neutron Network API) +- forwarding packets through virtual networks in data path +- filtering packets based on security rules and port security in data path +- dynamic network runtime operations through the life of a VNF (e.g. attach/detach, + enable/disable, read stats) +- correct behavior after common virtual machine life cycles events (e.g. + suspend/resume, reboot, migrate) +- simple virtual machine resource scheduling on multiple nodes + +OPNFV mainly supports OpenStack as the VIM up to the Danube release. The +VNFs used in the OVP program, and features in scope for the program which are +considered to be basic to all VNFs, require commercial OpenStack distributions +to support a common basic level of cloud capabilities, and to be compliant +to a common specification for these capabilities. This requirement significantly +overlaps with OpenStack community's Interop working group's goals, but they are not +identical. The OVP runs the OpenStack Refstack-Compute test cases to verify +compliance to the basic common API requirements of cloud +management functions and VNF (as a VM) management for OPNFV. +Additional NFV specific requirements are added in network data path validation, +packet filtering by security group rules and port security, life cycle runtime events of +virtual networks, multiple networks in a topology, validation +of VNF's functional state after common life-cycle events including reboot, pause, +suspense, stop/start and cold migration. In addition, the +basic requirement also verifies that the SUT can allocate VNF resources based +on simple anti-affinity rules. + +The combined test cases help to ensure that these basic operations are always +supported by a compliant platform and they adhere to +a common standard to enable portability across OPNFV compliant platforms. + +2. NFV specific functional requirements + +NFV has functional requirements beyond the basic common cloud +capabilities, esp. in the networking area. Examples like SDNVPN, IPv6, SFC may +be considered additional NFV requirements beyond general purpose cloud +computing. These feature requirements expand beyond common OpenStack (or other +VIM) requirements. OPNFV OVP will incorporate test cases to verify +compliance in these areas as they become mature. Because these extensions +may impose new API demands, maturity and industry adoption is a prerequisite for +making them a mandatory requirement for OPNFV compliance. At the time of Danube, +we have not identified a new functional area that is mandatory for OVP. +In the meantime, OVP +intends to offer tests in some of these areas as an optional extension of the test +report to be submitted for review, noting that passing these tests will not be +required to pass OPNFV compliance verification. + +SDNVPN is relevant due to the wide adoption of MPLS/BGP based VPNs in wide area +networks, which makes it necessary for data centers hosting VNFs to be able to +seamlessly interconnect with such networks. IPv6 is a high priority service provider +requirement to ease IP addressing and operational issues. SFC is also an important +NFV requirement, however its implementation has not yet been accepted or adopted +in the upstream at the time of Danube. + +3. High availability + +High availability is a common carrier grade requirement. Availability of a +platform involves many aspects of the SUT, for example hardware or lower layer +system failures or system overloads, and is also highly dependent on +configurations. The current OPNFV high availability verification focuses on +OpenStack control service failures and resource overloads, and verifies service +continuity when the system encounters such failures or resource overloads, and +also verifies the system heals after a failure episode within a reasonable time +window. These service HA capabilities are commonly adopted in the industry +and should be a mandatory requirement. + +The current test cases in HA cover the basic area of failure and resource +overload conditions for a cloud platform's service availability, including all +of the basic cloud capability services, and basic compute and storage loads, +so it is a meaningful first step for OVP. We expect additional high availability +scenarios be extended in future releases. + +4. Resiliency + +Resiliency testing involves stressing the SUT and verifying its ability +to absorb stress conditions and still provide an acceptable level of service. +Resiliency is an important requirement for end-users. + +The OPNFV testing projects have started testing +OPNFV system resiliency in +the Danube release that can be used to provide limited coverage in this area. +However, this is a relatively new test methodology in OPNFV, additional study +and testing experiences are still needed. We defer the resiliency testing to +future OVP releases. + +5. Security + +Security is among the top priorities as a carrier grade requirement by the +end-users. Some of the basic common functions, including virtual network isolation, +security groups, port security and role based access control are already covered as +part of the basic cloud capabilities that are verified in OVP. These test cases +however do not yet cover the basic required security capabilities expected of an end-user +deployment. It is an area that we should address in the near future, to define +a common set of requirements and develop test cases for verifying those +requirements. + +Another common requirement is security vulnerability scanning. +While the OPNFV security project integrated tools for security vulnerability +scanning, this has not been fully analyzed or exercised in Danube release. +This area needs further work to identify the required level of security for the +purpose of OPNFV in order to be integrated into the OVP. End-user inputs on +specific requirements in security is needed. + +6. Service assurance + +Service assurance (SA) is a broad area of concern for reliability of the NFVI/VIM +and VNFs, and depends upon multiple subsystems of an NFV platform for essential +information and control mechanisms. These subsystems include telemetry, fault management +(e.g. alarms), performance management, audits, and control mechanisms such as security +and configuration policies. + +The current Danube release implements some enabling capabilities in NFVI/VIM +such as telemetry, policy, and fault management. However, the specification of expected +system components, behavior and the test cases to verify them have not yet +been adequately developed. We will therefore not be testing this area at this time +but defer to future study. + +7. Use case testing + +Use-case test cases exercise multiple functional capabilities of a platform in +order to realize a larger end-to-end scenario. Such end-to-end use cases do +not necessarily add new API requirements to the SUT per se, but exercise +aspects of the SUT's functional capabilities in more complex ways. For +instance, they allow for verifying the complex interactions among multiple VNFs +and between VNFs and the cloud platform in a more realistic fashion. End-users +consider use-case-level testing as a significant tool in verifying OPNFV +compliance because it validates design patterns and support for the types of +NFVI features that users care about. + +There are a lot of projects in OPNFV developing use cases and sample VNFs, +however most are still in early phase and require further enhancements to +become useful additions to the OVP. Examples such as vIMS, or those which are +not yet available in Danube release, e.g. vCPE, will be valuable additions to +the OVP. These use cases need to be widely accepted, and since they are more +complex, using these VNFs for OVP demands a higher level of community resources +to implement, analyze and document these VNFs. Hence, use case testing is not +ready for OVP at the time of Danube, but can be incorporated in Euphrates or as +a future roadmap area. + +8. Additional capabilities + +In addition to the capabilities analyzed above, there are further system +aspects which are of importance for the OVP. These comprise operational and +management aspects such as platform in-place upgrades and platform operational +insights such as telemetry and logging. Further aspects include API backward +compatibility / micro-versioning, workload migration, multi-site federation and +interoperability with workload automation platforms, e.g. ONAP. Finally, +efficiency aspects such as the hardware and energy footprint of the platform +are worth considering in the OVP. + +OPNFV is addressing these items on different levels of details in different +projects. However, the contributions developed in these projects are not yet +considered widely available in commercial systems in order to include them in +the OVP. Hence, these aspects are left for inclusion in future releases of the +OVP. + + + +Scope of the Danube-release of the OVP +-------------------------------------- + +Summarizing the results of the analysis above, the scope of the Danube-release +of the OVP is as follows: + +- Test Area: Basic cloud capabilities + + - **OpenStack interoperability test cases excluding object storage**\* + - **OPNFV-Functest/vPing, including both user data and ssh** + - *Port security and security groups* + - *VM life-cycle events* + - *VM networking* + - *VM resource scheduling* + - *Forwarding packets in the data path* + +\* The OPNFV OVP utilizes the same set of test cases as the OpenStack +interoperability program *OpenStack Powered Compute*. Passing the OPNFV OVP +does **not** imply that the SUT is certified according to the *OpenStack Powered +Compute* program. *OpenStack Powered Compute* is a trademark of the OpenStack +foundation and the corresponding certification label can only be awarded by the +OpenStack foundation. + + + +- Test Area: SDNVPN + + - *OPNFV-SDNVPN* + + +- Test Area: IPv6 + + - *OPNFV-IPv6 + (Limited to overlay tests, v6Ping)* + + +- Test Area: High Availability + + - **OPNFV-Yardstick/HA** + (Limited to service continuity verification of control services) + +[Highlighting: **Mandatory test cases**, *Optional test cases*] + + +These tested areas represent significant advancement in the direction to meet +the OVP's objectives and end-user expectations, and is a good basis for the +initial phase of OVP. + +Note: The SUT is limited to NFVI and VIM functions. While testing MANO +component capabilities is out of scope, certain APIs exposed towards MANO are +used by the current OPNFV compliance testing suite. MANO and other operational +elements may be part of the test infrastructure; for example used for workload +deployment and provisioning. + + +Scope considerations for future OVP releases +-------------------------------------------- + +Based on the previous analysis, the following items are outside the scope of +the Danube release of the CV but are being considered for inclusion in future +releases: + +- service assurance +- use case testing +- platform in-place upgrade +- API backward compatibility / micro-versioning +- workload migration +- multi-site federation +- service function chaining +- platform operational insights, e.g. telemetry, logging +- efficiency, e.g. hardware and energy footprint of the platform +- interoperability with workload automation platforms e.g. ONAP +- resilience +- security and vulnerability scanning +- performance measurements + + +Criteria for Awarding Compliance +================================ + +This section provides guidance on compliance criteria for each test area. The +criteria described here are high-level, detailed pass/fail metrics are +documented in Dovetail test specifications. + +1. All mandatory test cases must pass. + +Exceptions to this rule may be legitimate, e.g. due to imperfect test tools or +reasonable circumstances that we can not foresee. These exceptions must be +documented and accepted by the reviewers. + +2. Optional test cases are optional to run. Its test results, pass or fail, + do not impact compliance. + +Applicants who choose to run the optional test cases can include the results +of the optional test cases to highlight the additional compliance. + +.. References +.. _`OPNFV OVP Guidelines`: https://wiki.opnfv.org/display/dovetail/OVP+document +.. _`Pharos specification`: https://wiki.opnfv.org/display/pharos/Pharos+Specification + -- cgit 1.2.3-korg