diff options
| -rw-r--r-- | docs/testing/user/cvpaddendum/index.rst | 229 | ||||
| -rw-r--r-- | dovetail/compliance/debug.yml | 6 | ||||
| -rw-r--r-- | dovetail/compliance/proposed_tests.yml | 7 | ||||
| -rw-r--r-- | dovetail/conf/dovetail_config.yml | 2 | ||||
| -rw-r--r-- | dovetail/testcase/osinterop.tc001.yml (renamed from dovetail/testcase/defcore.tc001.yml) | 8 | ||||
| -rw-r--r-- | dovetail/testcase/osinterop.tc002.yml (renamed from dovetail/testcase/defcore.tc002.yml) | 8 |
6 files changed, 143 insertions, 117 deletions
diff --git a/docs/testing/user/cvpaddendum/index.rst b/docs/testing/user/cvpaddendum/index.rst index ee356ff9..69c10b95 100644 --- a/docs/testing/user/cvpaddendum/index.rst +++ b/docs/testing/user/cvpaddendum/index.rst @@ -29,15 +29,15 @@ networks. Meaning of Compliance ===================== -OPNFV Compliance indicates adherence to NFV platform behavior defined as -various platform capabilities or features to prepare, instantiate, operate and remove -VNFs running on the NFVI. Danube compliance evaluates the ability of a platform -to support Service Provider network capabilities and workloads that are -supported in the OPNFV platform as of this release. Compliance test cases shall -be designated as compulsory or optional based on the maturity of OPNFV -capabilities as well as industry expectations. Compulsory test cases may for -example include NFVI management capabilities whereas tests for certain -high-availability features may be deemed as optional. +OPNFV Compliance indicates adherence of an NFV platform to behaviors defined +through specific platform capabilities, allowing to prepare, instantiate, +operate and remove VNFs running on the NFVI. Danube compliance evaluates the +ability of a platform to support Service Provider network capabilities and +workloads that are supported in the OPNFV platform as of this release. +Compliance test cases are designated as compulsory or optional based on the +maturity of OPNFV capabilities as well as industry expectations. Compulsory +test cases may for example include NFVI management capabilities whereas tests +for certain high-availability features may be deemed as optional. Test coverage and pass/fail criteria are designed to ensure an acceptable level of compliance but not be so restrictive @@ -86,6 +86,7 @@ Similarly, performance benchmarking related testing is also out of scope or for further study. Newer functional areas such as MANO (outside of APIs in the NFVI and VIM) are still developing and are for future considerations. + General Approach ---------------- @@ -95,7 +96,7 @@ then analyzing what OPNFV and the upstream communities can effectively test and presently to derive an initial working scope for CVP, and to recommend what the community should strive to achieve in future releases. -The overall requirements for CVP can be cateorized by the basic cloud +The overall requirements for CVP can be categorized by the basic cloud capabilities representing common operations needed by basic VNFs, and additional requirements for VNFs that go beyond the common cloud capabilities including functional extensions, operational capabilities and additional carrier grade @@ -106,10 +107,10 @@ leverage or improve upon existing test cases in OPNFV projects and upstream projects whenever we can, and bridge the gaps when we must, to meet these basic requirements. -We are not yet ready to support requirements -such as hardware portability, carrier grade performance, fault management and other -operational features, security, MANO and VNF verification. -These areas are being studied for future considerations. +We are not yet ready to include compliance requirements for capabilities such +as hardware portability, carrier grade performance, fault management and other +operational features, security, MANO and VNF verification. These areas are +being studied for consideration in future CVP releases. In some areas, we will start with a limited level of verification initially, constrained by what community resources are able to support at this @@ -127,12 +128,16 @@ area will not have to be run or passed in order to achieve compliance. Optional tests provide an opportunity for vendors to demonstrate compliance with specific OPNFV features beyond the mandatory test scope. + Analysis of Scope ----------------- -Following on these high level objectives and the identified general approach, -we seek to define the initial verification scope by the analysis summarized -in the following categories: +In order to define the scope of the Danube-release of the compliance and +verification program, this section analyzes NFV-focused platform capabilities +with respect to the high-level objectives and the general approach outlined in +the previous section. The analysis determines which capabilities are suitable +for inclusion in this release of the CVP and which capabilities are to be +addressed in future releases. 1. Basic Cloud Capabilities @@ -158,19 +163,19 @@ services. Running such basic VNF leads to a set of common requirements, includin suspend/resume, reboot, migrate) - simple virtual machine resource scheduling on multiple nodes -OPNFV mainly supports Openstack as the VIM up to the Danube release. The +OPNFV mainly supports OpenStack as the VIM up to the Danube release. The VNFs used in the CVP program, and features in scope for the program which are -considered to be basic to all VNFs, require commercial Openstack distributions +considered to be basic to all VNFs, require commercial OpenStack distributions to support a common basic level of cloud capabilities, and to be compliant to a common specification for these capabilities. This requirement significantly -overlaps with Openstack community's Interop working group's goals, but they are not -identical. The CVP runs the Openstack Refstack-Compute test cases to verify +overlaps with OpenStack community's Interop working group's goals, but they are not +identical. The CVP runs the OpenStack Refstack-Compute test cases to verify compliance to the basic common API requirements of cloud management functions and VNF (as a VM) management for OPNFV. Additional NFV specific requirements are added in network data path validation, packet filtering by security group rules and port security, life cycle runtime events of virtual networks, multiple networks in a topology, validation -of VNF's functional state after common lifecylce events including reboot, pause, +of VNF's functional state after common life-cycle events including reboot, pause, suspense, stop/start and cold migration. In addition, the basic requirement also verifies that the SUT can allocate VNF resources based on simple anti-affinity rules. @@ -184,7 +189,7 @@ a common standard to enable portability across OPNFV compliant platforms. NFV has functional requirements beyond the basic common cloud capabilities, esp. in the networking area. Examples like SDNVPN, IPv6, SFC may be considered additional NFV requirements beyond general purpose cloud -computing. These feature requirements expand beyond common Openstack (or other +computing. These feature requirements expand beyond common OpenStack (or other VIM) requirements. OPNFV CVP will incorporate test cases to verify compliance in these areas as they become mature. Because these extensions may impose new API demands, maturity and industry adoption is a prerequisite for @@ -208,7 +213,7 @@ High availability is a common carrier grade requirement. Availability of a platform involves many aspects of the SUT, for example hardware or lower layer system failures or system overloads, and is also highly dependent on configurations. The current OPNFV high availability verification focuses on -Openstack control service failures and resource overloads, and verifies service +OpenStack control service failures and resource overloads, and verifies service continuity when the system encounters such failures or resource overloads, and also verifies the system heals after a failure episode within a reasonable time window. These service HA capabilities are commonly adopted in the industry @@ -230,7 +235,7 @@ The OPNFV testing projects have started testing OPNFV system resiliency in the Danube release that can be used to provide limited coverage in this area. However, this is a relatively new test methodology in OPNFV, additional study -and testing experiences are still needed. We defer the resilency testing to +and testing experiences are still needed. We defer the resiliency testing to future CVP releases. 5. Security @@ -267,104 +272,126 @@ but defer to future study. 7. Use case testing -More complex interactions among multiple VNFs and between VNFs and the cloud -platform can be better exercised through selected more realistic use cases. +Use-case test cases exercise multiple functional capabilities of a platform in +order to realize a larger end-to-end scenario. Such end-to-end use cases do +not necessarily add new API requirements to the SUT per se, but exercise +aspects of the SUT's functional capabilities in more complex ways. For +instance, they allow for verifying the complex interactions among multiple VNFs +and between VNFs and the cloud platform in a more realistic fashion. End-users +consider use-case-level testing as a significant tool in verifying OPNFV +compliance because it validates design patterns and support for the types of +NFVI features that users care about. + +There are a lot of projects in OPNFV developing use cases and sample VNFs, +however most are still in early phase and require further enhancements to +become useful additions to the CVP. Examples such as vIMS, or those which are +not yet available in Danube release, e.g. vCPE, will be valuable additions to +the CVP. These use cases need to be widely accepted, and since they are more +complex, using these VNFs for CVP demands a higher level of community resources +to implement, analyze and document these VNFs. Hence, use case testing is not +ready for CVP at the time of Danube, but can be incorporated in Euphrates or as +a future roadmap area. + +8. Additional capabilities + +In addition to the capabilities analyzed above, there are further system +aspects which are of importance for the CVP. These comprise operational and +management aspects such as platform in-place upgrades and platform operational +insights such as telemetry and logging. Further aspects include API backward +compatibility / micro-versioning, workload migration, multi-site federation and +interoperability with workload automation platforms, e.g. ONAP. Finally, +efficiency aspects such as the hardware and energy footprint of the platform +are worth considering in the CVP. + +OPNFV is addressing these items on different levels of details in different +projects. However, the contributions developed in these projects are not yet +considered widely available in commercial systems in order to include them in +the CVP. Hence, these aspects are left for inclusion in future releases of the +CVP. + + + +Scope of the Danube-release of the CVP +-------------------------------------- + +Summarizing the results of the analysis above, the scope of the Danube-release +of the CVP is as follows: + +- Test Area: Basic cloud capabilities + + - **OpenStack interoperability test cases excluding object storage**\* + - **OPNFV-Functest/vPing, including both user data and ssh** + - *Port security and security groups* + - *VM life-cycle events* + - *VM networking* + - *VM resource scheduling* + - *Forwarding packets in the data path* + +\* The OPNFV CVP utilizes the same set of test cases as the OpenStack +interoperability program *OpenStack Powered Compute*. Passing the OPNFV CVP +does **not** imply that the SUT is certified according to the *OpenStack Powered +Compute* program. *OpenStack Powered Compute* is a trademark of the OpenStack +foundation and the corresponding certification label can only be awarded by the +OpenStack foundation. -End-users see use case level testing as a significant tool in -verifying OPNFV compliance because it validates design patterns and support -for the types of NFVI features that users -care about. There are a lot of projects in OPNFV developing use cases -and sample VNFs, however most are still in early phase and require further -enhancements to become useful additions to the CVP. -Many of these use case test cases do not add new API requirements to the SUT per se, but -exercise aspects of the SUT's functional capabilities in more complex ways. -Other use cases, such as SDNVPN, will require additional API level extension, -and to clearly separate the two, we will categorize the latter as -NFV specific functional requirements and not simply as use cases. -Examples such as vIMS, or those which are not yet available -in Danube release e.g. vCPE, -will be valuable additions to the CVP. These use cases need to -be widely accepted, and since they are more complex, using these VNFs for CVP demands -higher level of community resources to implement, analyze and document these VNFs. +- Test Area: SDNVPN -Use case testing is not ready for CVP at the time of Danube, but can be incorporated -in Euphrates or as a future roadmap area. + - *OPNFV-SDNVPN* -Finally, we take a preliminary look at future roadmap ideas that may be helpful -for the community to plan and pull resources around. -8. Future CVP scope enhancements +- Test Area: IPv6 -Some possible areas of enhancement for the CVP scope in subsequent releases include: + - *OPNFV-IPv6 + (Limited to overlay tests, v6Ping)* - - service assurance, as discussed above - - use case testing, as discussed above - - platform in-place upgrade - - API backward compatibility / micro-versioning - - workload migration - - multisite federation - - platform operational insights, e.g. telemetry, logging - - efficiency, e.g. hardware and energy footprint of the platform - - interoperability with workload automation platforms e.g. ONAP -And enhancements may also be made to the existing test areas as well, -particularly those with limited coverage in this release. - -Summary of Test Scope ---------------------- +- Test Area: High Availability -The above analysis concludes with the following scope summarized below. -These tested areas represent significant advancement in the -direction to meet the CVP's objectives and end-user expectations, and is a -good basis for the initial phase of CVP. + - **OPNFV-Yardstick/HA** + (Limited to service continuity verification of control services) -- Test Area: common cloud capabilities - - Openstack Refstack-compute test cases - Image, Identity, Compute, Network, Storage - - OPNFV-Functest/vPing, including both user data and ssh - - Port security and security groups - - VM lifecycle events - - VM networking - - VM resource scheduling - - Mandatory +[Highlighting: **Mandatory test cases**, *Optional test cases*] -- Test Area: SDNVPN - - OPNFV-SDNVPN - - Optional -- Test Area: IPv6 - - OPNFV-IPv6 - - Limited to overlay tests, v6Ping - - Optional +These tested areas represent significant advancement in the direction to meet +the CVP's objectives and end-user expectations, and is a good basis for the +initial phase of CVP. -- Test Area: High Availability - - OPNFV-HA - - OPNFV-Yardstick - - Limited to service continuity verification on control services - - Mandatory - -- Test Area: Resilency with Load - - OPNFV-Bottlenecks - - OPNFV-Yardstick - - Limited to compute resource load testing - - Optional - -Note 1: While the current scope of compliance includes functional verification -of certain performance-enhancing NFVI features, no performance measurements or -assessment of performance capabilities are included as of this release. - -Note 2: The SUT is limited to NFVI and VIM functions. While testing MANO +Note: The SUT is limited to NFVI and VIM functions. While testing MANO component capabilities is out of scope, certain APIs exposed towards MANO are used by the current OPNFV compliance testing suite. MANO and other operational elements may be part of the test infrastructure; for example used for workload deployment and provisioning. + +Scope considerations for future CVP releases +-------------------------------------------- + +Based on the previous analysis, the following items are outside the scope of +the Danube release of the CV but are being considered for inclusion in future +releases: + + - service assurance + - use case testing + - platform in-place upgrade + - API backward compatibility / micro-versioning + - workload migration + - multi-site federation + - service function chaining + - platform operational insights, e.g. telemetry, logging + - efficiency, e.g. hardware and energy footprint of the platform + - interoperability with workload automation platforms e.g. ONAP + - resilience + - security and vulnerability scanning + - performance measurements + + Criteria for Awarding Compliance ================================ -This section provides guidance on compliance criteia for each test area. The +This section provides guidance on compliance criteria for each test area. The criteria described here are high-level, detailed pass/fail metrics are documented in Dovetail test specifications. diff --git a/dovetail/compliance/debug.yml b/dovetail/compliance/debug.yml index 9f6d2389..cd70c4ec 100644 --- a/dovetail/compliance/debug.yml +++ b/dovetail/compliance/debug.yml @@ -1,13 +1,13 @@ --- # only used for dovetail tool development debug # nfvi.tc001 tc002 running time is shorter, about 3 minutes -# defcore.tc001 about 30 minutes +# osinterop.tc001 about 30 minutes debug: name: debug testcases_list: - dovetail.example.tc002 - - dovetail.defcore.tc001 -# - dovetail.defcore.tc002 + - dovetail.osinterop.tc001 +# - dovetail.osinterop.tc002 # - dovetail.ipv6.tc008 # - dovetail.ipv6.tc009 # - dovetail.ipv6.tc018 diff --git a/dovetail/compliance/proposed_tests.yml b/dovetail/compliance/proposed_tests.yml index 18b8cec9..83b97d53 100644 --- a/dovetail/compliance/proposed_tests.yml +++ b/dovetail/compliance/proposed_tests.yml @@ -2,9 +2,8 @@ proposed_tests: name: proposed_tests testcases_list: - # defcore,defcore.tc001 used for auto-method, defcore.tc002 used for manually method - - dovetail.defcore.tc001 - # - dovetail.defcore.tc002 + # osinterop + - dovetail.osinterop.tc001 # vping - dovetail.vping.tc001 - dovetail.vping.tc002 @@ -55,4 +54,4 @@ proposed_tests: - dovetail.tempest.tc004 - dovetail.tempest.tc005 # resiliency - - dovetail.resiliency.tc001 + # - dovetail.resiliency.tc001 diff --git a/dovetail/conf/dovetail_config.yml b/dovetail/conf/dovetail_config.yml index de849a52..e93bd64a 100644 --- a/dovetail/conf/dovetail_config.yml +++ b/dovetail/conf/dovetail_config.yml @@ -29,7 +29,7 @@ testsuite_supported: - debug # testarea supported, should adjust accordingly testarea_supported: - - defcore + - osinterop - example - ha - ipv6 diff --git a/dovetail/testcase/defcore.tc001.yml b/dovetail/testcase/osinterop.tc001.yml index 19092c09..f84f52e1 100644 --- a/dovetail/testcase/defcore.tc001.yml +++ b/dovetail/testcase/osinterop.tc001.yml @@ -1,7 +1,7 @@ --- -dovetail.defcore.tc001: - name: dovetail.defcore.tc001 - objective: > # Set of DefCore tempest test cases not flagged and required. It only contains OpenStack core (no object storage) +dovetail.osinterop.tc001: + name: dovetail.osinterop.tc001 + objective: > # Set of OpenStack interoperability test cases not flagged and required. It only contains OpenStack core (no object storage) # The approved guidelines (2016.08) are valid for Kilo, Liberty, Mitaka and Newton releases of OpenStack # The list can be generated using the Rest API from RefStack project: # https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false @@ -11,7 +11,7 @@ dovetail.defcore.tc001: pre_condition: - 'cp /home/opnfv/userconfig/pre_config/tempest_conf.yaml /usr/local/lib/python2.7/dist-packages/functest/opnfv_tests/openstack/tempest/custom_tests/tempest_conf.yaml' pre_copy: - src_file: defcore.txt + src_file: osinterop.txt dest_path: /usr/local/lib/python2.7/dist-packages/functest/opnfv_tests/openstack/refstack_client/defcore.txt cmds: - 'functest env prepare' diff --git a/dovetail/testcase/defcore.tc002.yml b/dovetail/testcase/osinterop.tc002.yml index ae7a1fb3..42346a53 100644 --- a/dovetail/testcase/defcore.tc002.yml +++ b/dovetail/testcase/osinterop.tc002.yml @@ -1,7 +1,7 @@ --- -dovetail.defcore.tc002: - name: dovetail.defcore.tc002 - objective: > # Set of DefCore tempest test cases not flagged and required. It only contains OpenStack core (no object storage) +dovetail.osinterop.tc002: + name: dovetail.osinterop.tc002 + objective: > # Set of OpenStack interoperability test cases not flagged and required. It only contains OpenStack core (no object storage) # The approved guidelines (2016.08) are valid for Kilo, Liberty, Mitaka and Newton releases of OpenStack # The list can be generated using the Rest API from RefStack project: # https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false @@ -11,7 +11,7 @@ dovetail.defcore.tc002: pre_condition: - 'echo test for precondition in testcase' cmds: - - 'cd /home/opnfv/repos/refstack-client; source .venv/bin/activate; ./refstack-client test -c /home/opnfv/userconfig/refstack_tempest.conf -v --test-list /home/opnfv/userconfig/defcore.txt' + - 'cd /home/opnfv/repos/refstack-client; source .venv/bin/activate; ./refstack-client test -c /home/opnfv/userconfig/refstack_tempest.conf -v --test-list /home/opnfv/userconfig/osinterop.txt' post_condition: - 'echo test for postcondition in testcase' report: |