diff options
Diffstat (limited to 'docs/testing/user/teststrategy')
| -rw-r--r-- | docs/testing/user/teststrategy/index.rst | 301 |
1 files changed, 0 insertions, 301 deletions
diff --git a/docs/testing/user/teststrategy/index.rst b/docs/testing/user/teststrategy/index.rst deleted file mode 100644 index db05e035..00000000 --- a/docs/testing/user/teststrategy/index.rst +++ /dev/null @@ -1,301 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) Ericsson AB - -====================================================== -Compliance and Verification program test specification -====================================================== - -.. toctree:: - :maxdepth: 2 - -Version history -=============== - -+------------+----------+------------------+----------------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+----------------------------------+ -| 2017-03-15 | 0.0.1 | Chris Price | Draft version | -| | | | | -+------------+----------+------------------+----------------------------------+ - - -Introduction -============ - -This test specification provides a detailed outline of the prerequisites for performing evaluation -testing, testing methods and procedures used in the evaluation testing process and the tools provided -for running OPNFV evaluation testing. - -A CVP system under test is assumed to be a stand alone cloud infrastructure running a virtualisation -software stack providing high availability, redundancy and resiliency. OPNFV CVP testing covers the -combination of hardware and software that provide an NFV platform for running virtual workloads, approximately -the VIM & VNFi as defined by ETSI NFV ISG. - -While the NFV platform is a composite system under test comprised of both hardware and software the VIM -& NFVi testing focuses on software evaluation where it is required and assumed the software is running on -a platform deemed to be Pharos compliant. A "Pharos compliant" stand alone hardware system can be summarised -as a POD of at least 3 control and two compute blades to exercise the minimum set of compliance testing. -Pharos compliance is further defined, and expected to be implemented according to the `"Pharos specification"`_. - - -------- -Purpose -------- - -This document is intended to be read by an engineer, intending to run or prepare a system for the evaluation -tests, prior to beginning the preparation for executing the evaluation tests. The document is also useful as -a reference to learn more about OPNFV CVP testing, it's assumptions, targets, methods & tools and expected outcomes. - -The engineer will be guided through configuring and populating an environment that is suitable for executing the -OPNFV compliance evaluation test suite. This includes interpretations of the Pharos specification and assumptions -made by the toolchain on how those interpretations are evaluated. - -In addition to system preparation the document provides a guide for working with the methods and tools associated -with the evaluation test procedure. - ----------------- -Scope of testing ----------------- - -The OPNFV CVP testing suite is implemented to evaluate the compliance of a virtualisation platform with standard -NFV, carrier and communications network, platform requirements. Testing focuses on evaluating the software layer -in the platform in it's ability to provide OPNFV features and behaviours required to host communication and networking -industry applications. This section will provide a brief overview of the target areas and scope addressed by the -testing suites - -Features --------- - -CVP testing addresses primarily features and capabilities of value to, and further developed by, the OPNFV community. -Target areas for CVP testing are to verify the presence and compliance of: - * Validation of common cloud environment requirements inherited from key communities such as OpenStack and ETSI - * Capabilities required to provide consistent application and workload on-boarding, lifecycle management, and scaling - * Networking features required to address a carrier environment including VPN, service chaining, Trunking and peering - * Others?? - -Resilience ----------- - -Availability of service and infrastructure are fundamental principals of carrier networking and as such form a -component of the compliance test suite aimed to establish the ability of a virtualisation infrastructure to identify -and recover from failures across the system. The evaluation criteria specifically target control plane resilience -and the ability to identify, accommodate and recover from hardware failures in the system. - -Security? ---------- - -https://jira.opnfv.org/browse/DOVETAIL-382 - -This section should outline the test strategy related to security, -representing the test cases and requirements on the SUT for security testing. - -Scale ------ - -The ability to scale is important for carrier networking infrastructure and applications. The first iteration of the -compliance evaluation test suites address the need to scale however do not enforce strict requirements on how scale is -achieved. - -Compliance to the Pharos specification is evaluated as a component of the test suite and provides an visibility into -the physical infrastructures ability to scale in accordance with OPNFV requirements. The test suite itself does not -require an infrastructure that is running to be deployed at scale in order to pass the tests. It is assumed the -compliance to Pharos provides enough infrastructure validation that the capability is inherent. - -Characteristics ---------------- - -The OPNFV community invests heavily in ensuring the features and capabilities of the stack are able to run in the -most performant manner according to the needs of the workloads. This can range from the ability to linearly scale -workloads to the ability to process traffic at line rates. - -While each of these is a critical factor in evaluating the performance of a virtualisation infrastructure the CVP -suite does not at this time specify strict requirements on any given capability as part of the evaluation. It is -expected that in future test suites concise performance metrics will be required to be achieved to achieve compliance -at this time the community has elected not to place pass/fail requirements on characteristics. - - ---------------------- -Definitions and terms ---------------------- - -This document uses a number of acronyms and terms the reader may not be familiar with. For a full glossary of -terms used in OPNFV, refer to the `OPNFV Glossary`_. - -+------------+----------------------------------------------------------------+ -| **Term** | **Description** | -| | | -+------------+----------------------------------------------------------------+ -| CVP | The OPNFVCompliance and Verification Program | -| | | -+------------+----------------------------------------------------------------+ -| SUT | System under test; the complete system targeted by the | -| | test cases, including software hardware and configuration. | -| | | -+------------+----------------------------------------------------------------+ -| More | Additional entries to be added to this table | -| | | -+------------+----------------------------------------------------------------+ - - -Overview -======== - -This section of the document will describe in details the processes and procedures required to perform OPNFV CVP -compliance testing. The section is structured to address; planning and preparation, the approach to testing, the -scope of test activities including details of the test areas, methods and tools used in the testing and the result, -reporting & output of the test suites when run. - -Test planning and preparation -============================= - -https://jira.opnfv.org/browse/DOVETAIL-384 - -Give an outline of the planning phase, refer to the detailed system prep guide and test guides here. - - ../../systempreparation/index.rst - -Feature testing scope and approach -================================== - -------------------- -Pre-test validation -------------------- - -Describe how to evaluate test readiness here. -I suggest this be a process of doing a "dry run" and evaluating the results on the DB. This should not -need to be reproduced later in the document. - - -------------- -Test approach -------------- - -Here we should describe the way we approach testing different areas. API through RefStack, resilience through -ETSI test implementations, security is done in xyz way. This should serve as an introduction to the following -feature test scope sections and provide common information not to be replicated further down. - - ------------------- -Feature test scope ------------------- - -Included test areas -------------------- - -CVP testing for the Danube release focuses on evaluating the ability of a platform to host basic carrier -networking related workloads. The testing focuses on establishing the ability of the SUT to perform -basic NFVi operations such as managing images, instatiating workload & networking these workloads in a -secure and resiliant manner. - -Many OPNFV features are derived from our target upstream communities resulting in the tests focusing -on exposing behaviour present through those development efforts. This approach to OPNFV development -is reflected in our CVP testing where upstream validation procedures are leveraged to validate the -composite platform. The OpenStack `RefStack <https://refstack.openstack.org/#/>`_ test suites are -leveraged in OPNFV for performing VIM validation according to expected behaviour in OPNFV. - -OPNFV CVP testing has explicit requirements on hardware, control plane and compute topologies which -are assumed to be in place, these are covered in the `Preparing the virtualisation infrastructure`_ -section of this document. Tests may fail if the system is not prepared and configured in accordance -the prpeparation guidelines. - -Excluded test areas -------------------- - -The CVP testing procedure in Danube do not cover all aspects of the available OPNFV system, nor feature -suites. To ensure the highest quality of evaluation that provides a trustworthy foundation for industry -compliance toward a common OPNFV standard tests and test areas are selected based on three key principals; -maturity of the test area and framework, availability of features and capabilities across OPNFV compositions, -relevance and value to the industry for evaluation. - -In the Danube release of the CVP we have elected to esbalish an evaluation suite for only the common base -features of the platform. Features areas that are optional or not yet mature in the platform are expluded. -This includes a number of optinal networking features such as BGP VPN networking and Service Chaining which -are intended to be included as optional evaluation areas in future CVP releases. - -The Danube release of the OPNFV CVP testing suite in addition dose not attempt to provide an evaluation criteria -for hardware. Any hardware used in the evaluation testing must comply to the pre-resuities outlined in the -`Preparing the virtualisation infrastructure`_ section of this document. Although the hardware is not tested -itself and no qualification metric has been established for the hardware in the Danube release. - -Test criteria and reporting ---------------------------- - -https://jira.opnfv.org/browse/DOVETAIL-389 - -This section should specify the criteria to be used to decide whether a test item has passed or failed. -As each area may have differences it is important to ensure the user can react to a failure in accordance -with it's relevance to the overall test suites running. - -Critical functions that are mandatory should be identified here. If any of these fail the system testing -should be halted. If we can having a grading or sorts described here would be helpful as a "guide" for a -tester to know if they should submit, resolve an item in their stack, or try again at another time. - - ---------------------- -Test design and tools ---------------------- - -This section needs to be done once we know the tools and areas covered by the CVP testing suites. Parked for now. - -VIM NBI testing ---------------- - -Describe the test areas addressed and how the tools are designed. It is important to understand the behaviour -of the testing framework when running these tests. Here we get into the details of behaviour for each of -the test frameworks we use, what they are testing and how the results are to be interpreted. - -Outline the tool in detail, in this case RefStack. How does it work, is it run to completion, is reporting -done per test case when do I as a user know I should do something? - -Summarise the tests to be executed by this test framework and the purpose of those tests in the evaluation -of the CVP. Are there dependancies between tests, does the tool expect a certain behaviour, do the test -cases have specific dependancies. This provides the overall context of the evaluation performed by this -toolchain / suite and I would not want to be surprised by something when I run the tests after reading this. - -Next test area --------------- - -What is the test area, what tools, how do they work what does it mean to me as a tester? - -Another test area ------------------ - -Again what is the test area, what tools, how do they work what does it mean to me as a tester? - - -CVP test execution -================== - -This section should identify the test procedure being executed. Include all people and -roles involved in the execution of a CVP test. Include procedures, mailing lists, escalation, support -and periods of waiting for results. - -The general workflow I assume should be something like this: - -* Log into the CVP test case staging computer -* Open a Browser and Log into the CVP tool. -* Select the CVP suite to run, agree to the questions (there should be only 1 for now) -* Run the tests (we should be able to launch this from the Web UI if they are on the hosting machine) -* Wait for the tool to report completed. -* Review your results, refer to trouble shooting or support as needed -* Submit your results for CVP evaluation - ------------- -Test reports ------------- - -Describe the process of producing and accessing the test report. This shoudl be a sub-section of CVP test execution I think. - -how do I connect a test suite to my account to get a report? How do I access the report when it is ready, -how do I identify one report from another in the toolchain? We should go into all the details here and point -to the tool, referring to the "preparation" section of this document if needed for context. - - -References -========== - -.. _`"Pharos specification"`: https://opnfv.org/ -.. _`OPNFV Glossary`: http://docs.opnfv.org/en/latest/glossary/index.html - |