summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/teststrategy
diff options
context:
space:
mode:
authorGeorg Kunz <georg.kunz@ericsson.com>2017-12-22 14:48:10 +0100
committerGeorg Kunz <georg.kunz@ericsson.com>2018-01-15 15:16:22 +0000
commit6c99abc7aa0d101388bb904a231ceccda6ace4d9 (patch)
tree55e3bf97c8412b7f725bf91e8a9012da34ea2410 /docs/testing/user/teststrategy
parentcd28f48393e20ae88ce0225cf45099dafe82b873 (diff)
Changing CVP to OVP in the documentation
Due to the name change from 'CVP' to 'OVP', the documentation has to be updated. Change-Id: Id934e8fc07f60802f7aa6b3d295d268c4c9b71f8 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com> (cherry picked from commit 8f16dae9dae6d13bc6f5934bae5d26d420efd892)
Diffstat (limited to 'docs/testing/user/teststrategy')
-rw-r--r--docs/testing/user/teststrategy/index.rst301
1 files changed, 0 insertions, 301 deletions
diff --git a/docs/testing/user/teststrategy/index.rst b/docs/testing/user/teststrategy/index.rst
deleted file mode 100644
index db05e035..00000000
--- a/docs/testing/user/teststrategy/index.rst
+++ /dev/null
@@ -1,301 +0,0 @@
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) Ericsson AB
-
-======================================================
-Compliance and Verification program test specification
-======================================================
-
-.. toctree::
- :maxdepth: 2
-
-Version history
-===============
-
-+------------+----------+------------------+----------------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+----------------------------------+
-| 2017-03-15 | 0.0.1 | Chris Price | Draft version |
-| | | | |
-+------------+----------+------------------+----------------------------------+
-
-
-Introduction
-============
-
-This test specification provides a detailed outline of the prerequisites for performing evaluation
-testing, testing methods and procedures used in the evaluation testing process and the tools provided
-for running OPNFV evaluation testing.
-
-A CVP system under test is assumed to be a stand alone cloud infrastructure running a virtualisation
-software stack providing high availability, redundancy and resiliency. OPNFV CVP testing covers the
-combination of hardware and software that provide an NFV platform for running virtual workloads, approximately
-the VIM & VNFi as defined by ETSI NFV ISG.
-
-While the NFV platform is a composite system under test comprised of both hardware and software the VIM
-& NFVi testing focuses on software evaluation where it is required and assumed the software is running on
-a platform deemed to be Pharos compliant. A "Pharos compliant" stand alone hardware system can be summarised
-as a POD of at least 3 control and two compute blades to exercise the minimum set of compliance testing.
-Pharos compliance is further defined, and expected to be implemented according to the `"Pharos specification"`_.
-
-
--------
-Purpose
--------
-
-This document is intended to be read by an engineer, intending to run or prepare a system for the evaluation
-tests, prior to beginning the preparation for executing the evaluation tests. The document is also useful as
-a reference to learn more about OPNFV CVP testing, it's assumptions, targets, methods & tools and expected outcomes.
-
-The engineer will be guided through configuring and populating an environment that is suitable for executing the
-OPNFV compliance evaluation test suite. This includes interpretations of the Pharos specification and assumptions
-made by the toolchain on how those interpretations are evaluated.
-
-In addition to system preparation the document provides a guide for working with the methods and tools associated
-with the evaluation test procedure.
-
-----------------
-Scope of testing
-----------------
-
-The OPNFV CVP testing suite is implemented to evaluate the compliance of a virtualisation platform with standard
-NFV, carrier and communications network, platform requirements. Testing focuses on evaluating the software layer
-in the platform in it's ability to provide OPNFV features and behaviours required to host communication and networking
-industry applications. This section will provide a brief overview of the target areas and scope addressed by the
-testing suites
-
-Features
---------
-
-CVP testing addresses primarily features and capabilities of value to, and further developed by, the OPNFV community.
-Target areas for CVP testing are to verify the presence and compliance of:
- * Validation of common cloud environment requirements inherited from key communities such as OpenStack and ETSI
- * Capabilities required to provide consistent application and workload on-boarding, lifecycle management, and scaling
- * Networking features required to address a carrier environment including VPN, service chaining, Trunking and peering
- * Others??
-
-Resilience
-----------
-
-Availability of service and infrastructure are fundamental principals of carrier networking and as such form a
-component of the compliance test suite aimed to establish the ability of a virtualisation infrastructure to identify
-and recover from failures across the system. The evaluation criteria specifically target control plane resilience
-and the ability to identify, accommodate and recover from hardware failures in the system.
-
-Security?
----------
-
-https://jira.opnfv.org/browse/DOVETAIL-382
-
-This section should outline the test strategy related to security,
-representing the test cases and requirements on the SUT for security testing.
-
-Scale
------
-
-The ability to scale is important for carrier networking infrastructure and applications. The first iteration of the
-compliance evaluation test suites address the need to scale however do not enforce strict requirements on how scale is
-achieved.
-
-Compliance to the Pharos specification is evaluated as a component of the test suite and provides an visibility into
-the physical infrastructures ability to scale in accordance with OPNFV requirements. The test suite itself does not
-require an infrastructure that is running to be deployed at scale in order to pass the tests. It is assumed the
-compliance to Pharos provides enough infrastructure validation that the capability is inherent.
-
-Characteristics
----------------
-
-The OPNFV community invests heavily in ensuring the features and capabilities of the stack are able to run in the
-most performant manner according to the needs of the workloads. This can range from the ability to linearly scale
-workloads to the ability to process traffic at line rates.
-
-While each of these is a critical factor in evaluating the performance of a virtualisation infrastructure the CVP
-suite does not at this time specify strict requirements on any given capability as part of the evaluation. It is
-expected that in future test suites concise performance metrics will be required to be achieved to achieve compliance
-at this time the community has elected not to place pass/fail requirements on characteristics.
-
-
----------------------
-Definitions and terms
----------------------
-
-This document uses a number of acronyms and terms the reader may not be familiar with. For a full glossary of
-terms used in OPNFV, refer to the `OPNFV Glossary`_.
-
-+------------+----------------------------------------------------------------+
-| **Term** | **Description** |
-| | |
-+------------+----------------------------------------------------------------+
-| CVP | The OPNFVCompliance and Verification Program |
-| | |
-+------------+----------------------------------------------------------------+
-| SUT | System under test; the complete system targeted by the |
-| | test cases, including software hardware and configuration. |
-| | |
-+------------+----------------------------------------------------------------+
-| More | Additional entries to be added to this table |
-| | |
-+------------+----------------------------------------------------------------+
-
-
-Overview
-========
-
-This section of the document will describe in details the processes and procedures required to perform OPNFV CVP
-compliance testing. The section is structured to address; planning and preparation, the approach to testing, the
-scope of test activities including details of the test areas, methods and tools used in the testing and the result,
-reporting & output of the test suites when run.
-
-Test planning and preparation
-=============================
-
-https://jira.opnfv.org/browse/DOVETAIL-384
-
-Give an outline of the planning phase, refer to the detailed system prep guide and test guides here.
-
- ../../systempreparation/index.rst
-
-Feature testing scope and approach
-==================================
-
--------------------
-Pre-test validation
--------------------
-
-Describe how to evaluate test readiness here.
-I suggest this be a process of doing a "dry run" and evaluating the results on the DB. This should not
-need to be reproduced later in the document.
-
-
--------------
-Test approach
--------------
-
-Here we should describe the way we approach testing different areas. API through RefStack, resilience through
-ETSI test implementations, security is done in xyz way. This should serve as an introduction to the following
-feature test scope sections and provide common information not to be replicated further down.
-
-
-------------------
-Feature test scope
-------------------
-
-Included test areas
--------------------
-
-CVP testing for the Danube release focuses on evaluating the ability of a platform to host basic carrier
-networking related workloads. The testing focuses on establishing the ability of the SUT to perform
-basic NFVi operations such as managing images, instatiating workload & networking these workloads in a
-secure and resiliant manner.
-
-Many OPNFV features are derived from our target upstream communities resulting in the tests focusing
-on exposing behaviour present through those development efforts. This approach to OPNFV development
-is reflected in our CVP testing where upstream validation procedures are leveraged to validate the
-composite platform. The OpenStack `RefStack <https://refstack.openstack.org/#/>`_ test suites are
-leveraged in OPNFV for performing VIM validation according to expected behaviour in OPNFV.
-
-OPNFV CVP testing has explicit requirements on hardware, control plane and compute topologies which
-are assumed to be in place, these are covered in the `Preparing the virtualisation infrastructure`_
-section of this document. Tests may fail if the system is not prepared and configured in accordance
-the prpeparation guidelines.
-
-Excluded test areas
--------------------
-
-The CVP testing procedure in Danube do not cover all aspects of the available OPNFV system, nor feature
-suites. To ensure the highest quality of evaluation that provides a trustworthy foundation for industry
-compliance toward a common OPNFV standard tests and test areas are selected based on three key principals;
-maturity of the test area and framework, availability of features and capabilities across OPNFV compositions,
-relevance and value to the industry for evaluation.
-
-In the Danube release of the CVP we have elected to esbalish an evaluation suite for only the common base
-features of the platform. Features areas that are optional or not yet mature in the platform are expluded.
-This includes a number of optinal networking features such as BGP VPN networking and Service Chaining which
-are intended to be included as optional evaluation areas in future CVP releases.
-
-The Danube release of the OPNFV CVP testing suite in addition dose not attempt to provide an evaluation criteria
-for hardware. Any hardware used in the evaluation testing must comply to the pre-resuities outlined in the
-`Preparing the virtualisation infrastructure`_ section of this document. Although the hardware is not tested
-itself and no qualification metric has been established for the hardware in the Danube release.
-
-Test criteria and reporting
----------------------------
-
-https://jira.opnfv.org/browse/DOVETAIL-389
-
-This section should specify the criteria to be used to decide whether a test item has passed or failed.
-As each area may have differences it is important to ensure the user can react to a failure in accordance
-with it's relevance to the overall test suites running.
-
-Critical functions that are mandatory should be identified here. If any of these fail the system testing
-should be halted. If we can having a grading or sorts described here would be helpful as a "guide" for a
-tester to know if they should submit, resolve an item in their stack, or try again at another time.
-
-
----------------------
-Test design and tools
----------------------
-
-This section needs to be done once we know the tools and areas covered by the CVP testing suites. Parked for now.
-
-VIM NBI testing
----------------
-
-Describe the test areas addressed and how the tools are designed. It is important to understand the behaviour
-of the testing framework when running these tests. Here we get into the details of behaviour for each of
-the test frameworks we use, what they are testing and how the results are to be interpreted.
-
-Outline the tool in detail, in this case RefStack. How does it work, is it run to completion, is reporting
-done per test case when do I as a user know I should do something?
-
-Summarise the tests to be executed by this test framework and the purpose of those tests in the evaluation
-of the CVP. Are there dependancies between tests, does the tool expect a certain behaviour, do the test
-cases have specific dependancies. This provides the overall context of the evaluation performed by this
-toolchain / suite and I would not want to be surprised by something when I run the tests after reading this.
-
-Next test area
---------------
-
-What is the test area, what tools, how do they work what does it mean to me as a tester?
-
-Another test area
------------------
-
-Again what is the test area, what tools, how do they work what does it mean to me as a tester?
-
-
-CVP test execution
-==================
-
-This section should identify the test procedure being executed. Include all people and
-roles involved in the execution of a CVP test. Include procedures, mailing lists, escalation, support
-and periods of waiting for results.
-
-The general workflow I assume should be something like this:
-
-* Log into the CVP test case staging computer
-* Open a Browser and Log into the CVP tool.
-* Select the CVP suite to run, agree to the questions (there should be only 1 for now)
-* Run the tests (we should be able to launch this from the Web UI if they are on the hosting machine)
-* Wait for the tool to report completed.
-* Review your results, refer to trouble shooting or support as needed
-* Submit your results for CVP evaluation
-
-------------
-Test reports
-------------
-
-Describe the process of producing and accessing the test report. This shoudl be a sub-section of CVP test execution I think.
-
-how do I connect a test suite to my account to get a report? How do I access the report when it is ready,
-how do I identify one report from another in the toolchain? We should go into all the details here and point
-to the tool, referring to the "preparation" section of this document if needed for context.
-
-
-References
-==========
-
-.. _`"Pharos specification"`: https://opnfv.org/
-.. _`OPNFV Glossary`: http://docs.opnfv.org/en/latest/glossary/index.html
-