Draft one of the test specification.

JIRA: DOVETAIL-352

Needs a lot more work, but the directory tree and layout are according to our
needs in order to have this published as a CVP document.  Will push patches
to restructure the repo according to the new requirements later.

Change-Id: I9d4316407c94bc3096d322f6206eccf236d349da
Signed-off-by: ChristopherPrice <christopher.price@ericsson.com>

1 files changed, 296 insertions, 0 deletions

diff --git a/docs/testing/user/teststrategy/index.rst b/docs/testing/user/teststrategy/index.rst
new file mode 100644
index 00000000..f5f2da80
--- /dev/null
+++ b/docs/testing/user/teststrategy/index.rst
@@ -0,0 +1,296 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Ericsson AB
+
+======================================================
+Compliance and Verification program test specification
+======================================================
+
+.. toctree::
+:maxdepth: 2
+
+Version history
+===============
+
++------------+----------+------------------+----------------------------------+
+| **Date**   | **Ver.** | **Author**       | **Comment**                      |
+|            |          |                  |                                  |
++------------+----------+------------------+----------------------------------+
+| 2017-03-15 | 0.0.1    | Chris Price      | Draft version                    |
+|            |          |                  |                                  |
++------------+----------+------------------+----------------------------------+
+
+
+Introduction
+============
+
+This test specification provides a detailed outline of the prerequisites for performing evaluation
+testing, testing methods and procedures used in the evaluation testing process and the tools provided
+for running OPNFV evaluation testing.
+
+A CVP system under test is assumed to be a stand alone cloud infrastructure running a virtualisation
+software stack providing high availability, redundancy and resiliency.  OPNFV CVP testing covers the
+combination of hardware and software that provide an NFV platform for running virtual workloads, approximately
+the VIM & VNFi as defined by ETSI NFV ISG.
+
+While the NFV platform is a composite system under test comprised of both hardware and software the VIM
+& NFVi testing focuses on software evaluation where it is required and assumed the software is running on
+a platform deemed to be Pharos compliant.  A "Pharos compliant" stand alone hardware system can be summarised
+as a POD of at least 3 control and two compute blades to exercise the minimum set of compliance testing.
+Pharos compliance is further defined, and expected to be implemented according to the `"Pharos specification"`_.
+
+
+-------
+Purpose
+-------
+
+This document is intended to be read by an engineer, intending to run or prepare a system for the evaluation
+tests, prior to beginning the preparation for executing the evaluation tests.  The document is also useful as
+a reference to learn more about OPNFV CVP testing, it's assumptions, targets, methods & tools and expected outcomes.
+
+The engineer will be guided through configuring and populating an environment that is suitable for executing the
+OPNFV compliance evaluation test suite.  This includes interpretations of the Pharos specification and assumptions
+made by the toolchain on how those interpretations are evaluated.
+
+In addition to system preparation the document provides a guide for working with the methods and tools associated
+with the evaluation test procedure.
+
+----------------
+Scope of testing
+----------------
+
+The OPNFV CVP testing suite is implemented to evaluate the compliance of a virtualisation platform with standard
+NFV, carrier and communications network, platform requirements.  Testing focuses on evaluating the software layer
+in the platform in it's ability to provide OPNFV features and behaviours required to host communication and networking
+industry applications.  This section will provide a brief overview of the target areas and scope addressed by the
+testing suites
+
+Features
+--------
+
+CVP testing addresses primarily features and capabilities of value to, and further developed by, the OPNFV community.
+Target areas for CVP testing are to verify the presence and compliance of:
+  * Validation of common cloud environment requirements inherited from key communities such as OpenStack and ETSI
+  * Capabilities required to provide consistent application and workload on-boarding, lifecycle management, and scaling
+  * Networking features required to address a carrier environment including VPN, service chaining, Trunking and peering
+  * Others??
+
+Resilience
+----------
+
+Availability of service and infrastructure are fundamental principals of carrier networking and as such form a
+component of the compliance test suite aimed to establish the ability of a virtualisation infrastructure to identify
+and recover from failures across the system.  The evaluation criteria specifically target control plane resilience
+and the ability to identify, accommodate and recover from hardware failures in the system.
+
+Security?
+---------
+
+https://jira.opnfv.org/browse/DOVETAIL-382
+
+This section should outline the test strategy related to security,
+representing the test cases and requirements on the SUT for security testing.
+
+Scale
+-----
+
+The ability to scale is important for carrier networking infrastructure and applications.  The first iteration of the
+compliance evaluation test suites address the need to scale however do not enforce strict requirements on how scale is
+achieved.
+
+Compliance to the Pharos specification is evaluated as a component of the test suite and provides an visibility into
+the physical infrastructures ability to scale in accordance with OPNFV requirements.  The test suite itself does not
+require and infrastructure that is running to be deployed at scale in order to pass the tests.  It is assumed the
+compliance to Pharos provides enough infrastructure validation that the capability is inherent.
+
+Characteristics
+---------------
+
+The OPNFV community invests heavily in ensuring the features and capabilities of the stack are able to run in the
+most performant manner according to the needs of the workloads.  This can range from the ability to linearly scale
+workloads to the ability to process traffic at line rates.
+
+While each of these is a critical factor in evaluating the performance of a virtualisation infrastructure the CVP
+suite does not at this time specify strict requirements on any given capability as part of the evaluation.  It is
+expected that in future test suites concise performance metrics will be required to be achieved to achieve compliance
+at this time the community has elected not to place pass/fail requirements on characteristics.
+
+
+---------------------
+Definitions and terms
+---------------------
+
+This document uses a number of acronyms and terms the reader may not be familiar with.  For a full glossary of
+terms used in OPNFV, refer to the `OPNFV Glossary`_.
+
++------------+----------------------------------------------------------------+
+| **Term**   | **Description**                                                |
+|            |                                                                |
++------------+----------------------------------------------------------------+
+| CVP        | The OPNFVCompliance and Verification Program                   |
+|            |                                                                |
++------------+----------------------------------------------------------------+
+| SUT        | System under test; the complete system targeted by the         |
+|            | test cases, including software hardware and configuration.     |
+|            |                                                                |
++------------+----------------------------------------------------------------+
+| More       | Additional entries to be added to this table                   |
+|            |                                                                |
++------------+----------------------------------------------------------------+
+
+
+Overview
+========
+
+This section of the document will describe in details the processes and procedures required to perform OPNFV CVP
+compliance testing.  The section is structured to address; planning and preparation, the approach to testing, the
+scope of test activities including details of the test areas, methods and tools used in the testing and the result,
+reporting & output of the test suites when run.
+
+Test planning and preparation
+=============================
+
+https://jira.opnfv.org/browse/DOVETAIL-384
+
+Give an outline of the planning phase.
+
+--------------
+Pre-requisites
+--------------
+
+Describe what needs to be in place before starting.
+
+Required Infrastructure, connectivity needs, LF & CVP accounts and any additional security or peripheral needs.
+Use sub-chapters for instance for any accounts etc that need to be created.
+
+-------------------------------------------
+Preparing the virtualisation infrastructure
+-------------------------------------------
+
+Briefly state what will be tested as an intro, then outline the required system state to be achieved prior to running the tests.
+
+Preparing the test staging host system
+--------------------------------------
+
+https://jira.opnfv.org/browse/DOVETAIL-385
+
+What is required from the system running the CVP test suites?
+
+Required network configuration
+-------------------------------
+
+https://jira.opnfv.org/browse/DOVETAIL-386
+
+VLAN configurations required for NFV deployments.  Non-VLAN configurations if needed.
+
+Preparing your system for CVP testing
+-------------------------------------
+
+https://jira.opnfv.org/browse/DOVETAIL-387
+
+Describe how to realise the "Pharos ready state" and the "Software ready state" in this section.
+
+-------------------
+Pre-test validation
+-------------------
+
+Describe how to evaluate test readiness here.
+I suggest this be a process of doing a "dry run" and evaluating the results on the DB.  This should not
+need to be reproduced later in the document.
+
+Feature testing scope and approach
+==================================
+
+-------------
+Test approach
+-------------
+
+Here we should describe the way we approach testing different areas.  API through RefStack, resilience through
+ETSI test implementations, security is done in xyz way.  This should serve as an introduction to the following
+feature test scope sections and provide common information not to be replicated further down.
+
+
+------------------
+Feature test scope
+------------------
+
+Included test areas
+-------------------
+
+This section should identify all the features areas and combinations of features that are to be tested.
+This should reference sections of test descriptions and test tooling documents to enable cross checking.
+A clear an concise description of the test areas and the purposes and content should be captured here, a
+little like an executive summary of the test case documentation.
+
+Excluded test areas
+-------------------
+
+Describe what is not tested here.  At a similar level to the above, not making excuses just being concise
+as to what is out of scope.
+
+Test criteria and reporting
+---------------------------
+
+This section should specify the criteria to be used to decide whether a test item has passed or failed.
+As each area may have differences it is important to ensure the user can react to a failure in accordance
+with it's relevance to the overall test suites running.
+
+Critical functions that are mandatory should be identified here. If any of these fail the system testing
+should be halted.  If we can having a grading or sorts described here would be helpful as a "guide" for a
+tester to know if they should submit, resolve an item in their stack, or try again at another time.
+
+
+---------------------
+Test design and tools
+---------------------
+
+VIM NBI testing
+---------------
+
+Describe the test areas addressed and how the tools are designed.  It is important to understand the behaviour
+of the testing framework when running these tests.  Here we get into the details of behaviour for each of
+the test frameworks we use, what they are testing and how the results are to be interpreted.
+
+Outline the tool in detail, in this case RefStack.  How does it work, is it run to completion, is reporting
+done per test case when do I as a user know I should do something?
+
+Summarise the tests to be executed by this test framework and the purpose of those tests in the evaluation
+of the CVP.  Are there dependancies between tests, does the tool expect a certain behaviour, do the test
+cases have specific dependancies.  This provides the overall context of the evaluation performed by this
+toolchain / suite and I would not want to be surprised by something when I run the tests after reading this.
+
+Next test area
+--------------
+
+What is the test area, what tools, how do they work what does it mean to me as a tester?
+
+Another test area
+-----------------
+
+Again what is the test area, what tools, how do they work what does it mean to me as a tester?
+
+
+CVP test execution
+==================
+
+This section should identify the test procedure being executed. Include all people and
+roles involved in the execution of a CVP test.  Include procedures, mailing lists, escalation, support
+and periods of waiting for results.
+
+Test reports
+============
+
+Describe the process of producing and accessing the test report.
+
+how do I connect a test suite to my account to get a report?  How do I access the report when it is ready,
+how do I identify one report from another in the toolchain?  We should go into all the details here and point
+to the tool, referring to the "preparation" section of this document if needed for context.
+
+
+References
+==========
+
+.. _`"Pharos specification"`: https://opnfv.org/
+.. _`OPNFV Glossary`: http://docs.opnfv.org/en/latest/glossary/index.html
+
+