Updating test case requirements document

This patchset addresses review comments which were not incorporated in
the previous version of the test case requirements description.
Additionally, the text has been wrapped to 80 character lines.

Change-Id: If817f6c44d94870d915791a1a11916ffe4871f06
Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>

1 files changed, 87 insertions, 47 deletions

diff --git a/docs/testing/developer/testcaserequirements/index.rst b/docs/testing/developer/testcaserequirements/index.rst
index 912f19da..ecb56a5e 100644
--- a/docs/testing/developer/testcaserequirements/index.rst
+++ b/docs/testing/developer/testcaserequirements/index.rst
@@ -9,71 +9,96 @@ Compliance Verification Program test case requirements
 .. toctree::
    :maxdepth: 2
 
+
 CVP Test Suite Purpose and Goals
 ================================
 
-The CVP test suite is intended to provide a method for validating the interfaces
-and behaviors of an NFVi platform according to the expected capabilities exposed in
-OPNFV.  The behavioral foundation evaluated in these tests should serve to provide
-a functional baseline for VNF deployment and portability across NFVi instances.
-All CVP tests are available in open source and are executed in open source test frameworks.
+The CVP test suite is intended to provide a method for validating the
+interfaces and behaviors of an NFVi platform according to the expected
+capabilities exposed in OPNFV.  The behavioral foundation evaluated in these
+tests should serve to provide a functional baseline for VNF deployment and
+portability across NFVi instances.  All CVP tests are available in open source
+and are executed in open source test frameworks.
+
 
 Test case requirements
 ======================
 
-The following requirements are mandatory for a test to be submitted for consideration in the
- CVP test suite:
+The following requirements are mandatory for a test to be submitted for
+consideration in the CVP test suite:
+
+- All test cases must be fully documented, in a common format.  Please consider
+  the existing :ref:`dovetail_testspecifications` as examples.
+
+  - Clearly identifying the test procedure and expected results / metrics to
+    determine a “pass” or “fail” result.
+
+- Tests must be validated for the purpose of CVP, tests should be run with both
+  an expected positive and negative outcome.
+
+- At the current stage of CVP, only functional tests are eligible, performance
+  testing is out of scope.
+
+  - Performance test output could be built in as “for information only”, but
+    must not carry pass/fail metrics.
+
+- Test cases should favor implementation of a published standard interface for
+  validation.
+
+  - Where no standard is available provide API support references.
 
-- All test cases must be fully documented, in a common format (please refer to the test
-  specification directory for examples).
+  - If a standard exists and is not followed, an exemption is required. Such
+    exemptions can be raised in the project meetings first, and if no consensus
+    can be reached, escalated to the TSC.
 
-  - Clearly identifying the test procedure and expected results / metrics to determine a “pass”
-    or “fail” result.
+- Test cases must pass on applicable OPNFV reference deployments and release
+  versions.
 
-- Tests must be validated for the purpose of CVP, tests should be run with both an expected
-  positive and negative outcome.
-- At the current stage of CVP, only functional tests are eligible, performance testing
-  is out of scope.
+  - Tests must not require a specific NFVi platform composition or installation
+    tool.
 
-  - Performance test output could be built in as “for information only”, but must not carry
-    pass/fail metrics.
+    - Tests and test tools must run independently of the method of platform
+      installation and architecture.
 
-- Test cases should favor implementation of a published standard interface for validation
+    - Tests and test tools must run independently of specific OPNFV components
+      allowing different components such as storage backends or SDN
+      controllers.
 
-  - Where no standard is available provide API support references
-  - If a standard exists and is not followed, an exemption is required. Such exemptions
-    can be raised in the project meetings first, and if no consensus can be reached, escalated
-    to the TSC.
+  - Tests must not require un-merged patches to the relevant upstream projects.
 
-- Test cases must pass on applicable OPNFV reference deployments and release versions.
+  - Tests must not require features or code which are out of scope for the
+    latest release of the OPNFV project.
 
-  - Tests must not require a specific NFVi platform composition or installation tool
+  - Tests must have a documented history of recent successful verification in
+    OPNFV testing programs including CI, Functest, Yardstick, Bottlenecks,
+    Dovetail, etc. (i.e., all testing programs in OPNFV that regularly validate
+    tests against the release, whether automated or manual).
 
-    - Tests and test tools must run independently of the method of platform installation
-      and architecture.
-    - Tests and tool must run independent of specific OPNFV components allowing different
-      components such as storage backends or SDN controllers.
+  - Tests must be considered optional unless they have a documented history for
+    ALL OPNFV scenarios that are both
+
+    - applicable, i.e., support the feature that the test exercises, and
+
+    - released, i.e., in the OPNFV release supported by the CVP test suite
+      version.
 
-  - Tests must not require un-merged patches to the relevant upstream projects
-  - Tests must not require features or code which are out of scope for the latest release of
-    the OPNFV project
-  - Tests must have a documented history of recent successful verification in OPNFV testing
-    programs including CI, Functest, Yardstick, Bottlenecks, Dovetail, etc (i.e. all testing
-    programs in OPNFV that regularly validate tests against the release, whether automated or manual)
-  - Tests must be considered optional unless they such have a documented history for ALL OPNFV
-    scenarios that are both
-    - applicable, i.e. support the feature that the test exercises
-    - released, i.e. in the OPNFV release supported by the CVP test suite version
 - Tests must run against a fully deployed and operational system under test.
-- Tests and test implementations must support stand alone OPNFV and commercial derived OPNFV based solution
+
+- Tests and test implementations must support stand alone OPNFV and commercial
+  OPNFV-derived solutions.
 
   - There can be no dependency on OPNFV resources or infrastructure.
 
+  - Tests must not require external resources while a test is running, e.g.,
+    connectivity to the Internet. All resources required to run a test, e.g.,
+    VM and container images, are downloaded and installed as part of the system
+    preparation and test tool installation.
+
 - The following things must be documented for the test case:
 
   - Use case specification
   - Test preconditions
-  - Basic test flow execution description
+  - Basic test flow execution description and test assertions
   - Pass fail criteria
 
 - The following things may be documented for the test case:
@@ -82,17 +107,32 @@ The following requirements are mandatory for a test to be submitted for consider
   - Fault/Error test case descriptions
   - Post conditions where the system state may be left changed after completion
 
-New test case proposals should complete a CVP test case worksheet to ensure that all
-of these considerations are met before the test case is approved for inclusion in the
-CVP test suite.
+New test case proposals should complete a CVP test case worksheet to ensure
+that all of these considerations are met before the test case is approved for
+inclusion in the CVP test suite.
+
 
 Dovetail Test Suite Naming Convention
 =====================================
 
-Test case naming and structure for dovetail needs to be done, it should consider:
+Test case naming and structuring must comply with the following conventions.
+The fully qualified name of a test case must comprise three sections:
+
+`<community>.<test_area>.<test_case_name>`
+
+- **community**: The fully qualified test case name must identify the community
+  or upstream project which developed and maintains the test case. For test
+  cases originating in OPNFV projects, the community identifier is 'opnfv'.
+  Test cases consumed from the OpenStack tempest test suite, are named
+  'tempest', for example.
+
+- **test_area**: The fully qualified test case name must identify the test case
+  area.  For test cases originating in OPNFV projects, the test case area must
+  identify the project name.
 
-- Identifying the test area
-- Identifying the purpose of the test case
+- **test_case_name**: The fully qualified test case name must include a concise
+  description of the purpose of the test case.
 
-To be discussed, agreed and amended to this document.
+An example of a fully qualified test case name is
+`opnfv.sdnvpn.router_association_floating_ip`.