Transfer Danube reviewer guidelines from wiki

The reviewer guidelines will be created per release, where the
CVP/Dovetail releases are aligned to OPNFV releases. The guidelines
outlined in this initial document are for the Danube release.

Updated reviewer guidelines

- Updated to account for issue where mandatory total always shows 215
in current version of web portal; removed one diagram and amended another
- First section title change and wording for test area results

- Removed white space
- Added SUT info section and diagrams
- Modified reference to cvp.opnfv.org with verified.opnfv.org
- Modified reference to cvp@opnfv.org with verified@opnfv.org
- Modified diagram filenames to reflect program name change
- Changed CVP references to OVP and Danube to 2018.01
- Remove TODO for granting marks and escalation - this info is in other docs

Change-Id: I01fc568ce6713d3c60daf339a2630e56e8dbcfcb
Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>

16 files changed, 190 insertions, 8 deletions

diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png
new file mode 100644
index 00000000..d5af068b
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png
new file mode 100644
index 00000000..395a2c3e
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png
new file mode 100644
index 00000000..6c363441
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png
new file mode 100644
index 00000000..7e2c6982
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png
new file mode 100644
index 00000000..74c99f37
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png
new file mode 100644
index 00000000..ad79f4ec
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png
new file mode 100644
index 00000000..fa12ddc7
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png
new file mode 100644
index 00000000..a8f988fc
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png
new file mode 100644
index 00000000..4b845df2
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png
new file mode 100644
index 00000000..1c9bcd30
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png
new file mode 100644
index 00000000..280d0c5f
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png
new file mode 100644
index 00000000..01384d02
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png
new file mode 100644
index 00000000..a82e1121
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png
diff --git a/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png
new file mode 100644
index 00000000..f11d531f
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png
diff --git a/docs/testing/user/reviewerguide/danube/images/sut_info.png b/docs/testing/user/reviewerguide/danube/images/sut_info.png
new file mode 100644
index 00000000..395f5f41
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/sut_info.png
diff --git a/docs/testing/user/reviewerguide/index.rst b/docs/testing/user/reviewerguide/index.rst
index fef318e9..bd615f19 100644
--- a/docs/testing/user/reviewerguide/index.rst
+++ b/docs/testing/user/reviewerguide/index.rst
@@ -2,9 +2,9 @@
 .. http://creativecommons.org/licenses/by/4.0
 .. (c) Ericsson AB
 
-==================================================
-Compliance and Verification program reviewer guide
-==================================================
+=============================================
+OPNFV Verified Program 2018.01 Reviewer Guide
+=============================================
 
 .. toctree::
    :maxdepth: 2
@@ -13,12 +13,194 @@ Compliance and Verification program reviewer guide
 Introduction
 ============
 
-This reviewer guide provides detailed guidance for reviewers on how to handle the result review process.
+This reviewer guide provides detailed guidance for reviewers on how to handle the result review
+process. Reviewers must follow the checklist below to ensure review consistency for the OPNFV
+Verified Program (OVP) 2018.01 (Danube) release at a minimum.
 
-TODO: add sections on
+#. **Mandatory Test Area Results** - Validate that results for all mandatory test areas are present.
+#. **Test-Case Count within Mandatory Test Area** - Check that the total number of test-cases are present in each mandatory test area.
+#. **Test-Case Pass Percentage** - Ensure all tests have passed (100% pass rate).
+#. **Log File Verification** - Inspect the log file for each test area (osinterop, ha, vping).
+#. **SUT Info Verification** - Validate the system under test (SUT) hardware and software endpoint info is present.
 
-    - specification of criteria for reviewing the results
 
-    - specification of criteria for granting the certification badge
+1. Mandatory Test Area Results
+==============================
 
-    - specification SLAs for responding to requests, questions, escalations, ...
+Validate that results for all mandatory test areas are included in the overall test suite. The
+required mandatory test areas are:
+ - **osinterop**
+ - **vping**
+ - **ha**
+
+Login to the OVP portal at:
+
+*https://verified.opnfv.org*
+
+Click on the 'My Results' tab in top-level navigation bar.
+
+.. image:: danube/images/ovp_top_nav.png
+    :align: center
+    :scale: 100%
+
+The OVP administrator will ask for review volunteers using the verified@opnfv.org email alias. The
+incoming results for review will be identified by the administrator with particular 'Test ID'
+and 'Owner' values. The corresponding OVP portal result will have a status of 'review'.
+
+.. image:: danube/images/ovp_result_review.png
+    :align: center
+    :scale: 100%
+
+In the example above, this information will be provided as:
+- Test ID: a00c47e8
+- Owner: jtaylor
+
+Click on the hyperlink within the 'Test ID' column.
+
+*Note, that the 'Test ID' column in this view condenses the UUID used for 'Test ID' to
+eight characters even though the 'Test ID' is a longer UUID in the back-end.*
+
+.. image:: danube/images/ovp_result_overview.png
+    :align: center
+    :scale: 100%
+
+The 'Test ID' hyperlink toggles the view to a top-level listing of the results displayed above.
+Validate that osinterop, vping and ha test area results are all present within the view.
+
+
+2. Test-Case Count within Mandatory Test Area
+=============================================
+
+Validate the test-case count within each test area. For the OVP 2018.01 release, this must break
+down as outlined in the table below.
+
+.. image:: danube/images/ovp_test_count.png
+    :align: center
+    :scale: 100%
+
+In the diagram above (from section 1), these counts can be gleaned from the numbers to the
+right of the test-cases. The total number is given for the osinterop (dovetail.osinterop.tc001)
+test area at 205. The vping (dovetail.vping.tc00x) and ha (dovetail.ha.tc00x) test-cases are
+broken down separately with a line for each test-case. Directly above the 'Test Result Overview'
+listing there's a summary labelled 'Test Run Results' shown below. For OVP 2018.01, a mandatory
+total of **215** test-cases must be present (205 osinterop + 8 ha + 2 vping).
+
+.. image:: danube/images/ovp_missing_ha.png
+    :align: center
+    :scale: 100%
+
+An example of a listing that should flag a negative review is shown above. The mandatory ha test
+area is missing one test case (dovetail.ha.tc008).
+
+3. Test-Case Pass Percentage
+============================
+
+All mandatory test-cases must pass. This can be validated in multiple ways. The below diagram of
+the 'Test Run Results' is one method and shows that 100% of the mandatory test-cases have passed.
+This value must not be lower than 100%.
+
+.. image:: danube/images/ovp_pass_percentage.png
+    :align: center
+    :width: 350 px
+
+Another method to check that all mandatory test-cases have passed is shown in the diagram below.
+The pass/total is given as a fraction and highlighted here in yellow. For the osinterop test area,
+the result must display [205/205] and for each of the test-cases under the vping and ha test areas
+[1/1] must be displayed.
+
+.. image:: danube/images/ovp_pass_fraction.png
+    :align: center
+    :width: 270 px
+
+4. Log File Verification
+========================
+
+Three log files must be verified for content within each mandatory test area. The log files for
+each of the test areas is noted in the table below.
+
+.. image:: danube/images/ovp_log_files.png
+    :align: center
+    :scale: 100%
+
+The three log files can be displayed by clicking on the setup icon to the right of the results,
+as shown in the diagram below.
+
+*Note, while the vping and ha test areas list multiple test-cases in the below diagram, there is
+a single log file for all test-cases within these test areas.*
+
+.. image:: danube/images/ovp_log_setup.png
+    :align: center
+    :scale: 100%
+
+Within the osinterop log (dovetail.osinterop.tc001.log), scroll down to the area of the log that
+begins to list the results of each test-case executed. This can be located by looking for lines
+prefaced with '**tempest.api**' and ending with '**... ok**'.
+
+.. image:: danube/images/ovp_log_test_count.png
+    :align: center
+    :scale: 100%
+
+The number of lines within the osinterop log for test-cases must add up according to the table
+above, where test-cases are broken down according to compute, identity, image, network and volume,
+with respective counts given in the table. The ha log (yardstick.log) must contain the 'PASS'
+result for each of the eight test-cases within this test area. This can be verified by searching
+the log for the keyword 'PASS'.
+
+
+
+The eight lines to validate are listed below:
+
+ - 017-10-16 05:07:49,158 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+   INFO The HA test case PASS the SLA
+ - 2017-10-16 05:08:31,387 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+   INFO The HA test case PASS the SLA
+ - 2017-10-16 05:09:13,669 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+   INFO The HA test case PASS the SLA
+ - 2017-10-16 05:09:55,967 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+   INFO The HA test case PASS the SLA
+ - 2017-10-16 05:10:38,407 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+   INFO The HA test case PASS the SLA
+ - 2017-10-16 05:11:00,030 yardstick.benchmark.scenarios.availability.scenario_general
+   scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+ - 2017-10-16 05:11:22,536 yardstick.benchmark.scenarios.availability.scenario_general
+   scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+ - 2017-10-16 05:12:07,880 yardstick.benchmark.scenarios.availability.scenario_general
+   scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+
+
+The final validation is for the vping test area log file (functest.log). The two entries
+displayed in the diagrams below must be present in this log file.
+
+ - vping_userdata
+ - vping_ssh
+
+.. image:: danube/images/ovp_vping_user.png
+    :align: center
+    :scale: 100%
+
+.. image:: danube/images/ovp_vping_ssh.png
+    :align: center
+    :scale: 100%
+
+5. SUT Info Verification
+========================
+
+SUT information must be present in the results to validate that all required endpoint services
+and at least two controllers were present during test execution. For the results shown below,
+click the '**info**' hyperlink in the **SUT** column to navigate to the SUT information page.
+
+.. image:: danube/images/sut_info.png
+    :align: center
+    :scale: 100%
+
+In the '**Endpoints**' listing shown below for the SUT VIM component, ensure that services are
+present for identify, compute, image, volume and network at a minimum by inspecting the
+'**Service Type**' column.
+
+.. image:: danube/images/sut_endpoints.png
+    :align: center
+    :scale: 100%
+
+Inspect the '**Hosts**' listing found below the Endpoints secion of the SUT info page and ensure
+at least two hosts are present, as two controllers are required the for the mandatory HA
+test-cases.