summaryrefslogtreecommitdiffstats
path: root/docs/testing/user/reviewerguide
diff options
context:
space:
mode:
authorEddie Arrage <eddie.arrage@huawei.com>2017-11-12 13:38:11 -0800
committerGeorg Kunz <georg.kunz@ericsson.com>2018-01-18 16:20:19 +0000
commitcac419a417f25a1c9e0de653e09d2f0ad32e4c42 (patch)
treefbe7725beb635f0621c0c59ee30bc679f7518c09 /docs/testing/user/reviewerguide
parent8f16dae9dae6d13bc6f5934bae5d26d420efd892 (diff)
Transfer Danube reviewer guidelines from wiki
The reviewer guidelines will be created per release, where the CVP/Dovetail releases are aligned to OPNFV releases. The guidelines outlined in this initial document are for the Danube release. Updated reviewer guidelines - Updated to account for issue where mandatory total always shows 215 in current version of web portal; removed one diagram and amended another - First section title change and wording for test area results - Removed white space - Added SUT info section and diagrams - Modified reference to cvp.opnfv.org with verified.opnfv.org - Modified reference to cvp@opnfv.org with verified@opnfv.org - Modified diagram filenames to reflect program name change - Changed CVP references to OVP and Danube to 2018.01 - Remove TODO for granting marks and escalation - this info is in other docs Change-Id: I01fc568ce6713d3c60daf339a2630e56e8dbcfcb Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
Diffstat (limited to 'docs/testing/user/reviewerguide')
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_log_files.pngbin0 -> 6504 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_log_setup.pngbin0 -> 37346 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.pngbin0 -> 6930 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.pngbin0 -> 80192 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.pngbin0 -> 117383 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.pngbin0 -> 75765 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_result_overview.pngbin0 -> 105270 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_result_review.pngbin0 -> 44980 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_run_results.pngbin0 -> 13149 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_test_count.pngbin0 -> 4012 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_top_nav.pngbin0 -> 23642 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.pngbin0 -> 5246 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/ovp_vping_user.pngbin0 -> 5667 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/sut_endpoints.pngbin0 -> 119897 bytes
-rw-r--r--docs/testing/user/reviewerguide/danube/images/sut_info.pngbin0 -> 29227 bytes
-rw-r--r--docs/testing/user/reviewerguide/index.rst198
16 files changed, 190 insertions, 8 deletions
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png
new file mode 100644
index 00000000..d5af068b
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png
new file mode 100644
index 00000000..395a2c3e
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png
new file mode 100644
index 00000000..6c363441
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png
new file mode 100644
index 00000000..7e2c6982
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png
new file mode 100644
index 00000000..74c99f37
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png
new file mode 100644
index 00000000..ad79f4ec
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png
new file mode 100644
index 00000000..fa12ddc7
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png
new file mode 100644
index 00000000..a8f988fc
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png
new file mode 100644
index 00000000..4b845df2
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png
new file mode 100644
index 00000000..1c9bcd30
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png
new file mode 100644
index 00000000..280d0c5f
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png
new file mode 100644
index 00000000..01384d02
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png
new file mode 100644
index 00000000..a82e1121
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png
new file mode 100644
index 00000000..f11d531f
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/danube/images/sut_info.png b/docs/testing/user/reviewerguide/danube/images/sut_info.png
new file mode 100644
index 00000000..395f5f41
--- /dev/null
+++ b/docs/testing/user/reviewerguide/danube/images/sut_info.png
Binary files differ
diff --git a/docs/testing/user/reviewerguide/index.rst b/docs/testing/user/reviewerguide/index.rst
index fef318e9..bd615f19 100644
--- a/docs/testing/user/reviewerguide/index.rst
+++ b/docs/testing/user/reviewerguide/index.rst
@@ -2,9 +2,9 @@
.. http://creativecommons.org/licenses/by/4.0
.. (c) Ericsson AB
-==================================================
-Compliance and Verification program reviewer guide
-==================================================
+=============================================
+OPNFV Verified Program 2018.01 Reviewer Guide
+=============================================
.. toctree::
:maxdepth: 2
@@ -13,12 +13,194 @@ Compliance and Verification program reviewer guide
Introduction
============
-This reviewer guide provides detailed guidance for reviewers on how to handle the result review process.
+This reviewer guide provides detailed guidance for reviewers on how to handle the result review
+process. Reviewers must follow the checklist below to ensure review consistency for the OPNFV
+Verified Program (OVP) 2018.01 (Danube) release at a minimum.
-TODO: add sections on
+#. **Mandatory Test Area Results** - Validate that results for all mandatory test areas are present.
+#. **Test-Case Count within Mandatory Test Area** - Check that the total number of test-cases are present in each mandatory test area.
+#. **Test-Case Pass Percentage** - Ensure all tests have passed (100% pass rate).
+#. **Log File Verification** - Inspect the log file for each test area (osinterop, ha, vping).
+#. **SUT Info Verification** - Validate the system under test (SUT) hardware and software endpoint info is present.
- - specification of criteria for reviewing the results
- - specification of criteria for granting the certification badge
+1. Mandatory Test Area Results
+==============================
- - specification SLAs for responding to requests, questions, escalations, ...
+Validate that results for all mandatory test areas are included in the overall test suite. The
+required mandatory test areas are:
+ - **osinterop**
+ - **vping**
+ - **ha**
+
+Login to the OVP portal at:
+
+*https://verified.opnfv.org*
+
+Click on the 'My Results' tab in top-level navigation bar.
+
+.. image:: danube/images/ovp_top_nav.png
+ :align: center
+ :scale: 100%
+
+The OVP administrator will ask for review volunteers using the verified@opnfv.org email alias. The
+incoming results for review will be identified by the administrator with particular 'Test ID'
+and 'Owner' values. The corresponding OVP portal result will have a status of 'review'.
+
+.. image:: danube/images/ovp_result_review.png
+ :align: center
+ :scale: 100%
+
+In the example above, this information will be provided as:
+- Test ID: a00c47e8
+- Owner: jtaylor
+
+Click on the hyperlink within the 'Test ID' column.
+
+*Note, that the 'Test ID' column in this view condenses the UUID used for 'Test ID' to
+eight characters even though the 'Test ID' is a longer UUID in the back-end.*
+
+.. image:: danube/images/ovp_result_overview.png
+ :align: center
+ :scale: 100%
+
+The 'Test ID' hyperlink toggles the view to a top-level listing of the results displayed above.
+Validate that osinterop, vping and ha test area results are all present within the view.
+
+
+2. Test-Case Count within Mandatory Test Area
+=============================================
+
+Validate the test-case count within each test area. For the OVP 2018.01 release, this must break
+down as outlined in the table below.
+
+.. image:: danube/images/ovp_test_count.png
+ :align: center
+ :scale: 100%
+
+In the diagram above (from section 1), these counts can be gleaned from the numbers to the
+right of the test-cases. The total number is given for the osinterop (dovetail.osinterop.tc001)
+test area at 205. The vping (dovetail.vping.tc00x) and ha (dovetail.ha.tc00x) test-cases are
+broken down separately with a line for each test-case. Directly above the 'Test Result Overview'
+listing there's a summary labelled 'Test Run Results' shown below. For OVP 2018.01, a mandatory
+total of **215** test-cases must be present (205 osinterop + 8 ha + 2 vping).
+
+.. image:: danube/images/ovp_missing_ha.png
+ :align: center
+ :scale: 100%
+
+An example of a listing that should flag a negative review is shown above. The mandatory ha test
+area is missing one test case (dovetail.ha.tc008).
+
+3. Test-Case Pass Percentage
+============================
+
+All mandatory test-cases must pass. This can be validated in multiple ways. The below diagram of
+the 'Test Run Results' is one method and shows that 100% of the mandatory test-cases have passed.
+This value must not be lower than 100%.
+
+.. image:: danube/images/ovp_pass_percentage.png
+ :align: center
+ :width: 350 px
+
+Another method to check that all mandatory test-cases have passed is shown in the diagram below.
+The pass/total is given as a fraction and highlighted here in yellow. For the osinterop test area,
+the result must display [205/205] and for each of the test-cases under the vping and ha test areas
+[1/1] must be displayed.
+
+.. image:: danube/images/ovp_pass_fraction.png
+ :align: center
+ :width: 270 px
+
+4. Log File Verification
+========================
+
+Three log files must be verified for content within each mandatory test area. The log files for
+each of the test areas is noted in the table below.
+
+.. image:: danube/images/ovp_log_files.png
+ :align: center
+ :scale: 100%
+
+The three log files can be displayed by clicking on the setup icon to the right of the results,
+as shown in the diagram below.
+
+*Note, while the vping and ha test areas list multiple test-cases in the below diagram, there is
+a single log file for all test-cases within these test areas.*
+
+.. image:: danube/images/ovp_log_setup.png
+ :align: center
+ :scale: 100%
+
+Within the osinterop log (dovetail.osinterop.tc001.log), scroll down to the area of the log that
+begins to list the results of each test-case executed. This can be located by looking for lines
+prefaced with '**tempest.api**' and ending with '**... ok**'.
+
+.. image:: danube/images/ovp_log_test_count.png
+ :align: center
+ :scale: 100%
+
+The number of lines within the osinterop log for test-cases must add up according to the table
+above, where test-cases are broken down according to compute, identity, image, network and volume,
+with respective counts given in the table. The ha log (yardstick.log) must contain the 'PASS'
+result for each of the eight test-cases within this test area. This can be verified by searching
+the log for the keyword 'PASS'.
+
+
+
+The eight lines to validate are listed below:
+
+ - 017-10-16 05:07:49,158 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+ INFO The HA test case PASS the SLA
+ - 2017-10-16 05:08:31,387 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+ INFO The HA test case PASS the SLA
+ - 2017-10-16 05:09:13,669 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+ INFO The HA test case PASS the SLA
+ - 2017-10-16 05:09:55,967 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+ INFO The HA test case PASS the SLA
+ - 2017-10-16 05:10:38,407 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81
+ INFO The HA test case PASS the SLA
+ - 2017-10-16 05:11:00,030 yardstick.benchmark.scenarios.availability.scenario_general
+ scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+ - 2017-10-16 05:11:22,536 yardstick.benchmark.scenarios.availability.scenario_general
+ scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+ - 2017-10-16 05:12:07,880 yardstick.benchmark.scenarios.availability.scenario_general
+ scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m
+
+
+The final validation is for the vping test area log file (functest.log). The two entries
+displayed in the diagrams below must be present in this log file.
+
+ - vping_userdata
+ - vping_ssh
+
+.. image:: danube/images/ovp_vping_user.png
+ :align: center
+ :scale: 100%
+
+.. image:: danube/images/ovp_vping_ssh.png
+ :align: center
+ :scale: 100%
+
+5. SUT Info Verification
+========================
+
+SUT information must be present in the results to validate that all required endpoint services
+and at least two controllers were present during test execution. For the results shown below,
+click the '**info**' hyperlink in the **SUT** column to navigate to the SUT information page.
+
+.. image:: danube/images/sut_info.png
+ :align: center
+ :scale: 100%
+
+In the '**Endpoints**' listing shown below for the SUT VIM component, ensure that services are
+present for identify, compute, image, volume and network at a minimum by inspecting the
+'**Service Type**' column.
+
+.. image:: danube/images/sut_endpoints.png
+ :align: center
+ :scale: 100%
+
+Inspect the '**Hosts**' listing found below the Endpoints secion of the SUT info page and ensure
+at least two hosts are present, as two controllers are required the for the mandatory HA
+test-cases.