From cac419a417f25a1c9e0de653e09d2f0ad32e4c42 Mon Sep 17 00:00:00 2001 From: Eddie Arrage Date: Sun, 12 Nov 2017 13:38:11 -0800 Subject: Transfer Danube reviewer guidelines from wiki The reviewer guidelines will be created per release, where the CVP/Dovetail releases are aligned to OPNFV releases. The guidelines outlined in this initial document are for the Danube release. Updated reviewer guidelines - Updated to account for issue where mandatory total always shows 215 in current version of web portal; removed one diagram and amended another - First section title change and wording for test area results - Removed white space - Added SUT info section and diagrams - Modified reference to cvp.opnfv.org with verified.opnfv.org - Modified reference to cvp@opnfv.org with verified@opnfv.org - Modified diagram filenames to reflect program name change - Changed CVP references to OVP and Danube to 2018.01 - Remove TODO for granting marks and escalation - this info is in other docs Change-Id: I01fc568ce6713d3c60daf339a2630e56e8dbcfcb Signed-off-by: Eddie Arrage --- .../reviewerguide/danube/images/ovp_log_files.png | Bin 0 -> 6504 bytes .../reviewerguide/danube/images/ovp_log_setup.png | Bin 0 -> 37346 bytes .../danube/images/ovp_log_test_count.png | Bin 0 -> 6930 bytes .../reviewerguide/danube/images/ovp_missing_ha.png | Bin 0 -> 80192 bytes .../danube/images/ovp_pass_fraction.png | Bin 0 -> 117383 bytes .../danube/images/ovp_pass_percentage.png | Bin 0 -> 75765 bytes .../danube/images/ovp_result_overview.png | Bin 0 -> 105270 bytes .../danube/images/ovp_result_review.png | Bin 0 -> 44980 bytes .../danube/images/ovp_run_results.png | Bin 0 -> 13149 bytes .../reviewerguide/danube/images/ovp_test_count.png | Bin 0 -> 4012 bytes .../reviewerguide/danube/images/ovp_top_nav.png | Bin 0 -> 23642 bytes .../reviewerguide/danube/images/ovp_vping_ssh.png | Bin 0 -> 5246 bytes .../reviewerguide/danube/images/ovp_vping_user.png | Bin 0 -> 5667 bytes .../reviewerguide/danube/images/sut_endpoints.png | Bin 0 -> 119897 bytes .../user/reviewerguide/danube/images/sut_info.png | Bin 0 -> 29227 bytes docs/testing/user/reviewerguide/index.rst | 198 ++++++++++++++++++++- 16 files changed, 190 insertions(+), 8 deletions(-) create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_log_files.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_result_review.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_run_results.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_test_count.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png create mode 100644 docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png create mode 100644 docs/testing/user/reviewerguide/danube/images/sut_endpoints.png create mode 100644 docs/testing/user/reviewerguide/danube/images/sut_info.png diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png new file mode 100644 index 00000000..d5af068b Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_log_files.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png new file mode 100644 index 00000000..395a2c3e Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_log_setup.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png new file mode 100644 index 00000000..6c363441 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_log_test_count.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png new file mode 100644 index 00000000..7e2c6982 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_missing_ha.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png new file mode 100644 index 00000000..74c99f37 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_pass_fraction.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png new file mode 100644 index 00000000..ad79f4ec Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_pass_percentage.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png new file mode 100644 index 00000000..fa12ddc7 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_result_overview.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png new file mode 100644 index 00000000..a8f988fc Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_result_review.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png new file mode 100644 index 00000000..4b845df2 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_run_results.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png new file mode 100644 index 00000000..1c9bcd30 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_test_count.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png new file mode 100644 index 00000000..280d0c5f Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_top_nav.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png new file mode 100644 index 00000000..01384d02 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_vping_ssh.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png new file mode 100644 index 00000000..a82e1121 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/ovp_vping_user.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png new file mode 100644 index 00000000..f11d531f Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/sut_endpoints.png differ diff --git a/docs/testing/user/reviewerguide/danube/images/sut_info.png b/docs/testing/user/reviewerguide/danube/images/sut_info.png new file mode 100644 index 00000000..395f5f41 Binary files /dev/null and b/docs/testing/user/reviewerguide/danube/images/sut_info.png differ diff --git a/docs/testing/user/reviewerguide/index.rst b/docs/testing/user/reviewerguide/index.rst index fef318e9..bd615f19 100644 --- a/docs/testing/user/reviewerguide/index.rst +++ b/docs/testing/user/reviewerguide/index.rst @@ -2,9 +2,9 @@ .. http://creativecommons.org/licenses/by/4.0 .. (c) Ericsson AB -================================================== -Compliance and Verification program reviewer guide -================================================== +============================================= +OPNFV Verified Program 2018.01 Reviewer Guide +============================================= .. toctree:: :maxdepth: 2 @@ -13,12 +13,194 @@ Compliance and Verification program reviewer guide Introduction ============ -This reviewer guide provides detailed guidance for reviewers on how to handle the result review process. +This reviewer guide provides detailed guidance for reviewers on how to handle the result review +process. Reviewers must follow the checklist below to ensure review consistency for the OPNFV +Verified Program (OVP) 2018.01 (Danube) release at a minimum. -TODO: add sections on +#. **Mandatory Test Area Results** - Validate that results for all mandatory test areas are present. +#. **Test-Case Count within Mandatory Test Area** - Check that the total number of test-cases are present in each mandatory test area. +#. **Test-Case Pass Percentage** - Ensure all tests have passed (100% pass rate). +#. **Log File Verification** - Inspect the log file for each test area (osinterop, ha, vping). +#. **SUT Info Verification** - Validate the system under test (SUT) hardware and software endpoint info is present. - - specification of criteria for reviewing the results - - specification of criteria for granting the certification badge +1. Mandatory Test Area Results +============================== - - specification SLAs for responding to requests, questions, escalations, ... +Validate that results for all mandatory test areas are included in the overall test suite. The +required mandatory test areas are: + - **osinterop** + - **vping** + - **ha** + +Login to the OVP portal at: + +*https://verified.opnfv.org* + +Click on the 'My Results' tab in top-level navigation bar. + +.. image:: danube/images/ovp_top_nav.png + :align: center + :scale: 100% + +The OVP administrator will ask for review volunteers using the verified@opnfv.org email alias. The +incoming results for review will be identified by the administrator with particular 'Test ID' +and 'Owner' values. The corresponding OVP portal result will have a status of 'review'. + +.. image:: danube/images/ovp_result_review.png + :align: center + :scale: 100% + +In the example above, this information will be provided as: +- Test ID: a00c47e8 +- Owner: jtaylor + +Click on the hyperlink within the 'Test ID' column. + +*Note, that the 'Test ID' column in this view condenses the UUID used for 'Test ID' to +eight characters even though the 'Test ID' is a longer UUID in the back-end.* + +.. image:: danube/images/ovp_result_overview.png + :align: center + :scale: 100% + +The 'Test ID' hyperlink toggles the view to a top-level listing of the results displayed above. +Validate that osinterop, vping and ha test area results are all present within the view. + + +2. Test-Case Count within Mandatory Test Area +============================================= + +Validate the test-case count within each test area. For the OVP 2018.01 release, this must break +down as outlined in the table below. + +.. image:: danube/images/ovp_test_count.png + :align: center + :scale: 100% + +In the diagram above (from section 1), these counts can be gleaned from the numbers to the +right of the test-cases. The total number is given for the osinterop (dovetail.osinterop.tc001) +test area at 205. The vping (dovetail.vping.tc00x) and ha (dovetail.ha.tc00x) test-cases are +broken down separately with a line for each test-case. Directly above the 'Test Result Overview' +listing there's a summary labelled 'Test Run Results' shown below. For OVP 2018.01, a mandatory +total of **215** test-cases must be present (205 osinterop + 8 ha + 2 vping). + +.. image:: danube/images/ovp_missing_ha.png + :align: center + :scale: 100% + +An example of a listing that should flag a negative review is shown above. The mandatory ha test +area is missing one test case (dovetail.ha.tc008). + +3. Test-Case Pass Percentage +============================ + +All mandatory test-cases must pass. This can be validated in multiple ways. The below diagram of +the 'Test Run Results' is one method and shows that 100% of the mandatory test-cases have passed. +This value must not be lower than 100%. + +.. image:: danube/images/ovp_pass_percentage.png + :align: center + :width: 350 px + +Another method to check that all mandatory test-cases have passed is shown in the diagram below. +The pass/total is given as a fraction and highlighted here in yellow. For the osinterop test area, +the result must display [205/205] and for each of the test-cases under the vping and ha test areas +[1/1] must be displayed. + +.. image:: danube/images/ovp_pass_fraction.png + :align: center + :width: 270 px + +4. Log File Verification +======================== + +Three log files must be verified for content within each mandatory test area. The log files for +each of the test areas is noted in the table below. + +.. image:: danube/images/ovp_log_files.png + :align: center + :scale: 100% + +The three log files can be displayed by clicking on the setup icon to the right of the results, +as shown in the diagram below. + +*Note, while the vping and ha test areas list multiple test-cases in the below diagram, there is +a single log file for all test-cases within these test areas.* + +.. image:: danube/images/ovp_log_setup.png + :align: center + :scale: 100% + +Within the osinterop log (dovetail.osinterop.tc001.log), scroll down to the area of the log that +begins to list the results of each test-case executed. This can be located by looking for lines +prefaced with '**tempest.api**' and ending with '**... ok**'. + +.. image:: danube/images/ovp_log_test_count.png + :align: center + :scale: 100% + +The number of lines within the osinterop log for test-cases must add up according to the table +above, where test-cases are broken down according to compute, identity, image, network and volume, +with respective counts given in the table. The ha log (yardstick.log) must contain the 'PASS' +result for each of the eight test-cases within this test area. This can be verified by searching +the log for the keyword 'PASS'. + + + +The eight lines to validate are listed below: + + - 017-10-16 05:07:49,158 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 + INFO The HA test case PASS the SLA + - 2017-10-16 05:08:31,387 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 + INFO The HA test case PASS the SLA + - 2017-10-16 05:09:13,669 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 + INFO The HA test case PASS the SLA + - 2017-10-16 05:09:55,967 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 + INFO The HA test case PASS the SLA + - 2017-10-16 05:10:38,407 yardstick.benchmark.scenarios.availability.serviceha serviceha.py:81 + INFO The HA test case PASS the SLA + - 2017-10-16 05:11:00,030 yardstick.benchmark.scenarios.availability.scenario_general + scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m + - 2017-10-16 05:11:22,536 yardstick.benchmark.scenarios.availability.scenario_general + scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m + - 2017-10-16 05:12:07,880 yardstick.benchmark.scenarios.availability.scenario_general + scenario_general.py:71 INFO [92m Congratulations, the HA test case PASS! [0m + + +The final validation is for the vping test area log file (functest.log). The two entries +displayed in the diagrams below must be present in this log file. + + - vping_userdata + - vping_ssh + +.. image:: danube/images/ovp_vping_user.png + :align: center + :scale: 100% + +.. image:: danube/images/ovp_vping_ssh.png + :align: center + :scale: 100% + +5. SUT Info Verification +======================== + +SUT information must be present in the results to validate that all required endpoint services +and at least two controllers were present during test execution. For the results shown below, +click the '**info**' hyperlink in the **SUT** column to navigate to the SUT information page. + +.. image:: danube/images/sut_info.png + :align: center + :scale: 100% + +In the '**Endpoints**' listing shown below for the SUT VIM component, ensure that services are +present for identify, compute, image, volume and network at a minimum by inspecting the +'**Service Type**' column. + +.. image:: danube/images/sut_endpoints.png + :align: center + :scale: 100% + +Inspect the '**Hosts**' listing found below the Endpoints secion of the SUT info page and ensure +at least two hosts are present, as two controllers are required the for the mandatory HA +test-cases. -- cgit 1.2.3-korg