diff options
Diffstat (limited to 'docs/results')
| -rw-r--r-- | docs/results/overview.rst | 36 | ||||
| -rw-r--r-- | docs/results/results.rst | 86 |
2 files changed, 96 insertions, 26 deletions
diff --git a/docs/results/overview.rst b/docs/results/overview.rst index 30834d7f6..2515703bf 100644 --- a/docs/results/overview.rst +++ b/docs/results/overview.rst @@ -21,6 +21,10 @@ This document is part of deliverables of the OPNFV release brahmaputra.1.0 Scope ----- +.. _Dashboard: http://130.211.154.108/grafana/dashboard/db/yardstick-main +.. _Jenkins: https://build.opnfv.org/ci/view/yardstick/ +.. _Scenarios: http://130.211.154.108/grafana/dashboard/db/yardstick-scenarios + This document provides an overview of the results of test cases developed by the OPNFV Yardstick Project, executed on OPNFV community labs. @@ -30,8 +34,8 @@ automated via Jenkins Jobs. Test results are visible in the following dashboard: -* *Yardstick Dashboard*: uses influx DB to store test results and Grafana for - visualization +* *Yardstick* Dashboard_: uses influx DB to store test results and Grafana for + visualization (user: opnfv/ password: opnfv) References @@ -42,9 +46,35 @@ References * OPNFV Brahamputra release note for Yardstick. + General ======= +Yardstick Test Cases have been executed for scenarios and features defined in +this OPNFV release. + +The test environments were installed by one of the following: Apex, Compass, +Fuel or Joid; one single installer per POD. + +The results of executed tests are available in Dashboard_ and all logs stored +in Jenkins_. + +After one week of measurments, in general, SDN ONOS showed lower latency than +SDN ODL, which showed lower latency than an environment installed with pure +OpenStack. Additional time and PODs make this a non-conclusive statement, see +Scenarios_ for a snapshot and Dashboard_ for complete results. + +It was not possible to execute the entire Yardstick test cases suite on the +PODs assigned for release verification over a longer period of time, due to +continuous work on the software components and blocking faults either on +environment, feature or test framework. + +Four consecutive successful runs was defined as criteria for release. +It is a recommendation to run Yardstick test cases over a longer period +of time in order to better understand the behavior of the system. + + + Document change procedures and history -------------------------------------- @@ -58,7 +88,7 @@ Document change procedures and history | **Release designation** | Brahmaputra | | | | +--------------------------------------+--------------------------------------+ -| **Release date** | 2016-02-25 | +| **Release date** | Feb 25th, 2016 | | | | +--------------------------------------+--------------------------------------+ | **Purpose of the delivery** | OPNFV Brahmaputra release test | diff --git a/docs/results/results.rst b/docs/results/results.rst index fa6825ea3..08acab573 100644 --- a/docs/results/results.rst +++ b/docs/results/results.rst @@ -1,8 +1,7 @@ .. This work is licensed under a Creative Commons Attribution 4.0 International .. License. .. http://creativecommons.org/licenses/by/4.0 -.. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd, Intel Corporation and -.. others. + ====================== Yardstick Test Results @@ -15,44 +14,85 @@ Yardstick Test Results Scenario Results ================ +.. _Dashboard: http://130.211.154.108/grafana/dashboard/db/yardstick-main +.. _Jenkins: https://build.opnfv.org/ci/view/yardstick/ + The following documents contain results of Yardstick test cases executed on -OPNFV labs, triggered by OPNFV CI pipeline, documented per verified scenario. +OPNFV labs, triggered by OPNFV CI pipeline, documented per scenario. + + +Ready scenarios +--------------- + +The following scenarios run at least four consecutive times Yardstick test +cases suite: .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - apex-os-odl_l2-sfc-ha.rst apex-os-odl_l2-nofeature-ha.rst - apex-os-odl_l3-nofeature-ha.rst - apex-os-onos-nofeature-ha.rst compass-os-nosdn-nofeature-ha.rst compass-os-odl_l2-nofeature-ha.rst compass-os-onos-nofeature-ha.rst - fuel-os-nosdn-kvm-ha.rst fuel-os-nosdn-nofeature-ha.rst - fuel-os-nosdn-ovs-ha.rst fuel-os-odl_l2-nofeature-ha.rst - fuel-os-odl_l3-nofeature-ha.rst fuel-os-onos-nofeature-ha.rst - joid-os-nosdn-nofeature-ha.rst joid-os-odl_l2-nofeature-ha.rst - joid-os-onos-nofeature-ha.rst + + +Limitations +----------- + +The following scenarios run at least one time Yardstick test cases suite, +partially or complete: + + * fuel-os-odl_l2-sfc-ha + + * fuel-os-odl_l2-bgpvpn-ha + + * fuel-os-odl_l3-nofeature-ha + + * joid-os-nosdn-nofeature-ha + + +Test results of executed tests are avilable in Dashboard_ and logs in Jenkins_. + + +Not executed +------------ + +For the following scenarios, Yardstick test cases were not run: + + + * apex-os-odl_l2-sfc-noha + + * apex-os-odl_l3-nofeature-ha + + * apex-os-onos-nofeature-ha + + * fuel-os-nosdn-kvm-ha + + * fuel-os-nosdn-ovs-ha + + * fuel-os-odl_l3-nofeature-ha + + * joid-os-onos-nofeature-ha + +Logs are available in Jenkins_. Feature Test Results ==================== -The following documents contain results of Yardstick test cases executed on -OPNFV labs which were not dedicated for OPNFV CI release verification -activities. +The following features were verified by Yardstick test cases: -Results of tests for OPNFV features verified as part a scenario are -documented in the previous section. + * IPv6 -.. toctree:: - :maxdepth: 2 + * HA (see :doc:`yardstick-opnfv-ha`) + + * KVM + + * Parser - yardstick-opnfv-ha.rst - yardstick-opnfv-kvm.rst - yardstick-opnfv-parser.rst - yardstick-opnfv-vtc.rst +.. note:: The test cases for IPv6 and Parser Projects are included in the + compass scenario. |