diff options
Diffstat (limited to 'docs/testing/developer')
| -rw-r--r-- | docs/testing/developer/design/apidocs/index.rst | 13 | ||||
| -rw-r--r-- | docs/testing/developer/design/apidocs/qtip_restful_api.rst | 10 | ||||
| -rw-r--r-- | docs/testing/developer/design/compute_QPI.rst | 69 | ||||
| -rw-r--r-- | docs/testing/developer/design/dashboard.rst | 151 | ||||
| -rw-r--r-- | docs/testing/developer/design/integration_with_yardstick.rst | 92 |
5 files changed, 0 insertions, 335 deletions
diff --git a/docs/testing/developer/design/apidocs/index.rst b/docs/testing/developer/design/apidocs/index.rst deleted file mode 100644 index 241a2680..00000000 --- a/docs/testing/developer/design/apidocs/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -################ -QTIP Configguide -################ - -.. toctree:: - :maxdepth: 2 - - ./qtip_restful_api.rst diff --git a/docs/testing/developer/design/apidocs/qtip_restful_api.rst b/docs/testing/developer/design/apidocs/qtip_restful_api.rst deleted file mode 100644 index 06c01292..00000000 --- a/docs/testing/developer/design/apidocs/qtip_restful_api.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -**************** -Qtip restful api -**************** - -You can get all the Qtip restful api by http://restful_api.qtip.openzero.net/api/spec.html. diff --git a/docs/testing/developer/design/compute_QPI.rst b/docs/testing/developer/design/compute_QPI.rst deleted file mode 100644 index 2e5aa87c..00000000 --- a/docs/testing/developer/design/compute_QPI.rst +++ /dev/null @@ -1,69 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -*********** -Compute QPI -*********** - -The compute QPI gives user an overall score for system compute performace. - -Summary -======= - -The compute QPI are calibrated a ZTE `E9000 <http://www.zte.com.cn/global/products/cocloud/cloud_computing/cloud_infrastructure/cloud_hw/429552>`_ server as a baseline with score of 2500 points. -Higher scores are better, with double the score indicating double the performance. -The compute QPI provides three different kinds of scores: - -* Workload Scores -* Section Scores -* Compute QPI Scores - -Baseline -======== - -ZTE E9000 server with an 2 Deca core Intel Xeon CPU processor,128560.0MB Memory. - -Workload Scores -=============== - -Each time a workload is executed QTIP calculates a score based on the computer's performance -compared to the baseline performance. - -Section Scores -============== - -QTIP uses a number of different tests, or workloads, to measure performance. -The workloads are divided into five different sections: - -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Section | Detail | Indication | -+=================+==============================================================+==========================================+ -| Integer | Integer workloads measure the integer instruction performace | All app relies on integer | -| | of host or vm by performing Dhrystone test. | performance | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Floating point | Floating point workloads measure the floating pointperfo | Floating point performance is especially | -| | rmance by performing Whetstone test. | important in video games,digital content | -| | | creation applications. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| Memory | Memory workloads measure memory bandwidth by performing | Software working with cipher large | -| | RamSpeed test. | amounts data relies on SSL Performace. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| DPI | DPI workloads measure deep-packet inspection speed by | Software working with network packet | -| | performing nDPI test. | anlysis relies on DPI performance. | -+-----------------+--------------------------------------------------------------+------------------------------------------+ -| SSL | SSL Performance workloads measure cipher speeds by | Software working with cipher large | -| | using the OpenSSL tool. | amounts data relies on SSL Performace | -+-----------------+--------------------------------------------------------------+------------------------------------------+ - -A section score is the `geometric mean <https://en.wikipedia.org/wiki/Geometric_mean>`_ of all the workload scores for workloads -that are part of the section. These scores are useful for determining the performance of -the computer in a particular area. - -Compute QPI Scores -================== - -The compute QPI score is the `weighted arithmetic mean <https://en.wikipedia.org/wiki/Weighted_arithmetic_mean>`_ of the five section scores. -The compute QPI score provides a way to quickly compare performance across different -computers and different platforms without getting bogged down in details. diff --git a/docs/testing/developer/design/dashboard.rst b/docs/testing/developer/design/dashboard.rst deleted file mode 100644 index 60c4720d..00000000 --- a/docs/testing/developer/design/dashboard.rst +++ /dev/null @@ -1,151 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -********* -Dashboard -********* - -The dashboard gives user an intuitive view of benchmark result. - -Purpose -======= - -The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it -is also important to show user - -#. How is the final score calculated? -#. Under what condition is the test plan executed? -#. How many runs of a performance tests have been executed and is there any deviation? -#. Comparison of benchmark result from different PODs or configuration - -Templates -========= - -Different board templates are created to satisfy the above requirements. - -Composition ------------ - -QTIP gives a simple score but there must be a complex formula behind it. This -view explains the composition of the QPI. - -Condition ---------- - -The condition of a benchmark result includes - -* System Under Test - - * Hardware environment - * Hypervisor version - * Operation System release version - * System Configuration - -* Test Tools - - * Release version - * Configuration - -* Test Facility - - * Laboratory - * Engineer - * Date - -Conditions that do NOT have an obvious affect on the test result may be ignored, -e.g. temperature, power supply. - -Stats ------ - -Performance tests are actually measurement of specific metrics. All measurement -comes with uncertainty. The final result is normally one or a group of metrics -calculated from many repeats. - -For each metric, the stats board shall consist of a diagram of all measured -values and a box of stats:: - - ^ +------------+ - | | count: ? | - | |average: ? | - | | min: ? | - | X | max: ? | - | XXXX XXXX X XXXXX | | - |X XX XX XX XXX XXX XX | | - | XXXXXX X XXXXX XX | | - | | | - | | | - | | | - | | | - | | | - +---------------------------------------------> +------------+ - -The type of diagram and selection of stats shall depend on what metric to show. - -Comparison ----------- - -Comparison can be done between different PODs or different configuration on the -same PODs. - -In a comparison view, the metrics are displayed in the same diagram. And the -parameters are listed side by side. - -Both common parameters and different parameters are listed. Common values are -merged to the same cell. And user may configure the view to hide common rows. - -A draft design is as following:: - - ^ - | - | - | - | XXXXXXXX - | XXX XX+-+ XXXXXXXXXX - | XXX +XXXX XXXXX - +-+XX X +--+ ++ XXXXXX +-+ - | X+-+X +----+ +-+ +----+X - |X +--+ +---+ XXXXXX X - | +-------+ X - | - | - +-----------------------------------------------------> - - +--------------------+----------------+---------------+ - | different param 1 | | | - | | | | - +-----------------------------------------------------+ - | different param 2 | | | - | | | | - +-------------------------------------+---------------+ - | common param 1 | | - | | | - +-------------------------------------+---------------+ - | different param 3 | | | - | | | | - +-------------------------------------+---------------+ - | common param 2 | | - | | | - +--------------------+--------------------------------+ - +------------+ - | HIDE COMMON| - +------------+ - -Time line ---------- - -Time line diagram for analysis of time critical performance test:: - - +-----------------+-----------+-------------+-------------+-----+ - | | | | | | - +-----------------> | | | | - | +-----------> | | | - | ? ms +-------------> | | - | ? ms +------------>+ | - | ? ms ? ms | - | | - +---------------------------------------------------------------+ - -The time cost between checkpoints shall be displayed in the diagram. diff --git a/docs/testing/developer/design/integration_with_yardstick.rst b/docs/testing/developer/design/integration_with_yardstick.rst deleted file mode 100644 index a8298d6f..00000000 --- a/docs/testing/developer/design/integration_with_yardstick.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -************************** -Integration with Yardstick -************************** - -Problem description -=================== - -For each specified QPI [1]_, QTIP needs to select a suite of test cases and collect -required test results. Based on these results, Qtip calculates the score. - -Proposed change -=============== -Qtip has a flexible architecture [2]_ to support different mode: standalone and agent. -It is recommended to use **agent mode** to work with existing test runners. Yardstick will -act as a runner to generate test result and trigger Qtip agent on the completion of test. - - -Work Items in Yardstick ------------------------ - -1. Create a customized suite in Yardstick - -Yardstick not only has many existing suites but also support customized suites. Qtip could -create a suite named **Qtip-PoC** in Yardstick repo to verify workflow of Qtip agent mode. - -2. Launch Qtip in Yardstick - -Whether to launch Qtip will be determined by checking the existence of OS environment -variable *QTIP*. If it exists, Qtip will be launched by using Yardstick CLI -`yardstick plugin install` [3]_. - -3. Yardstick interacts with Qtip - -See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Work Items in Qtip ------------------- - -1. Provide an API for Yardstick to post test result and environment info - -After completing test execution, Yardstick will post test result and enviroment info with -JSON format via QTIP API. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -2. Parse yardstick test result - -When Qtip agent receive Yarstick test result and enviroment info, Qtip agent will extract -metrics which is definded in metric spec configuration file. Based on these metrics, Qtip -agent will caculate QPI. - -3. Provide an API for querying QPI - -Qtip will provide an API for querying QPI. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Implementation -============== - -Assignee(s) ------------ - -*Primary assignee:* - wu.zhihui - -*Other contributors* - TBD - -Testing -======= - -The changes will be covered by new unit test. - -Documentation -============= - -TBD - -Reference -========= - -.. [1] QTIP performance index -.. [2] https://wiki.opnfv.org/display/qtip/Architecture -.. [3] https://wiki.opnfv.org/display/yardstick/How+to+install+a+plug-in+into+Yardstick |