diff options
Diffstat (limited to 'docs/testing/developer')
| -rw-r--r-- | docs/testing/developer/design/apidocs/index.rst | 13 | ||||
| -rw-r--r-- | docs/testing/developer/design/apidocs/qtip_restful_api.rst | 10 | ||||
| -rw-r--r-- | docs/testing/developer/design/dashboard.rst | 151 | ||||
| -rw-r--r-- | docs/testing/developer/design/integration_with_yardstick.rst | 92 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/_assets/sequence.png | bin | 0 -> 44132 bytes | |||
| -rw-r--r-- | docs/testing/developer/devguide/_assets/standalone.png | bin | 0 -> 47515 bytes | |||
| -rw-r--r-- | docs/testing/developer/devguide/api.rst | 2 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/arch.rst | 24 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/cli.rst (renamed from docs/testing/developer/design/cli.rst) | 0 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/compute-qpi.rst (renamed from docs/testing/developer/design/compute_QPI.rst) | 0 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/index.rst (renamed from docs/testing/developer/design/index.rst) | 7 |
11 files changed, 30 insertions, 269 deletions
diff --git a/docs/testing/developer/design/apidocs/index.rst b/docs/testing/developer/design/apidocs/index.rst deleted file mode 100644 index 241a2680..00000000 --- a/docs/testing/developer/design/apidocs/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -################ -QTIP Configguide -################ - -.. toctree:: - :maxdepth: 2 - - ./qtip_restful_api.rst diff --git a/docs/testing/developer/design/apidocs/qtip_restful_api.rst b/docs/testing/developer/design/apidocs/qtip_restful_api.rst deleted file mode 100644 index 06c01292..00000000 --- a/docs/testing/developer/design/apidocs/qtip_restful_api.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2015 Dell Inc. -.. (c) 2016 ZTE Corp. - -**************** -Qtip restful api -**************** - -You can get all the Qtip restful api by http://restful_api.qtip.openzero.net/api/spec.html. diff --git a/docs/testing/developer/design/dashboard.rst b/docs/testing/developer/design/dashboard.rst deleted file mode 100644 index 60c4720d..00000000 --- a/docs/testing/developer/design/dashboard.rst +++ /dev/null @@ -1,151 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -********* -Dashboard -********* - -The dashboard gives user an intuitive view of benchmark result. - -Purpose -======= - -The basic element to be displayed is QPI a.k.a. QTIP Performance Index. But it -is also important to show user - -#. How is the final score calculated? -#. Under what condition is the test plan executed? -#. How many runs of a performance tests have been executed and is there any deviation? -#. Comparison of benchmark result from different PODs or configuration - -Templates -========= - -Different board templates are created to satisfy the above requirements. - -Composition ------------ - -QTIP gives a simple score but there must be a complex formula behind it. This -view explains the composition of the QPI. - -Condition ---------- - -The condition of a benchmark result includes - -* System Under Test - - * Hardware environment - * Hypervisor version - * Operation System release version - * System Configuration - -* Test Tools - - * Release version - * Configuration - -* Test Facility - - * Laboratory - * Engineer - * Date - -Conditions that do NOT have an obvious affect on the test result may be ignored, -e.g. temperature, power supply. - -Stats ------ - -Performance tests are actually measurement of specific metrics. All measurement -comes with uncertainty. The final result is normally one or a group of metrics -calculated from many repeats. - -For each metric, the stats board shall consist of a diagram of all measured -values and a box of stats:: - - ^ +------------+ - | | count: ? | - | |average: ? | - | | min: ? | - | X | max: ? | - | XXXX XXXX X XXXXX | | - |X XX XX XX XXX XXX XX | | - | XXXXXX X XXXXX XX | | - | | | - | | | - | | | - | | | - | | | - +---------------------------------------------> +------------+ - -The type of diagram and selection of stats shall depend on what metric to show. - -Comparison ----------- - -Comparison can be done between different PODs or different configuration on the -same PODs. - -In a comparison view, the metrics are displayed in the same diagram. And the -parameters are listed side by side. - -Both common parameters and different parameters are listed. Common values are -merged to the same cell. And user may configure the view to hide common rows. - -A draft design is as following:: - - ^ - | - | - | - | XXXXXXXX - | XXX XX+-+ XXXXXXXXXX - | XXX +XXXX XXXXX - +-+XX X +--+ ++ XXXXXX +-+ - | X+-+X +----+ +-+ +----+X - |X +--+ +---+ XXXXXX X - | +-------+ X - | - | - +-----------------------------------------------------> - - +--------------------+----------------+---------------+ - | different param 1 | | | - | | | | - +-----------------------------------------------------+ - | different param 2 | | | - | | | | - +-------------------------------------+---------------+ - | common param 1 | | - | | | - +-------------------------------------+---------------+ - | different param 3 | | | - | | | | - +-------------------------------------+---------------+ - | common param 2 | | - | | | - +--------------------+--------------------------------+ - +------------+ - | HIDE COMMON| - +------------+ - -Time line ---------- - -Time line diagram for analysis of time critical performance test:: - - +-----------------+-----------+-------------+-------------+-----+ - | | | | | | - +-----------------> | | | | - | +-----------> | | | - | ? ms +-------------> | | - | ? ms +------------>+ | - | ? ms ? ms | - | | - +---------------------------------------------------------------+ - -The time cost between checkpoints shall be displayed in the diagram. diff --git a/docs/testing/developer/design/integration_with_yardstick.rst b/docs/testing/developer/design/integration_with_yardstick.rst deleted file mode 100644 index a8298d6f..00000000 --- a/docs/testing/developer/design/integration_with_yardstick.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 -.. (c) 2016 ZTE Corp. - - -************************** -Integration with Yardstick -************************** - -Problem description -=================== - -For each specified QPI [1]_, QTIP needs to select a suite of test cases and collect -required test results. Based on these results, Qtip calculates the score. - -Proposed change -=============== -Qtip has a flexible architecture [2]_ to support different mode: standalone and agent. -It is recommended to use **agent mode** to work with existing test runners. Yardstick will -act as a runner to generate test result and trigger Qtip agent on the completion of test. - - -Work Items in Yardstick ------------------------ - -1. Create a customized suite in Yardstick - -Yardstick not only has many existing suites but also support customized suites. Qtip could -create a suite named **Qtip-PoC** in Yardstick repo to verify workflow of Qtip agent mode. - -2. Launch Qtip in Yardstick - -Whether to launch Qtip will be determined by checking the existence of OS environment -variable *QTIP*. If it exists, Qtip will be launched by using Yardstick CLI -`yardstick plugin install` [3]_. - -3. Yardstick interacts with Qtip - -See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Work Items in Qtip ------------------- - -1. Provide an API for Yardstick to post test result and environment info - -After completing test execution, Yardstick will post test result and enviroment info with -JSON format via QTIP API. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -2. Parse yardstick test result - -When Qtip agent receive Yarstick test result and enviroment info, Qtip agent will extract -metrics which is definded in metric spec configuration file. Based on these metrics, Qtip -agent will caculate QPI. - -3. Provide an API for querying QPI - -Qtip will provide an API for querying QPI. See -`Yardstick-Qtip+integration <https://wiki.opnfv.org/display/yardstick/Yardstick-Qtip+integration>`_ -for details. - -Implementation -============== - -Assignee(s) ------------ - -*Primary assignee:* - wu.zhihui - -*Other contributors* - TBD - -Testing -======= - -The changes will be covered by new unit test. - -Documentation -============= - -TBD - -Reference -========= - -.. [1] QTIP performance index -.. [2] https://wiki.opnfv.org/display/qtip/Architecture -.. [3] https://wiki.opnfv.org/display/yardstick/How+to+install+a+plug-in+into+Yardstick diff --git a/docs/testing/developer/devguide/_assets/sequence.png b/docs/testing/developer/devguide/_assets/sequence.png Binary files differnew file mode 100644 index 00000000..b15458bf --- /dev/null +++ b/docs/testing/developer/devguide/_assets/sequence.png diff --git a/docs/testing/developer/devguide/_assets/standalone.png b/docs/testing/developer/devguide/_assets/standalone.png Binary files differnew file mode 100644 index 00000000..50222b5b --- /dev/null +++ b/docs/testing/developer/devguide/_assets/standalone.png diff --git a/docs/testing/developer/devguide/api.rst b/docs/testing/developer/devguide/api.rst new file mode 100644 index 00000000..eb2b0d67 --- /dev/null +++ b/docs/testing/developer/devguide/api.rst @@ -0,0 +1,2 @@ +- Which framework has been used and why +- How to extend to more api diff --git a/docs/testing/developer/devguide/arch.rst b/docs/testing/developer/devguide/arch.rst new file mode 100644 index 00000000..d95faba6 --- /dev/null +++ b/docs/testing/developer/devguide/arch.rst @@ -0,0 +1,24 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) 2017 ZTE Corp. + + +######################## +QTIP Architecture Design +######################## + +In Danube, QTIP releases its standalone mode, which is also know as ``solo``: + +.. figure:: _assets/standalone.png + :alt: QTIP standalone mode + +The runner could be launched from CLI (command line interpreter) or API +(application programming interface) and drives the testing jobs. The generated +data including raw performance data and testing environment are fed to collector. +Performance metrics will be parsed from the raw data and used for QPI calculation. +Then the benchmark report is rendered with the benchmarking results. + +The execution can be detailed in the diagram below: + +.. figure:: _assets/sequence.png + :alt: QTIP execution sequence diff --git a/docs/testing/developer/design/cli.rst b/docs/testing/developer/devguide/cli.rst index 72d1fbaf..72d1fbaf 100644 --- a/docs/testing/developer/design/cli.rst +++ b/docs/testing/developer/devguide/cli.rst diff --git a/docs/testing/developer/design/compute_QPI.rst b/docs/testing/developer/devguide/compute-qpi.rst index 2e5aa87c..2e5aa87c 100644 --- a/docs/testing/developer/design/compute_QPI.rst +++ b/docs/testing/developer/devguide/compute-qpi.rst diff --git a/docs/testing/developer/design/index.rst b/docs/testing/developer/devguide/index.rst index b6dd0c01..2b4bd9b0 100644 --- a/docs/testing/developer/design/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -10,6 +10,7 @@ QTIP Design Specifications .. toctree:: :maxdepth: 2 - dashboard.rst - compute_QPI.rst - integration_with_yardstick.rst + arch.rst + cli.rst + api.rst + compute-qpi.rst |
